Home

Senior Hadoop Administrator

Richardson, TX. Posted 2 months ago

Resource Informatics Group Inc
400 E Royal Lane
Posted By
Santosh Bollepalli
Duration
12 Months
Start Date
Immediate
Tech Category
NOSQL - Hadoop
Pay
Negotiable
Experience
6 - 12 Years
Tech Sub-Category
Administration
Work Permit
Any Work Permit encouraged to apply
Tax Terms
N/A

Job Description

Location: 

 

Downton Chicago IL 

Richardson, TX

 

Administrators, the primary responsibilities are:

 Manage large scale Hadoop/Spark cluster environments, handling all environment builds, including design, capacity planning, cluster setup, performance tuning, scaling (adding/removing nodes), scheduling and monitoring (including log reviews and job tracking/reporting), setting up Kerberos principals and testing HDFS, Hive, Pig and Map Reduce access for the new users.

 Understanding of interactions between applications, HW, and UNIX/Linux operating system and to provide recommendations and suggestions related to troubleshooting and performance improvement

 Deploy Hadoop cluster(s), add and remove nodes, keep track of jobs, monitor critical parts of the cluster, configure name-node high availability, schedule and configure.

 

Minimum Job Responsibilities:

1. Manage large scale Hadoop/Spark cluster environments, handling all environment builds, including design, capacity planning, cluster setup, performance tuning, scaling (adding/removing nodes), scheduling and monitoring (including log reviews and job tracking/reporting), setting up Kerberos principals and testing HDFS, Hive, Pig and MapReduce access for the new users.

2. Understanding of interactions between applications, HW, and UNIX/Linux operating system and to provide recommendations and suggestions related to troubleshooting and performance improvement

3. Deploy Hadoop cluster(s), add and remove nodes, keep track of jobs, monitor critical parts of the cluster, configure name-node high availability, schedule and configure.

4. Implement automated approaches for system administration and operation tasks, provisioning new servers and deployment of artifacts for multiple applications.

5. Evaluate and recommend systems software and hardware for the Big Data enterprise system, including collaborating with application teams to install operating system and Hadoop updates, patches, version upgrades when required.

6. Contribute to the evolving architecture of our Big Data services to meet changing requirements for scaling, reliability, performance, manageability, and price.

7. Develop metrics and measures of utilization and performance.

8. Manage cluster resource to meet application jobs’ SLAs, including support of DR, Backup/Recovery and archiving operations)

9. Handling job queues and workload mapping for multi-tenant environment with different level of priorities and SLAs.

10. Management and creation of support documentation for standard procedures.

11. Liaison with partner technology providers (ex: Cloudera, Hortonworks, Azure, AWS, etc…) on escalating, logging and resolving issues, managing enhancements, upgrades and patches.

 

 

 

 

 

Core Skills/Experience (minimum):

1. Professional experience supporting production Linux environments (2 years mid-level minimum; 4 years senior level minimum).

2. Expert knowledge of Hadoop/Spark design principals, cluster connectivity, security (ex: Kerboros, LDAP and Active Directory integration) and the factors that affect distributed system performance.(3 years)

3. Extensive experience and understanding across various stack components (HDFS, Map Reduce, Spark Core, Pig, Hive, Zookeeper, Mahout, Elastic Stack, workflow tools such as Oozie and Azkaban)

4. Advanced hands-on Linux skill is a must (2 years mid-level; 4 years senior level minimum). Understanding of shell, debugging things etc.

5. Experience with performance tuning, capacity planning, and workload mapping

6. Experience with handling job queues for multi-tenant environment with different level of priorities and SLAs.

7. Experience with complex networking infrastructure including firewalls, VLANs, and load balancers.

8. Experience integrating third party tools with Hadoop such as Tableau, PowerBI, Jupyter Hub, Tensorflow, H2O.ai, etc. (1 years)

9. Scripting Language experience (ex: Python, Ruby, Bash) (3 years)

10. Experience in using automation and container tools such as Ansible, Docker, Jenkins, Puppet and Chef

11. Manage content repositories and structures for support code, scripting and versioning. (ex: Github)

12. DR, Backup/Recovery and archiving experience

13. BS/equivalent or higher degree (Computer Science, Math or Engineering preferred)

 

Value-add Skills:

1. Experience working on NoSQL and Search technologies (Ex: Hbase, Mongo)

2. Knowledge of best practices related to security, performance, and disaster recovery.

3. Experience with monitoring frameworks like Ganglia and Nagios

4. Experience in Healthcare

5. Preferred: Hadoop Administrator Certifications

6. Preferred: Experience in Cloud platforms

7. Preferred: Experience in specific Cloud-based Hadoop deployments (AWS-EMR, Azure HDInsight, HDP or CDP on Cloud, etc)

8. Container deployment/management experience

9. Basic SQL and relational database experience

10. Basic skills in ETL (data ingestion, transformation and harmonization, metadata )

11. Java/Scala experience

12. Direct end-user/ business-client facing interaction experience

Soft-Skills:

1. Good collaboration & communication (oral and written) skills, the ability to participate in an interdisciplinary team.

2. Strong written communications and documentation experience

3. Excellent time management and organizational management skills

4. Able to lead end to end initiative

Key Skills
Hadoop Spark Scaling Scheduling Kerberos Testing UNIX Linux Operating System Troubleshooting Performance Improvement High Availability MapReduce System Administration Manageability Support Documentation Resolving Issues Managing LDAP Active Directory System Performance Shell Manage Content Versioning Disaster Recovery Collaboration Communication Able to lead Initiative Reliability Big Data Collaborating Operating System Production Linux Hive Mahout Elastic Oozie Linux Performance Tuning Capacity Planning Networking Infrastructure Firewalls VLANs Load Balancers Tableau HUB Tensorflow Scripting Python Ruby Automation Docker Jenkins Puppet NoSQL Hbase Mongo Frameworks Ganglia Nagios Cloud Platforms Cloud AWS EMR Azure SQL Java Scala Written Communications Documentation Time Management Organizational Management

Similar Jobs