AWS & Hadoop Consultant

Job Description

What are my responsibilities?

Implements complete automated environment on cloud providers like AWS using tools like terraform/ansible or any other.

Installation and configuration management of complete Hadoop clusters including security components like Ranger/Kerberos and vulnerability assessment.

Ensures adherence to defined guidelines, processes and procedures, within the team.

What else do I need to know?

Knowledge and Experience

* Planning, implementing and maintaining Hadoop Cluster using Cloudera Manager, Hortonworks and AWS EMR.

* Should have good experience on terraform/ansible.

* Installing and maintaining Hive, Spark,Kafka,flume along with Cloudera Manager.

* Optimizing Hadoop by managing memory for different tools in ecosystem.

* Proficient in Memory management (JVM, Heap, YARN).

* Cluster health monitoring and troubleshooting using Cloudera Manager.

* Actively participate with Hadoop Development team during project deployment and access management in the Hadoop environment.

* User Authentication management and Configuration.

* Creation and modification of Automation Script if any modification required.

* Sentry, Hue, Kerberos and Kerberos Administration.

* Provide response for functional queries related to application.

* Queries related to application and service workflow.

* Actively participate during release management and deployment.

* Commissioning and Decommissioning nodes to an existing Hadoop cluster

* Involved in bench marking Hadoop/Hbase cluster file systems various batch jobs and workloads.

* Good in analysing log files for Hadoop and ecosystem services and finding root cause.

* Verify HDFS data storage and running map-reduce jobs.

* Experience in importing and exporting the data using Sqoop to HDFS to Relational Database systems/mainframe and vice-versa.

* Good in scheduling Hadoop/Hive/Sqoop/Hbase jobs using Oozie.

* Experience building, configuring, and maintaining production environments, including standard networking preferably RHEL as OS.

* Expert knowledge of administration of Redhat Products, especially RHEL 6 & 7

* Expert knowledge of server side networking (TCP/UDP/Multicast).

* Experience using Amazon Web Services console, CLI, APIs. AWS Certifications are especially helpful

* Expert in AD integration and management and a little windows server management.

* * Experience developing solutions using infrastructure orchestration tools (Chef, Puppet, Salt, Ansible, Terraform etc.)

* Experience with UI/UX design for complex workflows and/or for technical end users / domain experts

* Experience working with common application architectures (app/web servers, containerization, etc.)

* Knowledge of IP Networking including VPC?s, DNS, load-balancing, and firewalls

* Understanding of microservice development and best practices

* Good to have hands-on experience with tools like Jenkins, Liquibase, Splunk, ELK

* Incident management

* Problem management like providing workaround in case cluster and services are not working as expected

* Service requests like user creation, user administration, automation job issues resolution.

* Troubleshooting in case of an application outage or failure of some critical component. Must be able to perform a detailed RCA and provide his findings/observation to management.

* Experience and knowledge on blockchain is preferred.

We are looking forward to receiving your online application. Please ensure you complete all areas of the application form to the best of you ability as we will use the data to review your suitability for the role.


Job ID: 111415

Organization: Siemens Operations

Experience Level: Experienced Professional

Job Type: Full-time

Can't find what you are looking for?

Let's stay connected

Can't find what you are looking for?