Big Data Engineer
The big data engineer implements and maintains the information architecture for the mobility data business based on the end-to-end vision of the data architect.
The big data engineer develops, supervises and maintains data integration processes into the data lake by building data pipelines and using state of the art ETL tools in alignment with business representatives and the business information needs.
The big data engineer executes master data management policies developed by the data architect. He also executes the data quality evaluations and works with business representatives in improving the data quality to the required levels
What are my responsibilities?
- Responsible for the integration of large, structured and unstructured data volumes into the existing cloud platforms
- Development of scalable end-to-end data pipelines for batch and stream processing
- Execution of the data integration activities (ETL / ELT) for populating the data lake and integrating diverse data sources
- Execution an further development of the physical implementation of the logical data model into a physical implementation in the data lake
- Implementation of solutions for reference data and master data management within the context of the mobility data business
- Execution of data quality measurements and implementation of data quality improvement activities to the required levels of data quality
- Support of build-up and maintenance of a data directory for all data relevant to the mobility data business
- Representation of the Data Architecture team in selected data architecture, data modeling, and metadata management work teams inside Mobility
What do I need to qualify for this job?
- University degree in an appropriate area (e.g. informatics)
- At least 2 years of relevant work experience
- Experience with modern big data technologies like Hadoop, MapReduce, Kafka, Hive, Presto, Spark, etc.
- Experience with cloud solutions like AWS
- Experience with programming languages like SQL, Scala, Python, Java
- Experience with enterprise application integration and with ETL approaches in one of the leading tool suites (e.g. Informatica, Talend)
- Creativity and lateral thinking
Job ID: 81370
Experience Level: Experienced Professional
Job Type: Full-time