Data Engineer

Midis Group

Main Duties and Responsibilities

• Design and build data pipelines to collect, store and process large and complex data sets from various sources including data ingestion, data cleansing, data validation and data transformation
• Work on Big-Data ecosystem including Hadoop, HDFS, Spark, Spark SQL, Hive, Impala, Talend and CFM
• Implement data storage and processing technologies such as data lakes, data warehouses and NoSQL databases to support data analytics and machine learning
• Optimize data pipeline performance, troubleshoot and debug related issues while monitoring data pipeline health
• Collaborate with Data Scientists and other stakeholders to understand data requirements and design data pipelines accordingly
• Create and maintain documentation for data pipelines, data models and data dictionaries
• Ensure data quality by implementing data validation and data cleansing processes
• Implement security measures to protect data and ensure compliance with regulations such as GDPR, HIPAA and PCI-DSS
• Research and evaluate new technologies and tools to improve data pipeline performance and scalability on a regular basis
• Apply performance optimization techniques and technical guidance to other Application Developers
• Provide advisory services for designing and implementing large, scalable distributed systems
• Use programming languages and technologies like Python, SQL, Java, Scala, Informatica, Hadoop, Spark and Hive
• Coordinate with multiple stakeholders on a regular basis.