Senior Data Engineer - Pyspark

Valuelabs

This role requires an understanding of various data engineering, management and processing related technologies and choices, has a deep understanding of both SQL/No-SQL technologies, distributed computing framework, and techniques to make right tools and technology choices.

Role & responsibilities

• More than 8+ years of experience in information technology, with 3+ years spent in data engineering, architecture and technology solutions definitions and implementations. Extensive experience in banking services or related domain
• Expertise in Data Architecture, Data Strategy and Roadmap for large and complex organization and systems and implemented large scale end-to-end Data Management & Analytics solutions
• Experience in transforming traditional Data Warehousing approaches to Big Data based approaches and proven track record of managing risks and data security
• Expertise with DW Dimensional modeling techniques, Star & Snowflake schemas, modeling slowly changing dimensions and role playing dimensions, dimensional hierarchies, and data classification
• Experiences in cloud native principals, designs and deployments.
• Extensive experience working with and enhancing Continuous Integration (CI) and Continuous Development (CD) environments Expertise in Data Quality, Data Profiling, Data Governance, Data Security, Metadata Management, and Data Archival
• Define workload migration strategies using appropriate tools
• Drive delivery in a matrixed environment working with various internal IT partners
• Demonstrated ability to work in a fast paced and changing environment with short deadlines, interruptions, and multiple tasks/projects occurring simultaneously
• Must be able to work independently and have skills in planning, strategy, estimation, scheduling,
• Strong problem solving, influencing, communication, and presentation skills, self-starter
• Experience with data processing frameworks and platforms (Hadoop, Presto, Tez, Hive, Spark etc.) Hands-on experience in work with bigdata platforms and related programming languages (e.g. Linux, Python, PySpark, GIT, Jenkins)