Data Engineer

Institute Of Foundation Models

As a Data Engineer specializing in Natural Language Processing (NLP) and large-scale data processing, you will quickly and effectively gather, curate, and prepare high-quality datasets to support cutting-edge NLP research. Your role will be instrumental in enabling researchers by delivering essential data through efficient and scalable engineering practices, including web crawling, LLM-generated content refinement, and robust data pipelines, primarily leveraging Python and related technologies.

Key Responsibilities

• Rapidly collect, curate, and preprocess datasets based on detailed specifications provided by NLP researchers, delivering data within tight timelines (typically within 1-2 days).
• Develop and maintain efficient web crawling solutions, APIs, and automated workflows to continuously improve data collection processes.
• Refine and evaluate outputs from Large Language Models (LLMs) to generate structured datasets suitable for model training and benchmarking.
• Implement scalable data pipelines, ensuring efficient data processing, storage, retrieval, and distribution to research teams.
• Collaborate closely with researchers and engineers to ensure collected data meets specified quality and relevance criteria.
• Document data collection methodologies, dataset characteristics, and pipeline architecture clearly and effectively.
• Engage with peer teams and participate in technical reviews to uphold best practices and data quality standards.
• Represent MBZUAI at industry and research forums, showcasing technical capabilities in large-scale data processing and AI data infrastructure.
• Perform all other duties as reasonably directed by the line manager commensurate with these functional objectives.