Lead Data Engineer

Alpheya

We're looking for a skilled Lead Data Engineer to contribute to the Python services that power our data platform. You'll help build and maintain high-throughput ETL pipelines that ingest real-time market data and expose it through secure CRUD APIs. Working alongside other backend and data engineers, you'll make sure data flows smoothly from external sources into our internal data platform.

Leadership: Lead a team of 2-3 Data Engineers to deliver production ready ETL pipelines and APIs exposing RAG and Agents capabilities.

• Market-Data ETL Pipelines: Implement, test, and improve scalable ETL workflows that handle large volumes of data.
• Async Orchestration & Scheduling: Develop and maintain task queues with Async tools, helping the team meet SLAs and surface operational metrics.
• API Implementation: Build REST endpoints with FastAPI Framework, providing clean CRUD access to market data and configuration entities.
• Observability & Reliability: Add structured logs, traces, and metrics; contribute alerts and dashboards (Prometheus/Grafana, OpenTelemetry) that keep services at 99.9 % + availability.
• Code Quality & Best Practices: Write type-safe, well-tested code, participate in code reviews, and help maintain CI/CD workflows.
• Cross-Functional Collaboration: Work closely with data scientists, and product managers to translate data requirements into robust backend capabilities.

• Education - Master's degree in Computer Science, Software Engineering, or a related field (or equivalent practical experience).
• Experience - 5+ years building production Python backends for data-intensive applications.
• Python Mastery - Modern Python (3.10 +) with asyncio, typing, dependency injection, and packaging best practices.
• ETL Skills - Experience designing batch or streaming pipelines; comfort with Kafka, RabbitMQ, or similar tools.
• Database Skills - Strong Postgres knowledge: migrations (Alembic, Django ORM), query tuning, partitioning, backup/restore.
• Async Orchestration - Hands-on with async tools like Airflow, Dagster, or Celery for scheduling and monitoring jobs.
• API Engineering - Proven ability to implement secure, versioned REST/GraphQL APIs with auth, rate-limiting, and RBAC.
• Testing & DevOps - Familiarity with Pytest, Docker, Kubernetes, Terraform, and GitHub Actions (or similar).
• Soft Skills - Clear written and verbal communication; collaborative mindset.

Nice-to-Haves

• Time-series or columnar DBs (TimescaleDB, ClickHouse, InfluxDB).
• Knowledge of financial market-data protocols (FIX, FAST) or regulatory feeds.