Quality Assurance Engineer

AI71

Non-Deterministic Testing: Architect automated frameworks to evaluate Generative AI outputs for hallucination, consistency, and factual accuracy against "Gold Standard" datasets.

RAG Evaluation: Implement automated metrics (e.g., RAGAS, faithfulness, answer relevance) to verify that Retrieval-Augmented Generation pipelines accurately cite technical and regulatory documentation.

Prompt Regression: Design regression suites to monitor "prompt drift," ensuring model updates do not degrade the quality of AI-generated engineering documents.

Enterprise Integration: Build robust tests to validate data consistency between AI agents and critical systems (e.g., SAP S/4HANA, Ariba), ensuring the integrity of Bill of Materials (BOM) and financial data.

Performance Benchmarking: Design tests to validate latency and throughput for forecasting models and risk-scoring engines using tools like Locust, JMeter, or K6.

API & Security Validation: Automate testing of secure API gateways, verifying Role-Based Access Control (RBAC) and PII redaction logic before data reaches AI models.

V-Model Alignment: Map automated test cases to "System Requirements" to create digital evidence for formal Verification and Validation (V&V) reports.

Stage Gate Compliance: Prepare "Test Readiness" packages for formal reviews, providing quantitative evidence that systems are stable enough to move from MVP to Production.

Defect Lifecycle Management: Manage the feedback loop between Requirements Quality Assistants and development teams, tracing AI logic defects back to specific model versions.

Core Automation: Expert proficiency in Python (Pytest) and standard libraries (Selenium/Playwright, Requests).

AI Evaluation: Hands-on experience with LLM evaluation frameworks (e.g., DeepEval, TruLens) and "Ground Truth" dataset management.

Performance Engineering: Proficiency in crafting Performance Test Plans and implementations (Locust, K6, etc.).

Data Validation: Expertise in SQL and data quality tools (e.g., Great Expectations) for Data Lakehouses and Vector Databases.

CI/CD & DevOps: Strong experience integrating quality gates into GitLab CI/CD pipelines.

Engineering Practices: Deep understanding of modern QE practices, including Shift Left, Test Pyramid, and Mono-repo architectures.