Quality Assurance Engineer

AI71

You will act as the bridge between Agile development and formal Systems Engineering. Your mandate is to build automated testing frameworks that not only verify software functionality but also validate AI behaviors against "Ground Truth" datasets. Working within a structured "Stage Gate" delivery model, you will ensure our AI agents pass the rigorous Test Readiness Reviews (TRR) and Functional Configuration Audits (FCA) required for deployment.

Key Responsibilities

• AI & LLM Validation (LeverEDGE)

• Non-Deterministic Testing: Architect automated frameworks to evaluate Generative AI outputs (e.g., drafted CONOPS, technical requirements) for hallucination, consistency, and factual accuracy against "Gold Standard" datasets.
• RAG Evaluation: Implement automated metrics (e.g., RAGAS, faithfulness, answer relevance, etc.) to verify that the Retrieval-Augmented Generation pipeline is accurately citing internal technical documentation and regulatory texts.
• Prompt Regression: Design regression suites that monitor "prompt drift," ensuring that changes to the underlying model or system instructions do not degrade the quality of AI-generated engineering documents.

• Integration & System Verification (Supply Chain)

• ERP Integration Testing: Build robust integration tests to validate data consistency between AI agents and critical enterprise systems (e.g., SAP S/4HANA, Ariba, etc.), ensuring no corruption of Bill of Materials (BOM) or financial data.
• Performance Benchmarking: Design performance tests to validate the latency and throughput of forecasting models and risk scoring engines, ensuring they meet the real-time requirements of supply chain dashboards.
• API Validation: Automate the testing of secure API gateways, verifying that Role-Based Access Control (RBAC) and PII redaction logic are functioning correctly before data reaches the AI models.

• Governance & Traceability

• V-Model Alignment: Map automated test cases directly to "System Requirements" and "User Needs," creating the digital evidence required for formal Verification and Validation (V&V) reports described in the Systems Engineering Handbook.
• Stage Gate Compliance: Prepare "Test Readiness" packages for formal Stage Gate reviews, providing quantitative evidence that the system is stable enough to move from MVP to Production.
• Defect Lifecycle Management: Manage the feedback loop between the "Requirements Quality Assistant" and the development teams, ensuring that defects found in AI logic are traced back to specific model versions or data sets.

Technical Requirements

• Core Automation: Expert proficiency in Python for building custom test harnesses (Pytest) and standard automation libraries (Selenium/Playwright for UI, Requests for API).
• Core Performance Testing: Expert proficiency in crafting Performance Test Plans + Implementations (e.g., Locust, Jmeter, K6, etc.)
• AI Evaluation: Experience utilizing frameworks for evaluating LLMs (e.g., DeepEval, TruLens, or custom Python evaluators). Understanding of "Ground Truth" dataset creation and management.
• Data Validation: Proficiency with SQL and data validation tools (e.g., Great Expectations) to verify data quality within Data Lakehouses and Vector Databases.
• CI/CD Integration: Strong experience integrating automated tests into GitLab CI/CD pipelines, enforcing "Quality Gates" that prevent non-compliant code or models from merging.
• Traceability Tools: Familiarity with requirements management tools (e.g., Jira, Linear, Jama, Polarion, etc.) and how to link automated test results to specific requirement IDs.
• Test Management/Reporting Tools: Strong hands-on on Managing Test Reports + Artifacts (e.g., TestRail, Allure, etc.)
• Version Control: String knowledge maintaining code based frameworks (e.g., Git, Gitlab, etc.)
• Quality Engineering practices: Strong hands-on knowledge on modern Quality Engineering best practices designated for fast-paced development environments (e.g., Shift Left Approaches, Test Pyramid, Mono-repo architecture for automation projects)