Agentic AI quality strategy

Lead QA - Agentic AI

Own the quality strategy for Agentic AI platforms: define AI test strategies, evaluate systemic risks, validate autonomous behaviors and ensure production readiness.

Role: Lead QA Analyst – Agentic AI
Designation: Principal Lead Engineer
Experience: 9 – 15 years

Apply on LinkedIn

What you will own. Select a card to jump to its details.

Test strategyEnd-to-end, for agentic and autonomous AI systems Risk and safetyHallucinations, bias, ethical risks and unsafe behaviors Advanced agentic AI testingFrameworks, orchestration and red-teaming Senior deep-dive testingNon-determinism, long-running agents and stateful workflows

Agentic AIMulti-agent systemsRed-teamingPrompt evaluationHuman-in-the-loopAI governanceGuardrailsModel risk managementTelemetryAPI automationRobot FrameworkPythonClaudeAWS BedrockAgentic AIMulti-agent systemsRed-teamingPrompt evaluationHuman-in-the-loopAI governanceGuardrailsModel risk managementTelemetryAPI automationRobot FrameworkPythonClaudeAWS Bedrock

Summary

The Senior Agentic AI QA Lead Principal owns the quality strategy for Agentic AI platforms.

This role focuses on defining AI test strategies, evaluating systemic risks, validating autonomous behaviors, and ensuring production readiness of agentic systems.

Roles & responsibilities

Define end-to-end testing strategy for agentic and autonomous AI systems
Design complex test scenarios for reasoning chains, tool orchestration, and multi-agent collaboration
Lead testing for hallucinations, bias, ethical risks, and unsafe behaviors
Validate human-in-the-loop controls, auditability, and traceability
Review and approve AI test cases, defect reports, and validation evidence
Partner with engineering, product, and compliance teams on risk mitigation
Mentor QA teams on agentic AI testing techniques
Provide release recommendations based on risk-based quality assessments
Define and conduct any scale/performance testing for AI Agents built as well
Train Sycamore resources with Agentic AI skills where applicable

Who we are looking for

Principal Lead Engineer with 9 – 15 years of experience. Bachelor’s degree, preferably in Computer Science, Electrical Engineering, Physics, Math or any other related discipline.

Python
Robot Framework
API automation
Claude
AWS Bedrock

Essential experience

Technical skills – advanced agentic AI testing9 items

Deep understanding of agent frameworks and orchestration layers
Expertise in testing autonomous workflows and decision policies
Experience evaluating planning failures, looping behaviors, and dead-ends
Advanced prompt evaluation, adversarial testing, and red-teaming
Strong experience testing API-driven and distributed agent architectures
Ability to design evaluation metrics for accuracy, reliability, and safety
Experience reviewing logs, traces, and telemetry for agent behavior analysis
Knowledge of AI governance, model risk management, and guardrails
Exposure to automation (Python, Robot Framework, API automation)

Deep-dive testing experience (senior)6 items

Designing AI test strategies under uncertainty and non-determinism
Validating autonomous escalation and self-correction mechanisms
Testing long-running agents and stateful workflows
Defining acceptance criteria for probabilistic outputs
Assessing business and compliance risks of agent decisions
Establishing testing standards and review processes for AI systems

Desired experience

Experience defining AI validation approaches under CSA or GxP
Automation framework ownership or strategy experience
Experience in AI governance, audits, or regulatory inspections
Working with any LLMs like Claude with AWS Bedrock

Ready to apply?

See the full job description and apply on LinkedIn.

Apply on LinkedIn