The Lead QA Analyst – Agentic AI is responsible for owning the quality strategy for Agentic AI platforms. The role focuses on defining AI testing strategies, evaluating systemic risks, validating autonomous behaviors and ensuring production readiness.
This is a senior QA leadership position requiring deep expertise in autonomous workflows, agent orchestration, probabilistic outputs, AI safety and risk-based quality assessment.
The strongest candidate will combine senior QA leadership with advanced Agentic AI testing expertise — particularly reasoning chains, tool orchestration, multi-agent collaboration, red teaming, autonomous decision-making and AI governance.
- Define end-to-end testing strategies for agentic and autonomous AI systems.
- Design complex test scenarios covering reasoning chains, tool orchestration and multi-agent collaboration.
- Lead testing for hallucinations, bias, ethical risks and unsafe AI behaviors.
- Validate human-in-the-loop controls, auditability and traceability.
- Review and approve AI test cases, defect reports and validation evidence.
- Partner with engineering, product and compliance teams on risk mitigation.
- Mentor QA teams on advanced Agentic AI testing techniques.
- Provide release recommendations based on risk-based quality assessments.
- Define and execute scale and performance testing for AI agents.
- Train and develop QA resources in Agentic AI testing skills.
Agent Frameworks
Deep understanding of agent frameworks and orchestration layers.
API Automation
Strong experience testing API-driven and distributed agent architectures.
Python
Automation and AI testing using Python.
Robot Framework
Exposure to Robot Framework-based test automation.
AI Evaluation
Design metrics for accuracy, reliability, safety and probabilistic outputs.
Observability
Analyze logs, traces and telemetry to understand agent behavior.
- Design AI test strategies that account for uncertainty and non-deterministic behavior.
- Define acceptance criteria for probabilistic AI outputs.
- Validate autonomous escalation and self-correction mechanisms.
- Test long-running agents and stateful workflows.
- Assess business and compliance risks associated with agent decisions.
- Establish testing standards and formal review processes for AI systems.
- Analyze logs, traces and telemetry for detailed agent behavior assessment.
Who Fits This Role?
A senior QA leader with 9–15 years of experience who has deep, hands-on expertise in testing Agentic AI and autonomous systems. The ideal candidate understands non-deterministic AI behavior, reasoning chains, agent orchestration, tool calling, multi-agent workflows, adversarial testing, AI safety and governance. Strong automation, risk assessment, stakeholder management and mentoring capabilities are essential.
Experience
9–15 years of professional experience with senior QA and AI testing leadership.
Designation
Principal Lead Engineer
Education
Bachelor's degree preferably in Computer Science, Engineering, Physics, Mathematics or a related discipline.