Lead QA Analyst – Agentic AI
Designation: Principal Lead Engineer
Role Overview
The Senior Agentic AI QA Lead / Principal is responsible for owning the quality strategy for Agentic AI platforms. The role focuses on defining AI testing strategies, evaluating systemic risks, validating autonomous behaviours, and ensuring production readiness of agentic systems.
Roles & Responsibilities
AI Quality Strategy & Test Leadership
- Define end-to-end testing strategies for agentic and autonomous AI systems.
- Design complex test scenarios for reasoning chains, tool orchestration, and multi-agent collaboration.
- Review and approve AI test cases, defect reports, and validation evidence.
- Establish testing standards and review processes for AI systems.
- Provide release recommendations based on risk-based quality assessments.
AI Risk, Safety & Governance
- Lead testing for hallucinations, bias, ethical risks, and unsafe behaviours.
- Validate human-in-the-loop controls, auditability, and traceability.
- Partner with engineering, product, and compliance teams on risk mitigation.
- Assess business and compliance risks associated with AI-driven decisions.
- Define evaluation metrics for accuracy, reliability, and safety.
Performance, Scale & Autonomous Workflows
- Define and conduct scale and performance testing for AI agents.
- Validate autonomous workflows, decision policies, and self-correction mechanisms.
- Evaluate planning failures, looping behaviours, and dead-end scenarios.
- Test long-running agents and stateful workflows.
- Validate autonomous escalation mechanisms and define acceptance criteria for probabilistic outputs.
Team Leadership & Capability Development
- Mentor QA teams on advanced Agentic AI testing techniques.
- Train Sycamore resources in Agentic AI skills, where applicable.
- Own or guide automation framework strategy and testing standards.
- Collaborate with cross-functional stakeholders to improve AI quality practices.
Essential Experience & Technical Skills
Advanced Agentic AI Testing
- Deep understanding of agent frameworks and orchestration layers.
- Expertise in testing autonomous workflows and decision policies.
- Experience evaluating planning failures, looping behaviours, and dead-ends.
- Advanced prompt evaluation, adversarial testing, and red-teaming.
- Strong experience testing API-driven and distributed agent architectures.
- Ability to design evaluation metrics for accuracy, reliability, and safety.
- Experience analysing agent behaviour through logs, traces, and telemetry.
- Knowledge of AI governance, model risk management, and guardrails.
- Exposure to automation using Python, Robot Framework, and API automation.
Senior-Level Deep-Dive Testing
- Designing AI test strategies under uncertainty and non-determinism.
- Validating autonomous escalation and self-correction mechanisms.
- Testing long-running agents and stateful workflows.
- Defining acceptance criteria for probabilistic outputs.
- Assessing business and compliance risks of agent decisions.
- Establishing testing standards and review processes for AI systems.
Desired Experience
The following are listed as desired experience rather than mandatory essential requirements in the supplied job description.
Preferred Qualifications
- Experience defining AI validation approaches under CSA or GxP.
- Automation framework ownership or strategy experience.
- Experience in AI governance, audits, or regulatory inspections.
- Experience working with LLMs such as Claude through AWS Bedrock.
Key Skill Areas
Agentic AI & Orchestration
Agent frameworks, orchestration layers, autonomous workflows, decision policies, multi-agent collaboration.
AI Testing & Evaluation
Prompt evaluation, adversarial testing, red-teaming, probabilistic output validation, accuracy and reliability metrics.
Automation & Architecture
Python, Robot Framework, API automation, API-driven systems, distributed agent architectures.
AI Risk & Governance
AI safety, bias, ethical risks, guardrails, model risk management, auditability, traceability, CSA and GxP exposure.
Performance & Reliability
Scale testing, performance testing, long-running agents, stateful workflows, planning failure and loop detection.
LLM Platforms
LLM exposure, Claude, and AWS Bedrock experience.
Ideal Candidate
A senior QA professional with 9–15 years of experience who can lead quality strategy for autonomous AI systems, evaluate complex agent behaviours, identify systemic risks, and establish reliable testing practices for production readiness.
- Comfortable leading AI quality strategy and reviewing validation evidence.
- Experienced in testing non-deterministic, stateful, and distributed systems.
- Able to assess AI safety, governance, compliance, and operational risks.
- Capable of mentoring teams and building Agentic AI testing capabilities.
Education
A Bachelor's degree is preferred, particularly in Computer Science, Electrical Engineering, Physics, Mathematics, or another related discipline.
Ready to explore this opportunity?
Review the complete job details and submit your application through LinkedIn.