Lead QA Analyst – Agentic AI
Designation: Principal Lead Engineer
Role Overview
The Senior Agentic AI QA Lead / Principal owns the quality strategy for Agentic AI platforms. This role focuses on defining AI testing strategies, evaluating systemic risks, validating autonomous behaviours, and ensuring production readiness of agentic systems.
The position combines advanced AI testing expertise with quality leadership, risk assessment, automation strategy, and team mentoring.
Roles & Responsibilities
AI Quality Strategy & Test Leadership
- Define end-to-end testing strategies for agentic and autonomous AI systems.
- Design complex test scenarios for reasoning chains, tool orchestration, and multi-agent collaboration.
- Review and approve AI test cases, defect reports, and validation evidence.
- Provide release recommendations based on risk-based quality assessments.
- Establish testing standards and review processes for AI systems.
AI Safety, Risk & Governance
- Lead testing for hallucinations, bias, ethical risks, and unsafe behaviours.
- Validate human-in-the-loop controls, auditability, and traceability.
- Partner with engineering, product, and compliance teams on risk mitigation.
- Assess business and compliance risks associated with AI decisions.
- Define evaluation metrics for accuracy, reliability, and safety.
- Apply knowledge of AI governance, model risk management, and guardrails.
Autonomous Workflow & Performance Testing
- Define and conduct scale and performance testing for AI agents.
- Evaluate planning failures, looping behaviours, and dead-end scenarios.
- Validate autonomous escalation and self-correction mechanisms.
- Test long-running agents and stateful workflows.
- Define acceptance criteria for probabilistic outputs.
Team Leadership & Capability Development
- Mentor QA teams on advanced Agentic AI testing techniques.
- Train Sycamore resources in Agentic AI skills, where applicable.
- Guide automation framework ownership and testing strategy.
- Support teams in adopting consistent AI validation practices.
Essential Experience – Advanced Agentic AI Testing
Agent Frameworks & Orchestration
- Deep understanding of agent frameworks and orchestration layers.
- Expertise in testing autonomous workflows and decision policies.
- Strong experience testing API-driven and distributed agent architectures.
- Experience validating tool orchestration and multi-agent collaboration.
Advanced AI Evaluation & Adversarial Testing
- Advanced prompt evaluation, adversarial testing, and red-teaming.
- Experience evaluating planning failures, looping behaviours, and dead-ends.
- Ability to design evaluation metrics for accuracy, reliability, and safety.
- Experience reviewing logs, traces, and telemetry for agent behaviour analysis.
- Knowledge of AI governance, model risk management, and guardrails.
Automation & Engineering Tools
- Exposure to Python-based automation.
- Exposure to Robot Framework.
- Experience or exposure to API automation.
Deep-Dive Testing Experience – Senior Level
Advanced Testing Capabilities
- Designing AI test strategies under uncertainty and non-determinism.
- Validating autonomous escalation and self-correction mechanisms.
- Testing long-running agents and stateful workflows.
- Defining acceptance criteria for probabilistic outputs.
- Assessing business and compliance risks of agent decisions.
- Establishing testing standards and review processes for AI systems.
Desired Experience
The following are listed as desired experience, separate from the essential technical requirements.
Preferred Skills & Exposure
- Experience defining AI validation approaches under CSA or GxP.
- Automation framework ownership or strategy experience.
- Experience in AI governance, audits, or regulatory inspections.
- Experience working with LLMs such as Claude through AWS Bedrock.
Key Technical Skill Areas
Agentic AI & Orchestration
Agent frameworks, orchestration layers, autonomous workflows, decision policies, multi-agent collaboration.
AI Testing & Evaluation
Prompt evaluation, adversarial testing, red-teaming, probabilistic output validation, accuracy and reliability metrics.
Automation & API Testing
Python, Robot Framework, API automation, API-driven systems, and distributed agent architectures.
AI Risk & Governance
AI safety, bias, ethical risks, guardrails, model risk management, auditability, and traceability.
Performance & Reliability
Scale testing, performance testing, long-running agents, stateful workflows, planning failures, and loop detection.
LLM Platforms & Validation
LLM evaluation, Claude, AWS Bedrock, CSA, GxP, and AI validation approaches.
Education
A Bachelor's degree is preferred, particularly in Computer Science, Electrical Engineering, Physics, Mathematics, or another related discipline.
Ideal Candidate
A senior QA professional with 9–15 years of experience who can own quality strategy for autonomous AI systems, evaluate complex agent behaviours, identify systemic risks, and establish reliable testing practices for production readiness.
- Can lead end-to-end quality strategy for agentic and autonomous AI platforms.
- Understands agent reasoning, planning, orchestration, and multi-agent collaboration.
- Can assess AI safety, governance, compliance, and operational risks.
- Has experience with non-deterministic AI testing and probabilistic output evaluation.
- Can mentor QA teams and build advanced AI testing capabilities.
Ready to lead Agentic AI quality engineering?
Review the complete opportunity and submit your application through LinkedIn.