Senior QA Engineer, AI Agents & Agentic Workflows
Own the continuous validation and quality of multi-agent AI products.
Role Overview
QAD is seeking a Senior QA Engineer to own the continuous validation and quality of Champion Assist, a multi-agent AI product with approximately 20 evolving agents.
The role focuses on testing the agents themselves: their reasoning, tool use, orchestration, handoffs, and response quality. It goes beyond traditional UI and API checks by requiring evaluation frameworks and regression suites designed for non-deterministic, LLM-driven systems.
Roles & Responsibilities
End-to-End Agentic Workflow Testing
- Design, run, and maintain end-to-end business scenarios that exercise multi-step agentic workflows.
- Test agent routing, tool/function calls, and multi-turn conversations.
- Identify cross-agent, orchestration, and integration issues.
- Detect incorrect agent selection, broken handoffs, tool-call failures, context loss, looping, and runaway behaviour.
AI Response Quality Evaluation
- Evaluate agent outputs for correctness, relevance, groundedness, consistency, tone, and completeness.
- Use both human review and automated evaluation methods.
- Define pass/fail criteria, evaluation rubrics, and acceptance thresholds for AI responses.
- Quantify quality changes and regressions using relevant metrics.
Evaluation Datasets & Regression Automation
- Build and maintain evaluation datasets, including golden sets, edge cases, and adversarial prompts.
- Maintain automated regression suites as agents, prompts, models, and tools evolve.
- Detect regressions caused by model upgrades, prompt changes, and retrieval or knowledge changes.
- Keep evaluation suites current as the product evolves.
AI Security & Responsible Behaviour Testing
- Test for hallucinations, prompt injection, jailbreaks, and data leakage.
- Identify unsafe or out-of-scope responses.
- Validate the quality and safety of agent responses across relevant business scenarios.
Defect Analysis & Stakeholder Collaboration
- Triage issues and perform root-cause analysis with Engineering and Product teams.
- Separate prompt, model, retrieval, tool, and orchestration causes.
- Re-test fixes and validate the end-user experience.
- Document test strategy, evaluation results, and quality trends clearly for global stakeholders.
Qualifications & Required Experience
Education & Professional Experience
- Bachelor's degree in Software Engineering, Computer Science, or a related field.
- 3–5 years of QA experience.
- At least 1–2 years of experience testing LLM-based applications, AI agents, or agentic workflows.
AI Evaluation & Agentic Concepts
- Hands-on experience evaluating non-deterministic outputs.
- Ability to define pass/fail criteria, evaluation rubrics, and acceptance thresholds for AI responses.
- Experience building or maintaining AI evaluation or regression suites.
- Familiarity with LLM-as-judge approaches, golden datasets, and scoring rubrics.
- Understanding of tool/function calling, multi-agent orchestration, planning, memory/context, and RAG.
Test Automation & API Testing
- Experience with test automation tools such as Selenium, Playwright, TestComplete, or similar tools.
- Experience in API testing.
Key Technical Skill Areas
AI Agents & Workflows
Agent routing, multi-agent orchestration, tool/function calling, planning, multi-turn conversations, memory and context.
LLM Evaluation
Non-deterministic output evaluation, LLM-as-judge, golden datasets, scoring rubrics, groundedness and response quality.
Regression & Test Automation
Evaluation datasets, automated regression suites, Selenium, Playwright, TestComplete or similar tools.
RAG & Retrieval
Retrieval and knowledge changes, context handling, response grounding, and retrieval-related regression detection.
AI Security Testing
Hallucination testing, prompt injection, jailbreaks, data leakage, unsafe responses, and adversarial prompts.
Defect Investigation
Root-cause analysis across prompts, models, retrieval, tools, orchestration, and integrations.
Ideal Candidate
A QA Engineer with 3–5 years of testing experience, including at least 1–2 years working with LLM-based applications, AI agents, or agentic workflows. The ideal candidate can evaluate non-deterministic AI responses, build repeatable evaluation and regression suites, and investigate failures across multi-agent systems.
- Understands how agents select tools, exchange context, and coordinate workflows.
- Can assess response quality using measurable criteria and human review.
- Can identify security and reliability issues specific to LLM-powered applications.
- Can collaborate with Engineering and Product teams to investigate and resolve defects.
About QAD
QAD is building a SaaS platform focused on solving real-world challenges in manufacturing and supply chains. Together with Redzone, QAD is developing an intelligent platform connecting people, processes, and data.
Its platform includes Redzone for frontline empowerment, Adaptive Applications as the intelligent backbone, and Champion AI for Agentic AI in manufacturing.
QAD describes itself as a virtual-first company. The primary work experience is virtual or home-based, with occasional travel to a physical office potentially required for collaboration, design, strategy, and alignment.
What QAD Offers
Work Environment & Growth Opportunities
- Health and well-being programmes that support work-life balance.
- Opportunity to join a growing business entering its next phase of expansion and transformation.
- A collaborative culture where colleagues support one another.
- An environment that encourages idea-sharing regardless of level or hierarchy.
- Compensation packages based on experience and desired skill set.
Equal Opportunity Employer
QAD states its commitment to an inclusive workplace that values employee contributions and diverse perspectives. The company identifies itself as an Equal Opportunity Employer and states that it does not discriminate against employees or applicants on the protected grounds described in its job listing.
Ready to apply to QAD?
Explore the opportunity and submit your application through LinkedIn.