Agentic AI quality assurance

QA Agentic AI

Validate the quality, reliability, safety and correctness of Agentic AI systems, including autonomous agents, multi-step reasoning flows, tool orchestration and human-in-the-loop controls.

Role: QA – Agentic AI
Designation: Engineer - Testing
Experience: 2 - 5 years total experience

Apply on LinkedIn

What you will test. Select a card to jump to its details.

Agentic workflowsPlanning, reasoning, action and reflection loops Tool calling and oversightAPIs, databases, RPA and human-in-the-loop checkpoints Technical skillsAgentic architectures, multi-agent systems and RAG Deep-dive testingMemory, tool misuse, consistency and guardrails

Summary

The Agentic AI QA Analyst is responsible for validating the quality, reliability, safety, and correctness of Agentic AI systems. This includes testing autonomous agents, multi-step reasoning flows, tool orchestration, and human-in-the-loop controls.

The role requires strong analytical skills to evaluate non-deterministic AI behavior.

Roles & responsibilities

Test agentic AI workflows involving planning, reasoning, action, and reflection loops
Validate autonomous decision-making paths and goal completion behavior
Test prompt chaining, context persistence, and memory handling
Validate tool calling (APIs, databases, search, RPA, internal services)
Design tests for hallucinations, partial failures, retries, and recovery logic
Validate human-in-the-loop checkpoints, approvals, overrides, and escalation paths
Perform functional, exploratory, regression, and scenario-based testing
Document defects with reproducible prompts, agent state, tools invoked, and outputs
Support risk-based testing and compliance requirements (CSA / GxP where applicable)
Under the Sycamore Product and coordinate with different teams to get the work done

Who we are looking for

Engineer - Testing with 2 - 5 years of total experience. Bachelor’s degree, preferably in Computer Science, Electrical Engineering, Physics, Math or any other related discipline.

Python
Robot Framework
API automation
REST/GraphQL
Vector databases
RAG

Essential experience

Technical skills – Agentic AI testing9 items

Understanding of Agentic AI architectures (Planner–Executor, ReAct, AutoGPT-style agents)
Experience testing multi-agent systems and agent-to-agent communication
Prompt testing and prompt-chain validation techniques
Testing non-deterministic outputs using evaluation heuristics and acceptance bands
Experience validating LLM tool calling and function calling
API testing (REST/GraphQL) for agent-integrated services
Ability to inspect logs, traces, and agent execution graphs
Familiarity with vector databases, embeddings, and retrieval-augmented generation (RAG)
Understanding of model limitations, hallucination patterns, and AI failure modes

Deep-dive testing experience (Agentic AI)7 items

Testing planning accuracy and goal decomposition across multi-step tasks
Validating agent memory (short-term vs long-term) and context carryover
Testing tool misuse, incorrect tool selection, and fallback mechanisms
Evaluating output consistency across multiple runs
Testing edge cases such as ambiguous prompts, conflicting goals, and incomplete data
Validating explainability and reasoning traces where available
Assessing bias, unsafe outputs, and guardrail effectiveness

Desired experience

Exposure to automation (Python, Robot Framework, API automation)
Experience with AI evaluation frameworks or red-teaming
Experience in regulated environments (GxP, CSA, 21 CFR Part 11)

Ready to apply?

See the full job description and apply on LinkedIn.

Apply on LinkedIn