QA and AI Experience
6+ years in Software QA/Testing, including 2+ years in AI/ML or LLM-based systems and hands-on agentic AI testing.
Qentelli Solutions · AI Quality Engineering
AI Quality Assurance Engineer
Own end-to-end quality engineering for agentic and multi-agent AI systems, validating reliability, resiliency, accuracy, latency, orchestration, and production readiness.
Apply on LinkedIn
Qentelli Solutions · AI Quality Assurance
Job Summary
Quality Engineering for Production-Grade Agentic AI
Qentelli Solutions is seeking a hands-on AI Engineer to design and execute end-to-end testing strategies for agentic AI solutions, including multi-agent systems in production-grade environments.
Working with the Agentic Operations Team, this role focuses on resiliency, reliability, accuracy, latency, orchestration correctness, and scale. The position also establishes QA frameworks, reusable test artifacts, and macro-level validations across complex workflows.
What You Will Do
Key Responsibilities
Lead QA strategy and engineering practices across development, staging, and production.
QA Strategy and Team Leadership
- Define and own the QA strategy for agentic and multi-agent AI systems across development, staging, and production.
- Mentor QA engineers and establish testing standards, coding guidelines for test harnesses, and review practices.
- Partner with Agentic Operations, Data Science, MLOps, and Platform teams to embed QA in the SDLC and incident response.
- Lead the QA function for Agentic AI from development through production readiness.
Agentic and Multi-Agent System Testing
- Design tests for agent orchestration, tool calling, planner-executor loops, and inter-agent coordination.
- Validate task decomposition, handoff integrity, and convergence toward intended goals.
- Validate state management, context windows, memory and knowledge stores, and prompt or graph correctness under varying conditions.
- Implement scenario fuzzing using adversarial inputs, prompt perturbations, tool latency spikes, and degraded APIs.
Reliability, Resiliency and Latency
- Create resilience testing suites covering chaos experiments, failover, retries and backoff, circuit-breaking, and degraded-mode behaviour.
- Establish latency SLOs and measure end-to-end response times across orchestration layers, including LLM calls, tool invocations, and queues.
- Ensure reliability through soak tests, canary verifications, and automated rollbacks.
- Test distributed-system behaviour, latency profiling, and resiliency patterns under failure conditions.
Accuracy and Macro-Level Validation
- Define ground-truth and reference pipelines for task accuracy using exact match, semantic similarity, and factuality checks.
- Build macro validation frameworks to validate task outcomes across multi-step agent workflows.
- Validate complex workflows such as data pipelines and content-generation plus verification-agent loops.
- Instrument guardrail validations for toxicity, PII, hallucinations, and policy compliance.
Scale, Orchestration and Test Automation
- Design load and stress tests for multi-agent graphs, covering concurrency, throughput, queue depth, and backpressure.
- Validate orchestrator correctness, including DAG execution, retries, branching, timeouts, and compensation paths.
- Engineer reusable test artifacts such as scenario configurations, synthetic datasets, prompt libraries, agent graph fixtures, and simulators.
- Integrate tests into CI/CD through pre-merge gates, nightly runs, and canary workflows.
- Connect production monitoring with alerting tied to relevant KPIs.
Production Readiness and Release Governance
- Define release criteria and conduct operational readiness assessments.
- Validate performance, security, compliance, and cost and latency budgets before release.
- Work with cross-functional teams to ensure quality outcomes throughout the software delivery lifecycle.
Required Qualifications
Technical Skills and Experience
Core capabilities expected for hands-on AI quality engineering.
Python Engineering
Strong Python programming skills for building test harnesses, simulators, and reusable fixtures.
LLM Evaluation
Exact and soft match, BLEU/ROUGE, BERTScore, embedding-based semantic similarity, guardrails, and prompt testing.
Distributed Systems
Latency profiling, resiliency patterns, circuit breakers, retries, chaos engineering, and message queues.
AI Orchestration
Familiarity with LangChain, LangGraph, LlamaIndex, DSPy, OpenAI Assistants/Actions, Azure OpenAI orchestration, or similar.
CI/CD and Observability
GitHub Actions or Azure DevOps, OpenTelemetry, Prometheus/Grafana, Datadog, feature flags, and canaries.
AI Security and Privacy
Understanding of PII handling, content policies, privacy, compliance, and model safety.
Leadership and Collaboration
Strong communication and leadership skills, with cross-functional experience across Operations, Data, and Engineering.
Preferred Qualifications
Additional Experience That Helps
These are preferred capabilities rather than stated mandatory requirements.
Agent Simulation and Graph Testing
- Experience with multi-agent simulators and agent graph testing.
- Knowledge of tooling latency emulation.
MLOps and LLM Experimentation
- Knowledge of model versioning, datasets, and evaluation pipelines.
- Experience with A/B experimentation for LLMs.
Cloud and Modern Architecture
- Background in AWS and serverless technologies.
- Experience with containerization and event-driven architectures.
Production Cost, Latency and SLAs
- Prior ownership of cost, latency, or service-level agreements for AI workloads in production.
Ideal Candidate
Who Should Apply?
A hands-on QA professional with 6+ years of software testing experience and at least 2 years working with AI/ML or LLM-based systems, including agentic and multi-agent architectures.
The strongest fit combines Python test engineering, LLM evaluation, distributed-system resilience testing, orchestration validation, and the ability to lead QA practices from development to production.
Core Focus Areas
What the Role Emphasizes
Agentic workflow correctness, multi-agent coordination, reliability, resiliency, latency, macro-level accuracy, scale, and production readiness.
The role also emphasizes reusable test assets, CI/CD integration, observability, operational readiness, and cross-functional ownership.
Ready to Apply?
Review the full job description and submit your application through LinkedIn.