AI Quality Engineer, Safety and RAG
Assistant Vice President · AI/ML Quality Engineering
Role Overview
Citi is seeking a highly motivated and experienced AI Quality Engineer to join its Retail and Wealth Risk Engineering team under the Enterprise Risk Technology platform.
This role spans modern AI quality engineering, including Agentic AI flow testing, RAG pipeline validation, AI safety, test automation, and performance and reliability engineering.
The successful candidate will help ensure complex autonomous AI systems are safe, accurate, explainable, resilient, and production-ready at scale. The position requires depth in AI/ML and broad expertise across testing disciplines.
Roles & Responsibilities
Agentic AI Testing
- Design and execute end-to-end test strategies for Agentic AI pipelines, including single-agent and multi-agent workflows.
- Validate agent reasoning, planning, and decision-making chains, including ReAct, Chain-of-Thought, Plan-and-Execute, and Reflexion.
- Test tool-use correctness, ensuring agents invoke the right tools with correct parameters at the right time.
- Evaluate short-term, long-term, and episodic agent memory for accuracy and context retention across sessions.
- Validate agent handoff and delegation logic in orchestration frameworks such as AutoGen, CrewAI, and LangGraph.
- Test termination conditions, loop detection, and infinite-loop prevention in autonomous agent loops.
RAG – Retrieval-Augmented Generation Testing
- Design comprehensive end-to-end RAG test strategies covering ingestion, chunking, embedding, retrieval, reranking, and generation.
- Validate retrieval accuracy and relevance by checking whether the correct context chunks are retrieved for a given query.
- Test embedding model quality and vector similarity thresholds across different document corpora.
- Evaluate faithfulness, groundedness, and answer relevance using RAGAS, TruLens, and DeepEval.
- Test fixed, semantic, and hierarchical chunking strategies and their impact on retrieval quality.
- Validate context-window management to prevent token-limit issues and degradation in generation quality.
- Conduct end-to-end regression testing when the knowledge base, embedding model, or LLM changes.
- Test multi-turn conversational RAG for context coherence and citation accuracy across turns.
Test Automation
- Build and maintain automated test harnesses for Agentic AI and RAG systems.
- Implement agent trajectory replay, tool mock injection, and prompt simulation.
- Develop automated evaluation pipelines integrated into CI/CD workflows for continuous model and agent validation.
- Create data validation and data quality frameworks using Great Expectations, Deequ, or custom tooling.
- Build prompt regression suites to detect behavioural drift across LLM versions or prompt changes.
- Implement determinism and reproducibility tests for stochastic LLM-based decisions.
- Automate vector database validation, including index integrity, embedding drift, and retrieval consistency checks.
AI Safety & Security Testing
- Conduct red-teaming and adversarial testing to uncover jailbreaks, prompt injection vulnerabilities, and goal misalignment.
- Test output guardrails and content filters for unsafe, biased, toxic, or out-of-scope model behaviour.
- Validate privilege escalation controls to ensure agents do not exceed permitted actions or access unauthorized resources.
- Perform data poisoning and backdoor attack simulations to assess model robustness.
- Evaluate bias, fairness, and discrimination using AI Fairness 360 and Aequitas.
- Test PII leakage and data privacy controls in RAG and agent pipelines in accordance with GDPR, CCPA, and internal data governance policies.
- Conduct security testing aligned with the OWASP Top 10 for LLM Applications.
- Validate Constitutional AI constraints, RLHF-aligned behaviour boundaries, and system prompt integrity.
- Collaborate with cybersecurity teams on AI-specific threat modelling and vulnerability management.
- Maintain safety testing playbooks and document red-team findings with severity ratings and remediation recommendations.
Direct and indirect prompt injection, insecure output handling, training data poisoning, insecure plugin/tool design, and sensitive information disclosure.
Performance & Reliability Testing
- Define and execute load, stress, soak, and spike testing for AI-powered APIs, inference endpoints, and agent orchestration services.
- Measure and optimize end-to-end latency across RAG and agentic pipelines, from query to final response.
- Benchmark LLM inference throughput in tokens per second and identify bottlenecks across model-serving infrastructure.
- Test auto-scaling behaviour of AI services under variable load conditions.
- Validate circuit breaker, retry, and fallback mechanisms for graceful degradation.
- Test vector database performance, including query latency, index build time, and retrieval accuracy under high concurrency.
- Conduct cost-efficiency analysis of token consumption, API call costs, and infrastructure spend per agent task.
- Establish SLOs and SLAs for availability, P50/P95/P99 latency percentiles, and error rates.
- Collaborate with MLOps teams on observability dashboards, monitoring alerts, and automated anomaly detection.
- Perform chaos engineering experiments to validate resilience under infrastructure failures.
Domain Knowledge
RAG Architecture & Agent Design Patterns
- Deep understanding of naive RAG, advanced RAG, and modular RAG architecture patterns.
- Solid grasp of ReAct, Plan-and-Execute, Reflexion, MRKL, and Mixture-of-Agents design patterns.
- Familiarity with AI safety and alignment principles, including RLHF, Constitutional AI, and guardrail layers.
- Knowledge of token economics, context management, and LLM cost optimization.
- Proficiency in performance engineering methodologies for distributed AI systems.
Preferred Qualifications
These are preferred qualifications in the supplied description, rather than stated mandatory experience thresholds.
Preferred Skills & Exposure
- Experience with MCP (Model Context Protocol) or similar agentic communication standards.
- Exposure to multimodal agent testing involving text, images, code, and documents.
- Experience in regulated industries such as banking, finance, or healthcare with strict compliance requirements.
- Familiarity with chaos engineering tools such as Chaos Monkey, Gremlin, and LitmusChaos.
Education & Experience
Education
Bachelor's degree in Computer Science, Engineering, or a related field. A Master's degree is a plus.
Overall Experience
8+ years of experience in software or AI/ML quality engineering.
RAG / Agentic AI Experience
3+ years of hands-on experience with RAG systems or Agentic AI.
Specialist Experience
Proven experience building automated test frameworks for non-deterministic AI systems, with a strong background in performance testing and AI safety/security assessments.
Key Technical Skill Areas
Agentic AI Frameworks
AutoGen, CrewAI, LangGraph, ReAct, Plan-and-Execute, Reflexion, MRKL, Mixture-of-Agents.
RAG Evaluation
RAGAS, TruLens, DeepEval, chunking, embeddings, retrieval, reranking, groundedness, and citation accuracy.
Test Automation & Data Quality
CI/CD evaluation pipelines, agent trajectory replay, tool mock injection, prompt regression, Great Expectations, Deequ.
AI Safety & Security
Red-teaming, prompt injection, jailbreaks, AI Fairness 360, Aequitas, GDPR, CCPA, OWASP Top 10 for LLM Applications.
Performance Engineering
Load, stress, soak, spike, latency, tokens per second, vector database performance, SLOs, SLAs, chaos engineering.
MLOps & Reliability
Observability, anomaly detection, monitoring, autoscaling, circuit breakers, retries, fallbacks, resilience testing.
Ideal Candidate
An experienced AI/ML Quality Engineering professional who can take ownership of testing complex autonomous AI systems across agent orchestration, RAG pipelines, safety, automation, and performance. The candidate should be able to build repeatable validation frameworks for non-deterministic AI systems and assess whether those systems are reliable and safe for production use.
- Can validate single-agent and multi-agent workflows, including reasoning, tool use, memory, and delegation.
- Can measure RAG quality across retrieval, context relevance, faithfulness, and generated answers.
- Can identify AI security vulnerabilities and evaluate safety controls.
- Can engineer performance, scalability, cost, and resilience tests for distributed AI systems.
- Can work with Engineering, Product, Cybersecurity, and MLOps teams.
Job Classification
Job Family Group
Technology
Job Family
Technology Quality
Time Type
Full time
Job Level
Assistant Vice President
Equal Opportunity & Accessibility
Citi is an equal opportunity employer. Qualified candidates receive consideration without regard to the protected characteristics specified in its employment policy.
Candidates with disabilities who require reasonable accommodation to use Citi's search tools or apply for a career opportunity can refer to Citi's Accessibility information and the relevant employment policy resources.
Ready to explore this Citi opportunity?
Review the complete job listing and submit your application through LinkedIn.