AI Quality Engineer – Safety & RAG | Assistant Vice President

Role Overview

Citi is seeking an experienced AI Quality Engineer for its Retail and Wealth Risk Engineering team under the Enterprise Risk Technology platform. This highly technical role is responsible for validating complex autonomous AI systems to ensure they are safe, accurate, explainable, resilient and production-ready at scale.

Agentic AI Quality

Validate single-agent and multi-agent workflows, reasoning, memory, tool use and orchestration.

RAG Quality Engineering

Test ingestion, chunking, embeddings, retrieval, reranking, groundedness and answer relevance.

AI Safety & Security

Identify prompt injection, jailbreaks, data leakage, bias, privilege escalation and alignment failures.

Performance & Reliability

Measure latency, throughput, cost, scalability, resilience and production service objectives.

Key Responsibilities

Explore the five major engineering responsibility areas.

01. Agentic AI Testing

Design and execute end-to-end test strategies for single-agent and multi-agent AI pipelines.
Validate reasoning, planning and decision-making patterns such as ReAct, Chain-of-Thought, Plan-and-Execute and Reflexion.
Verify tool-use correctness, including the right tool, correct parameters and appropriate invocation timing.
Evaluate short-term, long-term and episodic memory for accuracy and context retention across sessions.
Validate agent handoff and delegation in AutoGen, CrewAI and LangGraph orchestration frameworks.
Test termination conditions, loop detection and prevention of infinite autonomous loops.

02. Retrieval-Augmented Generation (RAG) Testing

Design end-to-end test strategies covering ingestion, chunking, embedding, retrieval, reranking and generation.
Validate retrieval accuracy and relevance of context chunks for each query.
Test embedding model quality and vector similarity thresholds across document corpora.
Measure faithfulness, groundedness and answer relevance using RAGAS, TruLens and DeepEval.
Compare fixed, semantic and hierarchical chunking strategies.
Validate context-window management, token limits and generation quality.
Run regression testing when knowledge bases, embedding models or LLMs change.
Test multi-turn conversational RAG for context coherence and citation accuracy.

03. Test Automation & Evaluation Pipelines

Build automated test harnesses for Agentic AI and RAG systems.
Implement agent trajectory replay, tool mock injection and prompt simulation.
Integrate automated evaluation pipelines into CI/CD workflows for continuous model and agent validation.
Create data quality frameworks using Great Expectations, Deequ or custom tooling.
Build prompt regression suites to detect behavioral drift across model versions and prompt changes.
Implement determinism and reproducibility testing for stochastic LLM decisions.
Automate vector database validation for index integrity, embedding drift and retrieval consistency.

04. AI Safety & Security Testing

Conduct red-teaming and adversarial testing for jailbreaks, prompt injection and goal misalignment.
Test output guardrails and content filters against unsafe, biased, toxic and out-of-scope responses.
Validate privilege escalation controls and ensure agents cannot exceed permitted actions or access unauthorized resources.
Simulate data poisoning and backdoor attacks to assess model robustness.
Evaluate bias, fairness and discrimination using AI Fairness 360 and Aequitas.
Test PII leakage and privacy controls in RAG and agent pipelines, including GDPR and CCPA requirements.
Test OWASP Top 10 for LLM Applications risks, including insecure output handling, insecure tool design and sensitive information disclosure.
Validate Constitutional AI constraints, RLHF-aligned boundaries and system prompt integrity.
Collaborate with cybersecurity teams on AI threat modelling and vulnerability management.
Maintain safety playbooks and document red-team findings with severity ratings and remediation recommendations.

05. Performance & Reliability Engineering

Execute load, stress, soak and spike testing for AI APIs, inference endpoints and agent orchestration services.
Measure end-to-end latency across RAG and agentic workflows.
Benchmark LLM inference throughput in tokens per second and identify model-serving bottlenecks.
Validate auto-scaling under variable traffic and workloads.
Test circuit breakers, retries and fallback mechanisms for graceful degradation.
Measure vector database query latency, index build time and retrieval accuracy under concurrency.
Analyse token consumption, API costs and infrastructure spending per agent task.
Establish SLOs and SLAs for availability, P50/P95/P99 latency and error rates.
Partner with MLOps teams on dashboards, monitoring alerts and anomaly detection.
Conduct chaos engineering experiments to validate resilience during infrastructure failures.

Ideal Candidate

A technically strong AI Quality Engineer for enterprise-grade autonomous systems

The ideal candidate has 8+ years of software or AI/ML quality engineering experience, including 3+ years of hands-on experience with RAG systems or Agentic AI. They can independently design automated test frameworks for non-deterministic AI systems, validate retrieval and agent behavior, conduct AI safety and security assessments, and engineer performance and reliability tests. Experience in regulated environments such as banking and finance is particularly relevant.

Technical Skills & Tools

Agentic AI Patterns

ReAct, Plan-and-Execute, Reflexion, MRKL, Mixture-of-Agents, multi-agent orchestration and tool use.

Agent Frameworks

AutoGen, CrewAI and LangGraph.

RAG Evaluation

RAGAS, TruLens, DeepEval, retrieval relevance, faithfulness, groundedness and citation accuracy.

Vector & Embedding Validation

Embedding quality, similarity thresholds, chunking strategies, index integrity and retrieval consistency.

Data Quality Engineering

Great Expectations, Deequ and custom data validation frameworks.

AI Safety & Fairness

AI Fairness 360, Aequitas, guardrails, RLHF, Constitutional AI, toxicity and bias testing.

AI Security Testing

Prompt injection, jailbreaks, data poisoning, backdoor attacks, PII leakage and privilege escalation.

Performance Engineering

Load, stress, soak, spike and chaos testing; latency, throughput and cost analysis.

Observability & MLOps

Production monitoring, dashboards, alerting, anomaly detection and AI service SLOs.

Distributed AI Systems

Inference endpoints, vector databases, auto-scaling, retries, circuit breakers and fallback mechanisms.

AI Governance & Privacy

GDPR, CCPA, internal data governance, fairness objectives and AI alignment principles.

Agent Communication

MCP (Model Context Protocol) and similar agentic communication standards are preferred.

RAGAS TruLens DeepEval AutoGen CrewAI LangGraph Great Expectations Deequ AI Fairness 360 Aequitas MCP OWASP LLM Top 10

Qualifications & Experience

Education

Bachelor's degree in Computer Science, Engineering or a related field.
Master's degree is an added advantage.

Required professional experience

8+ years of experience in software or AI/ML quality engineering.
3+ years of hands-on experience with RAG systems or Agentic AI.
Proven experience building automated test frameworks for non-deterministic AI systems.
Strong background in performance testing and AI safety/security assessments.

Preferred additional experience

MCP or similar agentic communication standards.
Multimodal agent testing involving text, images, code and documents.
Regulated industry experience, particularly banking, finance or healthcare.
Chaos engineering tools such as Chaos Monkey, Gremlin and LitmusChaos.

AI domain knowledge

RAG architecture patterns: naive RAG, advanced RAG and modular RAG.
Agent design patterns: ReAct, Plan-and-Execute, Reflexion, MRKL and Mixture-of-Agents.
AI safety and alignment: RLHF, Constitutional AI and guardrail layers.
Token economics, context management and LLM cost optimization.
Performance engineering for distributed AI systems.

About Citi

This position is part of Citi's Retail and Wealth Risk Engineering team under the Enterprise Risk Technology platform.

The role belongs to the Technology Job Family Group and Technology Quality Job Family.

The opportunity focuses on improving the safety, reliability and quality of AI systems used in enterprise risk technology environments.

Citi is an equal opportunity employer. The supplied listing also references accessibility support and reasonable accommodations during the application process.

AI Quality Engineer, Safety and RAG