Job Overview
QAD is hiring an AI Quality and Evaluation Lead to build and own enterprise AI evaluation frameworks for LLM-powered products. The role focuses on AI quality measurement, decision evaluation, RAG validation, production drift monitoring, release governance, AI observability, and continuous improvement of Generative AI applications.
QAD
5–8 Years
AI Evaluation
Pune (Hybrid)
Core Skills
Key Responsibilities
- Build enterprise AI evaluation frameworks for GenAI products.
- Create golden datasets and LLM evaluation rubrics.
- Measure decision quality and recommendation effectiveness.
- Define AI release quality gates and evaluation criteria.
- Monitor production drift and AI quality degradation.
- Implement auditability and traceability for AI decisions.
- Collaborate with Product, Engineering, and AI teams.
- Drive continuous AI quality improvements across products.
Required Skills
- 5–8 years in AI Product, AI/ML, Quality Engineering, or Product Operations.
- Experience evaluating production LLM systems.
- Knowledge of RAG architectures and AI agents.
- Strong Python and SQL programming.
- AI observability and drift monitoring.
- Git and evaluation tooling experience.
- Strong analytical and communication skills.
- Experience working with cross-functional AI teams.
Technology Stack
- Python
- SQL
- Git
- LLMs
- RAG
- AI Agents
- LLM-as-Judge
- Golden Datasets
- AI Observability
- Production Monitoring
- Decision Auditability
- Drift Detection
Why Apply?
This role offers a unique opportunity to define how AI quality is measured across enterprise SaaS products. You'll own evaluation frameworks, influence AI release decisions, implement production monitoring, and work closely with AI product leaders to build reliable, explainable, and customer-focused Generative AI systems.
Disclaimer: This job summary is published for informational purposes only. Please apply through the employer's official recruitment portal using the Apply button above.