About the Role
QAD is hiring an AI Quality and Evaluation Lead to build enterprise-wide AI evaluation frameworks for AI agents and LLM-powered products. The role focuses on production-quality measurement, release governance, drift monitoring, and continuous AI improvement across the organization.
QAD
5–8 Years
Pune (Hybrid)
AI Quality & Evaluation Lead
Role Overview
Lead the quality strategy for AI-powered enterprise products by defining evaluation frameworks, AI release gates, decision-quality metrics, production monitoring, and governance processes. Collaborate with engineering, product, and operations teams to ensure AI systems remain reliable, measurable, and customer-focused.
Key Responsibilities
- Build enterprise AI evaluation frameworks using golden datasets and LLM-as-Judge rubrics.
- Define release criteria and AI quality gates for production deployments.
- Monitor production drift, regression, exception rates, and human override metrics.
- Ensure AI decision auditability with traceable recommendations and rationale.
- Create evaluation rubrics for correctness, latency, decision quality, compliance, and tone.
- Partner with product teams to standardize AI quality measurement.
- Lead recurring evaluation reviews and identify regressions before customer impact.
Required Skills
Must Have
- 5–8 years in AI/ML product quality, data engineering, or quality engineering.
- Hands-on experience evaluating production LLM applications.
- Experience with AI agents, traces, tool calls, and RAG systems.
- Strong SQL plus Python or JavaScript skills.
- Experience using AI evaluation and observability platforms.
Success Metrics
- Enterprise-wide AI evaluation framework adoption.
- Production drift detection before customer impact.
- Reliable AI release gating using measurable evidence.
- Decision-quality scoring for AI recommendations.
- Continuous improvement of AI product performance.
Why Join QAD?
Join an organization building AI-powered manufacturing solutions where evaluation quality directly impacts production systems. Work with enterprise AI agents, measurable decision intelligence, and next-generation AI governance.
Highlights
- ✔ AI Product Quality Leadership
- ✔ LLM Evaluation Frameworks
- ✔ AI Agents & Decision Intelligence
- ✔ Production Drift Monitoring
- ✔ AI Governance & Release Gates
- ✔ Hybrid Work – Pune
- ✔ Enterprise SaaS AI Platform
Disclaimer: This job post is shared for educational and career guidance purposes only. All company names, trademarks, and logos belong to their respective owners. Please apply through the official company job portal or LinkedIn posting.