About the Role
Kobie is hiring a Lead AI QA Engineer to build enterprise-quality testing frameworks for production AI systems. You'll lead quality engineering for LLM-powered applications, AI agents, ETL workflows, and automated evaluation pipelines supporting global loyalty platforms.
Kobie
6+ Years
Bengaluru (India Tech Hub)
LLM & Agentic AI
Role Overview
Design enterprise AI quality strategies for production-grade AI systems by validating prompts, RAG pipelines, tool calls, structured outputs, and AI agent workflows. Build automated evaluation pipelines and partner with Engineering, Data, Product, and AI teams to deliver reliable AI solutions.
Key Responsibilities
- Design AI quality strategies for enterprise LLM applications.
- Create golden datasets and automated regression suites.
- Validate prompts, RAG pipelines, tool/function calls, and structured outputs.
- Build LLM-as-Judge evaluation pipelines.
- Develop automated QA frameworks using Python and PyTest.
- Perform ETL validation and SQL-based data verification.
- Integrate testing into Git, Docker, Jenkins, and GitHub Actions pipelines.
- Collaborate with U.S. AI & Innovation, Engineering, Product, and Data teams.
Required Skills
Must Have
- 6+ years in QA and AI quality engineering.
- Strong Python programming experience.
- Hands-on experience testing LLM-powered applications.
- Experience with RAG evaluation and AI agent validation.
- Strong SQL and ETL testing skills.
- Automation using PyTest, Git, Docker, Jenkins, and GitHub Actions.
- Knowledge of RAGAS, DeepEval, LangSmith, or Langfuse.
Technology Stack
- Python
- PyTest
- SQL
- Postman
- REST APIs
- Git
- Docker
- GitHub Actions
- Jenkins
- Snowflake
- Dataiku
- RAGAS
- DeepEval
- LangSmith
- Langfuse
Why Join Kobie?
Join the founding engineering team of Kobie's India Tech Hub and help build AI-powered loyalty platforms used by global brands. Work alongside U.S. engineering teams while shaping quality standards for production AI systems.
Highlights
- ✔ Founding India Tech Hub Team
- ✔ Production AI Platform
- ✔ LLM & Agentic AI Testing
- ✔ Global Engineering Collaboration
- ✔ Python Automation & ETL Testing
- ✔ Enterprise Loyalty AI Solutions
- ✔ AI Quality Leadership
Disclaimer: This job information is shared for educational and career guidance purposes only. Company names, trademarks, and logos belong to their respective owners. Please apply through the official company careers page or LinkedIn job posting.