QA Engineer – Generative AI & Agentic AI

Role Overview

We are looking for an experienced QA Engineer to lead quality assurance for Generative AI and Agentic AI solutions. This role goes beyond traditional functional testing, focusing on AI output quality, multi-step agent workflows, Text-to-SQL pipelines, RAG systems and LLM-based applications. Candidates should have hands-on production experience and have supported at least one production release of a GenAI or Agentic AI solution.

Key Responsibilities

• Design and execute functional, non-functional, integration and AI output-quality test strategies.
• Validate agent orchestration, tool/function calls, task completion, chained decisions and outputs at each agent step.
• Evaluate generated responses for correctness, relevance, coherence, hallucination, consistency and reliability.
• Test Text-to-SQL generation, SQL syntax and logic, schema alignment, query execution and result accuracy against Snowflake data models.
• Validate RAG retrieval relevance, chunking, context utilization, grounding and faithfulness to retrieved information.
• Apply evaluation techniques such as embedding similarity, ROUGE, BLEU and LLM-as-a-Judge.
• Build LLM-as-a-Judge evaluation frameworks and scoring rubrics for scalable automated assessments.
• Create golden datasets, evaluation datasets, regression suites and reusable AI testing frameworks.
• Develop test harnesses and scripts for batch evaluation and automated validation of LLM outputs.
• Own end-to-end release QA, including test planning, execution, regression, release sign-off and post-production monitoring.
• Work with Data Science, Platform Engineering, Product and other teams to establish acceptance criteria and quality gates.
• Identify hallucinations, prompt injection risks, bias, inconsistent responses and agent/tool failures.
• Track defects, document evidence and maintain traceability of test coverage, evaluation results, quality metrics and release reports.

Mandatory Skills & Experience

• Demonstrable hands-on experience testing Generative AI and LLM applications.
• Experience testing Agentic AI, multi-step workflows, orchestration and tool/function calls.
• Practical experience with Text-to-SQL, RAG testing, prompt validation and LLM-as-a-Judge evaluation.
• Understanding of hallucination assessment, response grounding, context windows and AI output-quality metrics.
• Strong hands-on Dataiku experience is highly preferred and stated as the first platform preference.
• Exposure to OpenAI workspace, Leena AI or similar GenAI/Agentic AI platforms.
• Strong working knowledge of Snowflake, data models, schemas, generated SQL and query result validation.
• Practical experience with evaluation methods such as embedding similarity, ROUGE, BLEU or equivalent approaches.
• Ability to build evaluation datasets, automated test harnesses, regression suites and release-quality reports.

Job Information