Role Overview
The Senior AI Evals Engineer is a hands-on technical authority within Rubrik's Applied AI: Test Engineering team. The role focuses on defining, developing and operating evaluation frameworks for LLM-based agents built with Google ADK or similar agentic frameworks. It bridges AI development and reliable deployment by measuring correctness, task completion, latency, cost, safety and policy adherence.
Automation Testing
Build scalable automated evaluation frameworks, test harnesses and CI-integrated quality gates.
Manual Testing
Perform focused manual evaluation, adversarial testing and safety validation of AI agents.
Key Responsibilities
Expand each area to explore the role's technical and leadership expectations.
01. Agent Evaluation Frameworks
- Design and build end-to-end evaluation infrastructure for LLM-based agents.
- Measure correctness, latency, cost, safety and task completion rates.
- Build synthetic testing environments and simulation strategies for diverse real-world conditions.
- Focus on agents built using Google ADK, Vertex AI, Gemini models and related Google AI/ML services.
- Define evaluation taxonomies and scoring rubrics for tool-use accuracy, multi-step reasoning, hallucinations, policy adherence and error recovery.
02. Golden Datasets & Eval-as-a-Service
- Build and maintain golden datasets and test harnesses for Rubrik's product domains.
- Cover data security, SaaS protection, cyber recovery and compliance use cases.
- Establish a centralized evaluation platform that engineering teams can use independently.
- Support standardized benchmarks as well as custom evaluation requirements across agent projects.
03. Automated Regression & Production Monitoring
- Integrate evaluation pipelines into CI/CD to detect regressions after model or prompt changes.
- Implement automated drift detection and pre-production quality gates.
- Instrument production monitoring for agent success rates, user corrections, fallback triggers and escalation patterns.
- Analyze evaluation trends and communicate risks, findings and mitigation recommendations.
04. Responsible AI, Safety & Red-Teaming
- Design adversarial evaluation suites for jailbreak resistance and prompt injection.
- Test data leakage prevention, boundary enforcement and policy adherence.
- Evaluate fairness, bias, toxicity and safety behavior.
- Partner with policy and legal teams to address regulatory, ethical and AI governance requirements.
- Test robustness against ambiguous, misleading and malformed inputs.
- Investigate data poisoning risks and other adversarial failure modes.
05. End-to-End AI Agent Quality Assurance
- Define functional, integration, regression and user acceptance testing strategies for conversational and goal-oriented agents.
- Test tool invocation, function calling, memory persistence and multi-turn reasoning.
- Maintain test plans, scenarios, cases and automated suites across evolving model versions.
- Track defects, performance regressions and alignment failures using Jira.
- Collaborate with Applied AI Engineers to identify root causes and validate fixes.
06. Model Benchmarking & Evaluation Analytics
- Compare models using structured cost, quality and latency benchmarks.
- Interpret evaluation results and communicate quality trade-offs to stakeholders.
- Design experiments and interpret metrics with appropriate statistical confidence.
- Avoid benchmark overfitting and metric gaming when designing evaluation strategies.
- Provide actionable reports on quality, safety risks and mitigation priorities.
07. Technical Leadership & Cross-Functional Collaboration
- Provide technical direction through solution design, design reviews, code reviews and pair programming.
- Mentor junior and contract engineers.
- Run proofs of concept for emerging evaluation technologies and influence the platform roadmap.
- Partner with ML engineers, product teams, platform engineers, security and policy teams.
- Embed evaluation-driven quality gates throughout the agent development lifecycle.
Ideal Candidate
An AI evaluation engineer who builds production-grade testing systems
The ideal candidate combines strong software test engineering experience with practical experience evaluating LLM applications and AI agents. They can build Python-based evaluation infrastructure, work with DeepEval and LLM-as-a-Judge, design golden datasets, automate evaluations in CI/CD, and investigate prompt injection and safety failures. This is not simply a manual QA position or a research-only role: it requires hands-on engineering of repeatable evaluation systems that influence enterprise release decisions.
Technical Skills & Technology Stack
Programming
Python is mandatory. TypeScript and Java are also listed, with Java optional.
LLM Evaluation Tools
DeepEval and LLM-as-a-Judge are mandatory. Promptfoo, RAGAS, LangSmith and custom evaluation harnesses are also relevant.
Agentic AI Frameworks
Google ADK, LangGraph, Claude Agent SDK and custom agent orchestrators.
LLM Providers
Anthropic Claude, OpenAI, Azure OpenAI and open-source models.
Agent Architecture
Tool use, function calling, multi-step planning, ReAct/CoT reasoning chains and RAG.
Python Test Engineering
pytest, scripting, asynchronous patterns and large-scale data processing.
Automation Testing
Selenium and Playwright, along with custom agent evaluation test harnesses.
Cloud & AI Infrastructure
Google Cloud Platform, Vertex AI, MLOps workflows and cloud services relevant to AI/ML.
CI/CD & DevOps
Jenkins, GitHub Actions, Git workflows, automated testing, static analysis and deployment automation.
Observability
Datadog, Grafana and production monitoring for agent quality and operational behavior.
Data & Statistical Analysis
Pandas, SQL, evaluation data analysis, experiment design and statistical interpretation.
Security & Responsible AI
Prompt injection, jailbreak testing, data leakage prevention, fairness, bias, toxicity and adversarial testing.
Qualifications & Experience
Required education and experience
- Bachelor's or Master's degree in Computer Science, Information Systems, Engineering or equivalent practical experience.
- 8+ years of overall software test engineering experience.
- 5+ years focused on test engineering, QA automation or developer tooling.
- 2+ years working with ML/AI systems, including production LLM applications, prompt engineering, RAG pipelines or agentic workflows.
- 2+ years in a senior or lead capacity, owning evaluation strategy, quality frameworks and technical decisions for enterprise AI agent systems.
- 1+ years designing or operating generative AI/LLM agent evaluation frameworks, including benchmarking, red-teaming and production quality monitoring.
Required technical capabilities
- Hands-on experience with DeepEval and LLM-as-a-Judge.
- Strong Python skills and experience building automated, CI-integrated evaluation infrastructure.
- Understanding of agentic AI patterns, tool use, function calling, multi-step planning and RAG.
- Experience with cloud services and MLOps workflows.
- Hands-on CI/CD and Git-based development workflows.
- Prompt injection testing experience is specifically identified as mandatory in the preferred qualifications section of the listing.
Preferred qualifications and additional experience
- Deep learning and NLP fundamentals, including LLM behavior, prompting, RAG and alignment challenges.
- AI ethics principles and Responsible AI development standards.
- Data poisoning skills and robustness testing for malformed or misleading inputs.
- Complex API testing, particularly tool integration and function calling.
- Human-in-the-loop evaluation, annotation workflows and inter-rater reliability.
- Model fine-tuning evaluation and pre/post quality comparison.
- AI/ML, GCP or AWS certifications and software testing qualifications.
- Experience in cybersecurity, data protection, infrastructure or enterprise data management platforms.
Leadership and communication expectations
- Influence cross-functional teams without relying on direct authority.
- Drive technical decisions using evidence, clarity and evaluation data.
- Collaborate with distributed Agile teams and stakeholders across time zones.
- Present technical solutions, risks and trade-offs to engineers, managers and senior business stakeholders.
- Decompose complex business processes into practical, maintainable engineering solutions.
About Rubrik
Rubrik (NYSE: RBRK) describes itself as the Security and AI Operations Company, operating at the intersection of data, identity and AI.
Its platform focuses on agentic cyber resilience and trusted AI operations, helping enterprises protect data and recover from disruptions.
The company highlights an inclusive workplace, equal employment opportunities and reasonable accommodations for applicants and employees with disabilities.
Role focus: Applied AI, Test Engineering, AI agent evaluation and Responsible AI safety.
Ready to engineer safer, more reliable AI agents?
Review the original job listing and application requirements on LinkedIn.