Senior Manager – Testing & AI Quality Engineering
Lead enterprise testing transformation at Ramco Systems by combining Agentic AI evaluation, AI safety, automation engineering and organisation-wide quality strategy.
- Company
- Ramco Systems
- Location
- Chennai, India
- Experience
- 18+ Years
- Function
- Quality Assurance & Test Engineering
Explore the role
Role Overview
Ramco Systems is seeking an Associate Director / Senior Manager to lead Testing and AI Quality Engineering. Reporting to the Head of Quality Engineering, this role combines responsibility for enterprise quality strategy with hands-on AI testing transformation across copilots, assistants, RAG pipelines and autonomous agents.
The leader will establish evidence-driven AI release criteria, improve traditional test engineering, govern AI-generated test assets, manage quality budgets and build the capabilities required for AI-augmented testing across global product teams.
Key Responsibilities
01 Testing AI & Agentic Software 8 areas
- Own test strategy for copilots, assistants, LLM workflows, RAG pipelines and multi-step agents across Ramco products.
- Build evaluation frameworks using golden datasets, evaluation harnesses, LLM-as-a-Judge with human calibration and agreed release thresholds.
- Run model and prompt regression testing with version pinning, drift detection, re-baselining and A/B release comparisons.
- Validate RAG precision and recall, groundedness, citation correctness and hallucination rates.
- Test agent tool selection, execution trajectories, memory, retries, loops, handoffs, MCP integrations and tool contracts.
- Validate autonomous-action safeguards, including permissions, approval gates, human oversight, idempotency and rollback.
- Lead AI red teaming for prompt injection, jailbreaks, data exfiltration, PII leakage and tenant-data exposure.
- Extend quality into production through tracing, online evaluations and drift monitoring, with failures feeding back into regression suites.
02 AI-Led Testing Transformation 6 areas
- Define the AI-in-QA strategy across test design, automation, test data, triage, root-cause analysis and release readiness, including cases where AI should not be used.
- Build practical testing accelerators using Claude Code or comparable agentic tools.
- Establish agent-engineering practices covering prompts, context files, skills, sub-agents, MCP integrations and shared workflow repositories.
- Drive AI-assisted test design with mandatory human review, natural-language-to-script automation, self-healing locators and risk-based test selection.
- Govern AI-generated artefacts through review gates, traceability, intellectual-property hygiene, secure data handling and human ownership.
- Measure adoption, cycle time, coverage, escaped defects and cost per release; retire ineffective solutions and reskill 40–50 engineers in prompt, context and agent engineering.
03 Quality Strategy & Organisational Leadership 4 areas
- Set the quality vision and multi-year roadmap with Product, Engineering, Data Science, Support and Delivery.
- Run Quality Maturity Assessments with the Quality CoE and ensure adherence to the Quality Management System.
- Balance Development and QA capacity, and re-baseline QA sizing as AI capabilities are introduced.
- Own the operating budget, including AI tooling and token expenditure, with ROI tracking.
04 Test Engineering & Non-Functional Quality 4 areas
- Manage functional automation, API, performance, accessibility and security testing strategies.
- Improve testability and observability, including instrumentation needed to evaluate AI features.
- Build simulation environments with mocked model endpoints for deterministic CI execution.
- Run proofs of concept comparing AI testing platforms with internal solutions, and plan end-to-end testing across products and partners.
05 Delivery, Metrics & People Leadership 4 areas
- Track productivity, defect leakage, automation coverage, AI adoption, evaluation pass rates and hallucination trends.
- Report licensing, financials, delivery and quality to executives, and resolve production issues with global stakeholders.
- Lead managers, technical leads and engineers across performance, automation, accessibility, security and AI quality.
- Define career paths for AI quality and test tooling engineers, and build a culture of experimentation and psychological safety.
Ideal Candidate
A senior quality engineering leader with 18+ years in test engineering, 10+ years as a QA manager and experience leading 50+ engineers, including managers, in a matrixed global organisation. The candidate should combine enterprise-scale people leadership with practical production experience in LLM and Agentic AI testing, evaluation frameworks, AI safety and AI-assisted engineering transformation.
- Enterprise QA Strategy
- Agentic AI
- LLM Evaluation
- RAG Testing
- AI Red Teaming
- Claude Code
- MCP
- Playwright
- AI Governance
- People Leadership
Required Experience & Skills
Leadership & Experience
Required
- Bachelor’s or Master’s degree in Computer Science or Engineering.
- 18+ years in test engineering, including 10+ years as a QA manager on complex products.
- 5+ years leading 50+ engineers, including managers, in a matrixed global multinational organisation.
- Demonstrated progression from QA to test engineering to AI-augmented testing.
AI & Agentic Testing
Essential
- Hands-on tools built using Claude Code or an equivalent agentic tool, with concrete user and savings examples.
- Production experience testing LLM applications, copilots, RAG systems and multi-step agent workflows.
- Evaluation frameworks using golden datasets, LLM-as-a-Judge, human calibration and defensible release criteria.
- AI safety testing covering prompt injection, jailbreaks, data leakage and agent permission scoping.
- Practical judgment on AI economics and when deterministic testing is preferable.
AI Evaluation Tools
Relevant tooling
- LangSmith, Langfuse, Braintrust, DeepEval, Ragas and promptfoo.
- Agent-run tracing, RAG evaluation, regression, model and prompt versioning, and production drift monitoring.
- MCP integrations and agent-driven connections to ALM and CI systems.
Core Test Engineering
Required
- Functional and non-functional testing, Agile/Scrum, SQL validation and strategic quality metrics.
- Playwright, Selenium, Cucumber, Postman, JMeter and LoadRunner.
- REST, SOAP, GraphQL and event-driven APIs.
Languages & DevOps
Technical skills
- Python, TypeScript/JavaScript, Java or C#.
- Git, Jenkins, containers, Terraform or Ansible.
- LLM APIs and orchestration patterns, CI/CD quality gates and mocked model environments.
ALM & Certification
Required / preferred
- Jira/Xray, Azure DevOps, TestRail and Zephyr, including APIs for agent-driven integrations.
- ISTQB Advanced Test Manager certification.
- AI certification such as CT-AI is preferred; PMP or PRINCE2 is an advantage.
Responsible AI & Governance
Preferred
- Exposure to the EU AI Act, NIST AI RMF and ISO/IEC 42001.
- Human-in-the-loop controls for agents operating on enterprise data.
- AI enablement through training, technical guilds and hackathons.
What Success Looks Like
Success is defined by measurable quality evidence, safer AI releases and demonstrable improvements in testing productivity—not by AI adoption alone.
Key Performance Indicators
The job description states that targets are agreed annually with the Head of Quality Engineering and reviewed at the relevant cadence.
01 AI Release Confidence Per release
- Evaluation pass rates against agreed thresholds.
- Hallucination rates, groundedness and citation correctness for RAG features.
02 AI Safety & Blast-Radius Assurance Per release
- Red-team findings closed before release.
- Permission scoping, approval gates and rollback validated for autonomous actions.
03 AI-in-QA Impact Quarterly
- Adoption of AI testing accelerators, cycle-time reduction, coverage gain and cost per release against baseline.
04 Production Quality & Defect Leakage Monthly
- Production defect leakage across products and resolution within agreed timelines.
05 Automation & Governance Quarterly / half-yearly
- Functional, API, performance, accessibility and security test coverage against plan.
- Quality Maturity Assessment outcomes and QMS adherence.
- Engineer reskilling, QA capacity planning and operating-budget stewardship.
Why Join Ramco Systems?
- Lead quality for AI assistants, copilots and autonomous agents embedded in enterprise workflows used across 35+ countries.
- Own a dual mandate: testing AI while transforming the testing organisation around AI.
- Build internal tools and engineering capabilities with budget and authority to develop solutions rather than only purchase them.
- Contribute to an enterprise software company focused on AI/ML-based, bot-enabled and cloud-native applications.
Ready to apply?
Review the complete job description and submit your application through LinkedIn.