GenAI Testing Consultant/Architect

Role Overview

This role focuses on architecting, developing, testing and continuously improving enterprise Generative AI and ML solutions. The position combines AI architecture, GenAI testing, LLMOps, MLOps, cloud platforms and solution consulting.

The role requires the ability to design end-to-end AI solutions, establish quality and evaluation frameworks, implement agentic architectures and drive adoption across teams and portfolios.

Solution Delivery & Consulting

• Partner with client stakeholders to understand business goals and identify AI/ML and GenAI use cases.
• Translate business requirements into scalable AI solution architectures.
• Define standard and custom success metrics for AI initiatives.
• Lead discovery workshops and create solution roadmaps.
• Contribute to effort estimation, solution design and delivery planning.
• Communicate technical findings, trade-offs and recommendations through documentation and presentations.

Generative AI Development

• Build Python-based prototypes and production-ready components for GenAI workflows.
• Develop and refine prompts, templates and guardrails.
• Implement automated evaluation approaches for AI output quality.
• Improve response accuracy, relevance, safety and consistency.
• Develop LLM-powered applications using modern GenAI patterns.

RAG & Agentic AI

• Design RAG architectures and context-aware AI applications.
• Implement embeddings, retrieval and vector search solutions.
• Build MCP and agentic-based solutions.
• Implement tool and function calling.
• Design workflow orchestration and multi-step reasoning patterns.
• Develop AI agents aligned with business requirements and operational workflows.

AI Testing & Evaluation

• Test and validate GenAI applications using appropriate evaluation tools and frameworks.
• Measure relevance, groundedness, toxicity, latency and cost.
• Identify hallucinations and reliability issues.
• Build automated evaluation and validation frameworks.
• Capture, analyze and report standard and custom AI quality metrics.
• Drive continuous quality improvement across AI solutions.

MLOps & LLMOps

• Implement experiment tracking and model lifecycle management.
• Design model registry and versioning strategies.
• Implement CI/CD pipelines for AI and ML workloads.
• Establish observability and monitoring for AI services.
• Implement drift detection and incident-response mechanisms.
• Support reliable production deployment and operational stability.

Cloud AI Architecture

• Architect scalable cloud-based AI solutions.
• Design secure data and inference architectures.
• Optimize AI workloads for cost and performance.
• Collaborate with cloud and platform engineering teams.
• Work across AWS, Azure and GCP environments.

AI / ML Engineering

• Develop and maintain ML pipelines using Python.
• Support data preparation, training, inference and monitoring.
• Perform model experimentation and feature engineering.
• Apply performance tuning aligned with business requirements.
• Integrate AI services into enterprise applications and workflows.

Architecture Governance

• Define reference architectures and reusable AI patterns.
• Drive architecture governance across teams and portfolios.
• Evaluate emerging GenAI technologies and testing methodologies.
• Establish engineering practices focused on maintainability, security and reliability.
• Support adoption of standardized AI development and evaluation practices.

Required Experience

• 3–5 years of professional experience delivering Python-based solutions.
• Hands-on experience with AI/ML or Generative AI components.
• Strong Python programming, testing and debugging skills.
• Understanding of supervised and unsupervised machine learning.
• Understanding of model validation and evaluation metrics.
• Bachelor’s degree in Engineering, Computer Science or a related discipline.

Preferred Experience

• End-to-end AI/ML solution delivery in a consulting or client-facing environment.
• Experience with RAG, embeddings, vector search and vector databases.
• Knowledge of tool/function calling and agentic AI architectures.
• Experience with MLOps and production AI monitoring.
• Exposure to experiment tracking and model versioning.
• Experience with scalable data and ML platforms.
• Ability to balance rapid prototyping with production readiness.

Key Skills

Good to Have

• Agile methodologies
• CI/CD pipelines
• API testing
• SQL and database fundamentals
• Cloud AI platforms
• AI observability and monitoring

Location