Associate Lead - Testing (QA + MLOps)

Quantiphi


Date: 16 hours ago
City: Bengaluru, Karnataka
Contract type: Full time
While technology is the heart of our business, a global and diverse culture is the heart of our success. We love our people and we take pride in catering them to a culture built on transparency, diversity, integrity, learning and growth.

If working in an environment that encourages you to innovate and excel, not just in professional but personal life, interests you- you would enjoy your career with Quantiphi!

Role: Lead/Associate Lead – QA + MLOps & Generative AI

Experience: 10+ years

Location: Mumbai/Bangalore (Hybrid)

Key Responsibilities

AI/ML & GenAI Testing Strategy (AWS Ecosystem)

Define testing approaches for AI systems built on AWS services such as:

  • Amazon SageMaker
  • Amazon Bedrock
  • AWS Lambda
  • Amazon API Gateway
  • Amazon Kinesis
  • AWS Glue
  • Amazon S3
  • Amazon CloudWatch

Design Validation Frameworks Covering

  • Model accuracy & performance validation
  • Data drift & concept drift detection
  • Hallucination detection for LLMs
  • Prompt robustness testing
  • RAG validation (retrieval accuracy + grounding)
  • Bias & fairness validation
  • Safety & toxicity testing

MLOps Quality Engineering (AWS-Centric)

Validate The End-to-end ML Lifecycle Including

  • Data ingestion & feature pipelines
  • Model training & hyperparameter tuning
  • Model versioning & registry
  • Deployment validation
  • Canary & blue/green release validation

Work With AWS-native Services Such As

  • SageMaker Pipelines
  • SageMaker Model Monitor
  • SageMaker Feature Store
  • Bedrock model evaluation workflows
  • CloudWatch-based observability

Implement CI/CD quality gates for ML pipelines integrated with AWS DevOps tools.

GenAI & Agentic AI Testing

Define Quality Engineering Approaches For

  • LLM-based applications using Amazon Bedrock
  • Prompt engineering validation
  • Multi-agent orchestration testing
  • Chatbot & Voice bot conversational testing
  • Intent classification validation
  • Conversation drift & fallback validation
  • API contract validation for LLM integrations

Build Reusable Evaluation Harnesses For

  • BLEU / ROUGE scoring
  • Embedding similarity scoring
  • Response consistency
  • Safety scoring frameworks

Framework & Capability Development

  • Design reusable AI testing accelerators
  • Create AWS-aligned AI test automation frameworks (Python-first)
  • Develop synthetic data generation strategies
  • Establish AI quality scorecards
  • Build an internal AI QA Center of Excellence

Client Engagement & Leadership

  • Lead AI/ML quality strategy workshops
  • Perform AI risk & readiness assessments
  • Present quality architecture to CXOs
  • Drive QA transformation programs
  • Mentor QA teams on AWS-based AI testing
  • Own delivery for AI testing engagements end-to-end

Must Have Skills

Testing Expertise

  • 8–12+ years in Quality Engineering
  • Strong test strategy, automation & governance experience
  • Experience leading QA transformation initiatives
  • Experience building frameworks from scratch AI/ML & GenAI Expertise
  • Deep understanding of ML lifecycle
  • Experience testing ML models (NLP preferred)
  • Hands-on experience validating LLM applications
  • Strong understanding of:
  • Prompt engineering
  • RAG architecture
  • Embeddings
  • Bias & explainability AWS AI/ML Expertise
  • Hands-on experience with:
  • Amazon SageMaker (training, deployment, monitoring)
  • Amazon Bedrock (LLM integration & evaluation)
  • S3-based data pipelines
  • AWS IAM (security validation)
  • CloudWatch monitoring
  • Lambda & API Gateway integrations
  • AWS CI/CD (CodePipeline / CodeBuild preferred)

Understanding Of

  • Infrastructure as Code (Terraform / CloudFormation)
  • Observability in AI systems
  • Cost monitoring for ML workloads

Technical Skills

  • Python (mandatory)
  • Experience with ML libraries (Scikit-learn, TensorFlow, PyTorch)
  • Experience with LLM frameworks (LangChain, etc.)
  • API & automation testing frameworks
  • Git-based workflows
  • Leadership & Communication
  • Strong client-facing communication
  • Experience leading QA teams
  • Ability to create strategy decks & solution proposals
  • Strong stakeholder management

If you like wild growth and working with happy, enthusiastic over-achievers, you'll enjoy your career with us!

How to apply

To apply for this job you need to authorize on our website. If you don't have an account yet, please register.

Post a resume

Similar jobs

Design Engineer

nVent, Bengaluru, Karnataka
1 day ago
We’re looking for people who put their innovation to work to advance our success – and their own. Join an organization that ensures a more secure world through connecting and protecting our customers with inventive electrical solutions.What YOU WILL EXPERIENCE IN THIS POSITIONCreating and verifying technical drawings starting from project;Design for Manufacturing, align with plant capabilitiesParticipates at initial meeting projects;Maintain...

Senior Payroll Analyst

MillerKnoll, Bengaluru, Karnataka
6 days ago
Why join us? Our purpose is design for the good of humankind. It’s the ideal we strive toward each day in everything we do. Being a part of MillerKnoll means being a part of something larger than your work team, or even your brand. We are redefining modern for the 21st century. And our success allows MillerKnoll to support causes...

Program Associate

Youmatter HR, Bengaluru, Karnataka
1 week ago
Compensation:  4 LPA - 5 LPANote : Please do not apply if your expected compensation is outside the stated range.Program Associate - Sustainability & Social ImpactLocation: BangaloreExperience: 1-3 YearsCompensation: up to 5 LPA About the OpportunityWe are hiring Program Associate on behalf of a leading sustainability-focused organization working at the intersection of sustainability, climate action, innovation, youth engagement, and ecosystem development.Selected...