AI Benchmarking Specialist, Amazon

AI Benchmarking Specialist
Karnataka, India, Bengaluru
May 29, 2026
Full Time

Apply for this job

Email *
Executive Name *

Job Description

Amazon is hiring an AI Benchmarking Specialist in its Seller AI team to evaluate AI systems and large language models. The role involves designing benchmarks, validating datasets, auditing model performance, and ensuring accuracy, reliability, compliance, and unbiased outcomes to enhance the global seller experience.

Qualification: Bachelor’s Degree or equivalent

Experience: 2+ years preferred

Apply: Apply Now

Main Duties:

  • Create and implement AI benchmarking tests which will include test specifications and performance assessment criteria for accuracy testing and bias evaluation and system reliability testing. 
  • Examine datasets and model outcomes and data processing methods to assess their accuracy and relevance and their compliance with privacy standards. 
  • Accuracy of verified data through their verification process. 
  • Creates audit reports which include benchmarking results and error assessments and root cause evaluations and recommendations for future actions. 
  • Maintain complete audit files which contain all documentation and proof and benchmarking test results. 
  • Work together to find ways to improve business operations while identifying tasks that would benefit from automation.
  • Create better AI audit methods and testing procedures and evaluation checklists.

Essential Qualifications:

  • Require applicants to have at least a Bachelor’s degree or equivalent academic qualification. 
  • Obtained a minimum of two years work experience in AI benchmarking and auditing and related fields. 
  • Possess strong analytical abilities and problem-solving skills for evaluating AI models. 
  • Worked with datasets and annotations and model output validation. 
  • Demonstrates excellent communication skills which enable him to write reports and engage with stakeholders.

Preferred Skills:

  • Knowledge about Generative AI (Gen-AI) and Large Language Models (LLMs) which they require for their work. 
  • Acquired knowledge about agentic AI solutions and AI auditing frameworks through their professional training. 
  • Expertise in developing automation solutions which enhance process efficiency across multiple areas of their work.
  • Regulations governing AI systems along with privacy requirements and standards for data protection. 
  • Demonstrates the capability to work with international teams while delivering complex information through effective presentation methods.