AI Conversation Analyst

What You'll Do

• Analyse AI conversations: review AI-generated voice calls and in-app conversations and evaluate quality across accuracy, context understanding, relevance, naturalness, tone and empathy, conversation flow, goal completion, instruction adherence, objection handling, and customer satisfaction.
• Investigate AI failures: identify and document hallucinations, gibberish responses, incorrect information, context loss, repetitive replies, poor reasoning, incorrect pronunciation, speech interruptions, misread intent, weak objection handling, conversation loops, and failures to complete business objectives.
• Perform root cause analysis: pinpoint the underlying cause, whether prompt-related, knowledge base gaps, tool failures, context management problems, speech recognition errors, text-to-speech issues, workflow limitations, or product bugs.
• Identify patterns at scale: review hundreds or thousands of conversations to surface recurring failure modes, customer-specific issues, quality trends, and emerging edge cases.
• Build quality frameworks: contribute to conversation scorecards, evaluation rubrics, quality benchmarks, failure taxonomies, review guidelines, and human-in-the-loop evaluation processes.
• Deliver actionable insights: prepare recurring quality reports covering failure categories, severity, trends, agent performance, and improvement recommendations.
• Work with AI to improve AI: use LLMs, transcription tools, analytics platforms, and internal observability tools to make analysis faster and more scalable.

What We're Looking For

You're a great fit if you have:

• Strong analytical and problem-solving skills, with excellent attention to detail.
• Strong written and verbal communication, and the ability to work with both technical and business teams.
• The ability to spot subtle conversation issues and explain why they matter.
• Structured thinking and strong documentation skills.
• Genuine curiosity about AI and Large Language Models.
• The ability to work independently with high ownership, and to review large volumes of conversations.
• Comfort thriving in a fast-paced startup environment.

Nice to Have

• Experience with Voice QA or Customer Support QA.
• Experience with AI products, chatbots, or conversational AI.
• Familiarity with ChatGPT, Claude, Gemini, or similar LLMs, and concepts like prompting, hallucinations, context windows, and AI evaluation.
• Experience with Excel, Google Sheets, SQL, or Python.
• Experience designing QA scorecards or evaluation frameworks.
• Experience working with enterprise customers.

Success in This Role

Your impact will be measured by:

• The quality and accuracy of your conversation reviews and root-cause identification.
• The number of actionable insights you deliver, and the improvements implemented from them.
• A measurable reduction in recurring AI failures and improvement in AI quality scores.
• Healthy review turnaround time and conversation coverage.
• Client satisfaction with your quality reports.

Why Join RevRag AI

• Work on cutting-edge Voice and Conversational AI used in production.
• Learn directly from experienced AI engineers and product leaders.
• Influence how enterprise AI agents are built, evaluated, and improved.
• Gain exposure to LLMs, AI evaluation, prompt engineering, and observability.
• Be part of a fast-growing startup solving real customer problems with AI.