Evaluate AI Output Quality

Introduction

The AI Output Evaluator and Hallucination Checker is a workflow designed to rigorously assess answers generated by AI models. It evaluates the quality, faithfulness, and potential hallucinations within a model answer relative to the original prompt and any supplied context documents.

It performs these steps:

1. Accepts the original user prompt, optional context documents, and the AI-generated answer as input.
2. Parses this input safely to prepare it for analysis.
3. Runs a specialized AI evaluator model that judges the answer across multiple criteria such as helpfulness, relevance, grounding to facts, clarity, and evaluator confidence.
4. Identifies any hallucinated claims or unsupported facts within the answer.
5. Outputs a structured JSON report including numerical scores, flags indicating risks or hallucinations, detailed reasoning, and metadata about the evaluation.

What You Need

1. Access to the Needle platform to run the workflow.
2. Input data matching the required schema.

How The Flow Works

1. Manual Trigger: Starts the workflow with predefined inputs.
2. Parse Input Node: Safely parses the input data into a structured object, applying defaults to any missing fields.
3. AI Evaluator: Runs a strict AI-based evaluation model that acts as a judge rather than a generator. It identifies factual claims in the answer, checks them against the context or general knowledge, and scores multiple dimensions from 1 to 5. It also flags hallucinations, policy risks, the need for human review, and provides overall explanatory reasoning.
4. Post Processing Code: Cleans and clamps the numeric scores to valid ranges, organizes flags and explanations, and appends metadata such as evaluation timestamp and model version.
5. Output Node: Produces a nested JSON output summarizing all evaluation details.

Output Metrics

At the end, users get a detailed JSON object containing structured evaluation data.

Notes

1. The workflow strictly does not generate or improve answers; its sole purpose is assessment and governance.
2. It follows best practices such as chain of thought evaluation and quantitative rubrics for transparency.
3. The hallucination check is strict. Any unsupported or fabricated claims trigger a hallucination flag.
4. Users should provide relevant context documents if available to improve grounding accuracy.
5. The workflow flags answers needing human review if any scores are low or risks are detected.
6. This setup provides a strong foundation for automated quality control of AI outputs in sensitive applications.