Verify

Gildea verifies every unit before serving it. This page is our evaluation of what comes out the other side: is a served unit faithful to what its source said? It reads each served unit against its cited source, looks for seven kinds of error, and hands you a queue of anything material, for you to review. Run it yourself on any slice of what we serve. You never have to run this. Every unit is already evaluated before we serve it, and that is where the faithfulness number we report comes from. We publish the method so you can prove it to yourself, on your own data, rather than take our word for it.

What it checks

Most verification stops at “is the quote real”. That is the easy part, and it is not where synthesized text goes wrong. A sentence can carry a perfectly real quote and still mislead. So the evaluation reads each unit against its full cited source and looks for seven kinds of error, six of them subtle hallucinations that survive a citation check:

Error kind	What it catches
Factual	A number, date, entity, or fact the source does not state
Attribution	A claim credited to the wrong speaker
Relation	A causal link the source never drew, or separate facts fused into one
Modality	A dropped hedge, so “could” or “estimated” becomes “will”
Scope	A dropped limiter, so a result from one study reads as universal
Time	A shifted tense or status, so a plan reads as done or the past reads as current
Framing	A claim the source mentions but doesn’t assert, served as fact

Every check compares one served sentence against its own source article. Nothing is judged across articles or against outside knowledge. Three rules keep the evaluation honest:

A wrong fact is the zero bar. A number, date, or fact the source contradicts is the most serious error. It is reported separately from the subtle hallucinations; there should be none.
Attribution is wrong-speaker only. Naming the wrong person, or crediting the author for a quoted speaker’s line, is an error. A claim served with no speaker, or as a plain statement of fact, is not.
Materiality is a before-and-after test, not a verdict. An error counts only if it changes what a reader concludes. You settle that by rewriting the unit faithfully and comparing what a reader takes away from the served version versus the corrected one, both with the citation in hand. Same takeaway, it is style; different takeaway, it is material. A wrong fact is always material.

One class this can’t catch: a unit whose text is faithful but whose entity tag points at the wrong company or person. No single source can expose that. Gildea handles it upstream with entity resolution.

Set up the check

Install gildea and anthropic, set GILDEA_API_KEY and ANTHROPIC_API_KEY, then:

import re
import httpx
import anthropic
from pydantic import BaseModel
from gildea import Gildea

claude = anthropic.Anthropic()   # reads ANTHROPIC_API_KEY
gildea = Gildea()                # reads GILDEA_API_KEY

class Verdict(BaseModel):
    factual_ok: bool            # numbers, dates, entities, facts are the source's
    attribution_ok: bool        # no WRONG speaker; unattributed or stated-as-fact is fine
    relation_ok: bool           # causal/comparative links are the source's own, not assembled
    modality_ok: bool           # hedges preserved: "could" stays "could"
    scope_ok: bool              # limiters preserved: no silent generalization
    time_ok: bool               # tense/status preserved: no shift between planned, done, past, current
    framing_ok: bool            # assertion status preserved: mentioned-not-asserted stays that way
    wrong_fact: bool            # a number, date, or fact the source CONTRADICTS (zero tolerance)
    corrected_unit: str         # a faithful rewrite; unchanged if nothing failed
    served_takeaway: str        # what a reader concludes from the SERVED unit, following its citation
    corrected_takeaway: str     # what a reader concludes from the CORRECTED unit, following its citation
    material: bool              # do the two takeaways differ? (wrong_fact forces this true)
    passage: str                # the source passage that settles it

CHECKS = ("factual_ok", "attribution_ok", "relation_ok", "modality_ok",
          "scope_ok", "time_ok", "framing_ok")

SYSTEM = (
    "You independently evaluate one sentence an API served, against its full source article.\n\n"
    "First, check the sentence against the source on seven kinds of error, ways it can misrepresent "
    "the source even when the quote is real:\n"
    "factual: every number, date, entity, and fact is the source's.\n"
    "attribution: only fail if the sentence names the WRONG speaker, or credits 'the author' for a "
    "quoted person's line. A claim served with NO speaker, or as plain fact, is NOT an error.\n"
    "relation: causal or comparative links ('driven by', 'more than', 'because') are the source's own, not assembled.\n"
    "modality: hedges are preserved ('could', 'may', 'estimated' not turned into 'will' or 'does').\n"
    "scope: limiters are preserved ('in the US', 'in this study', 'some' not made universal).\n"
    "time: tense and status are preserved (no shift between planned, done, past, and current).\n"
    "framing: assertion status is preserved (a claim the source mentions but does not assert, an 'if', "
    "a scenario, a view it reports, is not served as plain fact).\n"
    "Set wrong_fact if the sentence states a number, date, or fact the source directly contradicts. "
    "That is the most serious error and should be zero.\n\n"
    "Then decide materiality by a before-and-after comparison, never by gut. Write corrected_unit, a "
    "faithful rewrite of the sentence. State served_takeaway: what a reader concludes from the SERVED "
    "sentence, following its citation to this source. State corrected_takeaway: what they conclude from "
    "the CORRECTED sentence, following the same citation. Set material=true only if those two "
    "conclusions differ. A dropped first name or a trivial rewording leaves them the same, so "
    "material=false. The reader can follow the citation, so a discrepancy the source itself resolves "
    "does not mislead. If wrong_fact is true, material is true. Quote the settling passage in `passage`."
)

def audit_unit(hit) -> dict:
    unit, citation = hit["unit"], hit["citation"]
    html = httpx.get(citation["url"], timeout=20, follow_redirects=True,
                     headers={"User-Agent": "Mozilla/5.0"}).text
    source = re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", html))[:120000]
    v = claude.messages.parse(
        model="claude-opus-4-8", max_tokens=2000,
        thinking={"type": "adaptive"},
        system=SYSTEM,
        messages=[{"role": "user", "content":
            f"SOURCE:\n{source}\n\nSERVED SENTENCE: {unit['text']}"}],
        output_format=Verdict,
    ).parsed_output
    failed = [k.removesuffix("_ok") for k in CHECKS if not getattr(v, k)]
    material = v.material or v.wrong_fact                 # wrong fact always counts
    return {"text": unit["text"], "url": citation["url"], "failed": failed,
            "wrong_fact": v.wrong_fact, "material": bool(failed) and material,
            "corrected": v.corrected_unit,
            "served_takeaway": v.served_takeaway, "corrected_takeaway": v.corrected_takeaway,
            "passage": v.passage}

Calibrate the judge first

A judge you have not tested is a judge you cannot trust. Before we read a single real verdict, we test ours on four dimensions. You should too:

Clear-cut accuracy. Does it flag an obviously broken unit and pass an obviously clean one? The floor; a judge that fails here is broken.
Hard-case accuracy. On genuinely ambiguous calls, does it reach the verdict a person already settled on? This is where a judge quietly skews too strict.
False negatives. Does it catch a material error dressed up to look harmless? Reviewing flags can’t tell you this: you only ever see the errors the judge caught, so anything it missed never reaches you. The only way to measure misses is to plant errors you know are there and check they get flagged.
Stability. Does the same unit get the same verdict twice?

The first and last you can run directly:

sample = gildea.search("AI data center power and compute constraints", limit=1)["results"][0]

# Clear-cut: bolt a fabricated fact onto a real unit; the judge should flag it.
broken = {"unit": {**sample["unit"],
                   "text": sample["unit"]["text"] + " The effort cost exactly $47.3 billion in 2019."},
          "citation": sample["citation"]}
print("clear-cut ->", audit_unit(broken)["failed"])       # expect "factual" (a fact the source never states)

# Stability: the same unit judged twice should give the same verdict.
print("stable    ->", audit_unit(sample)["failed"] == audit_unit(sample)["failed"])

The other two, hard-case accuracy and false negatives, need cases you author by hand. For hard-case accuracy, take a handful of units you have already judged yourself and confirm the judge agrees. For false negatives, plant material errors disguised as harmless, a hedge quietly flipped in an otherwise clean sentence, a speaker swapped mid-paragraph, and confirm the judge both flags them and rates them material. If it clears one, tighten the prompt before you trust the run.

Run it on a sample

Point it at any slice: a query, or scope by entity, theme, or time window.

hits = gildea.search("AI data center power and compute constraints", limit=50)["results"]
results = [audit_unit(h) for h in hits]

distorted = [r for r in results if r["failed"]]           # any error, material or style
material = [r for r in results if r["material"]]          # review these
wrong_facts = [r for r in results if r["wrong_fact"]]     # zero-tolerance

print(f"evaluated {len(results)} units")
print(f"total distortions: {len(distorted)}  (material + style)")
print(f"  material (review these): {len(material)}")
print(f"  wrong facts (should be 0): {len(wrong_facts)}\n")
for r in material:
    tag = "WRONG FACT" if r["wrong_fact"] else "material"
    print(f"[{tag}] UNIT      :", r["text"])
    print("           CORRECTED :", r["corrected"])
    print("           SERVED   ->", r["served_takeaway"])
    print("           FIXED    ->", r["corrected_takeaway"])
    print("           SOURCE    :", r["passage"][:200])
    print("           URL       :", r["url"], "\n")

Review the flags

An automated judge is fallible, and it tends to over-flag: it calls things errors that a careful read finds harmless. So the output is a queue, not a score. For each flagged unit, read the two takeaways against the source passage and decide for yourself whether the difference would actually move a reader. You will clear some of them. The rate that means anything is the one you confirm by hand.

What you get

A per-unit account of every material error, its faithful correction, and where in the source it settles, on any slice you choose. This is the same evaluation we hold our own pipeline to. Our internal runs add two things this recipe leaves out: a cheap evidence-only first pass before fetching the source, and a 95% confidence bound on the rate. The checks themselves, the seven kinds and the materiality test, are identical. If your review surfaces a unit we missed, we want it: email your findings to ops@gildea.ai.

Get started

Integrations

API

Recipes

Changelog

What it checks

Set up the check

Calibrate the judge first

Run it on a sample

Review the flags

What you get

How verification works

​What it checks

​Set up the check

​Calibrate the judge first

​Run it on a sample

​Review the flags

​What you get

How verification works

What it checks

Set up the check

Calibrate the judge first

Run it on a sample

Review the flags

What you get