How to Fact-Check AI Podcast Show Notes

AI show notes go wrong in two ways. The first is sounding generic, which we cover in our post on building a voice guide. The second is being wrong: a quote the guest never said, a number that drifted, a book title that does not exist. The second problem is worse, because a wrong quote attributed to a real guest is a mistake with their name on it.

Language models write fluent text by predicting what comes next, and they do not check facts as they go. Given a full transcript they tend to do better than without one. They can still smooth a quote into something more quotable, round a number, or fill a gap with a plausible detail. The advice most sources give is to verify each quote word for word. That is right, and doing it by eye across a dozen quotes and figures is slow enough that people skip it.

This post gives you a checklist for what to verify and a short script that does the mechanical part: it checks that every quoted line in the notes appears in the transcript, flags near-misses, and lists numbers it cannot find.

What to verify

The script handles the first two rows. The rest need your judgment, though a good glossary from the transcript cleanup post helps with names.

The script

Save this as check_quotes.py. It reads a show-notes file and a transcript, finds every passage in double quotes that is at least 20 characters long, and reports whether it appears in the transcript. It ignores case and punctuation so a comma or a capital letter does not cause a false alarm. If a quote is not found exactly, it looks for the closest stretch of the same length and reports how similar it is.

#!/usr/bin/env python3
"""Check that quotes and numbers in AI-written show notes really appear in the transcript."""
import difflib
import re
import sys
from pathlib import Path


def normalize(text):
    return " ".join(re.sub(r"[^a-z0-9%' ]+", " ", text.lower()).split())


def best_match(quote, words):
    size = len(quote.split())
    target = quote
    best = 0.0
    for i in range(max(1, len(words) - size + 1)):
        window = " ".join(words[i:i + size])
        best = max(best, difflib.SequenceMatcher(None, target, window).ratio())
        if best == 1.0:
            break
    return best


def main(notes_file, transcript_file):
    notes = Path(notes_file).read_text(errors="ignore")
    transcript = normalize(Path(transcript_file).read_text(errors="ignore"))
    words = transcript.split()
    quotes = re.findall(r'["“]([^"”]{20,})["”]', notes)
    for raw in quotes:
        quote = normalize(raw)
        if quote in transcript:
            status = "FOUND    "
        else:
            score = best_match(quote, words)
            status = f"CLOSE {score:.2f}" if score >= 0.85 else "NOT FOUND"
        print(f"{status}  \"{raw}\"")
    numbers = sorted(set(re.findall(r"\b\d[\d,.]*%?", notes)))
    missing = [n for n in numbers if n.strip(",.") not in transcript]
    if missing:
        print("\nNumbers in the notes but not in the transcript (check by hand):", ", ".join(missing))


if __name__ == "__main__":
    if len(sys.argv) != 3:
        sys.exit("usage: check_quotes.py <show-notes.md> <transcript.txt>")
    main(sys.argv[1], sys.argv[2])

Run it with python3 check_quotes.py show-notes.md transcript.txt. On a small invented example it prints:

FOUND      "I raised the price by 30 percent and nobody left"
NOT FOUND  "She said it was the worst decision ever made"
CLOSE 0.99  "it was a good decission for the shop"

Numbers in the notes but not in the transcript (check by hand): 200

Read the three statuses like this.

• FOUND means the quote appears in the transcript, apart from case and punctuation. You still need to check that it is attributed to the right speaker and is not pulled out of context.
• CLOSE means the quote is nearly there, usually with a word swapped or a spelling difference. Open the transcript, compare, and fix the notes to match what was said.
• NOT FOUND means no similar passage exists. Treat it as invented until you find it yourself, and delete or rewrite it.

What the script cannot do

It has limits, and knowing them is part of using it responsibly.

• It cannot judge meaning. A real quote can be trimmed so that it says the opposite of what the guest meant. Read each FOUND quote in its surrounding sentences.
• Numbers in transcripts are often spoken as words. A guest who says "thirty percent" may be written as "30 percent" or "30%" depending on the transcriber, so a number can be flagged even when it is correct. That is why the output says to check by hand, not that the number is wrong.
• Punctuation and filler words differ between spoken and written text. If the AI cleaned up "uh, we, we tried it," a close match is expected.
• It checks a single transcript at a time and does not check timestamps. If your notes include times, open the audio and land on a few of them.

Make the model easier to check

You can cut the fact-checking work by changing what you ask for.

1. Ask for quotes with the transcript line number or timestamp beside each one.
2. Tell the model to use only the transcript and to say "not in the transcript" instead of guessing.
3. Ask for links only when the URL was spoken in the episode, and list them separately so you can open each.
4. Keep a Known mistakes section in your setup, as described in our Claude and ChatGPT workflow post, and add every fabricated detail you catch.

A five-minute review routine

1. Run the script and fix every CLOSE and NOT FOUND.
2. Search the transcript for each flagged number.
3. Open every link.
4. Scrub to two or three timestamps.
5. Read the notes aloud once, thinking about whether the guest would recognize themselves in them.

The time limit is a target for a normal episode, not a rule. Longer episodes with many quotes take longer.

Checklist

• Script saved and run against every set of AI-written notes
• All CLOSE and NOT FOUND quotes fixed or removed
• Every number found in the transcript
• Every link opened
• Two or three timestamps spot-checked
• Each new fabrication added to your Known mistakes list