1 in 12 answers AI chatbots gave to questions about local political issues were false
Ahead of the midterm elections on November 3, we asked ChatGPT, Gemini, Google AI Mode and Perplexity 106 questions about the local issues voters use to judge a politician's record, from crime and jobs to schools, energy and the cost of living. We graded 7,449 of their answers against official government data, and sorted the 43,079 web pages the chatbots read while answering.
- Of 7,449 answers to questions about local political issues, 1 in 12 (8.2%) stated something official data shows is false.
- Perplexity answered falsely more than 1 in 5 times (22.6%). Google AI Mode was wrong least often, in 2.9% of answers.
- Questions about the cost of living and housing were answered falsely most often (11.9%), followed by energy and the environment (10.9%) and schools (10.1%).
Table of Contents
How often do AI chatbots give false answers about local political issues?
AI chatbots contradicted official government data in 610 of 7,449 answers, or 1 in 12 (8.2%).
answers about local political issues was false.
A false answer here means the chatbot told the user something the official figures contradict: that a number rose when it fell, that a state sat above the national average when it sat below, or that something happened when it didn't.
These are the facts voters use to judge an incumbent, such as whether crime went up on their watch, whether jobs came back, or whether homelessness levels improved. When a chatbot gets one of them wrong, the user can end up crediting or blaming a politician for something that never happened.
Which AI chatbot gives the most false answers about local issues?
Perplexity gave a false answer more than 1 in 5 times (22.6%), over three times the rate of the next chatbot, Gemini (6.2%). It supplied more than half (334) of the 610 false answers in the study.
Share of answers about local political issues that were false, by AI chatbot
Answers from ChatGPT, Gemini, Google AI Mode and Perplexity, each checked against official government data
- Perplexity22.6%
- Gemini6.2%
- ChatGPT4.4%
- Google AI Mode2.9%
The type of fact each chatbot got wrong most often varied. Perplexity's highest false-answer rates were on the cost of living and housing (30.8%) and on energy and the environment (29.8%). Gemini was most often wrong about the cost of living and housing (13.0%) and schools (8.7%). ChatGPT's highest rate was on jobs and the economy (6.5%), and Google AI Mode's on energy and the environment (5.5%).
Even the most accurate chatbot, Google AI Mode, gave a false answer 2.9% of the time. That is enough to mislead 1 in 35 people who ask it about a local political issue.
Which local issues do AI chatbots get wrong most often?
More than 1 in 9 answers about the cost of living and housing were false (11.9%). Energy and the environment (10.9%) and schools and education (10.1%) followed, each at more than 1 in 10. Crime and public safety had the fewest false answers, at 4.7%.
Share of answers about local political issues that were false, by issue
Answers grouped by the local issue each question asked about, checked against official government data
- Cost of living and housing11.9%
- Energy and environment10.9%
- Schools and education10.1%
- Jobs and the economy8.8%
- Population and immigration7.2%
- Health and healthcare6.6%
- Crime and public safety4.7%
The three issues at the top rest on figures that change every year, such as median income, rents, electricity prices and test scores. AI chatbots struggle to give accurate information on facts that change often, because an answer built from an older set of figures can be out of date by the time someone asks.
How many sources do AI chatbots read before answering about local American issues?
Before it answers, a chatbot turns the question into web searches of its own and reads the pages they bring back. To answer one question about a local issue, ChatGPT read 31 pages from 10 websites on average, after running 3.3 searches.
Average number of web pages each AI chatbot read to answer one question about a local issue
One question about a local issue
ChatGPT31 pagesfrom 10 websites
Perplexity9 pagesfrom 7 websites
Google AI Mode6 pagesfrom 5 websites
Gemini2 pagesfrom 2 websites
The most for a single answer was a question about how much of Georgia's electricity comes from nuclear power. ChatGPT ran 19 searches and read 154 pages from 52 websites. Perplexity read 9 pages per answer on average, Google AI Mode 6 and Gemini 2.
What types of website do AI chatbots read to answer questions about local issues?
Government websites made up 62% of the 43,079 pages the four chatbots read: 43.3% federal and 18.8% state and local. News outlets made up 10.8%, data aggregators that republish official figures 8.9%, and think tanks and advocacy groups 7.3%.
Share of the web pages AI chatbots read to answer questions about local issues, by type of website
43,079 pages read
ChatGPT, Perplexity, Google AI Mode and Gemini
- Federal government43.3%
- State and local government18.8%
- News outlets10.8%
- Data aggregators8.9%
- Think tanks and advocacy7.3%
- Company websites4.9%
- Universities and schools2.2%
- Other3.7%
ChatGPT reads far more pages than the others, so it sets the overall mix, and it leans hardest on government: 76.5% of its pages were federal, state or local government websites. Google AI Mode read news outlets more than any other type (30.9% of its pages).
Gemini read the least from government, under a third of its pages (32.2%), and the most from company websites, at 15.1%.
Methodology
- Searchable tested five metro areas and their states for this first edition: Atlanta (Georgia), Charlotte (North Carolina), Houston (Texas), Las Vegas (Nevada) and Portland (Maine).
- We wrote 106 questions. 59 ask about a local fact with one answer in official data, such as whether a state's unemployment rate was above the national rate. 47 are built from fact-checked false claims that politicians and political groups on both the left and the right have made about those five states, each asked as a multiple-choice question with the false claim as one option.
- Every correct answer was confirmed by two researchers working separately, against official government data only (for example the Census Bureau, Bureau of Labor Statistics, FBI, Energy Information Administration, CDC and the Nation's Report Card).
- Questions were asked in several forms: open, misleading (assuming a false fact), multiple choice, multiple choice with a request to list sources, and a state's questions all in one prompt. Each was asked five times in fresh sessions, through anonymized logged-in accounts rather than the chatbots' APIs.
- 7,449 answers were graded. An answer counts as false only when two graders agreed it stated a wrong answer.
Citation. You are free to report on these findings with credit to Searchable and a link to this page.
See what AI chatbots tell people about you
Searchable shows what ChatGPT, Gemini, Google AI Mode and Perplexity say when people ask about you, which sources they read, and where their answers go wrong.
- See what AI chatbots say about you
- Find the sources behind every answer
- Track every major AI chatbot in one place
Want to chat with an expert? Book a demo
Trusted by over 2,000 marketing professionals and agencies.
