“Bullshit Index” Tracks AI Misinformation

Despite their impressive language capabilities, today’s leading AI models have a patchy relationship with the truth. A new “bullshit index” could help quantify the extent to which they are making things up and also find ways to curtail the behavior. Large language models (LLMs) have a well-documented tendency to produce convincing sounding but factually inaccurate responses, a phenomenon which has been dubbed hallucinating. But this is just the tip of the iceberg, says Jaime Fernández Fisac, an assistant professor of electrical and computer engineering at Princeton University. In a recent paper, his group introduced the idea of “machine bullshit” to encompass the range of ways that LLMs skirt around the truth. As well as outright falsehoods, they found that these models often use ambiguous language, partial truths, or flattery to mislead users. And crucially, widely used training techniques appear to exacerbate the problem. IEEE Spectrum spoke to Fernández Fisac and the paper’s first a