AI is known to be confidently wrong, and now it’s influencing humans to be that way, too.
In a new study, researchers tested AI’s influence on humans reviewing innovation proposals, and found that AI recommender tools were persuasive enough to convince the evaluators to reject decisions made by independent human experts, thus causing them to pass on promising innovations. Similarly, they went along with AI approval of ideas that the human experts found sub-par.
Interestingly, reviewers were also more inclined to defer to an incorrect AI decision when the model explained itself. Narrative explanations degraded human judgment, rather than enhancing it. People did better when they weren’t given a reason for the AI’s decision.
“Our findings reveal that LLM explanations do not necessarily improve decision-making,” the researchers, associated with Harvard Business School, MIT, and the University of Washington explained in their findings. “Effective human-AI collaboration requires designs that preserve rather than supplant independent human judgment.”
Every enterprise screens proposed projects before pursuing them, but there is always uncertainty, and the risk of trade-offs like false positives (going forward with projects that ultimately fail) or false negatives (rejecting ideas that might have succeeded). For an example of the former, the researchers point to Google Glass or Amazon’s Fire Phone; for the latter, Xerox terminating early Ethernet and PostScript projects.
Because they have limited time and only basic information to go on, decision-makers are increasingly turning to LLMs that use predictive algorithms to generate recommendations and rationales based on context.
The researchers set out to explore AI’s role in what they called “early-stage innovation screening.” They judged how human evaluators were influenced by LLM recommendations, both with and without explanations from the model on how and why it reached its decision.
Their experiment asked 228 experienced evaluators to assess nearly 50 submissions to an MIT challenge. They tested three different scenarios: human-only proposals with no AI assistance; LLM evaluations with a written rationale for the decision; and black-box AI pass-fail recommendations with no accompanying explanation.
Evaluators’ decisions were then compared to those made by four human experts. Those decisions were considered the ‘correct’ baseline. They were judged on whether they outright complied with the LLM’s recommendations, overrode them, or productively overrode them, meaning they independently verified persuasive model outputs before making a decision.
Their decisions were classified as correct (agreeing with human experts’ positive/negative decisions), false positive (supporting submissions that experts would reject), and false negative (rejecting submissions experts would move forward with).
Overall, the evaluators accepted LLM recommendations 67% of the time. They agreed with both black-box and narrative LLM decisions roughly 75% of the time, but only agreed with human decisions 54% of the time.
Seemingly counterintuitively, black-box recommendations improved the quality of decisions (aligning them with human experts) but recommendations with narratives did not. When given an LLM recommendation to reject a submission and an accompanying reason why, evaluators disproportionately agreed, which reduced false positives, but “substantially” increased false negatives.
The researchers posit that this is because narrative explanations “suppress” productive overrides; LLMs provide a convincing argument that is easy to accept, essentially discouraging independent human verification. This contradicts a common assumption that LLM explanations augment human decision-making.
The researchers pointed out that people are cognitively predisposed to weigh negative information more heavily than positive information; the phenomenon is known as ‘negativity bias.’
“Rejection is an active, eliminative decision that feels more consequential and accountable than preserving optionality,” they wrote. It also maintains the status quo, avoids risk and bias, and requires no resource commitment.
