AI Agents in Behavioral Research

Recent research published on September 2 in Science Advances reveals that artificial intelligence agents, or digital twins designed to mimic human behavior, are not yet ready to completely replace human subjects in behavioral and social science research. Led by computational social scientist Olivier Toubia and his colleagues, the study evaluated digital twins generated by feeding extensive survey data from over 2,000 U.S. respondents into large language models to assess how accurately they could mirror individual human responses across 19 experiments. The authors emphasize that while these digital twins perform better than chance and outperform chatbots relying solely on demographic information by capturing real individual variations, they are still wrong on average about a quarter of the time. The primary finding indicates that AI surrogates distort human views, creating a funhouse mirror effect where responses become excessively homogenous, skewed toward demographic stereotypes, and artificially hyper-rational compared to actual human participants. The research highlights critical structural limitations in current large language models, including static questioning methodologies and biases toward higher trust in others and lower concern regarding technological threats. Despite these shortcomings, the authors note that digital twins hold practical value for pretesting experimental designs, saving human respondent bandwidth, and generating detailed essay-length responses where tired human subjects might provide single-sentence answers. Researchers such as Hadi Hosseini suggest that improving digital twins requires transitioning from static questioning to more dynamic approaches, such as continuous chatbot shadowing or regular conversational tracking. Ultimately, social scientists are urged to maintain realistic expectations regarding synthetic data, recognizing that predicting complex human behavior and psychological nuances through machine learning remains an extraordinary challenge.