Evaluating Large Language Models for Virtual Survey Responses

Large Language Models as Virtual Survey Respondents: Evaluating Sociodemographic Response Generation

Questionnaire-based surveys serve as the backbone of social science research and public policymaking. However, traditional survey methods are often characterized by high costs, significant time investments, and limitations in scalability. Recent advancements in artificial intelligence, particularly in large language models (LLMs), have prompted exploration into their potential as virtual survey respondents. Yet, existing studies have primarily focused on narrow task settings, specific sociological domains, or lack a cohesive evaluation framework for comprehensive comparisons across various datasets and models.

To bridge these gaps, researchers have introduced two innovative task abstractions: Partial Attribute Simulation (PAS) and Full Attribute Simulation (FAS). These frameworks aim to enhance the understanding of LLMs’ capabilities in generating sociological responses.

• Partial Attribute Simulation (PAS): In this approach, LLMs are tasked with predicting missing attributes from incomplete respondent profiles. This method assesses the models’ ability to infer demographic and sociological data based on limited information.
• Full Attribute Simulation (FAS): This framework involves LLMs generating complete synthetic datasets. It operates under two conditions: zero-context, where the model has no prior information, and context-enhanced, where the model is provided with additional background information. FAS serves as both a diagnostic and exploratory tool to analyze the LLMs’ performance in generating comprehensive datasets.

Recognizing the need for a structured evaluation, the researchers curated LLM-S³, a benchmark that encompasses 11 real-world public datasets across four distinct sociological domains. This benchmark enables systematic testing and evaluation of popular LLMs, specifically GPT-3.5/4 Turbo and LLaMA 3.0/3.1-8B, under both zero-shot and few-shot settings.

The findings from the evaluation reveal several critical insights:

• Performance Trends: Consistent performance trends were observed across different model families, indicating that certain models may excel in generating sociologically relevant data irrespective of the dataset.
• Failure Modes: The study highlighted specific failure modes in structured output generation, drawing attention to areas where LLMs struggle, such as maintaining logical coherence or accuracy in demographic representation.
• Impact of Context and Prompt Design: The research demonstrated how variations in context and the design of prompts significantly influence the fidelity of the simulation. This emphasizes the importance of carefully structuring inputs to maximize response quality.

Ultimately, the research positions LLMs not as replacements for human data collection but as complementary tools that can enhance and expedite the survey process. By integrating LLMs into sociological research, scholars and policymakers may be able to gather insights more efficiently, potentially transforming the landscape of data collection in social sciences.

The code and the datasets used in this research are accessible for further exploration at: https://github.com/dart-lab-research/LLM-S-Cube-Benchmark.

Related AI Insights

• Detecting Defective Task Descriptions in LLM Code Generation
• PwC’s AI-Powered Contract Insights on AWS
• Personalized Worked Examples from Student Code Patterns
• WinkTPG: Advanced Multi-Agent Path Finding with Temporal Reasoning
• Explainable AI Techniques for Food Quality Models
• DepthKV: Layer-Wise KV Cache Pruning for Efficient LLMs
• Eero Signal: Reliable Backup for Business Internet Outages
• Human-AI Governance: Building Trust and Utility in AI
• LLMs for Multi-File DSL Code Generation: BMW Case Study
• On-Device Small Language Models: Mobile Integration Challenges