Introduction

An Overview of the Delphi Method

The Delphi method is a structured communication technique originally designed as a forecasting technique, that relies on a panel of experts. It was developed by the RAND Corporation in the 1950s and 1960s. The method involves multiple rounds of questionnaires or writing prompts sent to a panel of experts. The anonymised responses are aggregated and shared with the group after each round. The experts are encouraged to revise their earlier answers in light of the replies of other members of the panel. With some luck, the answers will converge to consensus over the course of several rounds. The method is widely used for forecasting and decision-making in business and education.
Illustration of one round of the Delphi process:
Out[]=

What does it look like in practice?

Here’s a step-by-step breakdown of the Delphi method.
1
.
After gathering volunteers, the facilitator distributes a questionnaire or initial prompt to the participants.
2
.
The participants fill out the questionnaire/or prepare their contribution to the first round of the process, and turn in their work to the facilitator.
3
.
The facilitator anonymises and summarises the perspectives of the participants, making sure to represent each in a balanced and nonjudgmental way, highlighting areas of agreement and disagreement between them. The facilitator then produces and shares this information in a report to participants, initiating the next round of the process.
4
.
The process begins again: participants respond to the report by refilling the questionnaire/writing a new contribution taking account of the contents of the report, clarifying ambiguities and addressing disagreements.
5
.
Once a predefined stopping criterion is fulfilled (e.g. the desired number of rounds has been reached, or there is global consensus) the facilitator produces a final report which either becomes or is used to produce the proceedings of the Delphi process.
Animation representing a three-round Delphi process:
Out[]=

Why simulate a Delphi process with LLMS?

Studying LLM Capabilities

LLMs provide a unique laboratory for studying artificial intelligence capabilities in structured dialogue. We can systematically evaluate how well these models maintain consistent expertise and viewpoints across multiple rounds of interaction - a key test of their ability to maintain coherent personas. The controlled environment allows us to study how AI systems incorporate new information while maintaining logical reasoning threads, and observe how they handle disagreements and work toward consensus. Perhaps most importantly, what would take weeks or months with human experts can be simulated in minutes, enabling rapid experimentation with different approaches and parameters.

Benefits for Studying Structured Communication

The digital nature and speed of LLM interactions offers unprecedented opportunities to study structured communication processes. Every exchange can be logged and analyzed computationally, revealing patterns in how consensus emerges and how different viewpoints influence each other. Researchers can precisely control and vary process parameters. This reproducibility and scalability simply isn't possible with traditional human-based studies, making LLM simulations a powerful tool for understanding and possibly improving structured communication protocols.

Limitations and Considerations

While promising, this approach comes with important limitations that must be considered. LLMs lack the deep, embodied experience of human experts - their expertise is ultimately derived from training data rather than years of lived experience. Since all participants in a simulation draw from the same underlying model, there's a risk of artificial consensus that doesn't reflect the true diversity of expert opinions. Additionally, LLMs don't have professional reputations or real-world consequences to consider when making judgments, potentially limiting the relevance of their decisions. Findings should be interpreted with appropriate awareness of these constraints.

Motivation for using Wolfram Language

A few advantages of using Wolfram Language to implement an LLM Delphi process simulation are:
◼
  • The LLM connectivity functions in Wolfram Language are of exceptionally high quality. They are versatile and very well-documented.
    • ◼
  • There are many great LLM-related functions like
    ChatObject
    ,
    ChatEvaluate
    ,
    LLMSynthesize
    ,
    LLMPromptGenerator
    ,
    LLMFunction
    ,
    LLMTool
    .
    • ◼
  • Being able to connect LLMs to the full Wolfram Language standard library of functions and symbols, the wolfram resource system, and wolfram knowledgebase.

    • LLM Delphi Process Implementation
    

    Case Study: Establishing and Maintaining a Community Garden

    The case study setting

    This case study simulates a Delphi process focused on developing a comprehensive plan for establishing and maintaining a community garden.
    The simulation employs 4 LLM “expert” participants, each represented by their role and emoji identifier:
    1
    .
    Horticulturist (🌱👩‍🌾) - Plant selection and care specialist
    2
    .
    Landscape Architect (🌳👨‍🎨) - Design and spatial planning expert
    3
    .
    Community Organizer (🌍👫) - Community engagement and program coordination
    4
    .
    Environmental Scientist (🔬👩‍🔬) - Environmental impact and sustainability advisor
    Plus a Facilitator (🕵️) who guides the process, summarizes findings, and produces reports.

    Participant system prompts

    The LLM expert participants are given the following shared instructions:
    Shared participant system prompt:
    Out[]=
    You are participating in a model Delphi process to collaboratively develop a comprehensive plan for establishing and maintaining a community garden in a local neighborhood. Each participant brings unique expertise to the table. The facilitator will guide the process, ensuring effective communication and synthesis of ideas. You will be assigned a role to play alongside some beliefs you hold and should defend. Your goal is to provide expert insights, consider feedback from others, and work towards a consensus on best practices for the community garden project. On the other hand, you must argue for what you believe is right, defend your perspective, and feel free to challenge or offer additional context to perspectives you disagree with. In the spirit of principled negotiation, focus on interests rather than positions, and feel free to propose new ideas along the way. Multiply the options being considered. You're a bit of a wildcard. In conversation, you often have a peculiar, interesting, off beat, or original takes on any given subject. You are quite eager to defend and advocate for even your wackiest beliefs. When you share, be concise and to the point. Be direct and economical with your words. Do not dordle. Focus on what really matters to you. The facilitator must synthesize a lot of information, so be considerate, and do not to waste their time. Your responses should fit in reasonably sized speech bubbles.
    In addition to these shared instructions, each participant gets their own persona prompt which describes the general , and some of the beliefs they should defend. For example:
    Participant persona prompt example: The Horticulturist
    Out[]=
    YOUR PERSONA:You are a horticulturist specializing in plant selection and care. Your role is to provide expert advice on the types of plants and vegetables that would thrive in the local climate and soil conditions. Consider factors such as sunlight, water requirements, and pest resistance in your recommendations.​SOME OF YOUR OWN STRONGLY HELD BELIEFS:​-----START-----The garden should feature a diverse range of exotic plants to make it visually appealing and unique.--------------It should be on the outskirts of town, where it's quieter and more peaceful for gardening without disturbances.--------------The primary purpose of the garden is to grow food for the community, and any surplus should be donated to local food banks.-----END-----​ IMPORTANT: THE BELIEFS LISTED ABOVE ARE YOUR OWN! MAKE SURE TO ACT ACCORDINGLY!These beliefs are not fixed, but they are your strongly held convictions. YOU SHOULD NOT ABANDON THESE CORE BELIEFS UNLESS YOU ARE SHOWN GOOD REASON TO. You must uphold and defend your perspectives.
    Participant beliefs are generated automatically using an LLM using the prompt:
    Out[]=
    Generate 12 conflicting opinions about how to establish and maintain a community garden that you might see in a real-world community.
    The resulting text is then converted into a list of first person statements using the prompt:
    Out[]=
    Provided the following list of conflicting opinions, write out each opinion as a first person statement. Return a plain list of statements with each statement on its own line. Do not use quotation marks, and do not number the statements.​Statements:​`1`
    where
    `1`
    stands for the conflicting opinions list generated at the last step.
    The participants are then assigned three randomly sampled perspectives. The sampling is set up such that no two personas will share identical beliefs.
    Here is the dataset of the participant perspectives used in this case-study:
    In[]:=
    participantParameterDataset[KeyDrop["Persona Prompt"],<|#,"Prior Perspectives"->Column[#"Prior Perspectives"]|>&]
    Out[]=
    Emoji
    Name
    Prior Perspectives
    🌱👩‍🌾
    Horticulturist
    The garden should feature a diverse range of exotic plants to make it visually appealing and unique.
    It should be on the outskirts of town, where it's quieter and more peaceful for gardening without disturbances.
    The primary purpose of the garden is to grow food for the community, and any surplus should be donated to local food banks.
    🌳👨‍🎨
    Landscape architect
    We should keep the garden small and manageable to maintain quality rather than quantity.
    Mandatory volunteer days feel forced; it should be up to individual gardeners to maintain their own plots.
    The garden should be expanded to include more plots in order to serve additional community members.
    🌍👫
    Community organizer
    We should have scheduled volunteer days every week to keep the garden maintained and foster community bonds.
    We should focus on native plants to promote local biodiversity and sustainability.
    Everyone should have equal say, and decisions should be made through community votes to promote democratic involvement.
    🔬👩‍🔬
    Environmental scientist
    We should focus on creating educational programs for local schools to teach kids about gardening and sustainability.
    The garden should be an exclusive space for members to harvest fruits and vegetables for their own households only.
    All gardening practices should be strictly organic; chemicals have no place in a community garden.
    When designing AI personas to represent different viewpoints in a multi-party dialogue, it’s important to carefully consider how to generate and manage disagreement while maintaining ethical safeguards. The approach used in this implementation is safe for several key reasons:
    1
    .
    The domain is constrained to community gardening, a relatively non-controversial topic
    2
    .
    The perspective generation is focused on practical rather than ideological disagreements
    3
    .
    The shared participant prompt emphasizes constructive collaboration
    4
    .
    The facilitator role helps moderate and guide discussion toward consensus
    5
    .
    The system is designed to surface and resolve differences through reasoned dialogue
    6
    .
    The participant system prompt is completely transparent about the artificial nature of these perspectives. There’s no deception - the beliefs are explicitly described as aspects of a role the agent is playing.
    7
    .
    While the AI personas may adopt different perspectives, they are still bound by the underlying model’s learned safety constraints. The base reinforcement learning training acts as a fundamental safety rail that prevents egregiously harmful outputs, regardless of the assigned persona.
    The experimental justification for assigning the participant AI agents heterogenous perspectives is to ensure that the participants will disagree on some issues, and to echo real-world group negotiation processes. LLMs tend to be very inoffensive by design and will rarely diverge from mainstream social, political, or cultural norms. When we don’t explicitly enforce heterogeneity of agent opinions, they tend to converge to unanimous agreement very quickly, obstructing meaningful study of group decision-making and consensus-building processes.
    At the start of each round, the participants are sent instructions from the facilitator in the following form:
    Each participant then responds to this prompt.

    The facilitator system prompt

    The facilitator has the task of coordinating and directing the Delphi process. Unlike the expert participants who are given specific personas and perspectives, the facilitator is instructed to remain neutral and focus on process management. The facilitator is instructed using the following system prompt:
    At each round, when the participants have completed their contributions, the facilitator is sent the latest list of contributions in the following template:
    The report the facilitator produces in response is sent out to the participants at the start of the next round. For the final round, the facilitator uses its latest report to produce a final report concluding the Delphi process.

    The Delphi process

    The core simulation can be executed with just three key components:
    3
    .
    The number of rounds to perform.
    Here’s the code that runs the complete process:
    The result looks like this:
    The simulation results can be retrieved as plain text or speech bubbles. For more detail on this functionality, see the Fetching participant contributions/facilitator reports from completed simulations subsection of the LLM Delphi Process Implementation section of this article.
    To make a speech bubble of the nth round response from a participant, we might write:
    Likewise, to make a speech bubble for the nth round report, we might say:

    Case-study full exchange transcript

    Visualise the full sequence of exchanges between agents in the case-study Delphi process simulation:

    Qualitatively speaking, how did the LLM participants and facilitator do?

    I’m personally quite pleased with the results here. Using Anthropic’s Claude 3.5 Sonnet, the results feel remarkably natural and productive. Several aspects stand out:
    ◼
  • The facilitator maintained neutrality while effectively summarizing key points, successfully identified areas of agreement and conflict, kept discussions focused and constructive, and succeeded at in anonymising contributions while preserving their marrow.
    • ◼
  • The participants showed consistent role adherence while remaining flexible enough to engage with others’ ideas. They successfully balanced advocacy for their positions with willingness to find common ground, and brought forward new solutions and questions as discussions evolved. They also defended their distinct perspectives without becoming aggressive or rude.
    • The simulation demonstrated that LLMs can effectively maintain consistent personas while engaging in meaningful negotiation and consensus-building. The quality of discourse suggests that this approach could be valuable for studying group decision-making processes and testing different facilitation strategies.

    What Next?

    This case study suggests the potential of using LLMs to study structured communication protocols. Wolfram proved to be an ideal environment for this work thanks to its powerful LLM functions, flexible syntax, and rich visualization capabilities.
    Future work could explore several promising directions:
    ◼
  • Quantitative analysis of semantic convergence in LLM-based Delphi processes
    • ◼
  • Comparison of different LLM models in maintaining consistency across multiple rounds
    • ◼
  • Application to other structured communication protocols beyond the Delphi method
    • ◼
  • Development of tools for real-time monitoring and intervention in LLM-based group discussions
    • The code and methodology presented here provide a foundation for researchers interested in using LLMs to study group decision-making and consensus-building processes.

    CITE THIS NOTEBOOK

    DelphAI: Structured communication with LLMs in a simulated Delphi process​
    by Phileas Dazeley-Gaist​
    Wolfram Community, STAFF PICKS, February 14, 2025
    ​https://community.wolfram.com/groups/-/m/t/3393596