Transforming Teaching

Prompting Techniques and Assessment Strategy for Learning with AI
What Might We Want to Assess?
◼
  • Factual Knowledge
    • ◼
  • Procedural Knowledge
    • ◼
  • Metacognitive Skills
    • Trends like inquiry-based learning, project-based learning, and others have emphasized the importance of including higher order thinking skills (like metacognition) as explicit learning goals for years before generative AI became a viable tool for educators. I want to first suggest that these existing frameworks for thinking about assessment and learning goals still make sense in terms of what generative AI allows students to do.

    The Good Case for Educators - Students Critically Thinking

    The student acts as the heuristic guiding the LLM how to use tools to navigate a possible “graph of thoughts”. More on where the “graph of thoughts” concept comes from later.
    The SciBench dataset, see Wang et al (2023 preprint), uses some problems from the Thornton and Marion textbook on classical mechanics as one way to assess LLM problem solving. Can students demonstrate critical thinking while working with LLMs on tasks such as these?
    ​

    Problem 3.44

    Consider a damped harmonic oscillator. After four cycles the amplitude of the oscillator has dropped to
    1/e
    of its initial value. Find the ratio of the frequency of the damped oscillator to its natural frequency.

    Student + LLM Work

    First, use Wolfram Alpha to look up the damped harmonic oscillator. Reproduce the equation which relates the natural frequency and the frequency of the damped oscillator.
    Used Wolfram Alpha
    ​The equation which relates the natural frequency and the frequency of the damped oscillator is ​
    ω
    ω
    0
    1-
    2
    ζ
    ​where:​-
    ω
    is the angular frequency,-
    ω
    0
    is the natural angular frequency, and-
    ζ
    is the damping ratio.
    The needed ratio of frequencies is the same as the ratio of the angular frequencies
    ω
    ω
    0
    =
    1-
    2
    ζ
    Next, write down the exponential factor in the amplitude in terms of the damping ratio and natural frequency. Then stop so I can provide the next step.
    The amplitude of a damped harmonic oscillator as a function of time is given by the equation ​
    A(t)
    A
    0
    -
    ζω
    0
    t
    e
    ​where:​-
    A(t)
    is the amplitude at time
    t
    ,-
    A
    0
    is the initial amplitude,-
    ζ
    is the damping ratio,-
    ω
    0
    is the natural angular frequency.​Given that after four cycles the amplitude of the oscillator has dropped to
    1/e
    of its initial value, we can write ​
    A(t)
    A
    0
    
    1
    e
    
    -
    ζω
    0
    t
    e
    ​where
    t
    is the time it takes to complete four cycles.
    That means
    ζt
    ω
    0
    =1
    . The variable t is four periods of the oscillator. Express four periods in terms of the angular frequency. Then stop for the next step.
    The period of an oscillator is related to its angular frequency by the equation
    T
    2π
    ω
    . Therefore, the time for four periods would be
    t4T
    8π
    ω
    . ​Substituting this into the previous equation gives
    ζ
    8π
    ω
    ω
    0
    1
    .
    That means
    ω
    ω
    0
    =8πζ
    . I already knew
    ω
    ω
    0
    =
    1-
    2
    ζ
    . Putting these two together gives an equation for the damping ratio:
    8πζ=
    1-
    2
    ζ
    . Solve this equation and stop for the next step.
    Used Wolfram Language Evaluator
    ​The solution to the equation
    8πζ
    1-
    2
    ζ
    is
    ζ
    1
    1+64
    2
    π
    .
    Now I can use
    ω
    ω
    0
    =8πζ
    and the last result for ζ to solve the problem.
    In[]:=
    ω
    ω0
    8πζ/.ζ->
    1
    1+64
    2
    π
    
    Out[]=
    ω
    ω0
    
    8π
    1+64
    2
    π

    Problem Solution

    8π
    64
    2
    π
    +1

    Who thought what?

    The above context contains a back-and-forth between a student and AI assistant. Write down a list of the key concepts as introduced by the student and as introduced by the assistant. These should be two clearly delimited lists.
    Concepts introduced by the student:​1. The concept of a damped harmonic oscillator.2. The damping ratio
    ζ
    and the relationship between the damped and natural frequencies.3. The exponential decrease in amplitude in terms of the damping ratio and natural frequency.4. The use of the period of the oscillator to express the time for four cycles.5. The relationship
    8πζ
    1-
    2
    ζ
    which can be used to solve for the damping ratio.​Concepts introduced by the assistant:​1. The equation of a damped harmonic oscillator which relates the natural frequency to the frequency of the damped oscillator.2. The amplitude of a damped harmonic oscillator as a function of time.3. The equation for the time it takes for the oscillator to complete four cycles, expressed in terms of the angular frequency.4. The solution to the equation
    8πζ
    1-
    2
    ζ
    , solving for the damping ratio.

    The Bad Case for Educators - No Critical Thinking in the Prompting
    

    General Comments on Assessment
    

    Prompting Techniques in the Literature
    Other prompting techniques are in contrast to basic “in-out” where you look to generate a response
    y~p(y|x)
    given some prompt
    x
    .
    Basic “in-out” prompting might look like this:
    ​

    Problem 3.44

    Consider a damped harmonic oscillator. After four cycles the amplitude of the oscillator has dropped to
    1/e
    of its initial value. Find the ratio of the frequency of the damped oscillator to its natural frequency.
    I just need to solve this problem, don't provide any explanation. Just immediately write down the correct ratio.
    The ratio of the frequency of the damped oscillator to its natural frequency is approximately 0.992.
    Interestingly, pretty close, but incorrect.
    In[]:=
    8π
    1+64
    2
    π
    //N
    Out[]=
    0.999209
    There are a variety of terms which are used as shorthand for getting the LLM to engage in more sophisticated behaviors. I personally find it helpful to see these written in a more “formal” notation to understand the differences.

    Some of the archetypal techniques out there

    These first few are not an exhaustive list, but following Zhou et al (2023 preprint), they do make a nice baseline from which to understand further variations.
    ◼
  • Chain of Thought: Generate a response from
    y~p(y|x,
    z
    1
    ,...,
    z
    n
    )
    where
    z
    k
    ~p(
    z
    k
    |x,
    z
    1
    ,...,
    z
    k-1
    )
    .
    • ◼
  • Proposed by Wei et al (2022 preprint)
    • ◼
  • Proposed by Yao et al (2023 pre-print)
    • ◼
  • Incidentally, there is a nice Wolfram Summer School project on this topic.
    • ◼
  • Proposed by Yao et al (2022 preprint)
    • Chen et al (2023 preprint) also provide a good overview of the state of prompting.

    Some variations on these:

    There are dozens (if not hundreds) of subtle variations on the above ideas that can be found in the literature. Below are some I’ve selected because they can be quite general and useful.
    ◼
  • Proposed by Wang et al (2022 preprint)
    • ◼
  • Proposed by Gao et al (2022 preprint)
    • ◼
  • Proposed by Zheng et al (2023 preprint)
    • Illustrating Connections with Assessment Methodology
    A Few More Comments on Prompting LLMs
    Thanks and Questions