Transforming Teaching
Transforming Teaching
Prompting Techniques and Assessment Strategy for Learning with AI
What Might We Want to Assess?
◼
Factual Knowledge
◼
Procedural Knowledge
◼
Metacognitive Skills
Trends like inquiry-based learning, project-based learning, and others have emphasized the importance of including higher order thinking skills (like metacognition) as explicit learning goals for years before generative AI became a viable tool for educators. I want to first suggest that these existing frameworks for thinking about assessment and learning goals still make sense in terms of what generative AI allows students to do.
The Good Case for Educators - Students Critically Thinking
The Good Case for Educators - Students Critically Thinking
The student acts as the heuristic guiding the LLM how to use tools to navigate a possible “graph of thoughts”. More on where the “graph of thoughts” concept comes from later.
The SciBench dataset, see Wang et al (2023 preprint), uses some problems from the Thornton and Marion textbook on classical mechanics as one way to assess LLM problem solving. Can students demonstrate critical thinking while working with LLMs on tasks such as these?
Problem 3.44
Problem 3.44
Consider a damped harmonic oscillator. After four cycles the amplitude of the oscillator has dropped to of its initial value. Find the ratio of the frequency of the damped oscillator to its natural frequency.
1/e
Student + LLM Work
Student + LLM Work
First, use Wolfram Alpha to look up the damped harmonic oscillator. Reproduce the equation which relates the natural frequency and the frequency of the damped oscillator.
ω
ω
0
1-
2
ζ
ω
ω
0
ζ
The needed ratio of frequencies is the same as the ratio of the angular frequencies
ω
ω
0
1-
2
ζ
Next, write down the exponential factor in the amplitude in terms of the damping ratio and natural frequency. Then stop so I can provide the next step.
The amplitude of a damped harmonic oscillator as a function of time is given by the equation where:- is the amplitude at time ,- is the initial amplitude,- is the damping ratio,- is the natural angular frequency.Given that after four cycles the amplitude of the oscillator has dropped to of its initial value, we can write where is the time it takes to complete four cycles.
A(t)
A
0
-t
ζω
0
e
A(t)
t
A
0
ζ
ω
0
1/e
A(t)
A
0
1
e
-t
ζω
0
e
t
That means . The variable t is four periods of the oscillator. Express four periods in terms of the angular frequency. Then stop for the next step.
ζt=1
ω
0
The period of an oscillator is related to its angular frequency by the equation . Therefore, the time for four periods would be . Substituting this into the previous equation gives .
T
2π
ω
t4T
8π
ω
ζ1
8π
ω
ω
0
That means =8πζ. I already knew =. Putting these two together gives an equation for the damping ratio: . Solve this equation and stop for the next step.
ω
ω
0
ω
ω
0
1-
2
ζ
8πζ=
1-
2
ζ
8πζ
1-
2
ζ
ζ
1
1+64
2
π
Now I can use =8πζ and the last result for ζ to solve the problem.
ω
ω
0
In[]:=
ω
ω0
1
1+64
2
π
Out[]=
ω
ω0
8π
1+64
2
π
Problem Solution
Problem Solution
8π
64+1
2
π
Who thought what?
Who thought what?
The above context contains a back-and-forth between a student and AI assistant. Write down a list of the key concepts as introduced by the student and as introduced by the assistant. These should be two clearly delimited lists.
Concepts introduced by the student:1. The concept of a damped harmonic oscillator.2. The damping ratio and the relationship between the damped and natural frequencies.3. The exponential decrease in amplitude in terms of the damping ratio and natural frequency.4. The use of the period of the oscillator to express the time for four cycles.5. The relationship which can be used to solve for the damping ratio.Concepts introduced by the assistant:1. The equation of a damped harmonic oscillator which relates the natural frequency to the frequency of the damped oscillator.2. The amplitude of a damped harmonic oscillator as a function of time.3. The equation for the time it takes for the oscillator to complete four cycles, expressed in terms of the angular frequency.4. The solution to the equation , solving for the damping ratio.
ζ
8πζ
1-
2
ζ
8πζ
1-
2
ζ
The Bad Case for Educators - No Critical Thinking in the Prompting
The Bad Case for Educators - No Critical Thinking in the Prompting
General Comments on Assessment
General Comments on Assessment
Prompting Techniques in the Literature
Other prompting techniques are in contrast to basic “in-out” where you look to generate a response given some prompt .
y~p(y|x)
x
Basic “in-out” prompting might look like this:
Problem 3.44
Problem 3.44
Consider a damped harmonic oscillator. After four cycles the amplitude of the oscillator has dropped to of its initial value. Find the ratio of the frequency of the damped oscillator to its natural frequency.
1/e
I just need to solve this problem, don't provide any explanation. Just immediately write down the correct ratio.
The ratio of the frequency of the damped oscillator to its natural frequency is approximately 0.992.
Interestingly, pretty close, but incorrect.
In[]:=
8π
1+64
2
π
Out[]=
0.999209
There are a variety of terms which are used as shorthand for getting the LLM to engage in more sophisticated behaviors. I personally find it helpful to see these written in a more “formal” notation to understand the differences.
Some of the archetypal techniques out there
Some of the archetypal techniques out there
These first few are not an exhaustive list, but following Zhou et al (2023 preprint), they do make a nice baseline from which to understand further variations.
◼
Chain of Thought: Generate a response from where ~p(|x,,...,).
y~p(y|x,,...,)
z
1
z
n
z
k
z
k
z
1
z
k-1
◼
Proposed by Wei et al (2022 preprint)
◼
Proposed by Yao et al (2023 pre-print)
◼
◼
Proposed by Yao et al (2022 preprint)
Chen et al (2023 preprint) also provide a good overview of the state of prompting.
Some variations on these:
Some variations on these:
There are dozens (if not hundreds) of subtle variations on the above ideas that can be found in the literature. Below are some I’ve selected because they can be quite general and useful.
◼
Proposed by Wang et al (2022 preprint)
◼
Proposed by Gao et al (2022 preprint)
◼
Proposed by Zheng et al (2023 preprint)
Illustrating Connections with Assessment Methodology
A Few More Comments on Prompting LLMs
Thanks and Questions