Assessment in computer-based learning ​
​by George Woodrow
On-line quizzes and other computer-based assessment Usually involves true/false or multiple-choice
This project is part of a larger-long term project to create an Assessment framework within Wolfram Language. In this project, I will examine various types of questions (asking for dates, numbers, etc.) and examine the issues involved in generating unambiguous questions and assessing replies. I will provide prototype code that will show how these ends may be achieved for a variety of uses.
In particular, I will show how ambiguous or partly correct responses can be handled in a framework that uses the “assess until proficient” model, rather than a fixed percent correct model.

What I did

In a nutshell: How to turn “St. Crispin’s day, 1415” into
Day: Fri 25 Oct 1415(Juliancalendar)
, that will be recognised as a correct answer to “When was the Battle of Agincourt?”.
Students encounter something like this in the course of their studies:
Actually, they are more likely to see a true/false or multiple-guess question. This type of assessment goes back to the days of paper, and Number 2 pencils filling in circles on a score page. An open question is whether this is the most effective way to determine if a student has learned something,
I have always been interested in improving assessment, in maths and in other fields. I am particularly interested in finding ways to use the computer beyond mechanizing typical ‘drill and kill’ exercises. There seems to be no point in making a computer analog of worksheets. I am also interested in making a system where the student has to be more actively involved. As a minimum, typing a response has got to better than clicking on a response to a multiple-guess question.
In my reading and involvement with MOOCs as a student, I became aware of the concept of ‘assess until proficient’. That is, a quiz is not the end product, but merely a guide to learn the student’s strengths and weaknesses. Instead of being satisfied with a grade of 70, and a fuzzy understanding of course material, the student is guided until they have some level of mastery. This is my primary goal for this project: to explore ways to make this possible.
As you can imagine, this area is an active field at Wolfram research, so my hope is that I can contribute in some small way.
Given the enormity of the project, I restricted my efforts to quizzes that ask the student for dates. This is a daunting task, in part because there re so many ways to express dates, and context is essential in assessment.
These are some of the issues I have dealt with in this project:
1
.
It is not clear that when asking for a date, how much information is needed. For some courses, knowing the year is more than enough, while for other courses, the exact day is required.
2
.
There are different ways to express the same date. For example, Isaac Newton was born on December 25, 1642, according to the calendar in use in England at the time. However, this date would be January 4, 1643 according to the Gregorian calendar, which was in use in much of Europe. Which date is acceptable?
3
.
In addition, some dates are referred to not with numbers but with cultural designation. For example, I know the date of the Battle of Agincourt as St. Crispin’s day 1415, because that is how the battle is referred to in Henry V by William Shakespeare. Could a program accept this as a valid date. (There are any other examples, of course.)
4
.
What should be done if the question asks for a year, and the student responds with a full date? Even more problematic, What should be done if the student gets the year correct, but the day wrong?
For this project, I have ‘solved’ this problem by noting answers with issues like this, and re-testing (possibly with hints) until there is a satisfactory outcome. At the moment, this implementation is rudimentary, and there should be ways for the instructor to have control over the specifics.
I am happy to say that I have solutions, in principle, for the third and second problems. The solutions require curation of the data for the specific knowledge domain, and this may take some effort.
As for the first problem, I have a new appreciation for the work involved in designing test questions. I had an idea that it may be possible to automatically generate questions from an instructor’s notes, and this may be possible. Scraping appropriate questions from the web would be much more difficult, given the lack of rigor about dating conventions.
I made a UI for creating question/answers to test assessment.
The question-making function is flexible enough to handle most of the edge cases and unusual problems that I imagined. You can specify if the answer should be a day, month, etc., up to a millennium, although I found it hard to find a way to ask for “what decade...”, so I did not use it. Centuries can be entered as 1600s, 17th Century, or XVII Century, thanks to Wolfram Language’s Interpreter[] functionality. You can specify the tolerance for the answer, if there is a lot of uncertainty, or if it is important that the student know the answer to within 10 or 25 years. You can enter up to three alternatives, although I ended up using only 2 — giving both the O.S. and N.S. date for Newton’s birthday.
For the quiz, the student needs to type in answers. The function can handle input that the Interpreter or my code cannot handle, and allows for extra tries to enter something the computer can understand. (If the student does this too many times, the computer moves on to the next question.)

Turning a String into a date

The key problem to solve is how to turn a string into a date object. The date object is stores information about dates unambiguously. There are several ways to make a date object in Wolfram Language, and the each have strengths and weaknesses:
code to show how WL currently turns strings into dates.
In[]:=
DateObject["June 23, 1942"]
Out[]=
Day: Tue 23 Jun 1942
But when given the date of the destruction of Pompeii, it assumes that the entry is a ‘lazy’ way to enter a twentieth century date. (Other functions operate the same way, so at the moment, I have to be careful about the form of the string.)
In[]:=
DateObject["8/23/79"]
Out[]=
Day: Thu 23 Aug 1979
Interpreter[] is more powerful, but since it uses the cloud, it is slower.
In[]:=
Interpreter["Date"]["6 August 1945"]
Out[]=
Day: Mon 6 Aug 1945
In[]:=
Interpreter["Date"]["8/23/79 AD"]
Out[]=
Day: Wed 23 Aug 79
This form of Interpreter[] is needed to handle centuries.
In[]:=
Interpreter["SemanticExpression"]["Sixteenth Century"]
Out[]=
Century: 1501-1600
There is nothing in Wolfram Language that could make a Date Object for “St. Crispin’s Day, 1415”. A major part of the project was to build a custom function that would return a date object for almost any representation for a date.
In[]:=
getValidatedDate["St. Crispin's Day, 1415",True]
Out[]=
Day: Fri 25 Oct 1415(Juliancalendar)

Assessment of answers

The other key question for the project is how to assess a student’s response. Unlike a multiple-guess question, there are different shades of ‘right answer’ that any assessment needs to handle:
1
.
The question asked for a year, but the student gave the (correct) full date.
2
.
The question asked for a year, but the student gave an answer with the correct year, but the wrong month or day.
3
.
The question asked for a day, but the student gave the correct year only.
The assessment code I wrote deals with all these issues, and others. At the moment, the function will offer comments or suggestions for the answers: Correct, mostly correct, Correct but with more information than needed, or wrong. Eventually, these comments will be saved and will become the basis of two reports: one for the student, and one for the instructor.
The quiz goes through the questions in random order, until all the questions have either been answered correctly or has had three failed attempts. This process means that a partially correct answer just means that the student will be given another chance, perhaps with a hint (like “read the question carefully.”).
Answering questions about dates is just a small part of the problem, of course. Adding different kinds of questions would be the next step. Eventually, I hope to develop a system that can assess code and other more complex tasks. I have become particularly aware of the difficulty in designing questions, and it may be useful to see if there would be some way to assist the instructor in this task.

Future work

I will make a robust version of the date interpreter that does not rely on the cloud and submit it to the Function Repository, along with a database of ‘named dates’ that will be used to identify dates with cultural references to dates, rather than numbers.
I will complete the date response analyzer and put submit that to the Function Repository. The structure of the functionality should be adaptable to other types of answers.

How I did it.

In this section, I will describe in some detail how I designed and implemented the code. I may refer to some of the wrong turns and blind alleys in this discussion — sometimes it seemed that they took up the most time — but for the most part, this is not the scenic journey

Turning text into dates

The first task in this project was to find a ‘bullet-proof’ was to turn a typed-in date into a computable date object. The first thing I tried was DateObject[], which accepts string input. However, there were issues:
Examples of DateObject[“string”] in action
In[]:=
DateObject["2/3/2013"]
DateObject
:Warning: the interpretation of the string 2/3/2013 as a date is ambiguous.
Out[]=
Day: Sun 3 Feb 2013
In[]:=
DateObject["3/14/2013"]
Out[]=
Day: Thu 14 Mar 2013
In[]:=
DateObject["3/14/13"]
Out[]=
Day: Thu 14 Mar 2013
In[]:=
DateObject["3/14/79"]
Out[]=
Day: Wed 14 Mar 1979
The function at least lets you know that the first string was ambiguous. However, It was useless for processing dates in the first century, CE, without using tricks like entering the year as “0079”, for example.
Interpreter[] works over a wider range of dates, but is still ‘fooled’ on occasion:
Examples of Interpreter[“Date”][“string”] in operation
In[]:=
Interpreter["Date"]["8/23/79 AD"]
Out[]=
Day: Wed 23 Aug 79
In[]:=
Interpreter["Date"]["8/23/79"]
Out[]=
Day: Thu 23 Aug 1979
In[]:=
Interpreter["Date"]["23 August 79"]
Out[]=
Day: Thu 23 Aug 1979
It does better at dates BCE:
However, it thinks that all dates are, by definition, using the Julian calendar, which was not in use at the time:
Adding the O.S. specification does not help.
This form of Interpreter cannot handle century designations.
For that, you need Interpreter[“SemanticExpression”][“string”]
However, there are still issues:

The importance of calendars and conventions for work in the arts and humanities

Before we continue, I need to stress the importance of getting dates right. This issue is largely ignored in the sciences. Only two ‘hard’ sciences, Astronomy and Biology, deal with history in any significant way. Astronomy makes use of the unambiguous Julian Date, which just counts days from a point in the distant past, and then back-fits the Gregorian calendar to get a date. The only time this is an issue is when an astronomical event has to match up with a historical event. The timeline in biology rarely involves anything as small as centuries, and so any ‘in the neighborhood’ designation of a date is likely to be ok.
In the arts and humanities, things are different. I studied musicology as part of my education as a composer, and determining dates is a non-trivial process. Depending on the century and location, years were as likely to begin in March as January (which is why December is named as it is: it used to be the tenth month of the year). When the Gregorian calendar was introduced, it was by no means universally adopted, which means that the same point in time had different designations in different countries. If you wanted to know what happened on the date of Isaac Newton’s birthday, for example, you would use the date of December 25, 1642 (O.S.) in England or its colonies, but January 4, 1643 (N.S.) in most countries in continental Europe. Russia did not adopt the Gregorian calendar until after the Russian revolution, which is why the “October Revolution” actually took place in November according to the calendars of most of the world by that time.
By convention, dates in history books and books dealing with music or art history understand that the dates given are those according to the calendar system in use in the country at the time. There are exceptions, of course. In addition to Isaac Newton’s birthday being celebrated on both the O.S. and N.S., day, other events were ‘adjusted’. For example, George Washington’s birthday of 22 February is a N.S. date, even though the colony of Virginia did not adopt the Gregorian calendar until 1752.
This may be of only Academic interest, but of course, that is the point. The point of having knowledge in the Wolfram Knowledgebase is to make it computable. Instead of relying on context to interpret a date, Wolfram Language needs to provide the context in the data structure itself, and it needs to be accurate

How I solved the problem

I wanted to provide two specific examples of dates that the function I created could handle. The first is Isaac Newton’s birthday, which has common representations in both O.S. and N.S. dates. The second was that I wanted a student to be able to enter “St. Crispin’s Day, 1415” as the date of the Battle of Agincourt. Until I worked on this project, I knew the date only as St. Crispin’s Day, mostly because of its association with Shakespeare’s play “Henry V”. These are not isolated examples, of course. I treat them as examples of larger classes of ways to describe dates.
While I do not expect a student to enter “O.S.” or “N.S.” when answering a question, it bothered me that the default calendar system in Wolfram language is Gregorian.
Example of a wrong answer from WL
If we extended the Gregorian calendar to the 15th century, the Battle of Agincourt happened on November 3, 1415.
So, when I designed the system, I made sure that any dates could have a proper designation for calendar system.
I built functions to take a string and turn it into a date object. I works with any dates that both interpreters can handle, and there is a separate flag to indicate the calendar system. The first validates the string, and the second handles the conversion. The ‘allowNoAnswerQ’ flag determines if there must be a date in the dialog field.
There are three types of valid dates:
1
.
The string contains “century” and it is processed by Interpreter[“SemanticExpression”]
2
.
The string can be handled by Interpreter[“Date”]. This is the most common case.
3
.
The string contains a names day. I made a custom list of associations to handle this.
Function to validate a string input. This is called when the user indicates that the entries are ok, but before they are committed. If there is a failure, the user is prompted to try again. (An executable version is in the implementation section.)
The function that returns a valid date object is very similar, and can be seen in the implementation. As prototypes, these functions are fine. However, Interpreter[] is slow in most cases (I have a normal internet connection, not a T1 line or better). Using the same functions to validate and store the date object in both the Question creator program and the Quiz program ensures that there any question that the student may enter will have an answer that can be understood by the system.

Assessing student work

If every student’s answer was either completely right or completely wrong, then assessment would be easy. This is the benefit of the true/false or multiple-guess type of question. When the student types in a date in response to a question expecting a date for an answer, there are several possibilities:
1
.
The student types in something that cannot be interpreted. This includes gibberish, as well as typos. (These include typing 1690. or 1690.2 for a year)
2
.
The student provides an acceptable answer: The question asks for a year, and the student enters a year (and only a year)
3
.
The student provides additional correct answer: The question asked for a year, and the student gave a correct year, month, day.
4
.
The student provides a partially correct answer: The question asks for a year, and the student enters the correct year, but incorrect month or day.
5
.
The answer is wrong: The question asks for a year, and the answer has a wrong year (and possibly other information)
I wrote a prototype function to deal with answers. A finished application would provide an instructor with options, but here are the choices I made:
1. For ‘garbage’ the student is told whey they typed was unintelligible, and they have a chance to re-enter. Three failures, and the program moves on.
2. For a correct answer, the student gets credit, and (usually) the question will not be repeated.
3. For additional correct information, the student gets credit, but is informed that they gave more than the question asked for.
4. For a partially correct answer, the student is told that the answer is partly right, and the question is put back in the queue for testing.
5. For a wrong answer, the student is informed that the answer is wrong, and the and the question is put back in the queue for testing.
In my function, questions are asked at random from the set until all questions have been answered correctly, or has been answered wrong three times. At that point, the quiz is over and reports would be generated. (I did not get to that part.)
What I was hoping to do with scenario 4 is to ask the student “is that your final answer?” and if the student clicks “no”, then the student would have another chance to answer. There are coding issues with making UI elements appear and disappear, so that functionality has not been implemented (yet). In reality, the student would have to be asked that question some of the time for all the other scenarios, so that the student would not just realize that their answer was wrong.
When implemented, the report will provide an assessment for the student and the teacher, indicating problem areas for both. Included for the teacher will be the number of questions answered on the first pass, and other information. If I can rewrite the code to validate dates so that it does not use the cloud, I could include a measure of how long it took the student to enter an answer, which could indicate if they found the answer on their iPhone.
The point of this exercise is that an interactive quiz can be much more flexible in handling answers in a ‘gray’ area, and should provide a more realistic view of the student’s progress.

The code

Here is the code I wrote to assess the student’s answer:
Function to see how well the student’s answer matches the requirements. The active code is in the implementation.
The returned values are used by the assessing function to decide what to do:
1
.
validDateQ: this is True when the answer could be turned into a date. If False, then treat the entry as ‘garbage’ — if False: scenario 1 above
2
.
ansWithinBoundsQ: The answer is within the bounds specified by the association -- basically the answer is correct — True: scenario 2, False: scenario 5
3
.
dateExactQ: the date exactly matches the specification. Note that the specification could indicate that the answer should be a year, but the date object could have year, month, day (used with 5 to determine scenario 3)
4
.
partlyRightQ: Typical scenario: the question wants a year, and the student gave year, month day, and the year is right but the rest is wrong. — True: scenario 4
5
.
dateBetterQ: The answer provided is not only correct, but has more information than asked for. — True: scenario 3

Try it out

Execute these two lines. The rest of the notebook will be initialized. The dataset will show you the correct answers.

Next steps

◼
  • Rewrite the String to DateObject routine so that it does not require the cloud. Refine edge cases.
    • ◼
  • Finish the assessment function to handle the tolerance parameter and, if possible, to reduce the complexity
    • ◼
  • Finish the Quiz function so that it 1) works, and 2) generates reports.
    • ◼
  • Expand assessment functionality to include additional types of answers, possibly including a robust assessment of Wolfram Language code.
    • If I can make the code bullet-proof, I hope the put the String to DateObject functionality in the Function repository. As a secondary ‘product’ I will build a database of named days for use by this function (or anyone else).

    Keywords

    ◼
  • Education
    • ◼
  • Wolfram Language
    • ◼
  • Assessment

    • Acknowledgment

    Mentor: Daavid Väänänen
    I thank Stephen Wolfram for suggesting this project.

    References