Data Collection and Annotation

We recruited students from Japan and the Philippines. 11 Japanese and 12 Filipino students participated in the data collection sessions, resulting in a total of 23 subjects overall. All participants were university students who were near the completion or have just completed a course in introductory programming at the time of the sessions.

Each participant took part in the session individually. The data collection session was divided into three parts: the briefing phase, the coding phase, and the self-report phase.

Figure 3-3. Coding phase data collection setup.

Briefing Phase. In the briefing phase, the student was informed about the details of the data collection process. The student was also asked to sign an informed consent form that clearly indicated all data that will be collected and how it will be used. By signing the form, the student also gave us the permission to publish snapshots and video clips of their faces in scientific publications and venues. The briefing phase lasted for about 5 to 10 minutes.

Coding phase. The briefing phase was followed by the coding phase. The setup of the coding phase is shown in Figure 3-3. In the coding phase, the student was asked to use a custom application in which they must solve a series of coding exercises in increasing difficulty. The exercises required them to write some code in the Processing (Japan) or Java (Philippines) programming language according to some given specification. The session contained a total of 8 problems, a summary of which is shown in Table 3-1. The students must solve the exercises sequentially, as they were not allowed to move on to the next exercise until they have submitted a correct solution. This was an intentional choice to induce various emotions on the students.

Table 3-1. Problems Used in the First Round of Data Collection.

Number Exercise Concepts Covered

1 Return “Hello World”. I/O

2 Get the change given the price of the item and the amount given.

Arithmetic expression, I/O

3 Convert from Fahrenheit to Celsius. Arithmetic expression, I/O 4 Determine if a score is passing (at least

60) or failing (less than 60).

If else, I/O

5 Get the amount due after applying discount rules.

Arithmetic expression, if else, I/O

6 Get the average of a list of numbers. Arithmetic expression, loop, I/O 7 Get the sum of all odd digits of a

number.

Arithmetic expression, if else, loop, I/O

8 Determine if a number is prime or not prime.

Arithmetic expression, if else, loop, I/O

A screenshot of the application is shown in Figure 3-4. The application provides an interface for the student to write code, test the code using user-defined input, and submit the code for checking. When the code is submitted, the system automatically determined its correctness by comparing the output of the user’s program with the expected output on a set of pre-defined test cases.

Aside from this, the application also logged all the system interactions made during the session. This included: (1) all editor changes (insertions and deletions) done by the student in the editor, (2) all compilations, as well as information about the compilation result (syntax error, runtime error, etc.), and (3) all submissions, as well as information about the submission result (passed or failed). Each system log event was marked with a timestamp of when the event happened, allowing the entire session history to be reconstructed. The application also recorded a video of the student’s face through a standard web camera positioned on the center of the top edge of the monitor. The video was saved into a file after the session, allowing it to be played alongside the session history showing the progression of the student’s code.

The coding phase lasted for 45 minutes or until all problems have been solved correctly.

After 45 minutes have elapsed, the coding phase automatically ended.

Annotation Phase. After the coding phase, the participant proceeded to the annotation phase. The goal of this phase was to label the session data with the emotions that the students experienced while doing the coding exercises. To do this, we asked the participant to view a replay of the session through another custom application, a screenshot of which is shown in Figure 3-5. In this application, the user could use a slider representing the session timeline to watch different parts of the session history. Each point in the session history included a snapshot of the student’s face and a snapshot of the student’s code at that point in time.

In this interface, the student could put one of the four affective labels: “engaged”,

“confused”, “frustrated” or “bored” on a selected interval in the session. This would be treated as a single annotated instance in the dataset. An instance contained the starting timestamp of the interval, the ending timestamp of the interval, and the emotion label self-reported by the student. The student could put as many or as few labels as he or she wished.

However, for simplicity, we only allowed a single emotion label for each point in the session (i.e., we did not allow self-reports to overlap; attempting to add an annotation over another would result in the old annotation being overwritten).

Again, this round of data collection used a spontaneous affect judgment scheme. A spontaneous affect judgment scheme meant that the subject could freely select which parts of the session to annotate with emotion. It also meant that the subject could select as few or as many intervals as he or she wished.

Figure 3-4. Application used for first round of data collection.

Figure 3-5. Tool used for making self-report spontaneous affect judgments in the data collection.