An experimental study which was participated by real-users was constructed to evaluate our system and proposed techniques. The theme was “the problem of environment”. This theme was chosen, because it allows multiple perspective views on environmental issues such as pollution of the sea, global warming and etc. In addition, this theme reflects that of the real world type of problems so that allows learners to bring in their experiences. Moreover, the learners expect to solve meaningful problems which their interests would encounter. Learners could expand the inquiry task based on these issues. We implemented proposed e-NOTEBOOK system using PHP on Apache. At the beginning of each experiment, participants were introduced to the system.
6.2.1 Participant and procedure
The experimental design was based on two groups:
• An experimental group having recommendation of learning resources and recommendation of advanced learners’ tools.
• A control group without these recommendation tools.
And 8 learners that were students of masters’ degree and postgraduate studied as two groups (the experimental group: 4; the control group: 4) participated in the experiment.
The experiment flow was based on three periods during which the recommendation mechanisms were used or unused. It was designed to enable the testing of effect of our proposed tools in our proposed learning environment – e-NOTEBOOK system. The learners posted notes (web links, abstracts and remarks) after they sought useful web pages on the web. They also were allowed to post some comments to notes and respond to comments already posted in the comment space. The test was conducted in three periods. One period was 2 days.
• The first period was set as training period. In this period the learners registered themselves and they began their learning according to their own feelings or inclinations.
Chapter 6: Experimental Evaluation of e-NOTEBOOK System 127
• In the second period, the learners were asked to read the lists of learning resources recommendations and learners’ recommendations to them. At this point recommendation procedures were carried out. Naturally, they can also perform their learning according to their own feelings or inclinations.
• In the third period, the learners were asked to learn the learning resources according to their interests. S/He can learn according to the list of learning resources recommendation to her/
him or perform her/ his learning according to her/his own feelings or inclinations.
The two groups of all learners had to complete the same tasks: completing a pre-test, working through our e-NOTEBOOK system, and completing a post-test. The only difference between the control group and the experimental group was that learners in the experimental group were used recommendations mechanisms. The learning environment of experimental group was equipped with the tools: recommendation of learning resources and recommendation of advanced learners.
6.2.2 Analytical data
Data for this study were extracted from three resources: (i) the log data in the proposed e-NOTEBOOK system for answering question 1, (ii) quantitative data sources consisted of learners’ pre- and post-test for answering question 2 and (iii) recorded individual interviews and questionnaires with all the learners for answering question 3.
(i) Log data
To gain an impression of participants’ behaviors and activities in our system we analyzed the log data which provide statistical information on the learners’ behavior while contributing knowledge.
We collected several types of quantitative data: the number of notes (or comments) was employed for investigating the learners’ knowledge production in three periods; the number of keywords of everyday was employed for investigating the learners’ reflective use of learning resources and etc.
(ii) The quantitative data sources consisted of learners’ pre- and post-test, as well as their
128 Chapter 6: Experimental Evaluation of e-NOTEBOOK System
knowledge progressions.
The pre-test and post-test were performed before and after using of our system, respectively. A pre-test was kept at the beginning using our system to evaluate learners’ prior knowledge. And we conducted the post-test to evaluate learning achievements. The pre- and post-test assessed learners’ knowledge on general concepts of the problem of environment is designed. This helps us to know the learners’ knowledge progressions. Participants received the following pre- and post-tests. The items were chosen from large number of existing examples, and question format is single-choice. A 36-item test was used for this study. Each item was worth 1 point, and the maximum score in both tests was 36. The tests were designed to capture both the nomenclature knowledge of “the problem of environment”.
(iii) Recorded interviews and questionnaire
After the three-2 days period of experiment, we conducted interviews and questionnaire with all the learners to evaluate the effective and efficient of proposed system for group learning.
The framework of activity theory [Engestrom, 1987] provides a useful lens through which to study inter subjective learning in distance educational contexts, because it focuses upon the relationships that exist among those involved in an activity [Greenhow and Belbas, 2007]. An activity system is a framework placing human action in a meaningful context. The three main elements of the framework are: the subject (actor in the activity), the object (product acted on by subject) and the community (social cultural context in which the activity takes place)[Engestrom, 1987]. In this study, the e-NOTEBOOK system is seen as the activity system in which the group learners are subject, the “problem of environment” which will be learned is the object and the advanced learner group during the work placement is the community. Subject, object and community are interrelated by three other elements: tools, rules and division of labor [Engestrom, 1987]. The tools (in this study is e-NOTEBOOK system) mediate the relation between the subjects (group learners) and the objects of activity (the problem of environment learning), the rules (in this study: inquiry based collaborative learning) mediate the relation between the subjects and the community (advanced learners group during a work placement) and the division of labor
Chapter 6: Experimental Evaluation of e-NOTEBOOK System 129
mediates the relation between the community and the object (not explicitly used in this study).
Finally, an activity is motivated by the need to transform the object into an outcome. Fig.1.1 shows how an activity system is often depicted as a triangle containing the seven aforementioned elements and this framework applied to our experimental evaluation study.
In addition, a questionnaire was administered to understand learner’ perceived attitudes toward the use of e-NOTEBOOK system. According to [Drachsler et al., 2008] [Drachsler, 2009], evaluations measure of recommender system for e-learning include such as effectiveness, efficiency, and satisfaction. In our research, we adopt ISO’s broad definition of usability [ISO, 1998] as consisting of three distinct aspects:
(a) Effectiveness, which is the accuracy and completeness with which users achieve certain goals.
Indicators of effectiveness include quality of solution and error rates.
(b) Efficiency, which is the relation between (i) the accuracy and completeness with which users achieve certain goals and (ii) the resources expended in achieving them. Indicators of efficiency include task completion time and learning time. Note: According [Drachsler, 2009], in e-learning domain, there is a difference between the measured ‘elapsed time’ that learners took to complete a learning task and the actual ‘study time’ they needed to successfully complete a learning task. In this study, we use task completion time which evaluated by learners themselves as the primary indicator of efficiency.
(c) Satisfaction, which is the users’ comfort with and positive attitudes towards the use of the system. In this study, according to [Doll and Torkzadeh, 1998] ‘s Likert scale, we design the instruments. In addition, we added the new factor (interaction) besides the Doll and Torkzadeh’s five factors (content, accuracy, format, ease of use, timeliness).
Meanwhile, [Dewiyanti et al., 2007] emphasis five elements should be considered when developing and evaluating collaborative e-learning environments: environmental characteristics, environmental satisfaction, collaborative activities, learners’ characteristics, and environmental acceptance. A list of 23 statements was designed to test the effectiveness, efficiency, and user satisfaction of the system. The statements can be classified into eight factors: effectiveness,
130 Chapter 6: Experimental Evaluation of e-NOTEBOOK System
efficiency, content, accuracy, format, ease of use, timeliness and interaction. All learners were asked to answer a questionnaire concerning the overall performance of the system. A 5-point Likescale, in which 1 means “strongly disagree” and 5 means “strongly agree”, was used in this questionnaire.