6.3 Results and discussion
6.3.1 Results
130 Chapter 6: Experimental Evaluation of e-NOTEBOOK System
efficiency, content, accuracy, format, ease of use, timeliness and interaction. All learners were asked to answer a questionnaire concerning the overall performance of the system. A 5-point Likescale, in which 1 means “strongly disagree” and 5 means “strongly agree”, was used in this questionnaire.
Chapter 6: Experimental Evaluation of e-NOTEBOOK System 131
knowledge for improving learning outcome. This also implies that the breadth of learners’
knowledge may be increased by using recommendation. In addition, it implies that the recommendation may improve the reflective use of learning resources. In Fig.6.5, we can clearly see that collaborating with advanced learners improves collaboration.
0 5 10 15 20 25 30
1-2days period
2-2days period
3-2days period
The number of notes
experimental group control group
Fig.6.2 The number of notes of two groups
The phenomenon of experimental group is clear: the learners make the majority of the changes after providing the recommendation (2-2days period). On the contrary, the learners make little or no majority of the changes in control group.
0 10 20 30 40 50 60 70 80 90
1-2days period
2-2days period
3-2days period
The number of comments
experimental group control group
Fig.6.3 The number of comments of two groups
132 Chapter 6: Experimental Evaluation of e-NOTEBOOK System
37
47
37
25
32 30
18 17
15 17
13 15
9 10 11 12
5 6 4
1 0 0 0 0
0 5 10 15 20 25 30 35 40 45 50
a b c d e f g h
The number of comments The number of notes
The number of collaborating with others
Experimental group: a-d Learner Control group: e-h
Fig.6.5 Products (notes & comments) vs. the number of collaborating with advanced learners
We can clearly see that collaborating with advanced learners improves collaboration.
0 100 200 300 400 500 600
1st day 2nd day 3rd day 4th day 5th day 6th day
Total
Experimental group Control group
The number of new keywords
Fig.6.4 The number of new keywords generated everyday
The new keywords generated everyday is presented in this figure, showing that after using recommendation (3rd day), the generated new keywords are increased.This suggests that the recommendation can help learner construct new knowledge for improving learning outcome. This also implies that the breadth of learners’ knowledge may be increased by using recommendation.
Chapter 6: Experimental Evaluation of e-NOTEBOOK System 133
166.7% 106.3% 71.4% 169.2% 83.3% 53.3% 64.7% 63.6%
37
47
37
25
32
30
18 17
15
17
13
15
9 10 11 12
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50
a b c d e f g h
Learner
learning achievement(%) The number of comments The number of notes
Experimental group: a-d Control group: e-h
Fig.6.6 Products (notes & comments) vs. learning achievement
The learners’ learning achievements of experimental group (a-d) are almost higher than the ones of control group (e-h).
Tab.6.1 Means, standard deviations and paired t-test of the pre-test and post-test measures of the learners in the evaluation study
Experimental group
Measure Pre-test Post-test
paired t-test
M (SD) M (SD)
knowledge of environmental
problem 15.5 4.04 34 1.83 3.18
Control group
Measure Pre-test Post-test paired t-test
M (SD) M (SD)
knowledge of environmental
problem 18 2.94 30 5.72 3.18
Note: n = 4 for all measures p<0.05
Converting to standardized progression scores [Fraenkel & Wallen, 2003]:
(1) Experimental group: 6.30 (2) Control group: 2.77
134 Chapter 6: Experimental Evaluation of e-NOTEBOOK System
(2) The quantitative data sources consisted of learners’ pre- and post-test, as well as their knowledge progressions
Let us show that proposed e-NOTEBOOK system provide a useful learning model for supporting knowledge construction, knowledge production and facilitation of knowledge acquisition.
Tab.6.1 shows the means, standard deviations and paired t-test of the pre-test and post-test measures of the learners in the evaluation study. The paired t-test for the differences between each pair of the pre-test and post-test measures of the knowledge of environment problem shows statistically significant differences. We attempted to further use the t-test results to show the experimental effects of the tools in assisting learners to generate learning outcomes. The mean differences on the pre-test and post-test measures of the knowledge of environment problem converted to standardized progression scores was: (1) experimental group:
(34-15.5)/((4.04+1.83)/2) =6.30 (2) control group: (30-18)/((2.94+5.72)/2)=2.77 [Fraenkel and Wallen, 2003]. These findings provide evidence that the proposed system was developed in this study can assist learners to construct knowledge through the use of recommendation tools and collaboration tools during learning. In other words, the recommendation and collaboration allow learners to construct knowledge by working in e-NOTEBOOK system. In addition, the results imply that through exploration, interpretation and negotiation, understanding is deepened as multiple perspectives are considered in constructivist learning environment.
We also evaluated the relationship of learning achievement and the learners’ products. Fig.6.6 shows the learning achievement obtained by each learner with respect to their contributions. As Fig.6.6 shown, the learners’ learning achievements of experimental group (a-d) are almost higher than the ones of control group (e-h). Average percentages of achievements between pre- and post-test are 128.39% (experimental group) and 66.25% (control group). This implies that the learning achievement is relative to the productions. In other words, it could be normal to expect that the more active a learner is the better learning outcomes achieves. The results demonstrate that our system is not only as a share space of information and experiences but also to help learners reflect on their experiences.
Chapter 6: Experimental Evaluation of e-NOTEBOOK System 135
(3) Recorded interviews and questionnaire
We investigate effectiveness of the proposed method by considering following aspects: (1) learners’ opinions about the proposed system; (2) Whether the system provides a meaningful learning environment for group learning.
This section presents results of interview and questionnaire from all learners participating in the study which we conducted after the three-2 days period.
We analyzed the interviews under six elements of an activity system [Engestrom, 1987]: subject, object, community, tools, rules and outcome.
(a) Subject
Group learners played significant roles and contributed fair shares to the completion of the problem solution. Meanwhile, they learned new knowledge and skills from other learners. “I hope I have contributed with my own ideas and opinions regarding the problem of environment and through active participation in all experiment period.” “We had to be more active. I felt I was very productive.” “I was always reading the learning resources provided by other members and making sure my ideas were changed. I acquired new information.” “I enjoyed the experience since sometimes I felt that I was ‘teaching’ other learners and sharing knowledge with them.”
(b) Object
Learner felt that collaboration in our system provided an opportunity for them to efficiently build and enhance knowledge learning. “I have acquired more of knowledge about problem of environment via the system and I could gain maximum benefit through learning resources recommendation since the recommendation materials are almost my needed resources.” “I post many of the notes” “I give some of my opinions about ‘COP’” “my interests about environment problems are many, I post many ideas”.
(c) Community
Let us consider social cultural context in which the activity takes place.
136 Chapter 6: Experimental Evaluation of e-NOTEBOOK System
More than 75% of the learners commented that they worked together as a group in the process of problem solution in our system. More than half learners stated that through discussions with advanced learners recommended by our system they can learn smoothly. Moreover, they learned how to respect their partners’ ideas. “I have participated actively discussion with other group members who were recommended by the system and have often volunteered to take part in the discussion with other group members. I think it is impossible to achieve my work without discussion with the other group members.” “Collaborating with the advanced learners is in the advantages of quick learners feedback and fast resolution of requirement questions and problems.”
“Communicating with advanced learners also helps in resolving misunderstanding among us and also helps to ensure the problem which will be solution in the next step.” “I think advanced learners can increase interruptions when learning is not well performed, thus, this will avoid mistakes and produce only minimal mistakes.” “advanced learners who recommended by the system are more authentic…give me chance discussing with them, learning from them.” “I am surely that I learnt useful knowledge from relative users.” “the relative users transferred the knowledge to me.”
(d) Tools
Fundamental part (note-taking, note-reading, comment-taking, and comment-reading):
All of the learners said the system improve their understanding in problem solution processes, especially the transfer of knowledge to practice. “I have deepened my idea and knowledge by using note-taking.” “By reading notes and comments which wrote by other learners, I have chance to sharpen my thinking.” “I recorded my notes after integrating the knowledge of web page; this is a good approach that I can construct my learning with other group members through comment.” “I always replied to other members’ notes or comments, this give me a chance to rethinking the knowledge provided by other learners. I believe I also give other members contributions to improve their knowledge”
Recommendation of learning resources:
“By using recommendation, I can easily gain the knowledge which I needed” “By using the recommendation of learning resources, I can easily know what I do in the next step, nice!” “I
Chapter 6: Experimental Evaluation of e-NOTEBOOK System 137
always reference the recommended learning resources, it makes the work smoothly and I can simply use the materials I wanted” “I believe I have provided the good notes after I referenced the recommendation of other members’ notes”. The more contribution, the more profit. “I could profit more for other members’ through providing learning resources” “after reading the notes provided by others and commenting to them, I went ahead and wrote a new one”.
Recommendation of advanced learners based on analysis of social interactions:
“Through the support of recommended learners, it was easier for me to jump over my shadow and contribute actively”. Besides, “I find it easier to present my viewpoint in the system”. “I maybe the expert of ‘global warming’, since other members always invited me to discussing this problems. This gives me self confidence. I like collaborating and discussing with others.”
“Through discussing with other recommended advanced learners, I know the shortcomings of my knowledge about ‘CO2 and the environment’. This made me engage more investigative activities about that. ”
(e) Rules
The system provides a scaffolding of searching/solving/creating/sharing knowledge and ideas. In the process of group learning in our system, they learned how to share experience or knowledge, collaborate with other members, open their mind to positive and negative feedback, and contribute new ideas. “sharing and publishing information that I explored, it made me feel good” “I always post the information that I has already known, others replied on the knowledge I provided made me check and recheck the accuracy of what I wrote” “by commenting to others’ notes, I studied them in depth”.
(f) Outcome
“Through the learning in e-NOTEBOOK system, I understood the concept such as ‘Kyoto protocol’, ‘COP’…”. “I think I have understood the reason of global warming, so that I will do that I can do for avoiding global warming problem” “I felt that I was contributing to the knowledge about ‘CO2 and environment’” “I think this is very good. It is a win-win: the writer who deepens his knowledge, the readers who receive reliable information”. They believe that by
138 Chapter 6: Experimental Evaluation of e-NOTEBOOK System
being able to collaboratively solve authentic tasks online “the learning outcomes accomplished are of the highest quality and quantity”. “Through the learning, I have improved problem solving abilities, enhanced communication skills” “I began to learn to make the connection between not only the information provided by myself but also the ones that provided by rest of the group learners”.
However, the one learner said it was very feasible but suggested that it should be very useful if the system may emphasis or present the keyword of discussion topic. And the group needs to stay more focused on completing the ‘group’ task and not just on getting the points for individual knowledge construction.
Tab.6.2 shows the questionnaire items and the corresponding statistical results of the feedback from the learners. The statistical results indicate that most of the learners agreed that by using the recommendation list, they can easily find the learning resources which are needed (Effe2, Effi2).
In terms of the advanced learner recommendation performance (Effe3, Effi3), almost all of the learners agreed that the advanced learners recommended were helpful or very helpful to them in improving their learning performance. Moreover, it is obvious that almost all of the learners agreed that our system can help them complete their task effectively and efficiently (Effe1, Effi1).
All in all, the learners were satisfied with our system (C1 – I3). Through the system, the learners can not only sharpen their thinking and gain new knowledge, but also gain more opportunities to interact with other learners (I4, I5).
Chapter 6: Experimental Evaluation of e-NOTEBOOK System 139
Tab.6.2 Educational evaluation results
Statement Mean Mean
Deviation
Effectiveness
Effe1. I can effectively complete my work using the system. 3.50 0.50 Effe2. The recommendation of learning materials helps me complete the task. 4.00 0.50 Effe3. Communication with advanced learners recommended by the system is
useful. 3.75 0.38
Efficiency
Effi1. I can quickly complete my learning goal using the system. 3.00 0.00 Effi2. I can quickly find the learning materials I needed. 4.00 0.00 Effi3. I can quickly find the advanced learner I will collaborate with. 3.50 0.50
Satisfaction
Content
C1. Does the system provide the precise information you need? 4.00 0.00
C2. Does the information content meet your needs? 3.75 0.38
C3. Does the system provide reports that seem to be just about exactly what you
need? 3.50 0.50
C4. Does the system provide sufficient information? 4.00 0.00
Accuracy
A1. Is the system accurate? 3.50 0.50
A2. Are you satisfied with the accuracy of the system? 4.00 0.00
Format
F1. Do you think the output is presented in a useful format? 5.00 0.00
F2. Is the information clear? 4.75 0.38
Ease of Use
E1. Is the system use friendly? 4.50 0.50
E2. Is the system easy to use? 5.00 0.00
Timeliness
T1. Do you get the information you need in time? 5.00 0.00
T2. Does the system provide up-to-date information? 5.00 0.00
Interaction
I1. The system is easy to interact with. 5.00 0.00
I2. The system is friendly. 5.00 0.00
I3. I feel comfortable using the system. 5.00 0.00
I4. Through the system, I gain more opportunities to interact with other learners. 5.00 0.00 I5. Through the system, I can shape my thinking and gain new knowledge from
others’ notes and comments. 4.00 0.00