Learning Technology


publication of


IEEE Computer Society's

Technical Committee on Learning Technology (TCLT)


Volume 9 Issue 2

ISSN 1438-0625

April 2007



From the editor…... 2

Scaffolding Social Presence and Democratic Space: A Case Study on the Benefits of Student Mentors and “Safe House” Discussion Forums. 3

Learners’ profiling by data driven approaches. 11

The URO Project - A new curriculum to improve the “quality of life” for people with URinary incontinence by the use of ICT   14

HotKey Coach: Training users to interact efficiently with software applications. 17

Evaluation of Training – A Critique of Kirkpatrick’s Evaluation Model 20

Using Microsoft’s Homepage Starter Kit to teach Content Management Systems in Postgraduate Journalism Students  22


From the editor…




Welcome to the April 2007 issue of Learning Technology.


Many advances in learning technologies are happening throughout the world. This issue focuses in bringing these new developments and emerging technologies to the readers. This issue contains papers ranging from practical learning technology solutions to evaluation of training.

Wanless-Sobel and Alcorn looks at ways to increase the student participation in discussion forums in eLearning with the use of student mentors. Wanless-Sobel and Alcorn argues that by removing the
residual hierarchy in online instruction student participation in discussion forums will be increased. Hence, they use student mentors as facilitators of these forums instead of lecturers to find out if student participation increases. Viola et. al. looks at learner profiling using data driven approaches. Colace and DeSanto outlines the URO Project which is designed to use eLearning as a bridge between the patient and a health professional. eLearning bridge in this project provides the patients with the necessary answers from a professional who works at home, in the community and in the hospital.


Bose provides a critical analysis of Kirkpatrick’s Evaluation Model while Veglis describes how Microsoft Homepage Starter Kit can be used to teach Content Management Systems. Finally, Krisler and Alterman look at a system which trains users to become experts in using software.


This newsletter focuses publishing new and emerging technologies in education focussing on advanced learning technologies and its usage in different contexts. Please feel free to bring forward your ideas and views.


Besides, if you are involved in research and/or implementation of any aspect of advanced learning technologies, I invite you to contribute your own work in progress, project reports, case studies, and events announcements in this newsletter. For more details, please refer author guidelines at http://www.ieeetclt.org/content/authors-guidelines.






Ali Fawaz Shareef, PhD

Director, Centre for Open Learning

Maldives College of Higher Education




Scaffolding Social Presence and Democratic Space: A Case Study on the Benefits of Student Mentors and “Safe House” Discussion Forums


Abstract: Current pedagogical theory promotes high social presence and democratic space in online instruction. This case study shares an experiment with four online classes featuring an online student mentor to increase social presence and democratic contact zone quotients by way of the mentor’s coaching personality as well as the creation of a “safe house” discussion forum in each class, completely administered by the mentor. The preliminary trial indicates increased social presence, along with increased student satisfaction with the online learning experience, although a good portion of student participation was observational and peripheral, suggesting the safe-house design requires further development for nurturing an active democratic contact zone.


Keywords: Distance learning, social presence, democratic contact zone, online student mentor, pedagogical safe-house, lurking, peripheral participation 




21st century online educators need to situate students in prime and authentic learning environments (Herrington, 2003), and one feature of a prime learning environment is the social presence of the instructor and class members. Research shows that social presence in computerized instruction—the presence of humans—increases student satisfaction with courses as well as instructors’ satisfaction (Richardson & Swan, 2003; Tu & McIsaac, 2002). Prime social presence classes are often choreographed with a variety of social presence opportunities, allowing for students’ individual preferences (Reeves & Nass, 1996):  frequent and timely posting; online office hours in synchronous chat; embedded instructor audio files or other speakers; human avatars with synthesized voices (SitePal.com); and constructivist, collaborative work with peers in asynchronous or synchronous discussion.


High social presence humanizes online education and democratizes power relations (Foucault, 1979, 1980; Zembylas, 2007).   Establishing a “democratic contact zone” (Brookfield, 2001, p.206; Pratt, 1991) in online instruction entails creating and maintaining a course climate where students feel safe and empowered to engage in public discourse not only about course material and assignments but also emotional components of their online experiences, such as  frustration from unreliability of the Internet; stress from juggling school, work and personal life; apprehension and anxiety about class assignments; “public” exposure of course work; embarrassment on inability to perform tasks or perform tasks at a brisk pace; social isolation; and complaints about the course instructor (O’Regan, 2003.)  Because affective domains of knowledge are recognized and valued in a democratic contact zone, there is tacit encouragement for critical discourse, along with associational qualities for democracy, which, in turn, nurture autonomous and thoughtful class activity (Zembylas, 2007). 


In online instruction, high social presence and a democratic contact zone are worthy goals, but they are also utopian intentions that easily fall short, leaving typical dystopian outcomes (Miller, 2001), such as unsuccessful course completion rates; student dissatisfaction with the online learning; and the prevalence of the online silent or passive majority, whose lack of interaction, intellectually and affectively, inhibits their active, constructivist learning. Dystopian outcomes, especially in regard to student passivity, have been typical for Colette Wanless-Sobel, an online instructor and one of this essay’s authors.  In an effort to maximize social presence and a democratic contact zone, the over-riding question for Wanless-Sobel has been, What is hindering social presence and a more democratic, participatory culture in online instruction?”


After reflection, Wanless-Sobel hypothesized that, power relations in online classrooms are still residually hierarchical and, therefore, inhibit social presence and democratic activity, especially in terms of majority interaction. In other words, although Wanless-Sobel casts herself as learning facilitator and fellow knowledge-builder, students, who still cast themselves as subordinates and who rightfully question how much power students actually have vis a vis the instructor and the institution, may be wary of affective and democratic pronouncements, especially since Wanless-Sobel is still the person in charge--the individual who dispenses the grades. (This issue is an important one for social presence and democratic space in online instruction but is beyond the scope of this case study.) Furthermore, residual hierarchy in online instruction is attributable to students’ (and the instructor’s) years of built-up experience of how one is supposed to be and act in a class: knowledge resides with only the instructor, who maintains the dominant discourse; course work and activity is issued only from the instructor; students follow instructions; students do not create their own knowledge base; and emotional and affective expressions have strict rules of conduct (Zembylas, 2007).


Residual hierarchy, whatever the cause, is antithetical to social presence and a democratic zone.  Perhaps, Wanless-Sobel surmised, hierarchy or power needs to be disrupted, and a “safe house” zone needs to be actively nurtured in order to increase the level of social presence and “democracy.”  The adoption of the term safe house for this purpose is to connote online space where students can seek sanctuary and protection from academic authority—namely, the instructor’s-- as well as an infrastructure to safely experiment with “subversive,” non-hierarchical course activity and new norms of classroom behaviour.


Based on this idea, Wanless-Sobel devised a pedagogical design test to see in what ways the social presence-democratic zone quotient would increase by adding two inter-related components to the online classroom: an online student mentor as student coach–advocate and a “safe house” discussion forum completely administered by a student mentor, where students can post anonymous questions and concerns about the course all term, and where the course instructor is limited to lurking (ATIS Telecon Glossary).The student mentor’s job would be to respond to forum questions, using as a knowledge base her or his personal experience as a student in online instruction, along with instructor consults.  Wanless-Sobel conceived both mentor and safe-house forum as pedagogical elements that would subvert not only the instructor’s course dominance but also the teacher-student hierarchy, while simultaneously encouraging peer to peer dialogic activity. Thus, rather than information dispensed only from the instructor, students could opt to “informally” gather information from a peer, the online mentor, in a student platform that provides a supportive knowledge culture, with the instructor having no visible presence.


This short essay provides a case study on the design and test-run of these social presence-democratic zone components, bringing together lessons learned about how to choreograph online engagement, and a point of departure for asking questions about future instructional design directions.


Situating the Case:  Institutional and Student Frameworks


The online mentor and safe house discussion forums were tested in four online classes at Inver Hills Community College (hereafter, “Inver Hills”), Inver Grove Heights, MN, an eastern suburb of the Twin Cities, where one of the authors, Wanless-Sobel, teaches part-time and the other author, Amanda Alcorn, is a student in law enforcement. Innovation in online education is a priority at Inver Hills, whose course management system is Desire 2Learn (hereafter, “D2L.”).  Inver Hills supports faculty efforts in online course development through grant money, technology support, lap top computer allocation, continuing education (CTL, 2004), and, most recently, an Online Student Mentoring Program, developed by Landon Pirius, Director of Enrolment and Online Services at Inver Hills, who used Walden University’s Online Concierge Program and Lake Superior College’s Online Student Mentor Program as models while also allowing more flexibility in terms of how instructors can make use of the mentors in their classes (Pirius, 2007).The rationale for the Online Student Mentor Program is the online mentor is a D2L veteran and can share her or his experiences with current D2L students, making practical suggestions as the class proceeds and offering coaching support, although the online student mentor can also be used to assist with the heavier clerical and record-keeping workload accompanying online instruction (Cavanaugh, 2005; Doube, 2000).


Online Mentors at IHCC are recruited from the student body, which ranges in age from sixteen on up and comprises a diverse demographic:  high school students; two-year college students; paraprofessional students; four year degree candidates; displaced workers; and life-long learners.  African Americans, Asian Americans, and Mexican Americans often comprise the student body as do recent émigrés from various countries in Africa, Russia, and the Ukraine, among others. Despite the ranges in age, ethnicity, and circumstance, most students hold down full time jobs and have busy lives outside of school. Many of them are parents. A student who wishes to be an online mentor, which is paid work, should have successful D2L experience and also should be “nominated” by a faculty member whose classes he or she has taken, thereby, drawing on a previous working relationship. Students are not allowed to mentor classes they, themselves, are currently registered in.


In spring 2007, Wanless-Sobel had four D2L classes she planned on integrating a student mentor and safe-house:” Research Writing in the Disciplines,” two sections; “Creative Problem Solving”; and “Human Sexuality.” Alcorn, a former “Research Writing in the Disciplines” student, approached her about working as a student mentor, and Wanless-Sobel readily agreed, as Alcorn had previously demonstrated intellectual ability, a personable online presence, and a high comfort level with computers and online instruction. Until her work with Alcorn, Wanless-Sobel had no previous online student mentor experience, although she had worked with teaching assistants in classes she had taught at the University of Minnesota, Minneapolis. Likewise, Alcorn had no previous experience as an online student mentor.


The venture was new for both Alcorn and Wanless-Sobel, but the goals were certain: increased social presence and scaffolding for a democratic contact zone. The questions were, “How to make best use of Alcorn as the student mentor?” and “How should the safe-zone be constructed in the online environment?”


The Student Mentor’s Autogenesis and the Creation of Central Perk


Out of habit, Wanless-Sobel, as the educational “expert” and “professional,” solely set about answering these questions, bypassing Alcorn, the student mentor. When Wanless-Sobel realized she was falling into old teacher-student behaviour patterns of the analogue educational world, however, she paused and reflected:  In a digital and democratic educational environment, her role, as an educator, is not to oversee public access to knowledge; no longer does she act as gatekeeper to what is produced, what is shown and how it is interpreted.  In a digital world of infinitely replicable and malleable content, this behaviour pattern no longer applies. The full implications of digital environments for educators, educational institutions and students are not yet clear, although private, public and third sector innovations, from Amazon to Wikipedia to Second Life and MySpace, are revolutionizing the digital landscape and suggesting possible directions.


The conclusion from these ruminations is that, perhaps Wanless-Sobel is not the person who should solely direct the creation of the online safe space or the role of the online student mentor.  If she truly wishes to disrupt hierarchical relationships in the online classroom and maximize social presence, then her own behaviour and role in the online environment must change and evolve, and what better way to begin than by allowing Alcorn, the online mentor, autonomy and independence, both in her mentoring job and in the online safe-houses she would be operating, although Wanless-Sobel could oversee all activity behind the scene.  In other words, Wanless-Sobel needed to give up control and empower Alcorn by letting her characterize her mentoring role to some extent and also design the online safe-house.


Alcorn and Wanless-Sobel conferenced in-person and online for two weeks, discussing and negotiating Alcorn’s mentoring responsibilities and how she should conduct the safe-house. The actual design of the safe house was uncertain for a time, although both women agreed that Wanless-Sobel would create a D2L DISCUSSION forum in each of the classes for Alcorn, placing each forum as the top forum in each class DISCUSSION, increasing its visibility and accessibility for students. Alcorn would then conceptualize the safe-house design and Wanless-Sobel would then enter the design into D2L.


The design Alcorn came up with for the safe-house was ingenious.  Her idea was to integrate or “mash” the highly popular Generations X / Y television series Friends, which is syndicated world-wide (http://www.answers.com/topic/friends-1/), with the D2L Discussion forum, making use of the series’ coffee shop, Central Perk, as an auto-poetic expression of friendliness and connectedness, a student space that acknowledges affective and pleasurable dimensions of the class.  Accordingly, Alcorn christened her safe-house forum Central Perk and even included a colourful poster image of Central Perk (http://www.allposters.com/-sp/-Posters_i855117_.htm) from Friends to greet students and set the tone.  Essentially, Alcorn conceptualized Central Perk as a student forum with a heart beat, possessing a tangible positive social energy. She, Alcorn, would be Central Perk’s host and manager. Here is the text Alcorn created to accompany the poster:


Central Perk, a forum in DISCUSSION, is a place for students to ask Amanda (Mandy) Alcorn, "the online mentor," questions about the class, assignments, or anything that they feel like talking about within reason. The course instructor, Colette, does not participate in Central Perk. So, if you have a question, ask. Who knows, by posting a question in here, you (the student) may also be helping out a fellow student. :) Posts are set for anonymous. Central Perk and its transcripts will be available all term.


The student mentor and the online safe-house were ready to launch. At the beginning of spring term 2007, Wanless-Sobel advertised Central Perk on each class’s HOMEPAGE and also posted class E-mails, informing students about this new class accommodation.  Alcorn also posted class E-mails, introducing herself and providing a brief biographical statement. Central Perk and Alcorn were in operation.


Central Perk in Action


Quantitatively, in terms of posting and without attempting to account for lurker activity, which was high (see discussion below), Central Perk participation was active in each of the four online classes through the first eight weeks of the semester, with activity falling off around the twelfth week of the term and then with occasional use until the end of the term. Cumulatively, student posts to Central Perk were four to five times higher than when Wanless-Sobel had maintained anonymous question and answer forums in previous online classes.  Although almost all the posts in Central Perk were anonymous, frequency of student posts indicates the forum was perceived as a “safe zone,” where students would ask questions or network for information more comfortably than in an instructor-run question and answer forum.


Central Perk participants’ posts centred around five types of information and communication: factual and clarification information (e.g. When is this assignment due); cross-functional human skills, especially regarding technology (“How do you post a PDF file?”); access to others’ online class experiences (”Why does this class have so much collaborative learning?”); moral support and thanks (“Hang in there.”); and mild flaming and criticism about class peers, the instructor, and collaborative class work (“I haven’t heard from any of my team members. It’s been two weeks. What should I do?”).   Although most posts were addressed to the student mentor, who responded to questions within twenty-four hours (usually twelve), Central Perk also operated as an “open administration” forum, especially as the term progressed, where students responded to posted queries, too, if they thought they knew the answers.  One example:


Q.  Am I to understand correctly that for this week, we are to select a job ad and submit it in dropbox and also write a rhetorical essay? (Due on the 12th @ midnight) The actual writing of the resume comes later, right?  Or, are we supposed to post a resume now for team discussion?


A.  No resume.  The work this week is foundation work for the resume, which will be required in an upcoming assignment.


Central Perk’s Social Presence Efficacy


Tentatively, going by post frequency, Central Perk and the student mentor had social presence benefits for all four courses.  In an effort to qualitatively measure the benefits of Central Perk and the mentor, Alcorn and Wanless-Sobel created an online survey, using Survey Monkey.com’s free survey utility (http://surveymonkey.com/), and administered the survey to all four classes at the end of the semester.  Along with collecting some demographic information, such as age and gender, the survey asked students to gauge Central Perk for nineteen social presence indicators, such as impersonal or personal, insensitive or sensitive, helpful or unhelpful.


Out of the four classes with a total of ninety-seven students, forty students formally responded to the survey; seven responded informally to Wanless-Sobel in personal E-mails; ten responded to Wanless-Sobel in course feedback at the end of the term; and forty students chose not to respond in any way. Most respondents found Central Perk a positive feature and appreciated its safety zone features. Thirty-two survey respondents, however, commented that they, themselves, did not post questions to Central Perk, but, nonetheless, frequently checked in to see what questions had been posted and found Central Perk and the online mentor helpful. Four lurkers said they wanted to post, but certain circumstances deterred them. These included lack of time for posting; computer access issues, making posting difficult; and computer literacy.  Finally, three respondents said they did not find Central Perk or the mentor useful and stopped checking in to the forum early in the term, with reasons such as repetitious, inane questions; perceived mentor abrasiveness; and resistance to subverted class authority; meaning: some students preferred to acquire information only from the instructor.


Alcorn and Wanless-Sobel had not realized peripheral participation was as high as it was until reading the survey results.  As it turns out, the high incidence of lurking poses questions about Central Perk’s social presence and how well it scaffolds a democratic contact zone for students. These issues are discussed below. 


Lurkers and Peripheral Participation


Internet studies show that, much of the time people spend on any forum or list is with reading, not posting, and that the primary motivation for participating or frequently logging in to the forum-list is learning by reading the question / answer exchanges (Nonnecke & Preece, 2000; Lakhani & von Hippel, 2003). What this means is that, posting is performed by a small percentage of active forum / list participants, with a majority of people participating peripherally as lurkers. As it turns out, the same holds true with the Central Perk forums, where a small percentage of students in each class of twenty-five students, participated as active and frequent posters and a majority participated as observers or peripheral participants.


Not all lurking in Central Perk was voluntary as mentioned above. In the survey, some lurkers said they wanted to post, but certain circumstances deterred them, some of which Wanless-Sobel can not easily influence. These include lack of time for posting; computer access issues; computer skills literacy, such as how to cut and paste text into a Discussion post; and intimidation by hyperactive interactive class peers, who expected immediate responses to posts.


Since lurking was so high in Central Perk, what are the implications for social presence and the democratic zone as well as online course design?  Although the active Central Perk participants likely had a stronger sense of community and personal satisfaction than observers (Preece, Nonnecke, and Andrews, 2004), peripheral participation also entailed social presence, as evidenced by the survey responses. Central Perk lurkers offered a variety of reasons for experiencing social presence, ranging from satisfaction in receiving class “mail”; comfort in knowing other students were experiencing difficulties with an assignment; and ability to ask a peer questions about the course instructor and her expectations.


High lurking activity with Central Perk, then, does not necessarily negate its social presence efficacy. At the same time, in terms of the democratic zone efficacy, especially in regard to scaffolding a high level of interactivity, the silent student majority is an issue for constructivist and collaborative pedagogy, which is based on students functioning as active, individual learners. The high lurker activity indicates many students are still passive learners, perceiving knowledge acquisition and even interrogative posting as activity for only certain individuals. In this regard, Central Perk is not successful, and Wanless-Sobel’s future design efforts now need to focus on revising Central Perk to provide optimal conditions for an active learning democratic zone.


Design Challenges and Opportunities


In retrospect, after a one semester trial period with Central Perk, Wanless-Sobel sees a number of design challenges and opportunities for future classes.  One idea emerging from this case study is that, her conception of a democratic contact zone in Central Perk is incorrectly based on the notion of an online class as a “homogenous collective” (Foth, 2006), affectively, socially, and intellectually. As a result, Central Perk does not provide students with a choice for peer-to peer networking opportunities (Foth, 2006) or individual empowerment.  In other words, Central Perk, though somewhat successful in nurturing social presence, is interactively too one-dimensional and parochial; it does not offer students the choice of what to do, how to do it, whom to “socialize” with, and whom to gather information from. As an online intellectual and affective space, it cannot fulfill the full range of students’ needs for democratic empowerment (Satchell, 2003).


Additionally, although the goal of Central Perk is to subvert classroom hierarchy, Wanless-Sobel’s use of the discussion board to achieve this goal may need to be reviewed and re-evaluated.  For one thing, Central Perk’s current design does not guarantee there still are not asymmetrical power relations in the classroom (Miller, 2001). For another, its discussion board design may ironically foster hierarchical relationships, which, in turn, encourage lurking (Foth, 2006).  Marcus Foth, whose scholarship addresses computer-supported cooperative work in RL (real life) inner-city neighbourhoods, addresses the inherent hierarchy of discussion board architecture, and says:


A discussion board that affords collective many-to-many broadcast style interaction does not support true social networks that are personal, tacit, intricate, and peer-to-peer. Social networks possess tribal and swarming qualities which [prove] to be incompatible with the mostly hierarchical nature of tools that are designed to support dispersed online communities. (Foth, 2006)


Wanless-Sobel’s original concept of a democratic contact zone had not taken into account either online tribal or swarming behavior, meaning polarization of interests, personalities and learning styles.  Swarming and tribalism are not necessarily positive features (Kelly, 1995) in an online environment, where the goal is for more of a self-organized knowledge culture, but they are often entailed in active networking and relevant to individual and emotional control in the online environment (Zembylas, 2007) and need to be recognized and addressed in course design. Foth’s ideas for digital interactivity of inner-city neighborhood communities suggests that online instructional design for a democratic zone requires features that animate students’ interaction (Foth,2006), while acknowledging some degree of lurking will be inevitable.


Central Perk is still a useful online feature, but its present design needs to be revised and expanded.  Its discussion board format is too conventional, encouraging students to resort to old classroom behaviour patterns, such as looking for answers from one person or from the dominant online personalities and hiding out in the “crowd.” Instead, what students need is a variety of democratic zone choices, especially zones designed to accommodate different learning styles and personalities. The next logical step in Wanless-Sobel’s design experiment is to enlist other media and utilities as democratic zone designations — designations that will allow more tribal and swarming activity and provide individual empowerment: cell phones, with voice and text messaging (Satchell, 2003) and micro-blogging, using a utility like Twitter (http://twitter.com/) ; I-PODs; audio editors, such as Audacity (http://audacity.sourceforge.net/); file sharing facilities; mailing lists; digital video; and perhaps a virtual environment, such as Second Life (http://secondlife.com/), where students can take advantage of visual and spatial components to interact.  Individualizing the democratic contact zone will provide a stronger scaffold for class interaction and active learning in the online environment (Gilchrist, 2000).




In sum, the present case study suggests a student mentor and a safe house discussion forum do increase social presence in online instruction for students, although a considerable percentage takes advantage of these course amenities peripherally as lurkers. Furthermore, some students are unreceptive to these course accommodations. Central Perk is less successful for establishing a student majority democratic contact zone. To actively engage more students, especially lurkers and non-participants, a more complex and nuanced design of the online mentor and safe house forum is needed, including the creation of other safe-house zones using a variety of media and utilities to meet students’ social presence and democratic contact zone needs.




ATIS Telecon Glossary T1.523. (2001). Definition of Lurker. 

Accessed on 08/05/07 at http://www.atis.org/tg2k/_lurker.html


Audacity.Net. Accessed on 24/05/07 at http://audacity.sourceforge.net/


Brookfield, S. D. (2001). A political analysis of discussion groups: Can the circle be unbroken? In R. M. Cervero, & A. L. Wilson (Eds.), Power in Practice: Adult Education and the Struggle for Knowledge and Power in Society (pp. 206-225). San Francisco: Jossey-Bass.


Burns, Philip, J. (1999). Supporting deliberative democracy: Pedagogical arts of the contact zone of the electronic sphere. Rhetoric Review, 19 (1), 128-46.  Accessed on 25/04/07 at http://links.jstor.org/sici?sici=0735


Cavanaugh, Joseph. (2005). Teaching online—A time comparison. Online Journal of Distance Learning Administration, 8 (1). 

Accessed on 07/04/07 at http://distance.westga.edu/~distance/ojdla/spring81/cavanaugh81.pdf


Center for Teaching and Learning (CTL / MNSCU). (2004). CTL grant guidelines, 2004- 05.  http://www.ctl.mnscu.edu/programs/grants/guidelines-05.htm


“Central Perk.” Entry in Wikipedia. Accessed on 25/04/07 at http://en.wikipedia.org/wiki/Central_Perk


Central Perk Poster.  Accessed on 11/01/07 at http://www.allposters.com/-sp/-Posters_i855117_.htm


Doube, W. (2000). Distance teaching workloads. Association for Computing Machinery, Special Interest Group on Computer Science Education Bulletin, 32 (1), 347-351.


Foucault, M. (1979). Discipline & punish: The birth of the prison. Alan Sheridan trans. New York: Random House.


Foucault, M. (1980). Power/knowledge: Selected interviews and other writings 1972-1977. Colin Gordon, et al., trans. New York: Pantheon.


Foth, Marcus. (2006).  Facilitating social networking in inner city neighborhoods. Computer, 39 (9), 44-50.  Accessed on 13/05/07 at http://eprints.qut.edu.au/archive/00004750/01/4750.pdf


“Friends Television Series.” (1994-2004). Entry in Answers.com

Accessed on 20/04/07 at http://www.answers.com/topic/friends-1


Gilchrist, Alison. (2000). The well-connected community: networking to the "edge of chaos." Community Development Journal 35(3), 264-75


Gunawardena, C.N. (1995). Social presence theory and implications for interaction and collaborative learning in computer conferences. International Journal of Educational Telecommunications, 1(2/3), 147-166.


Gunawardena, C.N. and Zittle, F.J. (1997). Social presence as a predictor of satisfaction within a computer-mediated conferencing environment. The American Journal of Distance Education, 11(3), 8-26.


Herrington, J., Oliver, R., & Reeves, T.C. (2003). Patterns of engagement in authentic online learning environments. Australian Journal of Educational Technology. 19 (1), 59-71.


Kelly, Kevin. (1995). Out of control: The new biology of machines, social systems, and the economic world.  New York:  Perseus Books Group, Reprint Edition.

Accessed on 25/04/07 at http://www.kk.org/outofcontrol/ch2-f.html


Lakhani, K. and von Hippel, E.  (2003). How open source software works: “free” user-to-user assistance. Research Policy, 32, 923–943. 

Accessed on 13/05/07 at    http://opensource.mit.edu/papers/lakhanivonhippelusersupport.pdf


Miller, N. (2001). The politics of access and communication: Using distance learning technologies. In R. M. Cervero, & A. L. Wilson (Eds.), Power in Practice: Adult Education and the Struggle for Knowledge and Power in Society (pp. 187-205). San Francisco: Jossey-Bass.


Nonnecke, B., and J.  Preece. (2000). Lurker demographics: Counting the silent. CHI 2000, The Hague, ACM Press. pp. 73-80.


O’Regan, Kerry.  Emotion and e-learning. Journal of Asynchronous Learning Networks, 7 (3), 78-92.


Pirius, Landon, Director of Student Enrollment Services, Inver Hills Community College, Inver Grove Heights, MN  55076.  E-mail to Amanda Alcorn, 25 April 2007.


Pratt, Mary Louise. (1991). Arts of the contact zone. Profession, 91, 33-40.


Preece, J., Nonnecke, B.  and Andrews, D.C. (2004). The top five reasons for lurking: Improving community experiences for everyone,” Computers in Human Behavior, 20 (2), 201-223.


Reeves, B. and Nass, C. (1996). The media equation: How people treat computers, television, and new media like real people and places. Cambridge University Press, New York,


Richardson, Jennifer C. and Swan, Karen. (2003). Examining social presence in online courses in relation to students’ perceived learning and satisfaction. Journal of Asynchronous Learning Networks, 7 (1), 68-88.


Satchell, C. (2003). The swarm: Facilitating fluidity and control in young people’s use of mobile phones. Proc. OZCHI 2003: New Directions in Interaction, Information Environments, Media and Technology, Information Environments Program, Univ. of Queensland.


Second Life.Com.  Accessed on 17/05/07 at http://secondlife.com/


Short, J., Williams, E., and Christie, B. (1976). The social psychology of telecommunications. London: John Wiley and Sons.


SitePal.Com. Accessed on 22/04/07 at http://sitepal.com/


Suchman, L. A. (1987). Plans and situated action: The problem of human-machine communication: Cambridge University Press, New York.


Survey Monkey.Com.  Accessed on 20/04/07 at http://surveymonkey.com/


Swan, K., Shea, P., Frederickson, E., Pickett, A. Pelz, W., and Maher, G. (2000). Building knowledge building communities: Consistency, contact, and communication in the virtual classroom. Journal of Educational Computing Research, 23 (4), 389-413.


Tu, Chih-Hsiung and McIsaac, Marina. (2002). The relationship of social presence and interaction in online classes.  American Journal of Distance Education, 16 (3), 131-50.


Twitter.Com. Accessed on 22/05/05 at http://twitter.com/


Zembylas, Michalinos. (2007). A politics of passion in education: The Foucauldian legacy. Educational Philosophy and Theory, 39 (2),135–149.



Colette Wanless-Sobel, Ph.D.

Department of English and Interdisciplinary Studies

Inver Hills Community College

2500 East 80th Street,

Inver Grove Heights, Minnesota 55076-3224 USA



Amanda Alcorn

Student in Law Enforcement

Inver Hills Community College

2500 East 80th Street

 Inver Grove Heights, Minnesota 55076-3224 USA



Learners’ profiling by data driven approaches



Data driven approaches for learners’ profiling


In recent years, E-Learning became an opportunity for re-thinking learning itself in terms of personalized learning. Personalization in this context is meant as the capacity of the system to adapt to the individual differences and the needs of learners: this implies the capacity of representing these differences and needs and their evolution. “Personalized” learning is said both regarding individuals and regarding groups.


Differently from traditional educational settings, E-Learning allows tracking the users’ behaviour in the Electronic Learning Environments: these data are fully authentic. Therefore, to improve personalization, data driven approaches (e.g. [1]) that are accurate, precise and flexible, should be tested for analyzing and modelling online interactions. The approaches have to deal with the complexity of learning, taking into account:

  • the evolution of the behaviour during time;
  • the pedagogical, cognitive and metacognitive features of the learning process;
  • the variables affecting learning (social and situated learning, impact of each artefact on the learning process);
  • the uncertainty of the domain, and the unexpected patterns that can arise;
  • the individuality of each profile (or of each group in case of collaborative work).


In past years, an increasing attention has been dedicated to data driven approaches for analyzing educational data within different Research Communities (eg Data Mining, User Modelling and ITSs). Today, the field is growing and maturing [11], but there are still many open problems.


Experimental settings, mathematical settings and results


Our dataset comes from the V Framework European WINDS Project, and is made by sessions data. The attention was focused on establishing if data driven approaches – in particular the ones attested in the literature for modelling complex processes, such as biological, medical or economic processes - can be applied to learning to attain a useful and meaningful characterization of pedagogical, cognitive and metacognitive features.


From the introduction of data driven approches we expected improvements in accuracy and precision in modelling complex domains, and smaller cost in terms of human and material resources. From a mathematical viewpoint, sessions data are heterogeneous, both for length and for the number belonging to each individual; moreover, being they collected outside any experimental design settings, classical statistical frameworks premises do not hold; eventually, appropriate measures have to be selected in order to avoid biased results.


Data are the absolute and joint frequencies of usage of different kinds of objects provided by the WINDS ALE, including traditional objects (units, paragraphs), interactive objects (cases, exercises), collaborative objects (annotations, forums), non sequential navigation tools (glossaries and maps). Data are expressed on a numeral scale.


Two views have been considered: the first, called “profiles view”, focuses on the average preferences of each learner; the second, called “sessions view” focuses on the evolution and changes of the profiles during time.


We have applied:

  • multivariate methods, such as Principal Component Analysis (PCA, [3]) and Multiple Correspondence Analysis (MCA, [2, 4]), together with appropriate proximity measures and/or additional data (test scores) where available; MCA has been also compared with classical methods (p values and chi-square test);
  • “frequent episodes discovery” algorithms (FED, [5]) in sessions treated as sequence data;
  • time series methods, such as autocorrelation and cross correlation, in order to get synthetic indexes of the evolution of the profiles along time.


Results shows the effectiveness of these methods. In fact, PCA detects unobservable dimensions coherent with important features of technology-enhanced learning (sequential, non sequential, multimodal, collaborative strategies) [6]. MCA detects different strategies according to the learning outcomes. Moreover, these methods are more sensitive than classical statistical ones in detection of differences even when they are not statistically significant [7]. FED-based sessions analysis and time series methods show two different ways of non sequential navigation patterns whose difference is statistically significant [8, 9].



Figure 1 – The data driven approaches



Conclusions and further work


From our experience, data driven approaches are effective and promising for detecting learning process patterns; moreover, the unobservable dimensions revealed are coherent with some important features of learning, and helpful in individual differences detection. In order to improve precision and accuracy, it is remarkable to consider the need for:

  • multi-source heterogeneous data integration inside the profiles;
  • multi-models cross-validations to obtain profiles as coherent as possible;
  • efficient proximity measures for heterogeneous data;
  • techniques for high dimensional data visualization.


Improvement can be foreseen in: intelligent patterns discovery, monitoring systems, evaluation and assessment tasks, adaptivity/personalization of responsive environments, supporting collaborative learning tasks, and research; however, limitations should be taken into account as well (eg the easiness of use and of understanding by unskilled people in real-life tasks, and the impact on educational contexts).


More can be found at http://www.del.univpm.it:8080/del/Members/silvia/pubblicazioni_html  




Cherkassky & V. Mulier, F. (1998). Learning from data, Wiley.


Greenacre, M. (1984). Theory and Applications of Correspondence Analysis, Academic Press.


Jolliffe, T. I. (1986). Principal Component Analysis, Springer.


Lebart, L., Morineau, A.& Warwick, K. M. (1984). Multivariate Descriptive Statistical Analysis, Wiley.


Mannila, H., Toivonen, H. & Verkamo, A. I. (1997). “Discovery of Frequent Episodes in Event Sequences”, Data Mining and Knowledge Discovery, 1:259-289.


Viola, S. R., Giretti, A.& Leo, T.(2005). “Discovering learning process patterns by multivariate analysis of usage frequencies data in e-learning courses”. ICL 2005 Proceedings, 1B-4.


Viola, S. R., Giretti, A.& Leo, T. (2006). “Exploring attitudes of learners with respect to different learning strategies and performances using statistical methods”. International Journal of Emerging Technologies in Learning, 1(1), June 2006, ISSN 1863-0383, http://www.online-journals.org/index.php/i-jet (retrieved March 22, 2007)


Viola, S. R., Giretti, A.& Leo, T. (2006). “Differences in meaningful learning strategies of navigation: an empirical model.” IEEE ICALT 2006 Proceedings, pp.441-445.


Viola, S. R., Giretti, A.& Leo, T. (2006): “Non sequential pattern analysis along time for “meaningful learning” strategies of navigation detection”. ICL 2006 Proceedings, 6A–3.


Viola, S. R.(2006): Learning Process Patterns Extraction and Characterization using mathematical and statistical methods, Ph. D. Thesis, Universita’ Politecnica delle Marche,



http://www.educationaldatamining.org/ (retrieved March 22, 2007)


Silvia Rita Viola, PhD

Universita’ Politecnica delle Marche

DIIGA, Dip. Ing. Informatica,

Gestionale e dell’Automazione “Maurizio Panti”

Via delle Brecce Bianche – 60100 Ancona, Italy



Alberto Giretti

Universita’ Politecnica delle Marche

DACS – Dipartimento di Architettura,

Costruzioni e Strutture

Via delle Brecce Bianche – 60100 Ancona, Italy



Tommaso Leo

Universita’ Politecnica delle Marche

DIIGA – Dip. Di Ingegneria Informatica,

Gestionale e dell’Automazione “Maurizio Panti”

Via delle Brecce Bianche – 60100 Ancona, Italy



The URO Project - A new curriculum to improve the “quality of life” for people with URinary incontinence by the use of ICT





ICT technologies should be useful tools for the acquisition and the increase of knowledge and skills in the field of medicine. The application of ICT in the health sector could provide a better quality of life to the citizens and an easier job environment for physicians and other health care workers. ICT can be used wherever it has a clear benefit, such as reaching remote populations, offering the tools for building a national health network and providing continuous training for doctors and hospital attendants, sharing knowledge between institutions and countries. In this latter field ICT services, and in particular E-Learning, could allow an effective improvement of training, management and organization of medical staff. So the introduction of e-health (health services and information delivered or enhanced through the Internet and related technologies) represents the promise of information and communication technologies to improve health and the health care system. The Uro project is set in this scenario. In fact it aims to prepare a “new professional”, named Content Therapist, in the area of Urinary Incontinence by integrating specific preparation in the field of urology with disciplines principally concerned with gynecology, obstetrics, physical and rehabilitative medicine, neurology, coloproctology, geriatrics, paediatrics. It is foreseen the extensive use of New Educational Technologies and of E-Learning approach. The main activities are concentrated on the preparation of a curriculum for appropriate studies for the training of human resources, from both Asian and European countries and course material (content definition, teaching tools, production/printing of course material), their dissemination and the demonstration of the realized training services. In particular the target groups are medical attendants of Italy, Slovenia, India and China Hospitals. The project, by means of E-Learning approach and Information Communication Technologies, aims to provide and the test a mechanism for the exchange of information relating to state-of-the-art methodologies and technology in the various components of care and assistance. In this paper we describe the general aims of URO project and the results of the first activities.


Main aims of the Uro Project


ICT and its applications are increasingly looked upon as the potential answer to the requirement of a modern society, with demands for better healthcare, improvements in medical outcomes, and maintenance of relatively high quality of life, especially with the onset of chronic health conditions coming to the fore as a key issue. Furthermore, implying a view of utilizing the technology as a tool for readdressing the prevailing state of affairs, ICT tools and applications are also seen as having a potential to support an enhanced access to health information in general and indeed, to the health system itself, in particular [1]. Today, Continuous Medical Education (CME) becomes a crucial factor, because the life of knowledge and human skills in the field of medicine is shorter than ever. E-learning comes with solutions and methods, which can be very helpful in supporting doctors with access to the up-to-date medical knowledge and achievements. It allows creation of interactive model of learning, which stimulates knowledge acquisition. Another advantage is that e-learning provides flexibility in both time and location, while accessing medical curriculum presented online [2]. There is a possibility of collaboration between teachers and students from different universities and countries, which allows exchange of knowledge and experiences. In this field the URO project aims to give the main contribution. In fact it aims to make available a new curriculum, supported by ICT technologies, for appropriate studies for the training of human resources, from both Asian and European countries. The overall objective of the URO project is the preparation of a “new professional”, named “Continence Therapist”, who works at home, in the community and in the hospital and is the fulcrum of the interaction between the needs of the patient and the necessary social and specialist medical answers. We have to underline that the “Continence Therapist” highlights the need for a multi-dimensional and, above all, cross-discipline professional preparation able to offer integrated and sequential management in cooperation with specialists in the sector and/or centres in the world, for continence and the pelvic floor. In this scenario a central role is played by the ICT and in particular by the e-learning methods. Any e-learning platform and provision should give opportunities to improve the quality and the variety of teaching and learning not otherwise achieved through traditional methods. The particular requirement is the capacity of the e-learning provision to offer the additional services and tools for the needs of the target groups of the Uro project. We have two different kinds of target groups: ultimate target groups and direct target groups. The Ultimate Target Groups will be targeted within the participating countries. The indirect target groups are: Registered Nurses, Licensed Practical Nurses, Nurse Assessment Coordinators, Midwives and Physical Therapists, Occupational Therapists. Precise numbers of the above categories are impossible to quantify. In Italy approx 300.000 nurses operate together with a more limited number of midwives and other therapists. In Slovenia these numbers are much smaller (approx 10.000). In India and in China these numbers are of several thousand in each category but official reliable data are unavailable. Immediate (direct) target groups in the medium term are nurses and midwives freshly graduated at the 3 partners (estimated at 360 people) on the one hand, and the staff involved in the development of the project (22 people). These 360 neo-professionals will take advantage of the results of URO and will be involved in the phase of demonstration and dissemination of the learning services. In particular, there will be approx 120 people from Xiehe Hospital - Tongji Medical College, 100 from University of Ljubljana and 140 from NU Trust Hospital. We will start our display of evaluative considerations from a pedagogical and psychological point of view. The project, by means a virtual community, will provide a mechanism for the exchange of information relating to state-of-the-art methodologies and technology in the various components of care and assistance. As the project is multi-national and interdisciplinary, professionals from a range of professions, who may not always work together, will be able to share experience and knowledge. The following elements are of importance:


·       Students and their relationships are to be in the centre of attention

·       Enhancing the learning scenario by allowing communication richness

·       Focus on the social environment

·       Acknowledging the individuality of learning styles


Definition of Learning and training needs of the final users


The first activity of the project had the objective of showing the needs of the final users in terms of the characteristics of the learning service, interface and contents. The aim was to recognise in a clear manner the learning needs of each target group. In order to achieve this aim a series of interviews of members of the direct target groups was performed to highlight the needs of users with regard to the learning system to be implemented. We adopted a validated questionnaire, ILS – Index of Learning Styles questionnaire, for evaluation of preferred learning styles. ILS is an instrument used to assess preferences in four dimensions (active/reflective, sensing/intuitive, visual/verbal and sequential/global) of a learning style model and was designed by Richard M. Felder and Linda K. Silverman. We used the ILS questionnaire on two direct target groups: Slovenia (representative for European countries) and India (representative for Asian countries). The filled-in questionnaires were analyzed by the Italian partner and the results are as follows:


Slovenian Target group (N = 205)


The behaviour of the average Slovenian Target group:

[ACT/REF] = has a moderate preference on the active direction. It tends to discuss the information or explains it to others rather than to reflect on it.

[SNS/INT] = has a moderate preference on the sensing direction. It prefers to learn facts, to be patient with details rather than grasping new concepts. It is more practical than theoretical.

[VIS/VRB] =has a moderate preference on the visual direction. It remembers best what is seen - pictures, diagrams, films - rather than written or spoken explanations.

[SEQ/GLO] = has a moderate preference on the sequential direction. It tends to gain understanding in linear steps, with each step following logically from the previous one.


Indian Target group (N = 61)


The behaviour of the average Indian Target group:

[ACT/REF] = has a moderate preference on the active direction. It tends to discuss the information or explains it to others rather than to reflect on it.

[SNS/INT] = has a moderate preference on the sensing direction.  It prefers to learn facts, to be patient with details rather than grasping new concepts. It is more practical than theoretical.

[VIS/VRB] =has a moderate preference on the visual direction. It remembers best what is seen - pictures, diagrams, films - rather than written or spoken explanations.

[SEQ/GLO] = has a moderate preference on the sequential direction. It tends to gain understanding in linear steps, with each step following logically from the previous one.


Description of Slovenian Target Group

Description of Indian Target Group




The study content of the Continence Therapist study programme will be hence designed and structured to meet the recognised learning style preferences of the three Target groups. Given that the learning styles of Slovenian and Indian groups are strikingly similar, no major regional differences in form and structure of the study programme has to be taken in consideration for Europe and Asia. These results will be taken into consideration in the next phases of the project. In particular we will design, implement and customize an e-learning platform and the learning contents keeping in mind these indications.




The Euser Project. eUser – Workpackage 1: Conceptual and Analytical Framework. D1.1: eUSER Conceptual and Analytical Framework (Part A and Part C). European Commission, 6th Framework Program Contract number: IST-2002-507180


P. Rudowski, R. Rudowski (2006). E-Learning In Medicine – Myth Or Reality, in Malina Jordanova and Frank Lievens Proceedings of Med-e-Tel 2006, 5-7 April 2006, Luxembourg


R. M. Felder, L. K. Silverman (1988). Learning and Teaching Styles in Engineering Education, Engineering Education, 78(7), 674-681 (1988)





Francesco Colace

DIIIE-Università degli Studi di Salerno

Via Ponte Don Melillo, 1 84084 Fisciano (Sa)



Massimo De Santo

DIIIE-Università degli Studi di Salerno

Via Ponte Don Melillo, 1 84084 Fisciano (Sa)



HotKey Coach: Training users to interact efficiently with software applications






Once people become fluent with the basic workings of a software package, they turn their focus towards productivity. To achieve productivity, a user must become efficient with the execution of the most frequent tasks. While using a software application, a user can save a considerable amount of time over the course of a typical work session by substituting keyboard shortcuts for mouse actions. However, no useful method exists for helping an intermediate software user transition from relying on the mouse to using the keyboard. Consequently, many, if not most, users settle for less efficient methods of interaction with the application.


In our lab at Brandeis University, we are currently working on a method to ease the transition toward an expert’s efficiency using an “in-the-loop” training method. This method integrates training into the normal course of the user's activity. Thus the user can learn within the context of her normal tasks as it becomes relevant. By keeping the training closely associated with the task that is being performed, the user only learns what is required by the task at hand.




As outlined by Anderson [1], the acquisition of a skill progresses through three stages: cognitive stage, associative stage and autonomous stage.  Bederson [2] defines these three stages in terms of a graphical user interface (GUI) interaction as:


·         (Novice) Cognitive Stage:  Uses the GUI for interface exploration.

·         (Intermediate) Associative Stage:  Knows GUI elements, but still requires feedback.

·         (Expert) Autonomous Stage:  Requires no feedback from the interface.


When users execute a keyboard shortcut, they have minimal interaction with the interface. By reducing the feedback between user and user interface, while at the same time eliminating the need to transfer the hand between the keyboard and the mouse, users reach the autonomous stage and achieve maximum efficiency [3]. 


However, most software users fall into the associative stage, with some forays into the autonomous stage with the most repetitive tasks, such as save, copy and paste. Most users fail to achieve significant progress at autonomous interaction with the application because the learning required to advance using current methods is too time consuming. For example, cheat sheet cards and GUI widget tool-tips require extra time and effort to learn because they do not fit in with the normal flow of the user's tasks. This limitation results in only the most dedicated users developing a strong shortcut repertoire.


Our Approach


Our goal is to develop a method that helps the user become more efficient with their everyday software tasks. For example, if a user of Microsoft Word frequently switched between the Normal View and the Page Layout View via the mouse, learning the keyboard shortcuts for these options would, over the course of multiple operations, produce significant time savings. The net effect of this learning would be to replace user actions to move the hand from the keyboard, to the mouse, navigate to the menu option, select it, with a simple keystroke operation. It also reduces cognitive load.


We have devised a system that teaches keyboard shortcuts within the current task as the user proceeds with their work. When a user selects an operation from a menu item, a window displays the keyboard shortcut for the operation. At this point, the user has the option of performing the keyboard shortcut to reinforce the learning, or they can ignore the tip by either continuing with their task, or selecting a check box within the tip indicating that they no longer wish to be coached for that specific operation in the future.  See Figure 2 as an example.

Figure 2:  Coach Tip for Paragraph, Format in Word



By providing the shortcut for an operation directly after the operation is executed via the interface, the user sees the shortcut while the operation is still fresh in memory, making a direct link between the operation and the shortcut. Working within the typical usage patterns of the software, we ensure that only the frequently used commands are taught first. Allowing the user the ability to ignore the tip keeps the opportunity to learn from interfering with the workflow.


Preliminary Results


Currently, our software runs on the Mac OS X operating system. The HotKey Coach software is available to a limited test base and early results have demonstrated direct examples of learning. The data shows numerous instances where a user starts out solely using the mouse for a specific operation, but over time transitions into keyboard usage for common operations.  There is also evidence of transfer, where once a user learned the keyboard shortcut for an operation, they used that shortcut for the same operation in a different application.


Our data shows users making the first step towards autonomous software interaction after just a short amount of time using our coaching software. Because our method of coaching during the normal workflow provides a more natural approach to learning with minimal interruption, a user is more likely to spend the time required to learn the shortcuts for their commonly used commands. Our next steps are to expand the supported application features beyond menus to toolbar and palette operations. We are also investigating methods for assigning shortcuts for common operations that do not currently have a shortcut. Given the current trend to move towards network-centered applications, we are also working on applying our coaching method to web-based applications, such as GMail, that now support direct keyboard interaction.




Anderson, J.R. (1981). Acquisition of Cognitive Skill Dept. of Psychology, Carnegie-Mellon University


Bederson, B.B. (2004). Interfaces for staying in the flow. Vol. 5, ACM Press. 1-1


Lane, D.M., et al. (2005). Hidden Costs of Graphical User Interfaces: Failure to Make the Transition from Menus and Icon Toolbars to Keyboard Shortcuts. International Journal of Human-Computer Interaction. 16(2): p. 133-144




Brian Krisler

Brandeis University



Richard Alterman

Brandeis University



Evaluation of Training – A Critique of Kirkpatrick’s Evaluation Model



Evaluation of training means measuring the effectiveness of a training program. Evaluation helps in defining the learning outcomes more sharply, remove unnecessary training content, and ensure that the training method meets the training needs of the learners and consequently of the business.


This effectiveness is as much about short-term retention as about the long-term retention and application by learners. Today most organizations talk about measuring the effectiveness of the training programs. While most traditional institutions have mechanisms in place where they measure the effectiveness of the classroom training programs, most modern day training programs are still not evaluated objectively or accurately.


There are several approaches used to measure the effectiveness of training. One popular approach is Kirkpatrick's Four Levels of Evaluation. In this model, each successive evaluation level is built on information provided by the earlier lower level. Let’s look at a brief description of each of the four levels and the respective criticisms.


Level 1 Evaluations: Reactions


As the name suggests, evaluation at this level captures the immediate reaction of the participants about the training. It assesses learners’ satisfaction with the training. While this is not an accurate measure of effectiveness, it certainly can be a filter for bad training. Positive reactions may or may not result in good learning but negative reactions can certainly be roadblocks to learning.


Criticism: Research shows that reactions should be observed and not asked. Kirkpatrick’s model has not defined the mechanisms of capturing the reactions. The model depends on the participants to provide this information, which is not a measure of the effectiveness. At best, it can be an indicator.


Level 2 Evaluations: Learning


At this level, evaluation moves beyond learner satisfaction to assess the learning that has happened in terms of acquisition of skills, knowledge, or attitude. Common methods used to test this are pretest and post test assessments to determine the amount of learning that has occurred.


Criticisms: Pre-assessment and post-assessment can capture knowledge and comprehension level skills. Also this is a measure of how effective the training is in transferring this information to the learner’s short-term memory. It does not measure the effectiveness of training on the learner’s cognitive skill, which will ultimately determine whether the learning will be applied at the workplace or not. In short, this phase does not measure learning; it only measures “rememberability”.


The other point of criticism is that at no point is there a measurement against the specified learning goals and outcome. There is no mechanism to compare the % of fulfillment of the specified learning goals and outcome.


 Level 3 Evaluations: Transfer


This level is a critical measure of the amount of transfer of learning or amount of application of learning at the workplace. Evaluating at this level attempts to answer the question - Are the newly acquired skills, knowledge, or attitude being used in the everyday environment of the learner? According to Kirkpatrick, measuring at this level is difficult as it is often impossible to predict when the change in behavior will occur, and thus requires important decisions in terms of when to evaluate, how often to evaluate, and how to evaluate.


Criticisms: This is the most critical level of evaluation because it helps measure the most primary purpose of training. However, Kirkpatrick’s model does not specify the mechanisms of capturing and analyzing the information obtained in this level. While it is impossible to predict a change in behavior post training; information about learner behavior should have been captured during learner analysis (pre-training) so that the training is designed keeping in mind the said behavior. Once, this variable is factored in the evaluation metrics, evaluating the change in behavior or acquisition or transfer of skill will be much more effective and accurate.


Kirkpatrick’s model also ignores the other dependencies at this level of evaluation, such as “who will measure this information, what are the interventions needed, when to intervene, and what happens if intervention is delayed or absent…” The actual measurement of training effectiveness will depend on all these dependencies that are not defined in the Kirkpatrick’s model.


Level 4 Evaluations: Results


At this level, evaluation is done to measure the success of the program in terms that managers and executives can understand - increased production, improved quality, decreased costs, reduced frequency of accidents, increased sales, and even higher profits or return on investment. However, most organizations do not address this form of evaluation because they think determining results in financial terms is difficult to measure, and is hard to link directly with training.


Criticisms: This evaluation level is the most critical test for business and it’s the least explained in Kirkpatrick’s. The major criticism against Kirkpatrick is that its fourth level does not actually estimate the business impact of the training. It does not provide the outcome as ROI. While it makes an effort to measure the success of the program from the perspectives of managers and executives, it actually does not evaluate how the program relates to the measures of business success.


There is an interesting article Alternatives for Measuring Learning Success by Kaliym Islam that talks about how with the advent of the electronic delivery of training programs and the increased capital investment required to develop these programs, the Kirkpatrick model has become obsolete. Some interesting read about criticisms of Kirkpatrick’s Models:


  1. http://www.e-learningguru.com/articles/art5_1.htm
  2. http://www.sibm.edu/home/pdfs/research/srp/hr/SRP_Developing_a_quantitative_model_to_measure_training_effectiveness.pdf
  3. http://www.ltimagazine.com/ltimagazine/article/articleDetail.jsp?id=85206


The next article in this series will talk about an evaluation methodology evolved by Kern. This methodology not only overcomes some of Kirkpatrick’s deficiencies but also talks about a paradigm shift in the way eLearning should look at evaluation.




Geeta Bose
Director, Learning Solutions
Kern Communications Pvt. Ltd.



Using Microsoft’s Homepage Starter Kit to teach Content Management Systems in Postgraduate Journalism Students





A Content Management System (CMS) is a computer software system used to assist users in the process of content management. A CMS supports the organization, control, and publication of a large body of documents and other multimedia content.  CMS often facilitates the collaborative creation of documents. Usually a CMS is a web content management system that is a content management system with additional features to ease the tasks required to publish web content to websites (Wikipedia).


CMSs are often used for storing, controlling, versioning, and publishing industry-specific documentation such as news articles, operators' manuals, technical manuals, sales guides, and marketing brochures. This kind of environment is an essential part in a web design course for postgraduate Journalism students. The problem is that the majority of the available for downloading CMSs require installing additional software in the host server (a database -usually MSQL, - PERL, or PHP etc.) (CMS review). The installation and customization of such software is not an easy task. Media Informatics Lab at the Department of Journalism and Mass Communication in Aristotle University of Thessaloniki solved the previous problem with the deployment of the Microsoft Homepage Starter Kit. It is an ASP.NET 2.0 based Content Management System. No ASP knowledge is required. If the host server is running Microsoft Windows 2003 Server Edition, the ASP.NET 2.0 can be installed automatically with the optional updates. The only thing the administrator has to do is to activate the ASP.NET and add the read permission for the ASPNET user in a specific directory.


Description of the Homepage Starter Kit


First thing before a user starts creating content is to set some global parameters and information for the new website. Thus he is able to change some global parameters and even change the complete layout of your website, even after all content has been added. The user can customize the navigation and can create new pages. The pages are listed in the order in which they appear in the navigation. Indented items are subpages of the page above them and will be shown in a secondary navigation. The CMS seems to supports many level of navigation, although the user must not exceed three levels. After the third level the CMS does not display the navigation levels.  Each page can be moved within the navigation hierarchy or can be made a subpage of another page.The CMS usually allows by default everyone to visit the pages, but it also supports restricted area for members.



Page and Navigation customization



The CMS provides a simple yet powerful user management that can be used to create restricted areas. There are 2 levels of rights available:  Registered Users: Pages that are not visible to everyone can be visited by registered users only. There is only one level of registered users. Administrators: Among the registered users there can be one or more administrators. Administrators have full access to the environment and can edit the site, create and delete pages and mange user profiles.


Every new page is empty and can be customized by adding 10 different modules. You can add as many modules to your page as you like and arrange them in any order. Many modules will have a rich text editor to create new content. This editor allows the basic text formatting tasks and even table creation. If you are familiar with HTML, you can toggle views between the design view and the HTML view.





HTML editor





The Microsoft Homepage Starter Kit can be downloaded from Codeplex at http://www.codeplex.com/Wiki/View.aspx?ProjectName=MyWebPagesStarterKit. You can unpack it directly on the server and start building web sites. If you want to be more involved, using Visual Studio or the Visual Studio Web Developer Express Edition, you can customize and extend the features of the CMS. Our experience in using it in our lab courses has shown that the Homepage Starter Kit is an extremely useful tool in teaching web design. All data are stored in the App_Data folder. So you can easily collect lab exercises by simply copying the files from the App_Data folder. Also by deleting all files from the previous folder you can reset the CMS to its initial state, ready to be used by another student.




Wikipedia the free encyclopedia, Available at http://en.wikipedia.org/wiki/Content_management_system (retrieved on 04/01/07).


CMS review available at http://www.cmsreview.com/ (retrieved on 04/01/07).


The Microsoft Homepage Starter Kit,

Available at http://www.codeplex.com/Wiki/View.aspx?ProjectName=MyWebPagesStarterKit (retrieved on 04/01/07





Andreas Veglis

Media Informatics Lab. Dept of Journalism & MC

Aristotle University of Thessaloniki

54006 Thessaloniki