The Experts below are selected from a list of 10461 Experts worldwide ranked by ideXlab platform
Peter Gerjets - One of the best experts on this subject based on the ideXlab platform.
-
CogSci - The role of Thinking-Aloud instructions and prior domain knowledge in information processing and source evaluation during Web search
Cognitive Science, 2013Co-Authors: Yvonne Kammerer, Peter GerjetsAbstract:The role of Thinking-Aloud instructions and prior domain knowledge in information processing and source evaluation during Web search Yvonne Kammerer (y.kammerer@iwm-kmrc.de) Knowledge Media Research Center, Schleichstr. 6, 72076 Tuebingen, Germany Peter Gerjets (p.gerjets@iwm-kmrc.de) Knowledge Media Research Center, Schleichstr. 6, 72076 Tuebingen, Germany Abstract about information quality (i.e., the accuracy, authority, objectivity, or currency of information; e.g. Rieh, 2002; Tombros, Ruthven, & Jose, 2005). According to Tombros et al. (2005) also the structure of Web pages (i.e., the clarity of the Web page or the organization of the information therein) is evaluated extensively. The aim of the present paper was to identify potential reasons for the divergent findings. Specifically, we examined the impact of the instructions used in the studies as well as of participants' prior domain knowledge on information processing and source evaluation during Web search, using a rich multi-method approach including eye tracking methodologies, log file data, and verbal protocols. This paper examines the impact of Thinking-Aloud (TA) instructions as well as of individuals' prior domain knowledge on information processing and source evaluation during Web search on a health-related topic. With regard to TA instructions, prompted instructions that entailed evaluation prompts (as used in some previous Web search studies) were compared to neutral instructions (in line with the standards defined by Ericsson & Simon, 1993) and to a silent condition. To measure participants' (N = 44) information processing and source evaluation we used a rich multi-method approach including eye-tracking methodologies, log file data, and verbal protocols. Results indicate that prompted TA instructions as compared to neutral instructions significantly increased participants' verbal reflections on information quality and on structural aspects of Web pages, given that participants possessed at least a moderate level of prior domain knowledge. In addition, prompted instructions resulted in less linear viewing sequences on the search engine results pages than the silent condition. Finally, the higher participants' prior domain knowledge the more intensely they scrutinized the search results presented by the search engine and the smaller were their average pupil sizes, which indicated lower cognitive load. The significance of the results is considered in light of methodological as well as educational implications. The Role of Prompted Thinking-Aloud Instructions One reason for the divergent findings might be a methodological one, namely that in the studies by Rieh (2002) and Tombros et al. (2005) participants were instructed beforehand to explain what criteria they used to evaluate Web information (Tombros et al., 2005) or to select good or credible information during Web search and to explain their evaluation processes in the form of postsearch interviews including specific evaluation-related questions (Rieh, 2002). According to the seminal work by Ericsson and Simon (1993) and the meta-analysis by Fox, Ericsson, and Best (2011), however, procedures for verbal reporting are only nonreactive (i.e., do not alter thought processes and task performance) when instructions to think Aloud are given in a neutral way, by instructing participants to verbalize their thoughts per se. In contrast, procedures that entail describing or explaining thoughts and actions – as it was the case in the studies by Tombros et al. (2005) and Rieh (2002) – are significantly reactive, altering participants' course of cognitive processing and leading to higher task performance than silent conditions. Hence, the instructions used by Tombros et al. (2005) and Rieh (2002) might have increased participants’ awareness of the necessity of critically evaluating the information retrieved during Web search. Indirect evidence for this assumption comes from the studies by Hargittai et al. (2010) and Savolainen and Kari (2006) that used neutral Thinking-Aloud (TA) instructions and only found few utterances related to information quality or structural aspects of Web pages. A first aim of the present study, thus, was to further prove this assumption by directly comparing participants' verbal utterances when given TA Keywords: Web search; source evaluation; prior domain knowledge; Thinking-Aloud instructions; eye tracking Introduction In recent years, the World Wide Web (WWW) has evolved into a major information resource offering easy access to billions of Web pages on almost any topic. However, as anyone can publish virtually any information on the WWW, the quality of Web pages can vary widely. That is, misleading and low-quality Web pages, for example, in the field of medicine and health care, are as common as those providing neutral, high-quality information. Hence, to avoid the selection and use of doubtful or even false information, it is important that Web users themselves critically evaluate the quality of information they retrieve from the Web. Previous empirical findings about Web users' source evaluation as indicated by verbal reports, however, are inconclusive. Whereas some studies indicate that Web users mainly evaluate search results and Web pages only based on the topical relevance or their ranking in the search engine results page (e.g., Hargittai, Fullerton, Menchen-Trevino, & Thomas, 2010; Savolainen & Kari, 2006), others suggest that Web users are also concerned to a substantial extent
-
the role of Thinking Aloud instructions and prior domain knowledge in information processing and source evaluation during web search
Cognitive Science, 2013Co-Authors: Yvonne Kammerer, Peter GerjetsAbstract:The role of Thinking-Aloud instructions and prior domain knowledge in information processing and source evaluation during Web search Yvonne Kammerer (y.kammerer@iwm-kmrc.de) Knowledge Media Research Center, Schleichstr. 6, 72076 Tuebingen, Germany Peter Gerjets (p.gerjets@iwm-kmrc.de) Knowledge Media Research Center, Schleichstr. 6, 72076 Tuebingen, Germany Abstract about information quality (i.e., the accuracy, authority, objectivity, or currency of information; e.g. Rieh, 2002; Tombros, Ruthven, & Jose, 2005). According to Tombros et al. (2005) also the structure of Web pages (i.e., the clarity of the Web page or the organization of the information therein) is evaluated extensively. The aim of the present paper was to identify potential reasons for the divergent findings. Specifically, we examined the impact of the instructions used in the studies as well as of participants' prior domain knowledge on information processing and source evaluation during Web search, using a rich multi-method approach including eye tracking methodologies, log file data, and verbal protocols. This paper examines the impact of Thinking-Aloud (TA) instructions as well as of individuals' prior domain knowledge on information processing and source evaluation during Web search on a health-related topic. With regard to TA instructions, prompted instructions that entailed evaluation prompts (as used in some previous Web search studies) were compared to neutral instructions (in line with the standards defined by Ericsson & Simon, 1993) and to a silent condition. To measure participants' (N = 44) information processing and source evaluation we used a rich multi-method approach including eye-tracking methodologies, log file data, and verbal protocols. Results indicate that prompted TA instructions as compared to neutral instructions significantly increased participants' verbal reflections on information quality and on structural aspects of Web pages, given that participants possessed at least a moderate level of prior domain knowledge. In addition, prompted instructions resulted in less linear viewing sequences on the search engine results pages than the silent condition. Finally, the higher participants' prior domain knowledge the more intensely they scrutinized the search results presented by the search engine and the smaller were their average pupil sizes, which indicated lower cognitive load. The significance of the results is considered in light of methodological as well as educational implications. The Role of Prompted Thinking-Aloud Instructions One reason for the divergent findings might be a methodological one, namely that in the studies by Rieh (2002) and Tombros et al. (2005) participants were instructed beforehand to explain what criteria they used to evaluate Web information (Tombros et al., 2005) or to select good or credible information during Web search and to explain their evaluation processes in the form of postsearch interviews including specific evaluation-related questions (Rieh, 2002). According to the seminal work by Ericsson and Simon (1993) and the meta-analysis by Fox, Ericsson, and Best (2011), however, procedures for verbal reporting are only nonreactive (i.e., do not alter thought processes and task performance) when instructions to think Aloud are given in a neutral way, by instructing participants to verbalize their thoughts per se. In contrast, procedures that entail describing or explaining thoughts and actions – as it was the case in the studies by Tombros et al. (2005) and Rieh (2002) – are significantly reactive, altering participants' course of cognitive processing and leading to higher task performance than silent conditions. Hence, the instructions used by Tombros et al. (2005) and Rieh (2002) might have increased participants’ awareness of the necessity of critically evaluating the information retrieved during Web search. Indirect evidence for this assumption comes from the studies by Hargittai et al. (2010) and Savolainen and Kari (2006) that used neutral Thinking-Aloud (TA) instructions and only found few utterances related to information quality or structural aspects of Web pages. A first aim of the present study, thus, was to further prove this assumption by directly comparing participants' verbal utterances when given TA Keywords: Web search; source evaluation; prior domain knowledge; Thinking-Aloud instructions; eye tracking Introduction In recent years, the World Wide Web (WWW) has evolved into a major information resource offering easy access to billions of Web pages on almost any topic. However, as anyone can publish virtually any information on the WWW, the quality of Web pages can vary widely. That is, misleading and low-quality Web pages, for example, in the field of medicine and health care, are as common as those providing neutral, high-quality information. Hence, to avoid the selection and use of doubtful or even false information, it is important that Web users themselves critically evaluate the quality of information they retrieve from the Web. Previous empirical findings about Web users' source evaluation as indicated by verbal reports, however, are inconclusive. Whereas some studies indicate that Web users mainly evaluate search results and Web pages only based on the topical relevance or their ranking in the search engine results page (e.g., Hargittai, Fullerton, Menchen-Trevino, & Thomas, 2010; Savolainen & Kari, 2006), others suggest that Web users are also concerned to a substantial extent
-
Measuring spontaneous and instructed evaluation processes during Web search: Integrating concurrent Thinking-Aloud protocols and eye-tracking data
Learning and Instruction, 2011Co-Authors: Peter Gerjets, Yvonne Kammerer, Benita WernerAbstract:Abstract Web searching for complex information requires to appropriately evaluating diverse sources of information. Information science studies identified different criteria applied by searchers to evaluate Web information. However, the explicit evaluation instructions used in these studies might have resulted in a distortion of spontaneous evaluation processes. Accordingly, the present study compared explicit evaluation instructions and neutral Thinking-Aloud instructions. Data from Thinking-Aloud protocols, eye tracking, and information problem-solving were collected from 30 participants equally distributed to two experimental conditions, that is, the Instructed Evaluation condition and the Spontaneous Evaluation condition. Instructed evaluation, as compared to spontaneous evaluation, resulted in more verbal utterances of quality-related evaluation criteria, in an increased attention focus on user ratings displayed on Web pages, and in better quality of decision making, although participants in the Instructed Evaluation condition were not able to better justify their decision as compared to participants in the Spontaneous Evaluation condition.
Joke Verheij - One of the best experts on this subject based on the ideXlab platform.
-
learning styles self reports versus Thinking Aloud measures
British Journal of Educational Psychology, 2003Co-Authors: Marcel V. J. Veenman, Frans J. Prins, Joke VerheijAbstract:Background: Learning styles are often assessed through students' self-reports on instruments such as Vermunt's Inventory of Learning Styles (ILS). Recent research, however, has questioned the adequacy of questionnaires for the assessment of learning styles. Aims: The objective of this paper is to evaluate methods of learning style assessment as a means for identifying students at risk. Therefore, the ILS as a self-report instrument will be compared with the students' actual study processes, assessed through the Thinking-Aloud method. Sample: In the first study 1,060 students from the Technical University of Delft participated. Thirty-three of them were selected on the ILS for participation in the second study. Method: The ILS was administered to the 1,060 participants and their study results (GPA and credit points) were collected. Next, the selected 33 participants studied a technical text while Thinking Aloud. Knowledge acquisition was measured by means of a post-test. Thinking-Aloud protocols were analysed on frequencies of study activities, thus representing process measures of learning styles. Results: The ILS proved to be a rather weak predictor of study results in Study 1. Results from Study 2 show hardly any correspondence between ILS and study process measures, although principal component structures of both measures were highly similar. Furthermore, study process measures outweighed the ILS in the prediction of study results (post-test, GPA and credit points). Conclusions: Learning style theory was confirmed by results on the study process measures. The assessment of learning styles through self-report instruments such as the ILS, however, should be reconsidered.
-
Learning styles: Self‐reports versus Thinking‐Aloud measures
British Journal of Educational Psychology, 2003Co-Authors: Marcel V. J. Veenman, Frans J. Prins, Joke VerheijAbstract:Background: Learning styles are often assessed through students' self-reports on instruments such as Vermunt's Inventory of Learning Styles (ILS). Recent research, however, has questioned the adequacy of questionnaires for the assessment of learning styles. Aims: The objective of this paper is to evaluate methods of learning style assessment as a means for identifying students at risk. Therefore, the ILS as a self-report instrument will be compared with the students' actual study processes, assessed through the Thinking-Aloud method. Sample: In the first study 1,060 students from the Technical University of Delft participated. Thirty-three of them were selected on the ILS for participation in the second study. Method: The ILS was administered to the 1,060 participants and their study results (GPA and credit points) were collected. Next, the selected 33 participants studied a technical text while Thinking Aloud. Knowledge acquisition was measured by means of a post-test. Thinking-Aloud protocols were analysed on frequencies of study activities, thus representing process measures of learning styles. Results: The ILS proved to be a rather weak predictor of study results in Study 1. Results from Study 2 show hardly any correspondence between ILS and study process measures, although principal component structures of both measures were highly similar. Furthermore, study process measures outweighed the ILS in the prediction of study results (post-test, GPA and credit points). Conclusions: Learning style theory was confirmed by results on the study process measures. The assessment of learning styles through self-report instruments such as the ILS, however, should be reconsidered.
-
Learning styles: self-reports versus Thinking-Aloud measures.
The British journal of educational psychology, 2003Co-Authors: Marcel V. J. Veenman, Frans J. Prins, Joke VerheijAbstract:Learning styles are often assessed through students' self-reports on instruments such as Vermunt's Inventory of Learning Styles (ILS). Recent research, however, has questioned the adequacy of questionnaires for the assessment of learning styles. The objective of this paper is to evaluate methods of learning style assessment as a means for identifying students at risk. Therefore, the ILS as a self-report instrument will be compared with the students' actual study processes, assessed through the Thinking-Aloud method. In the first study 1,060 students from the Technical University of Delft participated. Thirty-three of them were selected on the ILS for participation in the second study. The ILS was administered to the 1,060 participants and their study results (GPA and credit points) were collected. Next, the selected 33 participants studied a technical text while Thinking Aloud. Knowledge acquisition was measured by means of a post-test. Thinking-Aloud protocols were analysed on frequencies of study activities, thus representing process measures of learning styles. The ILS proved to be a rather weak predictor of study results in Study 1. Results from Study 2 show hardly any correspondence between ILS and study process measures, although principal component structures of both measures were highly similar. Furthermore, study process measures outweighed the ILS in the prediction of study results (post-test, GPA and credit points). Learning style theory was confirmed by results on the study process measures. The assessment of learning styles through self-report instruments such as the ILS, however, should be reconsidered.
Bonnie Kaplan - One of the best experts on this subject based on the ideXlab platform.
-
deriving design recommendations through discount usability engineering ethnographic observation and Thinking Aloud protocol in usability testing for computer based teaching cases
American Medical Informatics Association Annual Symposium, 2003Co-Authors: Bonnie KaplanAbstract:Usability engineering often involves experimental evaluation designs carried out in special usability laboratories. Though becoming more popular in medical informatics, the approach is little used. Possibly this is because of the expense involved in set-up, data collection, and data analysis. Excellent results may be obtained, however, by employing discount usability engineering and qualitative methods of data collection and analysis to generate recommendations to improve usability. A prototype computer-based teaching case was evaluated in a discount usability engineering approach by combining modified ethnographic observation with simplified Thinking Aloud protocol. Data was collected and analyzed using standard approaches for qualitative data. This approach led to helpful recommendations for designing teaching cases. The project team believes this economical approach to usability testing may be helpful to others engaged in interface design.
-
AMIA - Deriving Design Recommendations Through Discount Usability Engineering: Ethnographic Observation and Thinking-Aloud Protocol in Usability Testing for Computer-Based Teaching Cases
AMIA ... Annual Symposium proceedings. AMIA Symposium, 2003Co-Authors: Bonnie KaplanAbstract:Usability engineering often involves experimental evaluation designs carried out in special usability laboratories. Though becoming more popular in medical informatics, the approach is little used. Possibly this is because of the expense involved in set-up, data collection, and data analysis. Excellent results may be obtained, however, by employing discount usability engineering and qualitative methods of data collection and analysis to generate recommendations to improve usability. A prototype computer-based teaching case was evaluated in a discount usability engineering approach by combining modified ethnographic observation with simplified Thinking Aloud protocol. Data was collected and analyzed using standard approaches for qualitative data. This approach led to helpful recommendations for designing teaching cases. The project team believes this economical approach to usability testing may be helpful to others engaged in interface design.
Morten Hertzum - One of the best experts on this subject based on the ideXlab platform.
-
What Do Thinking-Aloud Participants Say? A Comparison of Moderated and Unmoderated Usability Sessions
International Journal of Human-Computer Interaction, 2015Co-Authors: Morten Hertzum, Pia Borlund, Kristina Bonde KristoffersenAbstract:The value of Thinking Aloud in usability tests depends on the content of the users’ verbalizations. We investigated moderated and unmoderated users’ verbalizations during relaxed Thinking Aloud (i.e., verbalization at Levels 1–3). Verbalizations of user experience were frequent and mostly relevant to the identification of usability issues. Explanations and redesign proposals were also mostly relevant, but infrequent. The relevance of verbalizations of user experience, explanations, and redesign proposals showed the value of relaxed Thinking Aloud but did not clarify the trade-off between rich verbalizations and test reactivity. Action descriptions and system observations—two verbalization categories consistent with both relaxed and classic Thinking Aloud—were frequent but mainly of low relevance. Across all verbalizations, the positive or negative verbalizations were more often relevant than those without valence. Finally, moderated and unmoderated users made largely similar verbalizations, the main diffe...
-
Thinking Aloud Influences Perceived Time
Human factors, 2014Co-Authors: Morten Hertzum, Kristin Due HolmegaardAbstract:Objective: We investigate whether Thinking Aloud influences perceived time. Background: Thinking Aloud is widely used in usability evaluation, yet it is debated whether Thinking Aloud influences thought and behavior. If Thinking Aloud is restricted to the verbalization of information to which a person is already attending, there is evidence that Thinking Aloud does not influence thought and behavior. Method: In an experiment, 16 Thinking-Aloud participants and 16 control participants solved a code- breaking task 24 times each. Participants estimated task duration. The 24 trials involved two levels of time constraint (timed, untimed) and resulted in two levels of success (solved, unsolved). Results: The ratio of perceived time to clock time was lower for Thinking-Aloud than control participants. Participants overestimated time by an average of 47% (Thinking Aloud) and 94% (control). The effect of Thinking Aloud on time perception also held separately for timed, untimed, solved, and unsolved trials. Conclusion: Thinking Aloud (verbalization at Levels 1 and 2) influences perceived time. Possible explanations of this effect include that Thinking Aloud may require attention, cause a processing shift that overshadows the perception of time, or increase mental workload. Application: For usability evaluation, this study implies that time estimates made while Thinking Aloud cannot be compared with time estimates made while not Thinking Aloud, that ratings of systems experienced while Thinking Aloud may be inaccurate (because the experience of time influences other experiences), and that it may therefore be considered to replace concurrent Thinking Aloud with retrospective Thinking Aloud when evaluations involve time estimation.
-
Thinking Aloud in the Presence of Interruptions and Time Constraints
International Journal of Human-Computer Interaction, 2013Co-Authors: Morten Hertzum, Kristin Due HolmegaardAbstract:Thinking Aloud is widely used for usability evaluation, and its reactivity is therefore important to the quality of evaluation results. This study investigates whether Thinking Aloud (i.e., verbalization at Levels 1 and 2) affects the behavior of users who perform tasks that involve interruptions and time constraints, two frequent elements of real-world activities. The study finds that the presence of auditory, visual, audiovisual, or no interruptions interacts with Thinking Aloud for task solution rate, task completion time, and participants' fixation rate. Thinking-Aloud participants also spend longer responding to interruptions than control participants. Conversely, the absence or presence of time constraints does not interact with Thinking Aloud, suggesting that time pressure is less likely to make Thinking Aloud reactive than previously assumed. These results inform practitioners faced with the decision to either restrict verbalizations in usability evaluation to Thinking Aloud to avoid reactivity or...
-
scrutinising usability evaluation does Thinking Aloud affect behaviour and mental workload
Behaviour & Information Technology, 2009Co-Authors: Morten Hertzum, Kristin Due Hansen, Hans H K AndersenAbstract:Thinking Aloud is widely used for usability evaluation. The validity of the method is, however, debatable because it is generally used in a relaxed way that conflicts with the prescriptions of the classic model for obtaining valid verbalisations of thought processes. This study investigates whether participants that think Aloud in the classic or relaxed way behave differently compared to performing in silence. Results indicate that whereas classic Thinking Aloud has little or no effect on behaviour apart from prolonging tasks, relaxed Thinking Aloud affects behaviour in multiple ways. During relaxed Thinking Aloud participants took longer to solve tasks, spent a larger part of tasks on general distributed visual behaviour, issued more commands to navigate both within and between the pages of the websites used in the experiment, and experienced higher mental workload. Implications for usability evaluation are discussed.
-
ICIS - Cultural Cognition in the Thinking-Aloud Method for Usability Evaluation
2008Co-Authors: Torkil Clemmensen, Qing Xin Shi, Morten Hertzum, Kasper Hornbaek, Pradeep YammiyavarAbstract:We discuss the impact of cultural differences on usability evaluations that are based on the Thinking-Aloud method (TA). The term ‘cultural differences’ helps distinguish differences in the perception and Thinking of Westerners (people from Western Europe and US citizens with European origins) and Easterners (people from China and the countries heavily influenced by its culture). We illustrate the impact of cultural cognition on four central elements of TA: (1) instructions and tasks, (2) the user’s verbalizations, (3) the evaluator’s reading of the user, and (4) the overall relationship between user and evaluator. In conclusion, we point to the importance of matching the task presentation to users’ cultural background, the different effects of Thinking Aloud on task performance between Easterners and Westerners, the differences in nonverbal behaviour that affect usability problem detection, and, finally, the complexity of the overall relationship between user and evaluator when they have different cultural backgrounds.
Yvonne Kammerer - One of the best experts on this subject based on the ideXlab platform.
-
CogSci - The role of Thinking-Aloud instructions and prior domain knowledge in information processing and source evaluation during Web search
Cognitive Science, 2013Co-Authors: Yvonne Kammerer, Peter GerjetsAbstract:The role of Thinking-Aloud instructions and prior domain knowledge in information processing and source evaluation during Web search Yvonne Kammerer (y.kammerer@iwm-kmrc.de) Knowledge Media Research Center, Schleichstr. 6, 72076 Tuebingen, Germany Peter Gerjets (p.gerjets@iwm-kmrc.de) Knowledge Media Research Center, Schleichstr. 6, 72076 Tuebingen, Germany Abstract about information quality (i.e., the accuracy, authority, objectivity, or currency of information; e.g. Rieh, 2002; Tombros, Ruthven, & Jose, 2005). According to Tombros et al. (2005) also the structure of Web pages (i.e., the clarity of the Web page or the organization of the information therein) is evaluated extensively. The aim of the present paper was to identify potential reasons for the divergent findings. Specifically, we examined the impact of the instructions used in the studies as well as of participants' prior domain knowledge on information processing and source evaluation during Web search, using a rich multi-method approach including eye tracking methodologies, log file data, and verbal protocols. This paper examines the impact of Thinking-Aloud (TA) instructions as well as of individuals' prior domain knowledge on information processing and source evaluation during Web search on a health-related topic. With regard to TA instructions, prompted instructions that entailed evaluation prompts (as used in some previous Web search studies) were compared to neutral instructions (in line with the standards defined by Ericsson & Simon, 1993) and to a silent condition. To measure participants' (N = 44) information processing and source evaluation we used a rich multi-method approach including eye-tracking methodologies, log file data, and verbal protocols. Results indicate that prompted TA instructions as compared to neutral instructions significantly increased participants' verbal reflections on information quality and on structural aspects of Web pages, given that participants possessed at least a moderate level of prior domain knowledge. In addition, prompted instructions resulted in less linear viewing sequences on the search engine results pages than the silent condition. Finally, the higher participants' prior domain knowledge the more intensely they scrutinized the search results presented by the search engine and the smaller were their average pupil sizes, which indicated lower cognitive load. The significance of the results is considered in light of methodological as well as educational implications. The Role of Prompted Thinking-Aloud Instructions One reason for the divergent findings might be a methodological one, namely that in the studies by Rieh (2002) and Tombros et al. (2005) participants were instructed beforehand to explain what criteria they used to evaluate Web information (Tombros et al., 2005) or to select good or credible information during Web search and to explain their evaluation processes in the form of postsearch interviews including specific evaluation-related questions (Rieh, 2002). According to the seminal work by Ericsson and Simon (1993) and the meta-analysis by Fox, Ericsson, and Best (2011), however, procedures for verbal reporting are only nonreactive (i.e., do not alter thought processes and task performance) when instructions to think Aloud are given in a neutral way, by instructing participants to verbalize their thoughts per se. In contrast, procedures that entail describing or explaining thoughts and actions – as it was the case in the studies by Tombros et al. (2005) and Rieh (2002) – are significantly reactive, altering participants' course of cognitive processing and leading to higher task performance than silent conditions. Hence, the instructions used by Tombros et al. (2005) and Rieh (2002) might have increased participants’ awareness of the necessity of critically evaluating the information retrieved during Web search. Indirect evidence for this assumption comes from the studies by Hargittai et al. (2010) and Savolainen and Kari (2006) that used neutral Thinking-Aloud (TA) instructions and only found few utterances related to information quality or structural aspects of Web pages. A first aim of the present study, thus, was to further prove this assumption by directly comparing participants' verbal utterances when given TA Keywords: Web search; source evaluation; prior domain knowledge; Thinking-Aloud instructions; eye tracking Introduction In recent years, the World Wide Web (WWW) has evolved into a major information resource offering easy access to billions of Web pages on almost any topic. However, as anyone can publish virtually any information on the WWW, the quality of Web pages can vary widely. That is, misleading and low-quality Web pages, for example, in the field of medicine and health care, are as common as those providing neutral, high-quality information. Hence, to avoid the selection and use of doubtful or even false information, it is important that Web users themselves critically evaluate the quality of information they retrieve from the Web. Previous empirical findings about Web users' source evaluation as indicated by verbal reports, however, are inconclusive. Whereas some studies indicate that Web users mainly evaluate search results and Web pages only based on the topical relevance or their ranking in the search engine results page (e.g., Hargittai, Fullerton, Menchen-Trevino, & Thomas, 2010; Savolainen & Kari, 2006), others suggest that Web users are also concerned to a substantial extent
-
the role of Thinking Aloud instructions and prior domain knowledge in information processing and source evaluation during web search
Cognitive Science, 2013Co-Authors: Yvonne Kammerer, Peter GerjetsAbstract:The role of Thinking-Aloud instructions and prior domain knowledge in information processing and source evaluation during Web search Yvonne Kammerer (y.kammerer@iwm-kmrc.de) Knowledge Media Research Center, Schleichstr. 6, 72076 Tuebingen, Germany Peter Gerjets (p.gerjets@iwm-kmrc.de) Knowledge Media Research Center, Schleichstr. 6, 72076 Tuebingen, Germany Abstract about information quality (i.e., the accuracy, authority, objectivity, or currency of information; e.g. Rieh, 2002; Tombros, Ruthven, & Jose, 2005). According to Tombros et al. (2005) also the structure of Web pages (i.e., the clarity of the Web page or the organization of the information therein) is evaluated extensively. The aim of the present paper was to identify potential reasons for the divergent findings. Specifically, we examined the impact of the instructions used in the studies as well as of participants' prior domain knowledge on information processing and source evaluation during Web search, using a rich multi-method approach including eye tracking methodologies, log file data, and verbal protocols. This paper examines the impact of Thinking-Aloud (TA) instructions as well as of individuals' prior domain knowledge on information processing and source evaluation during Web search on a health-related topic. With regard to TA instructions, prompted instructions that entailed evaluation prompts (as used in some previous Web search studies) were compared to neutral instructions (in line with the standards defined by Ericsson & Simon, 1993) and to a silent condition. To measure participants' (N = 44) information processing and source evaluation we used a rich multi-method approach including eye-tracking methodologies, log file data, and verbal protocols. Results indicate that prompted TA instructions as compared to neutral instructions significantly increased participants' verbal reflections on information quality and on structural aspects of Web pages, given that participants possessed at least a moderate level of prior domain knowledge. In addition, prompted instructions resulted in less linear viewing sequences on the search engine results pages than the silent condition. Finally, the higher participants' prior domain knowledge the more intensely they scrutinized the search results presented by the search engine and the smaller were their average pupil sizes, which indicated lower cognitive load. The significance of the results is considered in light of methodological as well as educational implications. The Role of Prompted Thinking-Aloud Instructions One reason for the divergent findings might be a methodological one, namely that in the studies by Rieh (2002) and Tombros et al. (2005) participants were instructed beforehand to explain what criteria they used to evaluate Web information (Tombros et al., 2005) or to select good or credible information during Web search and to explain their evaluation processes in the form of postsearch interviews including specific evaluation-related questions (Rieh, 2002). According to the seminal work by Ericsson and Simon (1993) and the meta-analysis by Fox, Ericsson, and Best (2011), however, procedures for verbal reporting are only nonreactive (i.e., do not alter thought processes and task performance) when instructions to think Aloud are given in a neutral way, by instructing participants to verbalize their thoughts per se. In contrast, procedures that entail describing or explaining thoughts and actions – as it was the case in the studies by Tombros et al. (2005) and Rieh (2002) – are significantly reactive, altering participants' course of cognitive processing and leading to higher task performance than silent conditions. Hence, the instructions used by Tombros et al. (2005) and Rieh (2002) might have increased participants’ awareness of the necessity of critically evaluating the information retrieved during Web search. Indirect evidence for this assumption comes from the studies by Hargittai et al. (2010) and Savolainen and Kari (2006) that used neutral Thinking-Aloud (TA) instructions and only found few utterances related to information quality or structural aspects of Web pages. A first aim of the present study, thus, was to further prove this assumption by directly comparing participants' verbal utterances when given TA Keywords: Web search; source evaluation; prior domain knowledge; Thinking-Aloud instructions; eye tracking Introduction In recent years, the World Wide Web (WWW) has evolved into a major information resource offering easy access to billions of Web pages on almost any topic. However, as anyone can publish virtually any information on the WWW, the quality of Web pages can vary widely. That is, misleading and low-quality Web pages, for example, in the field of medicine and health care, are as common as those providing neutral, high-quality information. Hence, to avoid the selection and use of doubtful or even false information, it is important that Web users themselves critically evaluate the quality of information they retrieve from the Web. Previous empirical findings about Web users' source evaluation as indicated by verbal reports, however, are inconclusive. Whereas some studies indicate that Web users mainly evaluate search results and Web pages only based on the topical relevance or their ranking in the search engine results page (e.g., Hargittai, Fullerton, Menchen-Trevino, & Thomas, 2010; Savolainen & Kari, 2006), others suggest that Web users are also concerned to a substantial extent
-
Measuring spontaneous and instructed evaluation processes during Web search: Integrating concurrent Thinking-Aloud protocols and eye-tracking data
Learning and Instruction, 2011Co-Authors: Peter Gerjets, Yvonne Kammerer, Benita WernerAbstract:Abstract Web searching for complex information requires to appropriately evaluating diverse sources of information. Information science studies identified different criteria applied by searchers to evaluate Web information. However, the explicit evaluation instructions used in these studies might have resulted in a distortion of spontaneous evaluation processes. Accordingly, the present study compared explicit evaluation instructions and neutral Thinking-Aloud instructions. Data from Thinking-Aloud protocols, eye tracking, and information problem-solving were collected from 30 participants equally distributed to two experimental conditions, that is, the Instructed Evaluation condition and the Spontaneous Evaluation condition. Instructed evaluation, as compared to spontaneous evaluation, resulted in more verbal utterances of quality-related evaluation criteria, in an increased attention focus on user ratings displayed on Web pages, and in better quality of decision making, although participants in the Instructed Evaluation condition were not able to better justify their decision as compared to participants in the Spontaneous Evaluation condition.
