The Experts below are selected from a list of 52497 Experts worldwide ranked by ideXlab platform

Vacek Rachel - One of the best experts on this subject based on the ideXlab platform.

  • Search, Report, Wherever You Are: A Novel Approach to Assessing User Satisfaction with a Library Discovery Interface
    2020
    Co-Authors: Vacek Rachel, Smith Craig, Varnum, Kenneth J.
    Abstract:

    This is a virtual presentation given at the Library Assessment Conference on January 21, 2021. The presentation is in the Measurement & Methods track.PURPOSE AND GOALS In the summer of 2018, the University of Michigan Library launched a new discovery interface, Library Search (https://search.lib.umich.edu), for discovering the Library’s resources, collections, spaces, and expertise. Following our Assessment plan for Library Search, we have iteratively measured the success of Search since its launch across a variety of measures using a mixed-methods approach. We have tested for accessibility, monitored system performance, and conducted considerable usability testing as part of the design and development process. In general, our Assessments have indicated that Search works well according to many of our metrics, including accessibility, usability, design, and the general pattern of how searching for items, narrowing results sets, and accessing materials. There have been, however, a number of concerns about the catalog search in particular, and a general anecdotal sense that this important part of the interface is not quite meeting users’ needs. Therefore, we created a tool that we could use to first get a baseline measure of overall user satisfaction, and second, use again after we have made changes to Library Search to understand the degree to which our changes improved satisfaction. Our paper will report on our launch of our rather unique data collection tool and its use with multiple groups of stakeholders on campus. We will also detail how we plan to use the tool to gauge user satisfaction over time, and to regularly gather actionable data on the strengths and shortcomings of our catalog interface. DESIGN, METHODOLOGY, OR APPROACH An initial round of data collection focused on the search experiences of Library employees whose work involves using Library Search to assist members of our campus community. Focusing on this group allowed us to survey people who had clear expectations of how catalog searching should function for Library staff and users, and also to ensure that our data collection tool worked well before we used it to collect data from large numbers of faculty members and students on campus. The first full round of data collection, with faculty and students, is taking place in the winter of 2020; the results will be included in our paper and presentation. The invitation to participate in the initial round of data collection was sent in December 2019 to 96 Library employees. As an incentive, participants were given a chance to enter a drawing for one $50 gift card. Participants took part in the study online, via a survey on the Qualtrics platform. Forty people provided enough data to be included in some analyses (a 41% response rate); of those, 36 completed the whole survey. The final sample had decent variability with regard to respondents’ Library division and the number of years they had worked in the Library. A unique aspect of our data collection approach was to ask participants to conduct searches, to report on those searches, and to share the URLs associated with their search results. This allowed data from the survey to be interpreted while seeing exactly what participants were seeing as they did their searching. Thus, the first three sections of the survey asked participants to keep the survey tab open in their browsers while conducting specific types of catalog searches in a separate tab. The types of searches — known item, known set, and exploratory — were derived from another recent investigation of our Library search interface. Participants used the Qualtrics survey to answer questions about those three search experiences. Specifically, participants were asked to report on their satisfaction with the relevance of results, the speed with which results were returned, and the adequacy of various pieces of information contained in item records. When participants encountered unexpected results in their searches, they were given an opportunity to share more about what they expected to see, in relation to what they saw. A final section of the survey asked people to provide more global ratings and comments related to recent uses of Library Search (not limited to catalog searching; this could also include focused searches for articles, databases, etc.). For those that remembered using Search a year prior, a small set of questions also asked people to compare their current satisfaction with Search to what they remember feeling a year ago. These final questions, about recent experiences and comparisons to a year ago, were answered by most participants. FINDINGS We asked questions about three broad areas of search interactions: known item searches, known set searches, and exploratory searches. Known item searches are for specific, individual items. Known set searches are for collections of items (plays by Shakespeare, sonatas by Mozart, jazz CDs, etc.), from which the searcher would be more or less satisfied with any specific item. Exploratory searches are subject or topic-related. For known item searching, respondents were asked two questions about what they saw in the results: did the item appear in the results as expected, and was the position of the item in the result set satisfactory. Seventy-five percent saw the item in the results as expected, and the majority (92%) were either very or moderately satisfied with the position. We asked several additional questions for known item searches (we did not ask these questions for the other search categories, as we felt the responses would not be substantially different). When asked about the speed of search results, most (95%) expressed some level of satisfaction. In terms of ease of determining availability of print or online access to the items found, most were moderately or very satisfied (85%) but a notable minority were dissatisfied. And most people were moderately or very satisfied (86%) with identifying where physical items were located, with a notable minority expressing dissatisfaction. For known item searches -- and for the other two search types -- participants who saw unexpected search results were asked to share what they expected to see, and what they did see. Comments touched on concerns such as the relevance of results and the way that holdings were displayed. These findings serve as guides for the continued fine-tuning of Library Search. In the paper we will present examples of how such comments were paired with recreated searches in order to guide the work of our developers. For known set searches, just over half (58%) of the 36 participants who did this search saw what they expected; respondents were satisfied with the ranking of the results, about 50% each very or mostly satisfied. For exploratory searches, just over half (56%) saw what they expected in the results. Most (80%) were moderately or very satisfied, and a sizable minority (20%) reported some level of dissatisfaction. As noted, where participants saw unexpected results, they provided comments that shed additional light on their closed-ended survey responses. In the final section of the survey, participants were asked about their recent experiences with Search (not limited to Catalog Search), and their views on whether Search has improved or not compared to a year ago (for those with memories of Search at that time). Thirty-three participants had used Search within the previous two weeks. Of these, roughly three-quarters were moderately or very satisfied, with the rest expressing dissatisfaction. When asked about their satisfaction with the recent relevance of Search results, few were very satisfied (12%); most were moderately satisfied, and 30% expressed dissatisfaction. The same results were obtained when people were asked to rate their overall level of satisfaction with their recent experiences with Search. When asked to compare their current satisfaction with the speed of Search with what they remember from a year prior, most (81%) were somewhat or much more satisfied currently. When asked to compare their current satisfaction with the relevance of Search results with what they remember from a year prior, 72% were somewhat or much more satisfied currently. Finally, when asked to compare their current overall satisfaction with Search compared to a year ago, 82% were somewhat or much more satisfied. PRACTICAL IMPLICATIONS OR VALUE This study provides an example of how libraries can use an online data collection tool to reach key stakeholders when evaluating satisfaction with a web-based Library interaction. The general categorization of “known item”, “known set”, and “exploratory” searches, itself based on user research into kinds of searching, could easily be extended to other kinds of Library interactions in which a user is seeking something. The general method of the survey allows disintermediated user research to take place, with the efficiency of gaining detailed user feedback about specific interactions without the investment of a commensurate amount of staff time. Another key advantage of our methodology is that it facilitates the repeated evaluation of the search interface over time.The inclusion of both Library staff and campus users enables us to identify high-priority issues via staff insights and also to understand how users with a wide range of search expertise experience a core Library discovery interface.http://deepblue.lib.umich.edu/bitstream/2027.42/163532/3/Results of 2019 Survey on Catalog Search Satisfaction.pdfhttp://deepblue.lib.umich.edu/bitstream/2027.42/163532/2/Catalog Search Satisfaction Survey.pdfhttp://deepblue.lib.umich.edu/bitstream/2027.42/163532/1/Search, Report, Wherever You Are.pdfDescription of Search, Report, Wherever You Are.pdf : Presentation slide deckDescription of Catalog Search Satisfaction Survey.pdf : Survey from QualtricsDescription of Results of 2019 Survey on Catalog Search Satisfaction.pdf : Results of the SurveySEL

  • Search, Report, Wherever You Are: A Novel Approach to Assessing User Satisfaction with a Library Discovery Interface
    2020
    Co-Authors: Vacek Rachel, Smith Craig, Varnum, Kenneth J.
    Abstract:

    This is a virtual presentation given at the Library Assessment Conference on January 21, 2021. The presentation is in the Measurement & Methods track.PURPOSE AND GOALS In the summer of 2018, the University of Michigan Library launched a new discovery interface, Library Search (https://search.lib.umich.edu), for discovering the Library’s resources, collections, spaces, and expertise. Following our Assessment plan for Library Search, we have iteratively measured the success of Search since its launch across a variety of measures using a mixed-methods approach. We have tested for accessibility, monitored system performance, and conducted considerable usability testing as part of the design and development process. In general, our Assessments have indicated that Search works well according to many of our metrics, including accessibility, usability, design, and the general pattern of how searching for items, narrowing results sets, and accessing materials. There have been, however, a number of concerns about the catalog search in particular, and a general anecdotal sense that this important part of the interface is not quite meeting users’ needs. Therefore, we created a tool that we could use to first get a baseline measure of overall user satisfaction, and second, use again after we have made changes to Library Search to understand the degree to which our changes improved satisfaction. Our paper will report on our launch of our rather unique data collection tool and its use with multiple groups of stakeholders on campus. We will also detail how we plan to use the tool to gauge user satisfaction over time, and to regularly gather actionable data on the strengths and shortcomings of our catalog interface. DESIGN, METHODOLOGY, OR APPROACH An initial round of data collection focused on the search experiences of Library employees whose work involves using Library Search to assist members of our campus community. Focusing on this group allowed us to survey people who had clear expectations of how catalog searching should function for Library staff and users, and also to ensure that our data collection tool worked well before we used it to collect data from large numbers of faculty members and students on campus. The first full round of data collection, with faculty and students, is taking place in the winter of 2020; the results will be included in our paper and presentation. The invitation to participate in the initial round of data collection was sent in December 2019 to 96 Library employees. As an incentive, participants were given a chance to enter a drawing for one $50 gift card. Participants took part in the study online, via a survey on the Qualtrics platform. Forty people provided enough data to be included in some analyses (a 41% response rate); of those, 36 completed the whole survey. The final sample had decent variability with regard to respondents’ Library division and the number of years they had worked in the Library. A unique aspect of our data collection approach was to ask participants to conduct searches, to report on those searches, and to share the URLs associated with their search results. This allowed data from the survey to be interpreted while seeing exactly what participants were seeing as they did their searching. Thus, the first three sections of the survey asked participants to keep the survey tab open in their browsers while conducting specific types of catalog searches in a separate tab. The types of searches — known item, known set, and exploratory — were derived from another recent investigation of our Library search interface. Participants used the Qualtrics survey to answer questions about those three search experiences. Specifically, participants were asked to report on their satisfaction with the relevance of results, the speed with which results were returned, and the adequacy of various pieces of information contained in item records. When participants encountered unexpected results in their searches, they were given an opportunity to share more about what they expected to see, in relation to what they saw. A final section of the survey asked people to provide more global ratings and comments related to recent uses of Library Search (not limited to catalog searching; this could also include focused searches for articles, databases, etc.). For those that remembered using Search a year prior, a small set of questions also asked people to compare their current satisfaction with Search to what they remember feeling a year ago. These final questions, about recent experiences and comparisons to a year ago, were answered by most participants. FINDINGS We asked questions about three broad areas of search interactions: known item searches, known set searches, and exploratory searches. Known item searches are for specific, individual items. Known set searches are for collections of items (plays by Shakespeare, sonatas by Mozart, jazz CDs, etc.), from which the searcher would be more or less satisfied with any specific item. Exploratory searches are subject or topic-related. For known item searching, respondents were asked two questions about what they saw in the results: did the item appear in the results as expected, and was the position of the item in the result set satisfactory. Seventy-five percent saw the item in the results as expected, and the majority (92%) were either very or moderately satisfied with the position. We asked several additional questions for known item searches (we did not ask these questions for the other search categories, as we felt the responses would not be substantially different). When asked about the speed of search results, most (95%) expressed some level of satisfaction. In terms of ease of determining availability of print or online access to the items found, most were moderately or very satisfied (85%) but a notable minority were dissatisfied. And most people were moderately or very satisfied (86%) with identifying where physical items were located, with a notable minority expressing dissatisfaction. For known item searches -- and for the other two search types -- participants who saw unexpected search results were asked to share what they expected to see, and what they did see. Comments touched on concerns such as the relevance of results and the way that holdings were displayed. These findings serve as guides for the continued fine-tuning of Library Search. In the paper we will present examples of how such comments were paired with recreated searches in order to guide the work of our developers. For known set searches, just over half (58%) of the 36 participants who did this search saw what they expected; respondents were satisfied with the ranking of the results, about 50% each very or mostly satisfied. For exploratory searches, just over half (56%) saw what they expected in the results. Most (80%) were moderately or very satisfied, and a sizable minority (20%) reported some level of dissatisfaction. As noted, where participants saw unexpected results, they provided comments that shed additional light on their closed-ended survey responses. In the final section of the survey, participants were asked about their recent experiences with Search (not limited to Catalog Search), and their views on whether Search has improved or not compared to a year ago (for those with memories of Search at that time). Thirty-three participants had used Search within the previous two weeks. Of these, roughly three-quarters were moderately or very satisfied, with the rest expressing dissatisfaction. When asked about their satisfaction with the recent relevance of Search results, few were very satisfied (12%); most were moderately satisfied, and 30% expressed dissatisfaction. The same results were obtained when people were asked to rate their overall level of satisfaction with their recent experiences with Search. When asked to compare their current satisfaction with the speed of Search with what they remember from a year prior, most (81%) were somewhat or much more satisfied currently. When asked to compare their current satisfaction with the relevance of Search results with what they remember from a year prior, 72% were somewhat or much more satisfied currently. Finally, when asked to compare their current overall satisfaction with Search compared to a year ago, 82% were somewhat or much more satisfied. PRACTICAL IMPLICATIONS OR VALUE This study provides an example of how libraries can use an online data collection tool to reach key stakeholders when evaluating satisfaction with a web-based Library interaction. The general categorization of “known item”, “known set”, and “exploratory” searches, itself based on user research into kinds of searching, could easily be extended to other kinds of Library interactions in which a user is seeking something. The general method of the survey allows disintermediated user research to take place, with the efficiency of gaining detailed user feedback about specific interactions without the investment of a commensurate amount of staff time. Another key advantage of our methodology is that it facilitates the repeated evaluation of the search interface over time.The inclusion of both Library staff and campus users enables us to identify high-priority issues via staff insights and also to understand how users with a wide range of search expertise experience a core Library discovery interface.http://deepblue.lib.umich.edu/bitstream/2027.42/163532/1/Search, Report, Wherever You Are.pdfhttp://deepblue.lib.umich.edu/bitstream/2027.42/163532/2/Catalog Search Satisfaction Survey.pdfhttp://deepblue.lib.umich.edu/bitstream/2027.42/163532/3/Results of 2019 Survey on Catalog Search Satisfaction.pdfSEL

  • Diffusing Organizational Change through Service Design and Iterative Assessment
    Library Assessment Conference: 2018 Conference Proceedings, 2019
    Co-Authors: Vacek Rachel, Puckett Rodgers Emily, Sitar Meghan
    Abstract:

    This is the conference proceeding for the Library Assessment Conference that took place December 5-7, 2018, in Houston, TX. Proceedings are published as PDFs on this page: https://www.LibraryAssessment.org/past-conferences/2018-Library-Assessment-conference/2018-proceedings/At the University of Michigan Library, dozens of librarians and staff are engaged in a series of activities to reimagine the ways our organization designs and implements our services. Between 2016 and 2017, we collaborated with brightspot strategy to develop a service philosophy, framework, and principles to help us begin the process to transform our physical and digital spaces to better represent our expertise, collections, and tools, and to meet the evolving needs of our academic community today and tomorrow. As we look to transform our spaces to serve the needs of our research community, we are taking care to ensure that whatever form our buildings and web presence take, will follow the function and intent of our services. Our efforts in this work are collaborative and distributed in nature, diffusing the shift in design and evaluation across the organization. With it, we aim to facilitate organizational change that puts our users at the center of service design and delivery. It also fundamentally recognizes that our departments play a role in supporting the academic needs of our faculty, students, and staff at the University of Michigan. This work is structured by established approaches in design thinking and user-centered design. Multiple teams of librarians and staff are applying this approach to redesign services. Topics include consultation, digital scholarship, staff innovation, citation management, and developing a persona-based toolkit that any staff across our organization may use in efforts to design new or make improvements to existing services. While each team is using the same overall approach to its service design work, the application and outcomes are unique to each domain. Within four design cycles, each team engages in a retrospective to review the process, the impact of the work, and consider its potential effects on our organizational structures. Additionally, the service design efforts support our organization’s adoption of an Assessment-driven mindset through embedding evaluation into our processes. Once the service design phase is complete, each team will generate a series of pilots or prototypes to test aspects of their designs in the context of our organization. In the Summer and Fall of 2018, teams will implement those pilots and prototypes, testing their ability to scale effectively or meet our programmatic and mission-based goals yielding a series of small-scale, but impactful activities or processes that will further diffuse the design-thinking approach throughout our organization. Each of these sets of activities will be assessed before moving onto future stages of the work. We will also evaluate how they help us enact our service philosophy, framework, and principles in the practice of our everyday work. Ultimately this work will yield a culture shift within our organization enabling us to embrace a user-centered, service-based approach to how we develop services, and how we expect to collaborate and connect with colleagues within our Library and academic community. It will also enable us to embed Assessment practices into various facets of our work, from the beginning stages of design through its testing and into implementation at scale.Peer Reviewedhttps://deepblue.lib.umich.edu/bitstream/2027.42/151766/1/58-Vacek-PuckettRodgers-Sitar-DiffusingOrganizationalChange.pdfDescription of 58-Vacek-PuckettRodgers-Sitar-DiffusingOrganizationalChange.pdf : PDF of the Conference Proceedin

  • Diffusing Organizational Change through Service Design and Iterative Assessment
    2018
    Co-Authors: Vacek Rachel, Sitar Meghan, Puckett Rodgers Emily
    Abstract:

    This presentation was presented by Emily Puckett Rodgers and Rachel Vacek on December 6, 2018 at the Library Assessment Conference in Houston, TX.At the University of Michigan Library, dozens of librarians and staff are engaged in a series of activities to reimagine the ways our organization designs and implements our services. Between 2016 and 2017, we collaborated with brightspot strategy to develop a service philosophy, framework, and principles to help us begin the process to transform our physical and digital spaces to better represent our expertise, collections, and tools, and to meet the evolving needs of our academic community today and tomorrow. As we look to transform our spaces to serve the needs of our research community, we are taking care to ensure that whatever form our buildings and web presence take, will follow the function and intent of our services. Our efforts in this work are collaborative and distributed in nature, diffusing the shift in design and evaluation across the organization. With it, we aim to facilitate organizational change that puts our users at the center of service design and delivery. It also fundamentally recognizes that our departments play a role in supporting the academic needs of our faculty, students, and staff at the University of Michigan. This work is structured by established approaches in design thinking and user-centered design. Multiple teams of librarians and staff are applying this approach to redesign services. Topics include consultation, digital scholarship, staff innovation, citation management, and developing a persona-based toolkit that any staff across our organization may use in efforts to design new or make improvements to existing services. While each team is using the same overall approach to its service design work, the application and outcomes are unique to each domain. Within four design cycles, each team engages in a retrospective to review the process, the impact of the work, and consider its potential effects on our organizational structures. Additionally, the service design efforts support our organization’s adoption of an Assessment-driven mindset through embedding evaluation into our processes. Once the service design phase is complete, each team will generate a series of pilots or prototypes to test aspects of their designs in the context of our organization. In the Summer and Fall of 2018, teams will implement those pilots and prototypes, testing their ability to scale effectively or meet our programmatic and mission-based goals yielding a series of small-scale, but impactful activities or processes that will further diffuse the design-thinking approach throughout our organization. Each of these sets of activities will be assessed before moving onto future stages of the work. We will also evaluate how they help us enact our service philosophy, framework, and principles in the practice of our everyday work. Ultimately this work will yield a culture shift within our organization enabling us to embrace a user-centered, service-based approach to how we develop services, and how we expect to collaborate and connect with colleagues within our Library and academic community. It will also enable us to embed Assessment practices into various facets of our work, from the beginning stages of design through its testing and into implementation at scale.Peer Reviewedhttps://deepblue.lib.umich.edu/bitstream/2027.42/146754/1/Diffusing Organizational Change through Service Design and Iterative Assessment.pdfDescription of Diffusing Organizational Change through Service Design and Iterative Assessment.pdf : PDF of Presentatio

Larry Nash White - One of the best experts on this subject based on the ideXlab platform.

  • Assessment Planning for Distance Education Library Services: Strategic Roadmaps for Determining and Reporting Organizational Performance and Value
    Journal of Library Administration, 2010
    Co-Authors: Larry Nash White
    Abstract:

    ABSTRACT Through a review of the literature, the author sought to identify what Assessment planning practices exist in libraries, how the Assessment planning process works in libraries, and what components constitute a Library Assessment plan. The findings suggest that, while libraries determine and report a great many aspects of organizational performance, general Assessment planning (which encompasses total organizational impact and value) is not frequently used or integrated into the Library's comprehensive strategic planning process, and it does not always effectively cover the scope of the Assessment activities and information needed by organizational leaders and administrators to develop effective strategic plans and decisions. This lack of integration creates disconnects between the Library's ability to determine and report organizational value and the Library's need to provide accountability evidence to stakeholders and use the performance measurement results effectively in executing strategic pla...

  • an old tool with potential new uses return on investment
    The Bottom Line: Managing Library Finances, 2007
    Co-Authors: Larry Nash White
    Abstract:

    Purpose – The purpose of this paper is to examine potential new uses of return on investment (ROI) in alternative methods for Library Assessment and valuation.Design/methodology/approach – Uses examples of recent uses of ROI by libraries to identify potential new applications of ROI in Library Assessment and valuation.Findings – ROI investment has historically been used by libraries for the Assessment of past performance of Library organizations in a reactive/defensive nature or for assessing the value of customer services. ROI has not been widely used as a proactive Assessment tool, to introspectively assess administrative or other internal services, or as an offensive Assessment and valuation tool, especially for intangible performance Assessment and valuation.Originality/value – New applications of the ROI Assessment tool could provide Library administrators with proactive means of increasing the effectiveness of Library Assessment, valuation, and their results.

Varnum, Kenneth J. - One of the best experts on this subject based on the ideXlab platform.

  • Search, Report, Wherever You Are: A Novel Approach to Assessing User Satisfaction with a Library Discovery Interface
    2020
    Co-Authors: Vacek Rachel, Smith Craig, Varnum, Kenneth J.
    Abstract:

    This is a virtual presentation given at the Library Assessment Conference on January 21, 2021. The presentation is in the Measurement & Methods track.PURPOSE AND GOALS In the summer of 2018, the University of Michigan Library launched a new discovery interface, Library Search (https://search.lib.umich.edu), for discovering the Library’s resources, collections, spaces, and expertise. Following our Assessment plan for Library Search, we have iteratively measured the success of Search since its launch across a variety of measures using a mixed-methods approach. We have tested for accessibility, monitored system performance, and conducted considerable usability testing as part of the design and development process. In general, our Assessments have indicated that Search works well according to many of our metrics, including accessibility, usability, design, and the general pattern of how searching for items, narrowing results sets, and accessing materials. There have been, however, a number of concerns about the catalog search in particular, and a general anecdotal sense that this important part of the interface is not quite meeting users’ needs. Therefore, we created a tool that we could use to first get a baseline measure of overall user satisfaction, and second, use again after we have made changes to Library Search to understand the degree to which our changes improved satisfaction. Our paper will report on our launch of our rather unique data collection tool and its use with multiple groups of stakeholders on campus. We will also detail how we plan to use the tool to gauge user satisfaction over time, and to regularly gather actionable data on the strengths and shortcomings of our catalog interface. DESIGN, METHODOLOGY, OR APPROACH An initial round of data collection focused on the search experiences of Library employees whose work involves using Library Search to assist members of our campus community. Focusing on this group allowed us to survey people who had clear expectations of how catalog searching should function for Library staff and users, and also to ensure that our data collection tool worked well before we used it to collect data from large numbers of faculty members and students on campus. The first full round of data collection, with faculty and students, is taking place in the winter of 2020; the results will be included in our paper and presentation. The invitation to participate in the initial round of data collection was sent in December 2019 to 96 Library employees. As an incentive, participants were given a chance to enter a drawing for one $50 gift card. Participants took part in the study online, via a survey on the Qualtrics platform. Forty people provided enough data to be included in some analyses (a 41% response rate); of those, 36 completed the whole survey. The final sample had decent variability with regard to respondents’ Library division and the number of years they had worked in the Library. A unique aspect of our data collection approach was to ask participants to conduct searches, to report on those searches, and to share the URLs associated with their search results. This allowed data from the survey to be interpreted while seeing exactly what participants were seeing as they did their searching. Thus, the first three sections of the survey asked participants to keep the survey tab open in their browsers while conducting specific types of catalog searches in a separate tab. The types of searches — known item, known set, and exploratory — were derived from another recent investigation of our Library search interface. Participants used the Qualtrics survey to answer questions about those three search experiences. Specifically, participants were asked to report on their satisfaction with the relevance of results, the speed with which results were returned, and the adequacy of various pieces of information contained in item records. When participants encountered unexpected results in their searches, they were given an opportunity to share more about what they expected to see, in relation to what they saw. A final section of the survey asked people to provide more global ratings and comments related to recent uses of Library Search (not limited to catalog searching; this could also include focused searches for articles, databases, etc.). For those that remembered using Search a year prior, a small set of questions also asked people to compare their current satisfaction with Search to what they remember feeling a year ago. These final questions, about recent experiences and comparisons to a year ago, were answered by most participants. FINDINGS We asked questions about three broad areas of search interactions: known item searches, known set searches, and exploratory searches. Known item searches are for specific, individual items. Known set searches are for collections of items (plays by Shakespeare, sonatas by Mozart, jazz CDs, etc.), from which the searcher would be more or less satisfied with any specific item. Exploratory searches are subject or topic-related. For known item searching, respondents were asked two questions about what they saw in the results: did the item appear in the results as expected, and was the position of the item in the result set satisfactory. Seventy-five percent saw the item in the results as expected, and the majority (92%) were either very or moderately satisfied with the position. We asked several additional questions for known item searches (we did not ask these questions for the other search categories, as we felt the responses would not be substantially different). When asked about the speed of search results, most (95%) expressed some level of satisfaction. In terms of ease of determining availability of print or online access to the items found, most were moderately or very satisfied (85%) but a notable minority were dissatisfied. And most people were moderately or very satisfied (86%) with identifying where physical items were located, with a notable minority expressing dissatisfaction. For known item searches -- and for the other two search types -- participants who saw unexpected search results were asked to share what they expected to see, and what they did see. Comments touched on concerns such as the relevance of results and the way that holdings were displayed. These findings serve as guides for the continued fine-tuning of Library Search. In the paper we will present examples of how such comments were paired with recreated searches in order to guide the work of our developers. For known set searches, just over half (58%) of the 36 participants who did this search saw what they expected; respondents were satisfied with the ranking of the results, about 50% each very or mostly satisfied. For exploratory searches, just over half (56%) saw what they expected in the results. Most (80%) were moderately or very satisfied, and a sizable minority (20%) reported some level of dissatisfaction. As noted, where participants saw unexpected results, they provided comments that shed additional light on their closed-ended survey responses. In the final section of the survey, participants were asked about their recent experiences with Search (not limited to Catalog Search), and their views on whether Search has improved or not compared to a year ago (for those with memories of Search at that time). Thirty-three participants had used Search within the previous two weeks. Of these, roughly three-quarters were moderately or very satisfied, with the rest expressing dissatisfaction. When asked about their satisfaction with the recent relevance of Search results, few were very satisfied (12%); most were moderately satisfied, and 30% expressed dissatisfaction. The same results were obtained when people were asked to rate their overall level of satisfaction with their recent experiences with Search. When asked to compare their current satisfaction with the speed of Search with what they remember from a year prior, most (81%) were somewhat or much more satisfied currently. When asked to compare their current satisfaction with the relevance of Search results with what they remember from a year prior, 72% were somewhat or much more satisfied currently. Finally, when asked to compare their current overall satisfaction with Search compared to a year ago, 82% were somewhat or much more satisfied. PRACTICAL IMPLICATIONS OR VALUE This study provides an example of how libraries can use an online data collection tool to reach key stakeholders when evaluating satisfaction with a web-based Library interaction. The general categorization of “known item”, “known set”, and “exploratory” searches, itself based on user research into kinds of searching, could easily be extended to other kinds of Library interactions in which a user is seeking something. The general method of the survey allows disintermediated user research to take place, with the efficiency of gaining detailed user feedback about specific interactions without the investment of a commensurate amount of staff time. Another key advantage of our methodology is that it facilitates the repeated evaluation of the search interface over time.The inclusion of both Library staff and campus users enables us to identify high-priority issues via staff insights and also to understand how users with a wide range of search expertise experience a core Library discovery interface.http://deepblue.lib.umich.edu/bitstream/2027.42/163532/3/Results of 2019 Survey on Catalog Search Satisfaction.pdfhttp://deepblue.lib.umich.edu/bitstream/2027.42/163532/2/Catalog Search Satisfaction Survey.pdfhttp://deepblue.lib.umich.edu/bitstream/2027.42/163532/1/Search, Report, Wherever You Are.pdfDescription of Search, Report, Wherever You Are.pdf : Presentation slide deckDescription of Catalog Search Satisfaction Survey.pdf : Survey from QualtricsDescription of Results of 2019 Survey on Catalog Search Satisfaction.pdf : Results of the SurveySEL

  • Search, Report, Wherever You Are: A Novel Approach to Assessing User Satisfaction with a Library Discovery Interface
    2020
    Co-Authors: Vacek Rachel, Smith Craig, Varnum, Kenneth J.
    Abstract:

    This is a virtual presentation given at the Library Assessment Conference on January 21, 2021. The presentation is in the Measurement & Methods track.PURPOSE AND GOALS In the summer of 2018, the University of Michigan Library launched a new discovery interface, Library Search (https://search.lib.umich.edu), for discovering the Library’s resources, collections, spaces, and expertise. Following our Assessment plan for Library Search, we have iteratively measured the success of Search since its launch across a variety of measures using a mixed-methods approach. We have tested for accessibility, monitored system performance, and conducted considerable usability testing as part of the design and development process. In general, our Assessments have indicated that Search works well according to many of our metrics, including accessibility, usability, design, and the general pattern of how searching for items, narrowing results sets, and accessing materials. There have been, however, a number of concerns about the catalog search in particular, and a general anecdotal sense that this important part of the interface is not quite meeting users’ needs. Therefore, we created a tool that we could use to first get a baseline measure of overall user satisfaction, and second, use again after we have made changes to Library Search to understand the degree to which our changes improved satisfaction. Our paper will report on our launch of our rather unique data collection tool and its use with multiple groups of stakeholders on campus. We will also detail how we plan to use the tool to gauge user satisfaction over time, and to regularly gather actionable data on the strengths and shortcomings of our catalog interface. DESIGN, METHODOLOGY, OR APPROACH An initial round of data collection focused on the search experiences of Library employees whose work involves using Library Search to assist members of our campus community. Focusing on this group allowed us to survey people who had clear expectations of how catalog searching should function for Library staff and users, and also to ensure that our data collection tool worked well before we used it to collect data from large numbers of faculty members and students on campus. The first full round of data collection, with faculty and students, is taking place in the winter of 2020; the results will be included in our paper and presentation. The invitation to participate in the initial round of data collection was sent in December 2019 to 96 Library employees. As an incentive, participants were given a chance to enter a drawing for one $50 gift card. Participants took part in the study online, via a survey on the Qualtrics platform. Forty people provided enough data to be included in some analyses (a 41% response rate); of those, 36 completed the whole survey. The final sample had decent variability with regard to respondents’ Library division and the number of years they had worked in the Library. A unique aspect of our data collection approach was to ask participants to conduct searches, to report on those searches, and to share the URLs associated with their search results. This allowed data from the survey to be interpreted while seeing exactly what participants were seeing as they did their searching. Thus, the first three sections of the survey asked participants to keep the survey tab open in their browsers while conducting specific types of catalog searches in a separate tab. The types of searches — known item, known set, and exploratory — were derived from another recent investigation of our Library search interface. Participants used the Qualtrics survey to answer questions about those three search experiences. Specifically, participants were asked to report on their satisfaction with the relevance of results, the speed with which results were returned, and the adequacy of various pieces of information contained in item records. When participants encountered unexpected results in their searches, they were given an opportunity to share more about what they expected to see, in relation to what they saw. A final section of the survey asked people to provide more global ratings and comments related to recent uses of Library Search (not limited to catalog searching; this could also include focused searches for articles, databases, etc.). For those that remembered using Search a year prior, a small set of questions also asked people to compare their current satisfaction with Search to what they remember feeling a year ago. These final questions, about recent experiences and comparisons to a year ago, were answered by most participants. FINDINGS We asked questions about three broad areas of search interactions: known item searches, known set searches, and exploratory searches. Known item searches are for specific, individual items. Known set searches are for collections of items (plays by Shakespeare, sonatas by Mozart, jazz CDs, etc.), from which the searcher would be more or less satisfied with any specific item. Exploratory searches are subject or topic-related. For known item searching, respondents were asked two questions about what they saw in the results: did the item appear in the results as expected, and was the position of the item in the result set satisfactory. Seventy-five percent saw the item in the results as expected, and the majority (92%) were either very or moderately satisfied with the position. We asked several additional questions for known item searches (we did not ask these questions for the other search categories, as we felt the responses would not be substantially different). When asked about the speed of search results, most (95%) expressed some level of satisfaction. In terms of ease of determining availability of print or online access to the items found, most were moderately or very satisfied (85%) but a notable minority were dissatisfied. And most people were moderately or very satisfied (86%) with identifying where physical items were located, with a notable minority expressing dissatisfaction. For known item searches -- and for the other two search types -- participants who saw unexpected search results were asked to share what they expected to see, and what they did see. Comments touched on concerns such as the relevance of results and the way that holdings were displayed. These findings serve as guides for the continued fine-tuning of Library Search. In the paper we will present examples of how such comments were paired with recreated searches in order to guide the work of our developers. For known set searches, just over half (58%) of the 36 participants who did this search saw what they expected; respondents were satisfied with the ranking of the results, about 50% each very or mostly satisfied. For exploratory searches, just over half (56%) saw what they expected in the results. Most (80%) were moderately or very satisfied, and a sizable minority (20%) reported some level of dissatisfaction. As noted, where participants saw unexpected results, they provided comments that shed additional light on their closed-ended survey responses. In the final section of the survey, participants were asked about their recent experiences with Search (not limited to Catalog Search), and their views on whether Search has improved or not compared to a year ago (for those with memories of Search at that time). Thirty-three participants had used Search within the previous two weeks. Of these, roughly three-quarters were moderately or very satisfied, with the rest expressing dissatisfaction. When asked about their satisfaction with the recent relevance of Search results, few were very satisfied (12%); most were moderately satisfied, and 30% expressed dissatisfaction. The same results were obtained when people were asked to rate their overall level of satisfaction with their recent experiences with Search. When asked to compare their current satisfaction with the speed of Search with what they remember from a year prior, most (81%) were somewhat or much more satisfied currently. When asked to compare their current satisfaction with the relevance of Search results with what they remember from a year prior, 72% were somewhat or much more satisfied currently. Finally, when asked to compare their current overall satisfaction with Search compared to a year ago, 82% were somewhat or much more satisfied. PRACTICAL IMPLICATIONS OR VALUE This study provides an example of how libraries can use an online data collection tool to reach key stakeholders when evaluating satisfaction with a web-based Library interaction. The general categorization of “known item”, “known set”, and “exploratory” searches, itself based on user research into kinds of searching, could easily be extended to other kinds of Library interactions in which a user is seeking something. The general method of the survey allows disintermediated user research to take place, with the efficiency of gaining detailed user feedback about specific interactions without the investment of a commensurate amount of staff time. Another key advantage of our methodology is that it facilitates the repeated evaluation of the search interface over time.The inclusion of both Library staff and campus users enables us to identify high-priority issues via staff insights and also to understand how users with a wide range of search expertise experience a core Library discovery interface.http://deepblue.lib.umich.edu/bitstream/2027.42/163532/1/Search, Report, Wherever You Are.pdfhttp://deepblue.lib.umich.edu/bitstream/2027.42/163532/2/Catalog Search Satisfaction Survey.pdfhttp://deepblue.lib.umich.edu/bitstream/2027.42/163532/3/Results of 2019 Survey on Catalog Search Satisfaction.pdfSEL

Yoshihide Hayashizaki - One of the best experts on this subject based on the ideXlab platform.

  • computer based methods for the mouse full length cdna encyclopedia real time sequence clustering for construction of a nonredundant cdna Library
    Genome Research, 2001
    Co-Authors: Hideaki Konno, Yoshifumi Fukunishi, Piero Carninci, Kazuhiro Shibata, Masayoshi Itoh, Yuichi Sugahara, Yoshihide Hayashizaki
    Abstract:

    The collection of full-length genes requires libraries with a high content of full-length cDNA inserts, large-scale sequencing, Library Assessment, and high-speed sequence clustering (Adams et al. 1995). Here, we focus on computational methods, such as newly developed computer programs, because our experimental methods have been published previously (Carninci et al. 1996, 1997, 1998; Sasaki et al. 1998b; Seki et al. 1998; Carninci and Hayashizaki 1999; Mizuno et al. 1999). There have been many reports on the Library Assessments (Adams et al. 1995; Salamov et al. 1998) and clustering techniques (Boguski and Schuler 1995; Sutton et al. 1995; Schuler et al. 1996; Burke et al. 1999; Miller et al. 1999). Why did we develop new methods for old problems? Because there are relevant differences between the program of our sequencing system, which is running for the production of clusters in the course of a sequencing project, and the conventional programs, which are independent of the experiment and show clustering of sets of finished sequences. Raw sequence data typically contains vector sequences. Various programs have been developed for vector masking, such as GRASTA (Adams et al. 1995) and CrossMatch (http://bozeman.genome.washington.edu/), which is an easy task as long as a conventional slab-gel sequencer is used. RIKEN developed the RISA 384-capillary sequencer, which we use in our project (Shimadzu Corp.). In the beginning of our project the gel slipped occasionally, which introduced sequencing errors when conventional vector-masking techniques were applied. Our vector-masking program has been developed to accommodate such mechanical difficulties. Expressed sequence tag (EST) collections have been started from 1991 (Adams et al. 1991, 1992; Gieser and Swaroop 1992; Khan et al. 1992; Okubo et al. 1992). Currently 1,856,056 human ESTs and 1,426,107 mouse ESTs have been collected, and these ESTs were clustered into 84,130 and 72,669 groups by UniGene of NCBI on September 21, 2000 (Boguski and Schuler 1995; Schuler et al. 1996). Many clustering programs have been developed during the last few years. On a specialized computer, these programs can calculate the clustering of several million tag sequences in a few days (Boguski and Schuler 1995; Sutton et al. 1995; Schuler et al. 1996; Burke et al. 1999; Miller et al. 1999). Clustering, which is manipulating the known sequences in silico, involves static sets of sequences—the number of sequences is fixed while clustering is calculated. In the course of a sequencing project, the clustering result changes when the number of sequences increases; this change makes clone selection chaotic. In our cDNA collection project, once the sequences are clustered, we select one of the clustered groups for the following steps. We developed a new method because we must cluster daily to avoid dramatic changes in the clustering result—with conventional methods, a single clustering analysis may take several days. Our purpose is the collection of full-length cDNAs; therefore, the proportion of full-length inserts in each Library is an important issue. Predicting whether a clone contains coding sequence is a necessary step in Library Assessment. Prediction programs have been developed that assess sequences of known genes, as well as characteristics of the sequence near the initiation codon, such as GC ratio, codon preference, and Kozak consensus (Adams et al. 1995; Salamov et al. 1998). The average length of 5′ untranslated region (5′ UTR) is 150 nucleotides (nt), and these conventional programs typically require an input of several hundred nucleotides. Increased sequencing involves additional time. In our program, we limited the 5′ tag sequence requirement to 200–250 nt and we used a full-length Assessment program that is based on EST data (Sugahara et al. 2001). We would like to collect all cDNA clones as quickly as possible. Using sequencing of the 3′ ends of the inserts, we monitor the rate at which novel genes are identified in the Library. If this rate declines, we switch to the next Library. Nonredundant clones are generated by using previously prepared libraries, from which the drivers for the Library subtraction process is prepared, which removes known genes from the new Library (Carninci et al. 2000). Quality assurance and sample tracking are important issues for large-scale sequencing centers and clone banks. For instance, the IMAGE Consortium has been making a significant effort to keep their clone quality (see http://image.llnl.gov/image/html/quality.shtml). Because we supply clones to the cDNA microarray project, protein interaction project, and outside collaborators, we must ensure the identity of the clones we submit. Disembodied bioinformatics on desk does not resolve these problems. Several sample tracking systems from private companies can be the basis of an appropriate laboratory information management system (LIMS). In addition to the LIMS, we need a system-dependent error-Assessment method. We have developed our own LIMS and several error-checking schemes that reflect our experience; with these schemes, errors in identification are <3%, which has been confirmed by independent double sequencing.

  • Assessment of redundancy and full length rate of full length enriched cdna libraries
    Genome Informatics, 2000
    Co-Authors: Hideaki Konno, Yoshifumi Fukunishi, Piero Carninci, Yoshihide Hayashizaki
    Abstract:

    Collection of full-length genes requires libraries with full-length cDNA insert, large-scale sequencing, Library Assessment, and high-speed sequence clustering. Here we focus on computational methods, such as newly developed computer programs, since our experimental methods had been published previously. Our purpose is the collection of full-length cDNAs, therefore the proportion of full-length inserts in each Library is an important issue. Predicting whether a clone contains coding sequence is a necessary step in Library Assessment. We would like to collect all cDNA clones as quickly as possible. Using sequencing of the 3’ ends of the inserts, we monitor the rate at which novel genes are identified in the Library. If this rate declines, we switch to the next Library. Non-redundant clones are generated by using previously prepared libraries, from which the drivers for the libraries subtraction process is prepared, which removes known genes from the new libraries. 1st strand cDNA for these libraries were primed with oligo dT containing primer. There are possibility that it happens to miss-prime with A rich region instead of poly A tail. We estimated internal priming rate and redundancy of these libraries. In this issue, we report result of this analysis.

Puckett Rodgers Emily - One of the best experts on this subject based on the ideXlab platform.

  • Diffusing Organizational Change through Service Design and Iterative Assessment
    Library Assessment Conference: 2018 Conference Proceedings, 2019
    Co-Authors: Vacek Rachel, Puckett Rodgers Emily, Sitar Meghan
    Abstract:

    This is the conference proceeding for the Library Assessment Conference that took place December 5-7, 2018, in Houston, TX. Proceedings are published as PDFs on this page: https://www.LibraryAssessment.org/past-conferences/2018-Library-Assessment-conference/2018-proceedings/At the University of Michigan Library, dozens of librarians and staff are engaged in a series of activities to reimagine the ways our organization designs and implements our services. Between 2016 and 2017, we collaborated with brightspot strategy to develop a service philosophy, framework, and principles to help us begin the process to transform our physical and digital spaces to better represent our expertise, collections, and tools, and to meet the evolving needs of our academic community today and tomorrow. As we look to transform our spaces to serve the needs of our research community, we are taking care to ensure that whatever form our buildings and web presence take, will follow the function and intent of our services. Our efforts in this work are collaborative and distributed in nature, diffusing the shift in design and evaluation across the organization. With it, we aim to facilitate organizational change that puts our users at the center of service design and delivery. It also fundamentally recognizes that our departments play a role in supporting the academic needs of our faculty, students, and staff at the University of Michigan. This work is structured by established approaches in design thinking and user-centered design. Multiple teams of librarians and staff are applying this approach to redesign services. Topics include consultation, digital scholarship, staff innovation, citation management, and developing a persona-based toolkit that any staff across our organization may use in efforts to design new or make improvements to existing services. While each team is using the same overall approach to its service design work, the application and outcomes are unique to each domain. Within four design cycles, each team engages in a retrospective to review the process, the impact of the work, and consider its potential effects on our organizational structures. Additionally, the service design efforts support our organization’s adoption of an Assessment-driven mindset through embedding evaluation into our processes. Once the service design phase is complete, each team will generate a series of pilots or prototypes to test aspects of their designs in the context of our organization. In the Summer and Fall of 2018, teams will implement those pilots and prototypes, testing their ability to scale effectively or meet our programmatic and mission-based goals yielding a series of small-scale, but impactful activities or processes that will further diffuse the design-thinking approach throughout our organization. Each of these sets of activities will be assessed before moving onto future stages of the work. We will also evaluate how they help us enact our service philosophy, framework, and principles in the practice of our everyday work. Ultimately this work will yield a culture shift within our organization enabling us to embrace a user-centered, service-based approach to how we develop services, and how we expect to collaborate and connect with colleagues within our Library and academic community. It will also enable us to embed Assessment practices into various facets of our work, from the beginning stages of design through its testing and into implementation at scale.Peer Reviewedhttps://deepblue.lib.umich.edu/bitstream/2027.42/151766/1/58-Vacek-PuckettRodgers-Sitar-DiffusingOrganizationalChange.pdfDescription of 58-Vacek-PuckettRodgers-Sitar-DiffusingOrganizationalChange.pdf : PDF of the Conference Proceedin

  • Diffusing Organizational Change through Service Design and Iterative Assessment
    2018
    Co-Authors: Vacek Rachel, Sitar Meghan, Puckett Rodgers Emily
    Abstract:

    This presentation was presented by Emily Puckett Rodgers and Rachel Vacek on December 6, 2018 at the Library Assessment Conference in Houston, TX.At the University of Michigan Library, dozens of librarians and staff are engaged in a series of activities to reimagine the ways our organization designs and implements our services. Between 2016 and 2017, we collaborated with brightspot strategy to develop a service philosophy, framework, and principles to help us begin the process to transform our physical and digital spaces to better represent our expertise, collections, and tools, and to meet the evolving needs of our academic community today and tomorrow. As we look to transform our spaces to serve the needs of our research community, we are taking care to ensure that whatever form our buildings and web presence take, will follow the function and intent of our services. Our efforts in this work are collaborative and distributed in nature, diffusing the shift in design and evaluation across the organization. With it, we aim to facilitate organizational change that puts our users at the center of service design and delivery. It also fundamentally recognizes that our departments play a role in supporting the academic needs of our faculty, students, and staff at the University of Michigan. This work is structured by established approaches in design thinking and user-centered design. Multiple teams of librarians and staff are applying this approach to redesign services. Topics include consultation, digital scholarship, staff innovation, citation management, and developing a persona-based toolkit that any staff across our organization may use in efforts to design new or make improvements to existing services. While each team is using the same overall approach to its service design work, the application and outcomes are unique to each domain. Within four design cycles, each team engages in a retrospective to review the process, the impact of the work, and consider its potential effects on our organizational structures. Additionally, the service design efforts support our organization’s adoption of an Assessment-driven mindset through embedding evaluation into our processes. Once the service design phase is complete, each team will generate a series of pilots or prototypes to test aspects of their designs in the context of our organization. In the Summer and Fall of 2018, teams will implement those pilots and prototypes, testing their ability to scale effectively or meet our programmatic and mission-based goals yielding a series of small-scale, but impactful activities or processes that will further diffuse the design-thinking approach throughout our organization. Each of these sets of activities will be assessed before moving onto future stages of the work. We will also evaluate how they help us enact our service philosophy, framework, and principles in the practice of our everyday work. Ultimately this work will yield a culture shift within our organization enabling us to embrace a user-centered, service-based approach to how we develop services, and how we expect to collaborate and connect with colleagues within our Library and academic community. It will also enable us to embed Assessment practices into various facets of our work, from the beginning stages of design through its testing and into implementation at scale.Peer Reviewedhttps://deepblue.lib.umich.edu/bitstream/2027.42/146754/1/Diffusing Organizational Change through Service Design and Iterative Assessment.pdfDescription of Diffusing Organizational Change through Service Design and Iterative Assessment.pdf : PDF of Presentatio