View in EDS PDF Full Text

A Framework for Measuring Relevancy in Discovery Environments.

Saved in:
Bibliographic Details
Title: A Framework for Measuring Relevancy in Discovery Environments.
Authors: Galbreath, Blake L.1 blake.galbreath@wsu.edu, Merrill, Alex2 merrilla@wsu.edu, Johnson, Corey M.3 coreyj@wsu.edu
Source: Information Technology & Libraries. 2021, Vol. 40 Issue 2, p1-17. 17p. 10 Charts.
Subject Terms: *Academic libraries, *Bibliography, *Bibliographical citations, Internet, Ecology, Conceptual structures, Descriptive statistics, Questionnaires, Statistical sampling, Statistical models
Abstract: Discovery environments are ubiquitous in academic libraries but studying their effectiveness and use in an academic environment has mostly centered around user satisfaction, experience, and task analysis. This study aims to create a quantitative, reproducible framework to test the relevancy of results and the overall success of Washington State University's discovery environment (Primo by Ex Libris). Within this framework, the authors use bibliographic citations from student research papers submitted as part of a required university class as the proxy for relevancy. In the context of this study, the researchers created a testing model that includes: (1) a process to produce machine-generated keywords from a corpus of research papers to compare against a set of human-created keywords, (2) a machine process to query a discovery environment to produce search result lists to compare against citation lists, and (3) four metrics to measure the comparative success of different search strategies and the relevancy of the results. This framework is used to move beyond a sentiment or task-based analysis to measure if materials cited in student papers appear in the results list of a production discovery environment. While this initial test of the framework produced fewer matches between researcher-generated search results and student bibliography sources than expected, the authors note that faceted searches represent a greater success rate when compared to open-ended searches. Future work will include comparative (A/B) testing of commonly deployed discovery layer configurations and limiters to measure the impact of local decisions on discovery layer efficacy as well as noting where in the results list a citation match occurs. [ABSTRACT FROM AUTHOR]
Copyright of Information Technology & Libraries is the property of American Library Association and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Education Research Complete
FullText Links:
  – Type: pdflink
Text:
  Availability: 0
Header DbId: ehh
DbLabel: Education Research Complete
An: 151047206
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: A Framework for Measuring Relevancy in Discovery Environments.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Galbreath%2C+Blake+L%2E%22">Galbreath, Blake L.</searchLink><relatesTo>1</relatesTo><i> blake.galbreath@wsu.edu</i><br /><searchLink fieldCode="AR" term="%22Merrill%2C+Alex%22">Merrill, Alex</searchLink><relatesTo>2</relatesTo><i> merrilla@wsu.edu</i><br /><searchLink fieldCode="AR" term="%22Johnson%2C+Corey+M%2E%22">Johnson, Corey M.</searchLink><relatesTo>3</relatesTo><i> coreyj@wsu.edu</i>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22Information+Technology+%26+Libraries%22">Information Technology & Libraries</searchLink>. 2021, Vol. 40 Issue 2, p1-17. 17p. 10 Charts.
– Name: Subject
  Label: Subject Terms
  Group: Su
  Data: *<searchLink fieldCode="DE" term="%22Academic+libraries%22">Academic libraries</searchLink><br />*<searchLink fieldCode="DE" term="%22Bibliography%22">Bibliography</searchLink><br />*<searchLink fieldCode="DE" term="%22Bibliographical+citations%22">Bibliographical citations</searchLink><br /><searchLink fieldCode="DE" term="%22Internet%22">Internet</searchLink><br /><searchLink fieldCode="DE" term="%22Ecology%22">Ecology</searchLink><br /><searchLink fieldCode="DE" term="%22Conceptual+structures%22">Conceptual structures</searchLink><br /><searchLink fieldCode="DE" term="%22Descriptive+statistics%22">Descriptive statistics</searchLink><br /><searchLink fieldCode="DE" term="%22Questionnaires%22">Questionnaires</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+sampling%22">Statistical sampling</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+models%22">Statistical models</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Discovery environments are ubiquitous in academic libraries but studying their effectiveness and use in an academic environment has mostly centered around user satisfaction, experience, and task analysis. This study aims to create a quantitative, reproducible framework to test the relevancy of results and the overall success of Washington State University's discovery environment (Primo by Ex Libris). Within this framework, the authors use bibliographic citations from student research papers submitted as part of a required university class as the proxy for relevancy. In the context of this study, the researchers created a testing model that includes: (1) a process to produce machine-generated keywords from a corpus of research papers to compare against a set of human-created keywords, (2) a machine process to query a discovery environment to produce search result lists to compare against citation lists, and (3) four metrics to measure the comparative success of different search strategies and the relevancy of the results. This framework is used to move beyond a sentiment or task-based analysis to measure if materials cited in student papers appear in the results list of a production discovery environment. While this initial test of the framework produced fewer matches between researcher-generated search results and student bibliography sources than expected, the authors note that faceted searches represent a greater success rate when compared to open-ended searches. Future work will include comparative (A/B) testing of commonly deployed discovery layer configurations and limiters to measure the impact of local decisions on discovery layer efficacy as well as noting where in the results list a citation match occurs. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of Information Technology & Libraries is the property of American Library Association and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=ehh&AN=151047206
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.6017/ital.v40i2.12835
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 17
        StartPage: 1
    Subjects:
      – SubjectFull: Academic libraries
        Type: general
      – SubjectFull: Bibliography
        Type: general
      – SubjectFull: Bibliographical citations
        Type: general
      – SubjectFull: Internet
        Type: general
      – SubjectFull: Ecology
        Type: general
      – SubjectFull: Conceptual structures
        Type: general
      – SubjectFull: Descriptive statistics
        Type: general
      – SubjectFull: Questionnaires
        Type: general
      – SubjectFull: Statistical sampling
        Type: general
      – SubjectFull: Statistical models
        Type: general
    Titles:
      – TitleFull: A Framework for Measuring Relevancy in Discovery Environments.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Galbreath, Blake L.
      – PersonEntity:
          Name:
            NameFull: Merrill, Alex
      – PersonEntity:
          Name:
            NameFull: Johnson, Corey M.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 06
              Text: 2021
              Type: published
              Y: 2021
          Identifiers:
            – Type: issn-print
              Value: 07309295
          Numbering:
            – Type: volume
              Value: 40
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: Information Technology & Libraries
              Type: main
ResultId 1