This year, the Libraries reviewed 400+ subscriptions using a variety of criteria to capture a holistic snapshot of each resource’s value.
Traditionally, libraries have relied on quantitative metrics like cost-per-use to make cancellation decisions. However, as important as the bottom line can be, there’s more to exploration, research, and scholarship than economics. Higher education is also about values like equity, diversity, and openness.
Throughout the past year, librarians at Western have been developing a two-pronged methodology to better evaluate our subscriptions in alignment with the university’s values. The first prong involves building a portfolio of criteria that go beyond cost-per-use to capture a more complete picture of each subscription’s value. These criteria include quantitative metrics like annual price increase, as well as qualitative metrics like accessibility to users with disabilities, vendor privacy policies, and publisher commitment to Open Access. Together, these criteria provide a more balanced basis for decision-making, in line with Western’s liberal arts mission.
The second prong of our new methodology entails acknowledging and accounting for the differences in format and media among our many subscriptions. A 2,000-title journal package is very different from a collection of streaming videos, or an ebook package, or a statistical dataset. To compare them, apples-to-apples, isn’t really fair. Instead, each format deserves its own custom criteria and scoring rubrics. To that end, Western’s librarians have developed separate scorecards for each of our nine formats of subscription. Each scorecard is different, but each adds up to 100 points--allowing us to make apples-to-oranges comparisons across 400+ diverse subscriptions.
Throughout summer and fall, the Subscription Task Force collected data on Western’s 400+ subscriptions. Each component of the data contributed a number of points towards a resource’s final score. Ultimately, scores ranged from 6.25 to 96 out of 100 points. The Task Force used the lowest-scoring resources to generate the draft cancellation list, accumulating enough titles to meet the $330,000 reduction goal plus a margin for departmental retention requests. The majority of subscriptions on the list fall below 48 out of 100 points. You can see each subscription’s score in the “Total Review Score” column of the draft cancellation list, as well as the scores for each individual criterion broken out.
A complete list of the quantitative and qualitative criteria can be found at the bottom of this page, along with a list of the format categories, an example scorecard, and graphs showing how Western’s subscriptions scored.
Subscription Evaluation Criteria
- Cost per use (CPU). CPU is a commonly used indicator of value for library subscriptions. Calculated using three years of use and the most recent cost data. CPU constitutes 50 out of 100 points on each e-resource scorecard.
- Availability of content via ILL. This criterion protects material that cannot be obtained reliably via means other than a subscription. Availability via ILL is not a criterion across all categories; however, where present, it constitutes an average of 10.2 out of 100 points.
- Usage for course reserves. Curricular relevance is a core value of Western’s library collections; use of materials for course reserves is an indicator of curricular relevance. Use for course reserves is not a criterion across all categories; however, where present, it constitutes an average of 8.1 out of 100 points.
- Average annual price increase. Annual increase is a significant factor in the sustainability of library subscriptions, calculated over three years. Annual increase constitutes, on average, 7.6 out of 100 points on each scorecard.
- Support for small colleges. This criterion protects disciplines with few e-resources by privileging subscriptions that provide the bulk of the content for any given college. College support constitutes, on average, 7.8 out of 100 points on each scorecard.
- Accessibility for users with disabilities. Library resources should be accessible to all users, regardless of ability, with minimal mediation; this criterion operationalizes that value by holding vendors accountable for ongoing accessibility improvements. Accessibility constitutes, on average, 6.0 out of 100 points on each scorecard.
- Limitations on seats or number of users. Some e-resources limit access to a fixed number of concurrent users, interrupting access to our full user community and providing poor user experience; this criterion holds vendors accountable for providing broader access. Limitations on seats is not a criterion across all categories; however, where present, it constitutes an average of 4.7 out of 100 points.
- Support for large number of users. This criterion boosts resources that are highly interdisciplinary or support large departments or programs. Number of users supported constitutes, on average, 4.3 out of 100 points on each scorecard.
- Digital Rights Management (DRM) restrictions. This criterion refers to use restrictions on e-book platforms, which include limitations on printing, sharing, and downloading, as well as proprietary software and the requirement that users create usernames and passwords to access content. For e-books, DRM constitutes 4 out of 100 points.
- Overlap with other resources. This criterion privileges resources with largely unique content, while flagging e-resources with a high degree of overlap. Duplication is not a criterion across all categories; however, where present, it constitutes an average of 4.2 out of 100 points.
- Demonstrable commitment to Open Access. As library subscriptions become increasingly unsustainable, Western wishes to acknowledge and reward publishers taking real steps toward a more open scholarly landscape. This criterion gives a small boost to e-resources from publishers who make at least some content truly open. Commitment to Open Access is not a criterion across all categories; however, where present, it constitutes an average of 4.5 out of 100 points.
- Transparency of privacy policies. This criterion holds vendors accountable for having transparent privacy policies. Privacy policies constitutes, on average, 3.8 out of 100 points on each scorecard.
- Indexed in OneSearch. The ability of users to discover content by searching in OneSearch is a key, albeit small, piece of each e-resource’s value. This criterion acknowledges the vendors who make their content easy to find. Indexing is not a criterion across all categories; however, where present, it constitutes an average of 2.3 out of 100 points.
Format Categories
- E-Book Packages. Subscription databases containing exclusively e-books. Content is leased year to year, so upon cancellation all titles are lost. These are distinct from owned e-books, which may be purchased individually or in packages and represent a one-time expenditure.
- Full-Text Owned Journals. Packages or databases containing exclusively e-journals. Content is owned, so each year’s subscription fees in effect purchase a new year of content. Upon cancellation, content up to the cancellation date is retained (except for any content from prior to the original subscription start date).
- Full-Text Databases. Subscription databases containing exclusively full-text content, whether journals, monographs, conference proceedings, gray literature, etc. Content is leased year to year, so upon cancellation all titles are lost.
- Full-Text & Discovery Databases. Subscription databases containing a mix of full-text content and abstracting/indexing, whether journals, monographs, conference proceedings, gray literature, etc. Content is leased year to year, so upon cancellation all titles, including abstracts/indexing, are lost.
- Discovery Databases. Subscription databases containing exclusively abstracting/indexing, i.e. that provide a means of discovering content but not the content itself. This can include abstracting/indexing for journals, monographs, conference proceedings, gray literature, etc. The database is leased year to year, so upon cancellation all discovery is lost.
- Full-Text Reference. Databases that provide full-text access to a single title or reference work, e.g. an online encyclopedia, dictionary, or handbook. Purchase models for full-text reference works vary: in some cases, content is leased year to year and upon cancellation all material is lost; in other cases, content is owned and each year’s payment secures perpetual access to another year of material.
- Streaming Media. Databases consisting primarily of streaming music or video. Content is leased year to year, so upon cancellation all content is lost.
- Data, Other, Non-ILL. Databases that contain non-textual (or other non-traditional) scholarly material, e.g. images, chemical structures, data, etc. Because it does not follow standard bibliographic patterns, the content from these e-resources is more difficult to compare (or replace) on a one-to-one basis, and may not be available through ILL. Content is typically leased year to year, so upon cancellation all titles are lost.
- Single Subscriptions. Single e-journal titles purchased outside of packages. Content is owned, so each year’s subscription fees in effect purchase a new year of content. Upon cancellation, content up to the cancellation d ate is retained (except for any content from prior to the original subscription start date).
Example Scorecard
Distribution of Subscription Scores