View in EDS HTML Full Text PDF Full Text

Peer Review in Funding-by-Lottery: A Systematic Overview and Expansion

Saved in:
Bibliographic Details
Title: Peer Review in Funding-by-Lottery: A Systematic Overview and Expansion
Language: English
Authors: Shaw, Jamie
Source: Research Evaluation. Jan 2023 32(1):86-100.
Availability: Oxford University Press. Great Clarendon Street, Oxford, OX2 6DP, UK. Tel: +44-1865-353907; Fax: +44-1865-353485; e-mail: jnls.cust.serv@oxfordjournals.org; Web site: http://applij.oxfordjournals.org/
Peer Reviewed: Y
Page Count: 15
Publication Date: 2023
Document Type: Journal Articles
Reports - Evaluative
Descriptors: Peer Evaluation, Program Evaluation, Financial Support, Scientific Research, Taxonomy, Holistic Approach, Trust (Psychology), Social Responsibility
DOI: 10.1093/reseval/rvac022
ISSN: 0958-2029
1471-5449
Abstract: Despite the surging interest in introducing lottery mechanisms into decision-making procedures for science funding bodies, the discourse on funding-by-lottery remains underdeveloped and, at times, misleading. Funding-by-lottery is sometimes presented as if it were a single mechanism when, in reality, there are many funding-by-lottery mechanisms with important distinguishing features. Moreover, funding-by-lottery is sometimes portrayed as an alternative to traditional methods of peer review when peer review is still used within funding-by-lottery approaches. This obscures a proper analysis of the (hypothetical and actual) variants of funding-by-lottery and important differences amongst them. The goal of this article is to provide a preliminary taxonomy of funding-by-lottery variants and evaluate how the existing evidence on peer review might lend differentiated support for variants of funding-by-lottery. Moreover, I point to gaps in the literature on peer review that must be addressed in future research. I conclude by building off of the work of Avin in moving toward a more holistic evaluation of funding-by-lottery. Specifically, I consider implications funding-by-lottery variants may have regarding trust and social responsibility.
Abstractor: As Provided
Entry Date: 2023
Accession Number: EJ1382459
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwHkhgZWtLtGr4Ms97qUtZszAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDBT44EZNXc749hVj7gIBEICBmz86IG1bh00c6ZM6Fz7kIbDwQSKFa5q5iuHVytHPd_Wid-rqM8tBIX6TnwbagQIjH5gdXwf83h9XRtW6xfq-Rc79NH8zbi6Qf6VC1i9Do4xXiW9jMcDpGNUM9MSqMC52wtINTG_QNk3H9iM7Qjzlu0fumy8rkVK2I9rwnO9DoleL1ygxDA5beDcyWj-eM5NSh5KtFBiDFBZuiooy
Text:
  Availability: 1
  Value: <anid>AN0164246306;p9101jan.23;2023Jun14.05:22;v2.2.500</anid> <title id="AN0164246306-1">Peer review in funding-by-lottery: A systematic overview and expansion </title> <p>Despite the surging interest in introducing lottery mechanisms into decision-making procedures for science funding bodies, the discourse on funding-by-lottery remains underdeveloped and, at times, misleading. Funding-by-lottery is sometimes presented as if it were a single mechanism when, in reality, there are many funding-by-lottery mechanisms with important distinguishing features. Moreover, funding-by-lottery is sometimes portrayed as an alternative to traditional methods of peer review when peer review is still used within funding-by-lottery approaches. This obscures a proper analysis of the (hypothetical and actual) variants of funding-by-lottery and important differences amongst them. The goal of this article is to provide a preliminary taxonomy of funding-by-lottery variants and evaluate how the existing evidence on peer review might lend differentiated support for variants of funding-by-lottery. Moreover, I point to gaps in the literature on peer review that must be addressed in future research. I conclude by building off of the work of Avin in moving toward a more holistic evaluation of funding-by-lottery. Specifically, I consider implications funding-by-lottery variants may have regarding trust and social responsibility.</p> <p>Keywords: funding-by-lottery; peer review; social responsibility; trust</p> <hd id="AN0164246306-2">1. Preamble</hd> <p>Peer review has been and still is the most dominant form of research evaluation in science funding policy. Its widespread practice and adoption suggest that peer review has been a paradigm in the determination of research fund allocations largely to the exclusion of alternative methods. The coiner of the term 'paradigm', Thomas Kuhn, argued that the history of science moved from periods of normal science, where a single paradigm reigns supreme, to crisis when scientists lose faith in that paradigm, to revolution when alternative paradigms become proliferated ([<reflink idref="bib59" id="ref1">59</reflink>]). A similar story can be told for the development of research evaluation in modern science funding policy. The paradigm that dominated from (at least) the end of WWII dictated that research proposals should be evaluated by peer review ([<reflink idref="bib31" id="ref2">31</reflink>]). While peer review has always had its share of detractors ([<reflink idref="bib25" id="ref3">25</reflink>]), there has been a rapidly growing uneasiness with this paradigm since the 1970s. Some attempt to salvage peer review from its critics ([<reflink idref="bib72" id="ref4">72</reflink>]). Others have developed novel funding mechanisms in theory and practice.[<reflink idref="bib1" id="ref5">1</reflink>] Currently, there are several proposed alternatives to peer review including direct investment (see [<reflink idref="bib114" id="ref6">114</reflink>]), peer-to-peer distribution networks ([<reflink idref="bib14" id="ref7">14</reflink>]), crowdsourcing ([<reflink idref="bib103" id="ref8">103</reflink>]), and universal basic incomes ([<reflink idref="bib71" id="ref9">71</reflink>]). More broadly, RAND Europe commissioned a report outlining a variety of alternatives to peer review at different decision-making stages ([<reflink idref="bib46" id="ref10">46</reflink>]).</p> <p>One which has been discussed with increasing vigor is funding-by-lottery ([<reflink idref="bib42" id="ref11">42</reflink>]; [<reflink idref="bib17" id="ref12">17</reflink>]; [<reflink idref="bib39" id="ref13">39</reflink>]; [<reflink idref="bib37" id="ref14">37</reflink>]; [<reflink idref="bib32" id="ref15">32</reflink>]; [<reflink idref="bib5" id="ref16">5</reflink>]; [<reflink idref="bib78" id="ref17">78</reflink>]). Funding-by-lottery has been discussed abstractly and employed by bodies such as the New Zealand's Health Research Council (HRC) and Science for Technological Innovation Challenge, the Volkswagen Foundation's 'Experiment!',[<reflink idref="bib2" id="ref18">2</reflink>] the Swiss National Science Foundation, the New Frontiers Research Fund (for its 'Innovative Approaches to Research in the Pandemic Context' program), and the Foundational Questions Institute (for its 'mini grants'). This increased attention has other funding bodies, such as the Natural Sciences and Engineering Research Council of Canada, considering employing lotteries as well ([<reflink idref="bib102" id="ref19">102</reflink>]). However, the academic literature remains young and filled with questions that need answers.</p> <p>The purpose of this article is to contribute to the theoretical development of funding-by-lottery in a few related ways. First, I offer a preliminary taxonomy of different versions of funding-by-lottery. This taxonomy will organize existing models of funding-by-lottery, some of which remain theoretical while others are being practiced, and gesture toward novel variants of funding-by-lottery. I hope that subsequent research will further refine these categories and extend the taxonomy by developing novel variants. This taxonomy will be helpful for distinguishing between the versions of lotteries that may be implemented and allowing more fine-grained evaluations of particular funding allocation methods.</p> <p>Second, I survey and interpret much of the literature on peer review and suggest how it relates to the variants in the proposed taxonomy. This literature provides partial grounds for determining the appropriate scope of these variants and usages of peer review. Here, I follow in the footsteps of others by assessing how funding-by-lottery fares given findings on peer review, though I aim to show how distinct findings on the reliability of peer review relate to variants of funding-by-lottery. This shows the different ways and contexts in which peer review can be seen as a legitimate method of evaluating the promise of research (see also [<reflink idref="bib52" id="ref20">52</reflink>]). Moreover, I highlight gaps for evaluating between particular variants of funding-by-lottery. I also briefly discuss how considerations of cost may differ amongst variants of funding-by-lottery.</p> <p>Finally, I expand the scope of our analyses of funding-by-lottery. Some commentators have sought to contribute toward a more holistic assessment of funding-by-lottery by considering other desiderata of funding allocation methods such as fairness and transparency ([<reflink idref="bib30" id="ref21">30</reflink>]; [<reflink idref="bib5" id="ref22">5</reflink>]; [<reflink idref="bib44" id="ref23">44</reflink>]). I add to this literature by suggesting how different variants of funding-by-lottery relate to trust and social responsibility. This should help us move toward a more nuanced and multifaceted discussion of precisely when and where peer review and lottery elements can be fruitful. While I will not pretend to present a complete set of metrics for evaluating funding allocation methods, I hope to provide some steps toward a more holistic assessment of research evaluation.</p> <hd id="AN0164246306-3">2 Lotteries and peer review</hd> <p>Before elaborating upon a taxonomy of funding-by-lottery, it is worth taking a moment to understand how funding-by-lottery relates to peer review more broadly. Some passages within the literature on funding-by-lottery discuss it as if it were a single, homogenized mechanism. Moreover, it is sometimes portrayed as a full-blown alternative that should replace traditional peer review. For example, Fang and Casadevall write that the 'time-honored mechanism of allocating funds based on ranking of proposals by scientific peer review is no longer effective' ([<reflink idref="bib32" id="ref24">32</reflink>]: 1). Therefore, they claim, funding-by-lottery, which avoids the pitfalls of peer review, should be sought. Roumbanis claims that a 'a lottery could <emph>replace</emph> the peer review system' ([<reflink idref="bib90" id="ref25">90</reflink>], my emphasis, 1007), while Donald Gillies writes that:</p> <p>A widely-used method of research funding is through competitive grants, where the selection of which of the applications to fund is made using anonymous peer review. The aim of the present paper is to argue that the system would work more efficiently if the selection were made by random choice rather than peer review. ([<reflink idref="bib37" id="ref26">37</reflink>]: 1)</p> <p>A closer look at these commentators shows that none of them actually accept this framing of the debate, but still this simplistic formulation has been repeated by public figures and has become a part of the image of funding-by-lottery. For example, Warwick Anderson, ex-CEO of the Australian National Health and Medical Research Council, dismissed lotteries in a speech to the National Press Club of Australia stating that 'Some critics have been suggesting that peer review is just too much hard work and perhaps a lottery would be better. Mind you this is a suggestion from economists, so take that any way you want' (quoted in [<reflink idref="bib8" id="ref27">8</reflink>]). Much of the popular discussion of funding-by-lottery has followed suit and homogenized it as a single funding mechanism, to be contrasted with traditional methods of peer review.</p> <p>To speak more accurately, we should refer to funding-by-lottery as a family of proposals that retain elements of peer review but curtail its role in funding decisions. There are important similarities between these proposals that warrant being gathered under the heading of 'funding-by-lottery'. However, there are crucial differences which tend to be glossed over or ignored altogether. This is especially important when considering the relationship between funding-by-lottery and peer review, since one of the primary differentiators between variants is the appropriate role of peer review. Funding-by-lottery does not <emph>replace</emph> or offer an <emph>alternative</emph> to peer review. Rather, it <emph>supplements</emph> peer review. It is more accurate to speak modestly, as Shahar Avin does, where funding-by-lottery involves 'introducing a random element' ([<reflink idref="bib5" id="ref28">5</reflink>]: 16) into funding mechanisms. Elaborating these differences reflects differences of opinion about what the evidence on peer review reveals rather than a blanket condemnation of peer review as a whole. While this much is already implicitly recognized by most discussants of funding-by-lottery, it should be front and center to keep the debate as clear as possible.</p> <hd id="AN0164246306-4">3. A preliminary taxonomy of funding-by-lottery</hd> <p>One overview of funding-by-lottery can be found in [<reflink idref="bib90" id="ref29">90</reflink>]. Here, he distinguishes between peer review and funding-by-lottery by the roles played by reason and chance:</p> <p>What characterizes lotteries... is that 'they eschew rather than embrace identifiable elements of personal desert or social value; lotteries are driven by chance, not reason'... A lottery... can be seen as a 'second-order rationality,' which means that reason enters the decision making process during an earlier phase, then taking a step back to let chance be the determining factor—this being in many cases the most rational approach. ([<reflink idref="bib90" id="ref30">90</reflink>]: 995–996)</p> <p>Traditional peer-review ranks grant proposals via expert judgment ('reason'), while funding-by-lottery embraces randomness ('chance'). Funding-by-lottery does not <emph>entirely</emph> embrace chance and <emph>completely</emph> eschew reason. Rather, funding-by-lottery adds a 'second-order rationality' that circumscribes the appropriate roles of reason and chance. While Roumbanis does not distinguish amongst the variants of funding-by-lottery, we can distinguish variants of funding-by-lottery via the second-order rationality they employ.</p> <p>Before describing serious proposals for funding-by-lottery, let us consider a hypothetical funding-by-lottery approach that <emph>fully</emph> dissuades reason and <emph>entirely</emph> embraces chance. On this approach, no proposal could be rejected from the lottery. Even a preliminary skim of different proposals would require 'reason', even if minimally. Moreover, all proposals will enter the same lottery regardless of their promises, plans, or qualifications. To claim that a given proposal is not relevant to the mandate of the fund requires reason to determine what is relevant to the mandate and what is not. Employing boundary conditions, which can be controversial and difficult in practice, requires reason (see [<reflink idref="bib36" id="ref31">36</reflink>]; [<reflink idref="bib69" id="ref32">69</reflink>]; [<reflink idref="bib96" id="ref33">96</reflink>]). Looking at things this way, it is obvious that a <emph>full-blown</emph> lottery, one that <emph>entirely</emph> outcasts reason and embraces chance, cannot realistically inform science funding policy. As far as I can tell, no one has defended a full-blown lottery.</p> <p>All funding-by-lottery mechanisms employ boundary conditions to determine what proposals are relevant to their mission statement. Consider Roumbanis' description of the Volkswagen Foundation's 'Experiment!' program as an illustrative example:</p> <p>In Germany, the Volkswagen Foundation has recently started a pilot study called 'Experiment!', in which proposals are selected either by lot (just making sure this isn't an accident it's not supposed to be 'lottery') or peer review. All applicants are instructed to submit an application—a short sketch is enough. First, approximately 120–140 proposals are preselected through screening by the Foundation's staff. Next, an interdisciplinary jury of scientific experts evaluates these proposals double blinded and selects the fifteen to twenty most convincing applications. Thereafter, the same number of proposals is also drawn in a lottery from the same original pool of proposals. ([<reflink idref="bib90" id="ref34">90</reflink>]: 1012–1013)</p> <p>Here, there are multiple levels of screening.[<reflink idref="bib3" id="ref35">3</reflink>] The first is by the staff,[<reflink idref="bib4" id="ref36">4</reflink>] which focus on which projects are relevant to the aims and scope of the funding program. The Volkswagen Foundation invites proposals from 'researchers in science and engineering as well as in the life sciences (including immediately neighboring disciplines in the behavioral sciences)' rather than research outside of this scope.[<reflink idref="bib5" id="ref37">5</reflink>] Similarly, the New Zealand Health Research Council's mandate, determining their scope of possible public health projects, must be consonant with the HRC Act of 1990 and the New Zealand Health Research Strategy 2017–2027. These boundary conditions earmark funding priorities. Moreover, the Volkswagen Foundation is interested in 'radical' and 'possibly transformative' projects. Similarly, the New Zealand Health Research Council is interested in 'ideas considered to be transformative, innovative, exploratory or unconventional, and have potential for major impact' ([<reflink idref="bib53" id="ref38">53</reflink>]. Reason is needed not just to determine the <emph>relevance</emph> of proposals, but to characterize proposals as falling in certain categories (i.e. they are innovative rather than conventional, or transformative rather than orthodox). Finally, [<reflink idref="bib42" id="ref39">42</reflink>]: 686) suggests that lotteries should screen for 'scientists whose qualifications... have been certified as respectable'. A similar view is defended by [<reflink idref="bib37" id="ref40">37</reflink>]. This is another way proposals may be screened; not on the basis of content but the quality of the researchers.</p> <hd id="AN0164246306-5">3.1 Partial vs. non-partial lotteries</hd> <p>Variants of funding-by-lottery introduce peer review at different levels. The first sub-distinction is between partial lotteries and non-partial lotteries (see [<reflink idref="bib42" id="ref41">42</reflink>]). In partial lotteries, only some proposals are included in the lottery, whereas in non-partial lotteries, all proposals that pass screening are entered into the lottery. Partial lotteries are more commonly employed, but they come in different forms. Some partial lotteries allow for some proposals to be automatically funded. Others reverse this order by rejecting bottom ranked proposals. Others endorse both, where the top and bottom proposals can be accepted and rejected, respectively (e.g. [<reflink idref="bib17" id="ref42">17</reflink>]; [<reflink idref="bib18" id="ref43">18</reflink>]; [<reflink idref="bib40" id="ref44">40</reflink>]: 4; [<reflink idref="bib9" id="ref45">9</reflink>]: 7). Non-partial lotteries do not make these distinctions and allow every proposal, regardless of how strong or weak it may be, a chance to receive or be denied funds. As long as the aims and content of the proposal are relevant to the mandates of the funding agency, all proposals would be introduced into the lottery.</p> <hd id="AN0164246306-6">3.2 Weighted vs. non-weighted lotteries</hd> <p>Next, we can distinguish <emph>weighted</emph> from <emph>non</emph>-<emph>weighted</emph> lotteries. In weighted lotteries, different numbers of 'tickets' are awarded depending on where the proposal ranks amongst the pool of applications, whereas in non-weighted lotteries, all proposals receive an equal number of tickets. For weighted lotteries, each proposal must be reviewed and ranked. Unlike traditional peer review, funding-by-lottery transforms judgments from 'accept' or 'reject' to a number of tickets. Peer review maintains its function of ranking applications but softens the degree to which it impacts funding decisions. Non-weighted variants do not require this function of peer review. All proposals that enter the lottery are given equal amounts of tickets (see [<reflink idref="bib90" id="ref46">90</reflink>]: 1003). Note that both partial and non-partial lotteries can employ weighted or non-weighted approaches, respectively, allowing for four possible combinations (Table 1).</p> <p>Table 1. Weighted and partial lottery variants</p> <p> <ephtml> <table><thead><tr><th>. </th><th>Partial lotteries. </th><th>Non-partial lotteries. </th></tr></thead><tbody><tr><td>Weighted lotteries </td><td>Only some screened proposals enter the lottery. Those that do are given unequal amounts of tickets. </td><td>All screened proposals enter the lottery but are given unequal amounts of tickets </td></tr><tr><td>Non-weighted lotteries </td><td>Only some screened proposals enter the lottery. Those that do are given equal numbers of tickets. </td><td>All screened proposals enter the lottery and are given an equal number of tickets. </td></tr></tbody></table> </ephtml> </p> <p>Table 1. Weighted and partial lottery variants</p> <p> <ephtml> <table><thead><tr><th>. </th><th>Partial lotteries. </th><th>Non-partial lotteries. </th></tr></thead><tbody><tr><td>Weighted lotteries </td><td>Only some screened proposals enter the lottery. Those that do are given unequal amounts of tickets. </td><td>All screened proposals enter the lottery but are given unequal amounts of tickets </td></tr><tr><td>Non-weighted lotteries </td><td>Only some screened proposals enter the lottery. Those that do are given equal numbers of tickets. </td><td>All screened proposals enter the lottery and are given an equal number of tickets. </td></tr></tbody></table> </ephtml> </p> <p>The bottom right quadrant is likely what the phrase 'funding-by-lottery' evokes in popular discussions, though they are less popular amongst proponents of funding-by-lottery. Notice that non-partial weighted lotteries approximate partial lotteries depending on ticket assignment. In extreme variants, where, say, bottom ranked proposals are given a 0.01% chance, weighted non-partial lotteries are statistically nearly identical to a partial lottery.</p> <hd id="AN0164246306-7">3.3 Tiered vs. continuous lotteries</hd> <p>Next, we may distinguish tiered lotteries from continuous lotteries, both are kinds of weighted lotteries (see [<reflink idref="bib39" id="ref47">39</reflink>]). In tiered lotteries, there are multiple levels within the pool of applications. Each application <emph>within</emph> a tier is given the same number of tickets and, therefore, has the same chance of winning the lottery. Peer review is needed to place the proposal in a specific tier. The more tiers, the more fine-grained peer review judgments must be. A lottery with as many tiers as proposals would be a continuous lottery. More commonly, though, it is assumed that there will be a relatively small number of tiers—or at least a number of tiers that is smaller than the number of proposals. We can further distinguish two kinds of tiered lotteries, which we can call <emph>even tiered</emph> and <emph>uneven tiered</emph> lotteries. An uneven tiered lottery allows different numbers of proposals into different tiers whereas an even tiered lottery would have the same number of proposals in each tier (see e.g. Tables 2 and 3).</p> <p>Table 2. Example of uneven lottery</p> <p> <ephtml> <table><tbody><tr><td>Tier 1 </td><td>50 proposals </td></tr><tr><td>Tier 2 </td><td>100 proposals </td></tr><tr><td>Tier 3 </td><td>200 proposals </td></tr><tr><td>Tier 4 </td><td>50 proposals </td></tr><tr><td>Tier 5 </td><td>100 proposals </td></tr></tbody></table> </ephtml> </p> <p>Table 2. Example of uneven lottery</p> <p> <ephtml> <table><tbody><tr><td>Tier 1 </td><td>50 proposals </td></tr><tr><td>Tier 2 </td><td>100 proposals </td></tr><tr><td>Tier 3 </td><td>200 proposals </td></tr><tr><td>Tier 4 </td><td>50 proposals </td></tr><tr><td>Tier 5 </td><td>100 proposals </td></tr></tbody></table> </ephtml> </p> <p>Table 3. Example of even lottery</p> <p> <ephtml> <table><tbody><tr><td>Tier 1 </td><td>50 proposals </td></tr><tr><td>Tier 2 </td><td>50 proposals </td></tr><tr><td>Tier 3 </td><td>50 proposals </td></tr><tr><td>Tier 4 </td><td>50 proposals </td></tr><tr><td>Tier 5 </td><td>50 proposals </td></tr></tbody></table> </ephtml> </p> <p>Table 3. Example of even lottery</p> <p> <ephtml> <table><tbody><tr><td>Tier 1 </td><td>50 proposals </td></tr><tr><td>Tier 2 </td><td>50 proposals </td></tr><tr><td>Tier 3 </td><td>50 proposals </td></tr><tr><td>Tier 4 </td><td>50 proposals </td></tr><tr><td>Tier 5 </td><td>50 proposals </td></tr></tbody></table> </ephtml> </p> <p>This distinction has not been made, to my knowledge, but it is worth mentioning. We may contend that some upper or lower tiers should be more inclusive or exclusive, e.g. and therefore be more selective about which proposals should enter these tiers. More specifically, we could imagine further distinguishing between the various ways of curving uneven tiered lotteries. A few possible examples are shown in Tables 4–6.</p> <p>Table 4. Linear curving</p> <p> <ephtml> <table><tbody><tr><td>Tier 1 </td><td>100 tickets </td></tr><tr><td>Tier 2 </td><td>80 tickets </td></tr><tr><td>Tier 3 </td><td>60 tickets </td></tr><tr><td>Tier 4 </td><td>40 tickets </td></tr><tr><td>Tier 5 </td><td>20 tickets </td></tr></tbody></table> </ephtml> </p> <p>Table 4. Linear curving</p> <p> <ephtml> <table><tbody><tr><td>Tier 1 </td><td>100 tickets </td></tr><tr><td>Tier 2 </td><td>80 tickets </td></tr><tr><td>Tier 3 </td><td>60 tickets </td></tr><tr><td>Tier 4 </td><td>40 tickets </td></tr><tr><td>Tier 5 </td><td>20 tickets </td></tr></tbody></table> </ephtml> </p> <p>Table 5. Exponential curving</p> <p> <ephtml> <table><tbody><tr><td>Tier 1 </td><td>3,125 tickets </td></tr><tr><td>Tier 2 </td><td>625 tickets </td></tr><tr><td>Tier 3 </td><td>125 tickets </td></tr><tr><td>Tier 4 </td><td>25 tickets </td></tr><tr><td>Tier 5 </td><td>5 tickets </td></tr></tbody></table> </ephtml> </p> <p>Table 5. Exponential curving</p> <p> <ephtml> <table><tbody><tr><td>Tier 1 </td><td>3,125 tickets </td></tr><tr><td>Tier 2 </td><td>625 tickets </td></tr><tr><td>Tier 3 </td><td>125 tickets </td></tr><tr><td>Tier 4 </td><td>25 tickets </td></tr><tr><td>Tier 5 </td><td>5 tickets </td></tr></tbody></table> </ephtml> </p> <p>Table 6. Staggered curving</p> <p> <ephtml> <table><tbody><tr><td>Tier 1 </td><td>500 tickets </td></tr><tr><td>Tier 2 </td><td>125 tickets </td></tr><tr><td>Tier 3 </td><td>100 tickets </td></tr><tr><td>Tier 4 </td><td>15 tickets </td></tr><tr><td>Tier 5 </td><td>10 tickets </td></tr></tbody></table> </ephtml> </p> <p>Table 6. Staggered curving</p> <p> <ephtml> <table><tbody><tr><td>Tier 1 </td><td>500 tickets </td></tr><tr><td>Tier 2 </td><td>125 tickets </td></tr><tr><td>Tier 3 </td><td>100 tickets </td></tr><tr><td>Tier 4 </td><td>15 tickets </td></tr><tr><td>Tier 5 </td><td>10 tickets </td></tr></tbody></table> </ephtml> </p> <p>Mathematicians will be familiar with the wide variety of means of determining different ways of curving the number of tickets and the statistical patterns of funding allocation that follow therefrom. I will not try to outline them all here, or the reasons why we may adopt one over another. But, for now, it is worth mentioning that a variety of species of tiered lotteries are possible.</p> <p>In a continuous system, there is a spectrum of tickets that (presumably) increases as one moves up the rankings. Reviewers are more burdened in this system because the rankings must remain relatively precise to separate individual proposals. Again, many different functions can be used for determining the ways in which the number of tickets should be curved over a pool of proposals.</p> <hd id="AN0164246306-8">3.4 Summary</hd> <p>While this taxonomy is admittedly preliminary, it provides us with a clear way of determining the key differences between the variants of funding-by-lottery. Below are two visual representations of funding-by-lottery as outlined in this section. The flow chart illustrates the discussed variants, where we gradually become more specific as we go to the right. We can see there are currently 21 different variants of funding-by-lottery (Figure 1).</p> <p>Graph: Figure 1. Taxonomy of funding-by-lottery.</p> <p>This second graphic represents the same taxonomy, though it excludes the even/uneven variants for the ease of readability, but visualizes it in a distinctive manner that may be more intuitive for some (Figure 2).</p> <p>Graph: Figure 2. Taxonomy of funding-by-lottery.</p> <p>This taxonomy captures much of the variants of funding-by-lottery that have been entertained or implemented as well as introducing some as-of-yet unconsidered variants. It can be extended by adding more variants and refined as the definitions of the existing categories become more precise.</p> <p>Related to this taxonomy is an outline of a decision tree, which provides a series of questions and possible answers at different decision-making stages. Just like the taxonomy, this decision tree is somewhat simplistic and can be made more complex as research develops:</p> <p></p> <ulist> <item> Should we introduce lottery elements into a funding program?</item> <p></p> <item> Should grant applications short or detailed?</item> <p></p> <item> Who screens?</item> <p></p> <item> Staff, experts, stakeholders...</item> <p></p> <item> Screens for what?</item> <p></p> <item> Relevance, research characteristics, quality of researchers, quality...</item> <p></p> <item> How many rounds of screening should there be?</item> <p></p> <item> Should all post-screened applications be admitted to the lottery?</item> <p></p> <item> Partial or non-partial</item> <p></p> <item> If partial, which applications should not be admitted to the lottery?</item> <p></p> <item> Cut-off top, bottom, both...</item> <p></p> <item> Should all applications receive the same number of tickets?</item> <p></p> <item> Weighted or non-weighted</item> <p></p> <item> If weighted, how should the hierarchy be structured?</item> <p></p> <item> Tiered or continuous</item> <p></p> <item> If weighted, how should the distribution of tickets be curved?</item> <p></p> <item> Linear, staggered, exponential...</item> <p></p> <item> If tiered, how many tiers?</item> <p></p> <item> If tiered, should each tier contain the same number of tickets?</item> <p></p> <item> Even or uneven?</item> </ulist> <p>Questions 1–5 focus on questions of whether to use a lottery at all and, if so, how the screening process should proceed. Since the screening process is independent of which variant we choose, this is not represented in the taxonomy proper. Questions 6–12 refer to the selection of the specific lottery variant and are represented on flow chart 1.</p> <hd id="AN0164246306-9">4. Cost and reliability of lottery mechanisms</hd> <p>Roumbanis correctly asserts that 'before a community decides to apply a lottery as a method for making decisions, it is always necessary to consider the whole spectrum of pros and cons in order to agree upon its plausibility' ([<reflink idref="bib90" id="ref48">90</reflink>]: 996). As noted, though, we want to be careful since different variants may have different pros and cons. Moreover, while Roumbanis is optimistic that the appropriate scope of funding-by-lottery includes 'the realm of science' ([<reflink idref="bib90" id="ref49">90</reflink>]), a more cautious approach may suggest that different variants of funding-by-lottery are appropriate in different situations. In this section, I will review the two most referenced metrics, cost and reliability, as they pertain to different variants of funding-by-lottery.</p> <hd id="AN0164246306-10">4.1 Cost</hd> <p>Peer review is incredibly expensive. Most of this cost is not monetary, as administrative costs have steadily decreased since (at least) the 1990s, but opportunity costs of scientists who spend their time reviewing and (especially) writing grants ([<reflink idref="bib49" id="ref50">49</reflink>]). This has been exacerbated by increasingly large labor markets and hyper-competition ([<reflink idref="bib72" id="ref51">72</reflink>]). All commentators, for and against funding-by-lottery, are in favor of cost-effective funding allocation methods.[<reflink idref="bib6" id="ref52">6</reflink>] There are a few key determinants of cost that may be adjusted amongst different variants of funding-by-lottery. One concern is the requirements for the proposal itself. One option, exemplified by the Volkswagen Foundation, is to require minimal proposals that only provide a rough, coarse-grained overview of the proposed research. This decision is not wedded to any particular variant of funding-by-lottery as it can also be incorporated within peer review.[<reflink idref="bib7" id="ref53">7</reflink>] Because of this, despite the fact that it is (contraction in this paragraph) an important determinate of cost, it does not make funding-by-lottery intrinsically cheaper than a modified version of traditional peer review. Another determinate is the amount of peer review required. This suggests that lotteries that employ peer review less (e.g. non-partial non-weighted lotteries) will be cheaper than alternatives. Indeed, costs change dramatically depending on how many tiers are used ([<reflink idref="bib43" id="ref54">43</reflink>]). However, it is intuitive that a lottery that requires less information from the applicant will attract more grantees[<reflink idref="bib8" id="ref55">8</reflink>]—this would increase the total number of grants that would need to be reviewed. While I would suspect that the net costs will still decrease, this is an important consideration for any variant of funding-by-lottery.</p> <hd id="AN0164246306-11">4.2 Reliability and peer review</hd> <p>The cost of traditional peer review may be worth it if it ensured that time, energy, and resources were not being misused on fanciful or misguided research and the highest quality projects received support. But these assurances are unreliable according to critics. Many claim that peer review fails to bottleneck research into promising avenues and cannot reliably distinguish the relative fecundity of research proposals ([<reflink idref="bib23" id="ref56">23</reflink>]; [<reflink idref="bib89" id="ref57">89</reflink>]).[<reflink idref="bib9" id="ref58">9</reflink>] This purported failure of peer review has led some to abandon ship and explore funding-by-lottery as an alternative means of allocating grants. Since we have distinguished between variants of funding-by-lottery, we can now distinguish between the ways different studies on the reliability of peer review lend credence to particular variants over others. Before showing this, I want to briefly go over some limitations on reliability of peer review studies.</p> <p>4.2.1 Limitations of reliability studies</p> <p>I have argued elsewhere that reliability studies do not get us all the way to judgments of the objectivity of peer review ([<reflink idref="bib94" id="ref59">94</reflink>]). Inter-rater reliability studies, as considered below, only give us a local consensus between reviewers about whether a proposal should be accepted or rejected. Local consensuses can be unreliable on a different level if it could be shown that they do not reliably detect research that will be genuinely impactful. Indeed, as some have argued, even if peer review reliably identify the same proposals, it is doubtful that they will reliably pick out the proposals that will demonstrate their fecundity in the long term ([<reflink idref="bib37" id="ref60">37</reflink>]; [<reflink idref="bib93" id="ref61">93</reflink>]).</p> <p>Because of this, some have conducted predictive validity studies that evaluate the extent to which the peer evaluations predict the future impact of the research. Lutz Bornmann, amongst others, argues that reviews are often predicatively valid in the sense that papers that are initially accepted for publication are likely to receive higher citations ([<reflink idref="bib15" id="ref62">15</reflink>]: 221). Some studies on publications in various journals support this hypothesis ([<reflink idref="bib111" id="ref63">111</reflink>]; [<reflink idref="bib77" id="ref64">77</reflink>]; [<reflink idref="bib75" id="ref65">75</reflink>]). However, there are at least three problems with these studies as they pertain to funding practices. First, citation metrics provide a narrow, and often misleading, evaluation of research impact ([<reflink idref="bib48" id="ref66">48</reflink>]). Uptake in policy, popular culture, legal proceedings, curriculum, concrete practices, and so forth reflect research impact but are not captured in citation analysis ([<reflink idref="bib65" id="ref67">65</reflink>]). Second, most papers that are rejected for publication are re-published somewhere else (see e.g. [<reflink idref="bib47" id="ref68">47</reflink>]). The same cannot be said of funding bodies thus leading to a significant selection bias (see also [<reflink idref="bib3" id="ref69">3</reflink>]: 2–3). Third, there remains an outstanding question of how long of a retrospective period is necessary for establishing predictive validity. 'Sleeping beauty' papers are lowly cited for decades, and would score low on predictive validity if measured during this timeframe, but receive massive uptake at later points ([<reflink idref="bib105" id="ref70">105</reflink>]; [<reflink idref="bib57" id="ref71">57</reflink>]; see also [<reflink idref="bib28" id="ref72">28</reflink>]).[<reflink idref="bib10" id="ref73">10</reflink>] This problem has been raised abstractly by Imre Lakatos:</p> <p>An apparently defeated research program may, at some stage in the future, stage a comeback. At that point, the 'verdict of history' would seem to be reversed. How do we know that the hindsight we have the advantage of is <emph>long enough</emph> hindsight? [Genuine hindsight] is revealed only... on the date of Final Judgment, when we are all dead. ([<reflink idref="bib62" id="ref74">62</reflink>]: 139)[<reflink idref="bib11" id="ref75">11</reflink>]</p> <p>This is buttressed by historical literature that shows that many scientific findings that were initially rejected turned out to be significant later on ([<reflink idref="bib33" id="ref76">33</reflink>]; [<reflink idref="bib97" id="ref77">97</reflink>]; [<reflink idref="bib70" id="ref78">70</reflink>]). Even highly established consensus judgments, if followed, can eliminate research that is <emph>eventually</emph> transformative. With respect to funding policy, retrospective studies have been conducted. The American Institute of Biological Sciences found some mild, though highly variable, correlations between reviewer judgments in grant determinations and citation counts ([<reflink idref="bib35" id="ref79">35</reflink>]). Michael Lauer's study of National Institute of Health funding programs for cardiovascular research showed unambiguously that citation counts for lowly rated proposals were equal to highly rated proposals (see [<reflink idref="bib76" id="ref80">76</reflink>]; [<reflink idref="bib64" id="ref81">64</reflink>]; though see [<reflink idref="bib85" id="ref82">85</reflink>]). Reliable <emph>ex-ante</emph> longitudinal studies (past 13 years) are not available, as far as I know, and we can always request studies to take longer timeframes into account.[<reflink idref="bib12" id="ref83">12</reflink>] The idea that the sun was at the center of orbiting planets lied dormant for almost 2,000 years before it was revived by Copernicus. Examples like these behoove us to keep in mind that extremely long timeframes may be needed to properly appreciate the value of a research proposal (see also [<reflink idref="bib10" id="ref84">10</reflink>]).</p> <p>In my meta-study of predictive validity studies, it was revealed that current evidence suggests that peer reviewer judgments for ≤ 5 years are reliable and unreliable past 5 years ([<reflink idref="bib94" id="ref85">94</reflink>]). While this conclusion is extremely tentative, it suggests that inter-rater reliability scores only matter when they are also predictively valid. I argued that this has important implications for when judgments of pursuitworthiness can be made; namely, that they can only realistically be proclaimed in the context of what I called 'urgent science' ([<reflink idref="bib95" id="ref86">95</reflink>]). With this in mind, we will move forward with the literature engaging in inter-rater reliability for now while acknowledging this consideration.[<reflink idref="bib13" id="ref87">13</reflink>]</p> <p>4.2.2 <emph>Bias and peer review</emph></p> <p>With these limitations in mind, it is worth going over different findings about the reliability of peer review and how they relate to variants of funding-by-lottery. The primary class of studies concern bias. Meta-studies identify 25 distinct biases that distort peer review ([<reflink idref="bib79" id="ref88">79</reflink>]; [<reflink idref="bib71" id="ref89">71</reflink>]; [<reflink idref="bib51" id="ref90">51</reflink>]; [<reflink idref="bib113" id="ref91">113</reflink>]; [<reflink idref="bib55" id="ref92">55</reflink>]). Some biases stem from differences in perspectives concerning the research itself. This may come from differing disciplinary backgrounds, how reviewers are situated, or scientific disagreements between reviewers and reviewees. To no one's surprise, reviewers from different disciplinary backgrounds or perspectives within a discipline tend to disagree more often ([<reflink idref="bib108" id="ref93">108</reflink>]; [<reflink idref="bib63" id="ref94">63</reflink>]; [<reflink idref="bib66" id="ref95">66</reflink>]).[<reflink idref="bib14" id="ref96">14</reflink>] This can be especially problematic in cases of heated and deep-seated scientific disagreements, such as the 'String Wars' ([<reflink idref="bib88" id="ref97">88</reflink>]). In these cases, whether one receives funding is determined by whether one gets reviewers who happen to be on board with the presuppositions of the proposal. This privileges disciplinary or scientific perspectives that happen to be more popular and/or powerful in the moment. In [<reflink idref="bib45" id="ref98">45</reflink>] meta-study, they argue that the evidence that peer review discourages interdisciplinary research is unclear.</p> <p>In cases of intradisciplinary disagreement, we can accept or reject proposals that pass or fail <emph>all</emph> reviewers. Peer review can separate research that seems acceptable from multiple perspectives (even if for different reasons) from research that cannot command acceptance from either perspective. This suggests a partial lottery wherein the top and bottom tiers are respectively funded and rejected automatically and introduce lotteries into the middle tiers. This provides some initial reasons to think that partial lotteries will make room for scientific diversity though not research that appears problematic from multiple perspectives. Interdisciplinary cases are more difficult to assess, since research may employ mixed-methods that deviate from the norms of either reviewer and yet still be perfectly respectable research. Thus, interdisciplinary research may benefit from more minimal forms of peer review, such as non-partial and/or non-weighted lotteries.</p> <p>Additionally, there are several biases that discriminate against identifiable groups of scientists. While this is blatant during single-blind review, it is debatable that the situation is sufficiently improved by double-blind review ([<reflink idref="bib26" id="ref99">26</reflink>]). One of the more prominently discussed biases concerns gender. Findings are mixed on this front, with some claiming that women are less likely to receive funding for similar quality proposals while others claim that gender bias is negligible (see [<reflink idref="bib55" id="ref100">55</reflink>]; [<reflink idref="bib15" id="ref101">15</reflink>]: 215ff; [<reflink idref="bib45" id="ref102">45</reflink>]). Other studies focus on the impacts of race, seniority, linguistic conventions, prestige, and other biases ([<reflink idref="bib101" id="ref103">101</reflink>]). This grounds the worry that peer review allows power asymmetries to marginalize identifiable groups. Moreover, it can indirectly stifle innovative approaches (see more below) as marginalized groups are often innovative at greater rates ([<reflink idref="bib50" id="ref104">50</reflink>]).[<reflink idref="bib15" id="ref105">15</reflink>] As a result, some argue, funding-by-lottery is advantageous because it does not give as much room for these biases to stunt scientific diversity and inclusivity and can therefore offer a fairer funding process.</p> <p>Studies about bias and peer review support variants of funding-by-lottery that utilize peer review less. Thus, non-partial or non-weighted lotteries, e.g. would contain less possible bias than partial or weighted lotteries. Moreover, they suggest that the scope of funding-by-lottery can be partially determined by the identity of the applicant. That is to say there is a question of <emph>who</emph> should get to enter lottery funding mechanisms (as mentioned in the decision tree above). If identifiable groups differ in their reliability ratings, then funding-by-lottery may be fairer for particular groups. This has been contested by [<reflink idref="bib67" id="ref106">67</reflink>] who claim that racial biases in preliminary assessments that remain in lotteries still permeate funding decisions—though this study does not provide a differentiated analysis of how prominent racial biases would be in different funding-by-lottery variants. Moreover, some studies suggest that gender and racial biases can be significantly reduced by increasing diversity in the levels of reviewers ([<reflink idref="bib73" id="ref107">73</reflink>]). Perhaps, then, dividing social groups into different funding mechanisms is superfluous and can be addressed through reviewer quotas. Still, it is worth considering whether the scope of funding-by-lottery, in one of its variants, can be determined (even if partially) by the research<emph>ers</emph> rather than the research itself.</p> <p>4.2.3 <emph>Conservatism</emph></p> <p>Other studies focus on the epistemic situation in which a proposal is being judged. Inter-rater reliability scores are typically high when the research is <emph>conservative—</emph>i.e. when the proposal is consistent with a set of standards of evaluation that are likely to be employed by reviewers ([<reflink idref="bib18" id="ref108">18</reflink>]; [<reflink idref="bib90" id="ref109">90</reflink>]: 1005–1006; [<reflink idref="bib98" id="ref110">98</reflink>]; [<reflink idref="bib78" id="ref111">78</reflink>]; [<reflink idref="bib6" id="ref112">6</reflink>]). Since reviewers assess proposals relative to the 'current knowledge boundaries' ([<reflink idref="bib69" id="ref113">69</reflink>]: 51), there is a tendency to see research that departs from these boundaries as lacking in rigor ([<reflink idref="bib66" id="ref114">66</reflink>]). As a result, innovative research typically performs poorly using traditional peer review ([<reflink idref="bib87" id="ref115">87</reflink>]; [<reflink idref="bib41" id="ref116">41</reflink>]; [<reflink idref="bib54" id="ref117">54</reflink>]).[<reflink idref="bib16" id="ref118">16</reflink>] If this is right, we cannot identify what constitutes poor research without discouraging innovative research.[<reflink idref="bib17" id="ref119">17</reflink>] This is part of the reason why the HRC and Volkswagen Foundation utilize funding-by-lottery for innovative research specifically. Conservatism also stunts diversity, but against original rather than marginal research. Genuinely innovative research may fail peer review from multiple perspectives.[<reflink idref="bib18" id="ref120">18</reflink>] This suggests that a non-partial lottery may be more appropriate for innovative research.</p> <p>There may be times when we want to be more conservative. Kuhn famously declared that conservative science is the essence of normal science which comprises the majority of scientific practice ([<reflink idref="bib59" id="ref121">59</reflink>]). While this view is somewhat dated, there may be more limited cases in which conservatism may be welcomed ([<reflink idref="bib92" id="ref122">92</reflink>]; [<reflink idref="bib61" id="ref123">61</reflink>]). In these conservative cases, peer review seems to be reliable enough and can give consistent assessments of research quality. [<reflink idref="bib18" id="ref124">18</reflink>]: 692) similarly claims that 'when referees have a good understanding of the value of certain projects, their report will be final; but on projects that appear to be inventive, and which referees cannot evaluate, we propose that projects will be chosen randomly'. However, as mentioned above, consistency itself is not enough to justify conservatism—especially in the long run ([<reflink idref="bib93" id="ref125">93</reflink>]). However, it provides some initial reasons to maintain traditional forms of peer review evaluations in highly conservative cases.</p> <p>4.2.4 Level of quality</p> <p>Some make far-reaching judgments about the reliability of peer review and do not disambiguate situations where it is more or less reliable ([<reflink idref="bib12" id="ref126">12</reflink>]). There is some evidence to support the contention that peer review is unreliable across the board. In a meta-study of 48 papers on peer review, Bornmann et al. found that levels of agreements of reviewers range from 0.4 to 0.12κ (i.e. 'slight' to negligible levels of agreement) ([<reflink idref="bib16" id="ref127">16</reflink>]; see also [<reflink idref="bib72" id="ref128">72</reflink>]; [<reflink idref="bib55" id="ref129">55</reflink>]). This appears to lend support to non-partial and non-weighted lotteries since we should not rely on peer review to disambiguate which proposals should enter the lottery or, for those that do, how many tickets they should receive. However, others claim that inter-rater depending on the context. Some suggest that peer review is reliable at the level of agreeing which proposals should be <emph>rejected</emph>, but not the converse ([<reflink idref="bib60" id="ref130">60</reflink>]; [<reflink idref="bib21" id="ref131">21</reflink>], [<reflink idref="bib22" id="ref132">22</reflink>]; [<reflink idref="bib108" id="ref133">108</reflink>]: Chapter 6; [<reflink idref="bib37" id="ref134">37</reflink>]). Others, most notably Margit Osterloh, claim that both the <emph>top and bottom grants</emph> can be reliably assessed but not those in the middle ([<reflink idref="bib1" id="ref135">1</reflink>]; see also [<reflink idref="bib58" id="ref136">58</reflink>]). [<reflink idref="bib74" id="ref137">74</reflink>] argue that retrospectively highly rated research would have failed the funding cut-off with a frequency of 9 to 35%, depending on which pair of reviewers had been selected, and that 4 of the top 10 projects identified by ranking had > 50% chance of not being funded. Still others claim that we can reliably detect high quality research but not the converse ([<reflink idref="bib68" id="ref138">68</reflink>]). These studies suggest that there are different <emph>levels of quality</emph> at which we can make reliable judgments. These studies might support different versions of partial or weighted lotteries.</p> <p>4.2.5 Graininess</p> <p>There remains, however, an outstanding gap in the literature on peer review concerning the level of <emph>graininess</emph> at which proposals can be comparatively assessed. The essential question here would be: how fine-grained can we expect peer reviewer judgments to be reliable? Consider marking assignments as an analogy (see [<reflink idref="bib17" id="ref139">17</reflink>]; [<reflink idref="bib3" id="ref140">3</reflink>]). Are judgments about the difference between an 'A' and a 'B' (or 'A' and 'C', etc.) more, less, or equally reliable to judgments about the difference between an 'A' and an 'A+'? Similarly, we may ask whether differences between 'A' and 'B' are the same, larger, or smaller than differences between 'B' and 'C' or, more importantly, 'D' and 'F.' Some of the aforementioned studies can be interpreted as saying that, at a coarse-grained level, we can reliably discern which proposals should be accepted or rejected. But are these judgments about the differences between 'A+'s' and 'F's'? Answers to <emph>this</emph> question are unclear, from what I can tell, and seem indispensable for a detailed sense of what kinds of judgments are reliable.</p> <p>One difference between weighted and non-weighted lotteries as well as between tiered and continuous lotteries relies on reviewers' ability to reliably make fine-grained judgments. In partial non-weighted lotteries, reviewers only need to identify proposals that should not enter the lottery at all. We should expect, therefore, that reviewers can reliably distinguish some category of research. Weighted lotteries require finer-grained judgments to choose which proposals get how many tickets. Clearly, tiered lotteries require less fine-grained judgments than continuous lotteries and tiered lotteries will be more or less fine-grained depending on the number of tiers in the lottery. One criticism of tiered and partial lotteries is grounded in a skepticism of reviewers to be able to identify borderline cases. Barbara [<reflink idref="bib39" id="ref141">39</reflink>] makes this point clearly and emphatically: if we cannot make reliable boundary judgments, then those boundaries should not be employed. These boundary judgments are essential since 'a slight change in score can push a proposal below or above a funding line' ([<reflink idref="bib40" id="ref142">40</reflink>]: 2; see also [<reflink idref="bib27" id="ref143">27</reflink>]; [<reflink idref="bib82" id="ref144">82</reflink>]).[<reflink idref="bib19" id="ref145">19</reflink>] It may be the case that these boundary judgments are simple, affective differences (see [<reflink idref="bib91" id="ref146">91</reflink>]).</p> <hd id="AN0164246306-12">4.3 Summary</hd> <p>In this section, I speculated that overall costs increase depending on the amount of peer review utilized by the variant of funding-by-lottery. I also surveyed different dimensions of the literature on the reliability of peer review. Here is a table summarizing these findings where the left column states the result, the middle column states the variant of funding-by-lottery that this finding lends credence to, while the right column suggests what context the variant may be appropriate within (Table 7).</p> <p>Table 7. Evidence on reliability</p> <p> <ephtml> <table><thead><tr><th>Findings on peer review. </th><th>Variant(s) of funding-by-lottery. </th><th>Appropriate context. </th></tr></thead><tbody><tr><td>Peer review is unreliable <italic>tout court</italic></td><td>Non-partial, non-weighted lotteries </td><td>All contexts </td></tr><tr><td>Peer review is conservative </td><td>Non-partial, non-weighted lotteries </td><td>Innovative research </td></tr><tr><td>Peer review biases against identifiable groups </td><td>Non-partial, non-weighted lotteries </td><td>Marginalized researchers </td></tr><tr><td>Peer review is unreliable for interdisciplinary research </td><td>Non-partial, non-weighted lotteries </td><td>Interdisciplinary research </td></tr><tr><td>Peer review is unreliable for research fields within which there is disagreement </td><td>Partial lotteries that exclude the top and bottom proposals </td><td>Research within contested fields </td></tr><tr><td>Peer review can reliably detect high-quality research </td><td>Partial lotteries that exclude the top proposals </td><td>Transformative research </td></tr><tr><td>Peer review can reliably detect low-quality research </td><td>Partial lotteries that exclude the bottom proposals </td><td>Transformative research </td></tr><tr><td>Peer review can reliably detect high and low-quality research </td><td>Partial lotteries that exclude the top and bottom proposals </td><td>Transformative research </td></tr><tr><td>Peer review is reliable for conservative research </td><td>No lottery </td><td>Conservative research </td></tr></tbody></table> </ephtml> </p> <p>Table 7. Evidence on reliability</p> <p> <ephtml> <table><thead><tr><th>Findings on peer review. </th><th>Variant(s) of funding-by-lottery. </th><th>Appropriate context. </th></tr></thead><tbody><tr><td>Peer review is unreliable <italic>tout court</italic></td><td>Non-partial, non-weighted lotteries </td><td>All contexts </td></tr><tr><td>Peer review is conservative </td><td>Non-partial, non-weighted lotteries </td><td>Innovative research </td></tr><tr><td>Peer review biases against identifiable groups </td><td>Non-partial, non-weighted lotteries </td><td>Marginalized researchers </td></tr><tr><td>Peer review is unreliable for interdisciplinary research </td><td>Non-partial, non-weighted lotteries </td><td>Interdisciplinary research </td></tr><tr><td>Peer review is unreliable for research fields within which there is disagreement </td><td>Partial lotteries that exclude the top and bottom proposals </td><td>Research within contested fields </td></tr><tr><td>Peer review can reliably detect high-quality research </td><td>Partial lotteries that exclude the top proposals </td><td>Transformative research </td></tr><tr><td>Peer review can reliably detect low-quality research </td><td>Partial lotteries that exclude the bottom proposals </td><td>Transformative research </td></tr><tr><td>Peer review can reliably detect high and low-quality research </td><td>Partial lotteries that exclude the top and bottom proposals </td><td>Transformative research </td></tr><tr><td>Peer review is reliable for conservative research </td><td>No lottery </td><td>Conservative research </td></tr></tbody></table> </ephtml> </p> <p>I also pointed to a gap in the literature on peer review concerning the graininess of judgments of peer reviews, which must be conducted before we can make judgments concerning weighted, tiered, or continuous lotteries.</p> <hd id="AN0164246306-13">5. Toward a holistic evaluation of funding-by-lottery</hd> <p>It is easy for policy discussions to become too narrow at the expense of more well-rounded assessments of funding allocation methods. Funding-by-lottery is no exception: cost and reliability are only two dimensions for assessing its viability. [<reflink idref="bib5" id="ref147">5</reflink>], building off of the work of others, took some important steps in this direction. One important metric discussed is fairness.[<reflink idref="bib20" id="ref148">20</reflink>] Fairness has multiple dimensions. One is that funding mechanisms should not discriminate against members of particular groups. Funds should be equally available to all those whose research meets the standards of the call. As mentioned, some believe that fairness (or fairness along some axes) is increased in funding-by-lottery by being more inclusive of particular groups. More recently, [<reflink idref="bib44" id="ref149">44</reflink>] argues that lotteries are fairer than peer review insofar as they distribute funds across grantees more evenly—fairness is thus tethered to a kind of egalitarianism.</p> <p>Another metric [<reflink idref="bib5" id="ref150">5</reflink>] mentions is transparency. This would entail making the structure of the lottery available for the public and making the reasons applications were accepted or rejected clear to applicants. This demand does not seem to be a motivation for all variants of funding-by-lottery, since it can be equally met by traditional peer review. However, non-partial and non-weighted lotteries would allow for increased transparency since there would be no peer reviewers whose identity must be protected. Beginning in the Cold War, peer review came under increased scrutiny for its purported secrecy since those with the power to determine the direction of science via funding decisions could not be publicly identified ([<reflink idref="bib7" id="ref151">7</reflink>]; see also [<reflink idref="bib46" id="ref152">46</reflink>]). This is still a live issue since privacy is needed to protect peer reviewers. With fewer peer reviewers to protect, non-partial or non-weighted lotteries can be incrementally more transparent.</p> <p>In this section, I add two other dimensions to the discussion: trust and social responsibility. While I will not pretend that these dimensions complete the list of metrics for evaluating funding mechanisms, or that any definitive analysis of trust or social responsibility has been provided in such a small amount of space, I hope to take a few steps forward toward a more holistic framework for assessing variants of funding-by-lottery.</p> <hd id="AN0164246306-14">5.1 Trust</hd> <p>It is well-known that trust in scientific institutions is valuable for many reasons, including public acceptance of scientific findings. Trust, for many, is intimately connected to peer review. This was summarized and defended by Jeannette Wing and Ed Chi:</p> <p>In science, peer review matters not just for scientific truth, but, in the broader context, for society's perception of science. Peer review matters for the integrity of science. Scientific integrity is the basis for public trust in us, in our results, in science... When they read in the news that 'Scientists state X', there is an immediate trust that 'X' is true. They know that science uses peer review to vet results before they are published. They trust this process to work. It is important for us, as scientists, not to lose the public trust in science. That's why peer review matters. ([<reflink idref="bib112" id="ref153">112</reflink>]: 10)</p> <p>One worry with funding-by-lottery may be that the gatekeeping role of peer review, regardless of how problematic it may be, is widely used as a means to establish public trust. In hotly politicized debates surrounding issues like climate change, nuclear power, drug efficacy, or vaccination, reporters, journalists, public figures, and public-facing scientists routinely use peer review as a shorthand for establishing the legitimacy (or lack thereof for the illegitimacy) of a claim. While this normally concerns publications, criticism of funding sources is a common way to establish or destroy trust in research funded from that source (see e.g. [<reflink idref="bib24" id="ref154">24</reflink>]). It would not be far from the truth to claim that peer review is the public demarcation method between legitimate and illegitimate science. Given that peer review is deeply institutionally and culturally entrenched, heavy modifications of its practice will surely encounter resistance and confusion leaving public trust as a potential casualty. More specifically, [<reflink idref="bib8" id="ref155">8</reflink>] voices the worry that 'politicians and the public would react negatively, as a lottery might be interpreted as a lack of will to do a thorough assessment'. If we limit the function of peer review, might public trust be damaged? It is easy to imagine this possibility, where 'cranks and dabblers', to borrow [<reflink idref="bib84" id="ref156">84</reflink>]: 62) apt phrase, have no clear disambiguating mark.[<reflink idref="bib21" id="ref157">21</reflink>] It is possible, though, that funding-by-lottery may have the opposite effect, at least in the long term, and increase trust in science. Funding-by-lottery has the capability of being more inclusive. Many argue that trust and inclusivity come hand-in-hand since people trust science that they, or their perceived representatives, had a legitimate voice in producing (see [<reflink idref="bib56" id="ref158">56</reflink>]; [<reflink idref="bib13" id="ref159">13</reflink>]; [<reflink idref="bib38" id="ref160">38</reflink>]).</p> <p>There is also the trust that scientists may place in their granting institutions if funding-by-lottery becomes more widespread. Some preliminary findings summarized in [<reflink idref="bib86" id="ref161">86</reflink>] suggest that scientists, especially younger scientists, are skeptical of lotteries since it seems to run against their self-understanding of their trade where grants are awarded on the basis of merit. [<reflink idref="bib80" id="ref162">80</reflink>] reports that prestigious and acclaimed scholars are open to lotteries so long as it is combined with peer review which, as we have seen, is what proponents of lotteries support anyways. Similarly, as reported by [<reflink idref="bib55" id="ref163">55</reflink>] and [<reflink idref="bib45" id="ref164">45</reflink>], peer review is a well-trusted mechanism by scientists despite its <emph>de facto</emph> problems. It is not clear whether this attitude would change over time, whether it is the result of the improper framing previous discussed—where funding-by-lottery is seen as the opposite of peer review—or whether the increased distribution of funds may overall increase scientists trust in their funding institutions. Moreover, it is not clear whether the meritocratic self-image of scientists is one that should be upheld. Again, I have more questions than answers at this stage—but it surely requires consideration, nonetheless.</p> <p>Finally, it is possible that funding-by-lottery may increase trust by removing incentives for scientists to promise more than they can deliver. It is well-known that scientists are pressured to employ a range of rhetorical techniques, including hyperbole and exaggeration, while writing their grants. Funded grants are often publicly displayed, filled with incredible assurances and prospects. This is surely increased by hyper-competition. It is conceivable that the public may gradually lose trust in science if these promises, most of which are not obtained, continue. This argument was made long ago by Harvey Brooks:</p> <p>If the cost of being honest about one's ignorance of the potential benefits of one's research is the total loss of support, then a kind of Gresham's Law will set in which rewards those scientists most willing to stretch the bounds of intellectual honesty in explaining the benefits of their research, or in minimizing its possible adverse consequences or uses. This is not a healthy situation, and in the long run can only stimulate mistrust between scientists and the public. ([<reflink idref="bib19" id="ref165">19</reflink>]: 184)</p> <p>Brooks supports this with examples of research on possible consequences of research on recombinant DNA and nuclear energy, where speculations about possibilities disappointed the public who became less eager to support scientists and engineers in these areas. Such public pressure can and often does lead to changes in funding priorities, even if those speculations were really just tactical moves to get funding. Since funding-by-lottery decreases the competitive need for making these promises, it may remove an obstacle for establishing public trust.[<reflink idref="bib22" id="ref166">22</reflink>]</p> <hd id="AN0164246306-15">5.2 Social responsibility</hd> <p>Social responsibility concerns the ability of a funding mechanism to direct research toward socially desirable ends. Some commentators have argued that publicly funded scientific institutions should pursue knowledge that is shaped by and attentive to the desires and values of the people (or, at least, stakeholders). Peer review has historically been tied to the opposite view, that science should be autonomous from society and dictate its own agendas internally ([<reflink idref="bib84" id="ref167">84</reflink>]; [<reflink idref="bib20" id="ref168">20</reflink>]; [<reflink idref="bib110" id="ref169">110</reflink>]; [<reflink idref="bib7" id="ref170">7</reflink>]). Most accept a more moderate view: sometimes scientists should pursue science on its own terms while other times they should be on the frontlines of societal issues ([<reflink idref="bib19" id="ref171">19</reflink>]). Even 'basic science' institutions, like the NSF or the ERC, have a portion of their funds devoted to socially relevant research. Given this, we must ask how funding-by-lottery may help or impede funding socially responsible science. One straightforward way to ensure that science is accountable to the public is to include social impact metrics into the initial screening process: only projects that seek to answer specified questions, extend our knowledge of particular topics, employ specific methods, or meet demographic quotas can be included in the lottery. Another way may be to include stakeholders in the review process itself. Funding-by-lottery, it seems, has an especially wonderful opportunity to include stakeholders or the general public in screening decisions (see [<reflink idref="bib4" id="ref172">4</reflink>]: 18–19). The primary difficulty funding bodies face with including laypeople in funding decisions is that the learning curve for assessing proposals is too steep and requires too much deliberation leading to lagging funding decisions (see [<reflink idref="bib104" id="ref173">104</reflink>]). The cost and time lag of peer review is already inordinately high (see [<reflink idref="bib45" id="ref174">45</reflink>]) and public participation would certainly amplify this. But if the proposals are pitched at a coarse-grained enough level, they are much easier to review. This may facilitate incorporating the general public into the peer review process. This allows for greater inclusivity, trust, and social responsibility for establishing research priorities.[<reflink idref="bib23" id="ref175">23</reflink>]</p> <p>As we read above, the Volkswagen Foundation uses staff to screen which proposals enter the lottery while the HRC uses 'experts'. These are not the only options involved—stakeholders could be included in the screening process as well (as mentioned in the decision tree from Section 3). Obviously, there would be many questions that would arise from this suggestion. Would the proposals have to be more detailed in describing their methods or possible impacts? Which kinds of proposals would citizens be more able and/or willing to review? Would stakeholder decisions modify the amount of tickets, what enters the lottery, or both? Is the inclusion of stakeholders more apt for some variants of funding-by-lottery than others? Regardless of how these investigations turn out, we should be keen to consider another option at an early decision-making stage in our choice of who gets to screen grant proposals.</p> <p>But there is a worry that pushes in the opposite direction. The likelihood that large amounts of grant proposals will line up in their aims and methods with social desiderata seems slim. It could be the case that funding-by-lottery crowds out societally preferred research. This is especially the case if funding-by-lottery incentivizes research that is highly contentious—which may be valuable in the long term but mostly likely not helpful for pressing problems. It is also conceivable that members of the public will be more conservative than peer reviewers and disincentivize innovative solutions. Again, I have only questions and no definitive answers here. But as we further the discussion of funding-by-lottery, we must think about how funding-by-lottery can facilitate socially responsible science.</p> <hd id="AN0164246306-16">5.3 Summary</hd> <p>Of course, these considerations, and others, can crosscut which may lead to conflicts or happy coincidences in our motivations to adopt or reject particular variants of funding-by-lottery in a specific context. Careful reflection upon how to balance these considerations is sorely needed for well-justified concrete decisions. Moreover, I do not mean to suggest that these considerations are exhaustive. Anyone familiar with science funding policy is aware of the diversity and complexity of the considerations that go into choosing allocation methods. I hope readers will find steppingstones in this section for more nuanced and holistic metrics of assessment. To summarize the findings of the past two sections, the following graphic outlines the various motivations for adopting funding-by-lottery, while keeping in mind that these motivations may differ depending on which variant we are interested in (Figure 3).</p> <p>Graph: Figure 3. Motivations for funding-by-lottery.</p> <p>As discussed, there are reasons to be skeptical of funding-by-lottery on some of these points as well. Further research is needed to provide a more holistic assessment of the motivations for and against funding-by-lottery, with careful attention to how these assessments may differ amongst different variants.</p> <hd id="AN0164246306-17">6. Concluding remarks</hd> <p>Evaluating research is a difficult task and we should not pretend that blanket statements for or against peer review cloud what scholars already know: there are a variety of strengths and weaknesses of peer review that are exemplified in different contexts entailing a variety of ways lotteries can improve science funding policy. Since we do not know everything that we must regarding the specifics of peer review, there is a certain amount of guessing that must be done until further research is completed. This motivates the plea for experimentation in funding mechanisms. If we are to experiment with funding-by-lottery, we do not have to have all the details or evidence to implement it. I certainly do not want to stand in the way of this plea by demanding a complete assessment of the variants of funding-by-lottery before implementation, if this was even possible, but it is still worth gathering a strong evidentiary basis for clear-cut proposals as we continue to elaborate what different forms funding-by-lottery may take. As Freeman Dyson once remarked, '[w]hen the great innovation appears, it will almost certainly be in a muddled, incomplete, and confusing form' ([<reflink idref="bib29" id="ref176">29</reflink>]: 79–80). This comment can certainly be applied to funding-by-lottery. Working out all the kinks and bugs with any funding mechanism will take a great deal of effort, patience, and hope. Moreover, lotteries have been used or suggested for a wide variety of purposes and there may be helpful lessons for funding policy therein (see [<reflink idref="bib100" id="ref177">100</reflink>]). It is my contention that this article contains some steps in this direction for better understanding and evaluating funding-by-lottery as a means of allocating funds for scientific communities. As such, it provides a more detailed assessment of the ways in which peer review may be used to evaluate research.</p> <p>Acknowledgements</p> <p>I deeply appreciate the feedback received by Kevin Gross and Hakob Barseghyan, who was also extremely helpful in designing the visualizations. I also appreciate the detailed comments from two remarkably astute reviewers. Derek Liston read over the manuscript for grammar and word-choice.</p> <p> <emph>Conflict of interest statement</emph>. None declared.</p> <ref id="AN0164246306-18"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref5" type="bt">1</bibl> <bibtext> The Kuhnian story breaks down at two points. Specifically, novel paradigms are not incommensurable, at least in Kuhn's technical sense, and emerging alternatives do not vie to be the future dominant paradigm.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref18" type="bt">2</bibl> <bibtext> The Volkswagen Foundation discontinued its use of lotteries in 2021.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref35" type="bt">3</bibl> <bibtext> The literature on peer review does not distinguish between 'screening' peer review as and more detailed peer review. Perhaps this is justifiable if the reliability levels would be the same, but this important question requires further study.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref36" type="bt">4</bibl> <bibtext> Two funding programs in New Zealand, the Science for Technological Innovation challenge seed grants and New Zealand Health Research Council explorer grants, use 'experts' to conduct this screening review (Avin 2018 : 20–21).</bibtext> </blist> <blist> <bibl id="bib5" idref="ref16" type="bt">5</bibl> <bibtext> https://<ulink href="http://www.volkswagenstiftung.de/en/funding/our-funding-portfolio-at-a-glance/experiment">www.volkswagenstiftung.de/en/funding/our-funding-portfolio-at-a-glance/experiment</ulink></bibtext> </blist> <blist> <bibl id="bib6" idref="ref52" type="bt">6</bibl> <bibtext> Some critics of funding-by-lottery claim that its main (or perhaps only) virtue is that it is relatively cheap (e.g. Beattie 2020).</bibtext> </blist> <blist> <bibl id="bib7" idref="ref53" type="bt">7</bibl> <bibtext> The Templeton Foundation, e.g. uses a minimal coarse-grain grant application as a first step of some of their grants (see step 4, https://<ulink href="http://www.templeton.org/grants/apply-for-grant">www.templeton.org/grants/apply-for-grant</ulink>).</bibtext> </blist> <blist> <bibl id="bib8" idref="ref27" type="bt">8</bibl> <bibtext> This prediction is shared by scientists interviewed about lotteries in funding (see Philipps 2021 : 107). Some scientists disagree. One dissenting scientist, e.g. felt disincentivized to apply for funding by lottery because they worry it would diminish the quality of their research (108). von Hippel and von Hippel (2015) estimate that roughly half of active researchers in astronomy and physiology do not apply for grants due to the costs involved.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref45" type="bt">9</bibl> <bibtext> Before this, many studies suggested that while there was variance amongst disciplines, rates of agreement were high (at least for the natural sciences). See Zuckerman and Merton (1971 : fn. 3 67) and the citations contained therein. Zuckerman and Merton's study is more speculative, as it infers rates of consensus from rejection rates and journal space and is exclusively focused on manuscript reviews.</bibtext> </blist> <blist> <bibtext> Dynamic models of representing citation counts are equally unfit for providing <emph>predictions</emph> concerning future citation changes as trend analyses remain underdetermined. Moreover, the increasing visibility of old manuscripts and archives and historical self-awareness within many disciplines will likely exacerbate sleeping beauty papers.</bibtext> </blist> <blist> <bibtext> This issue is on display in the debate between the TRACEs project and Project Hindsight, where assessing the importance of 'basic science' for practical research morphed into debates about the proper timeframe to consider (see Pirtle and Moore 2019).</bibtext> </blist> <blist> <bibtext> See Wang, Song and Barabási (2013) and Stegehuis, Litvak and Waltman (2015) for admirable efforts, though neither provide <emph>ex-ante</emph> criteria of recognizing future impact.</bibtext> </blist> <blist> <bibtext> This is not to suggest that inter-rater reliability studies are without fault. Like many sciences, peer review studies suffer from worries about replicability. Moreover, peer review practices are sensitive to exogenous changes in scientific practice such as digitization, hyper competition, and blurred disciplinary and institutional boundaries which may lead to the increasing obsoleteness of older studies (see Whitley and Gläser 2007). Moreover, there are methodological challenges concerning the kappa coefficient which need to be resolved before the security of the criticisms of peer review can be adequately grounded (see Bornmann 2011 and citations contained therein).</bibtext> </blist> <blist> <bibtext> Attempts to combat this by increasing deliberation amongst disagreeing peer reviewers, perhaps counterintuitively, exacerbates disagreement (Fogelholm et al. 2012 ; see also Zollman 2012).</bibtext> </blist> <blist> <bibtext> AlShebli, Rahwan and Woon (2018) argue that ethnic diversity highly correlates with high impact publications.</bibtext> </blist> <blist> <bibtext> A survey of 288 NIH reviewers for the 'Director's Pioneer Award Program' suggested that 'the most innovative projects involved methodological risk' (Lee 2015 : 1277).</bibtext> </blist> <blist> <bibtext> Ismail, Farrands and Wooding's (2009) meta-study claims that the evidence that peer review is unduly conservation is unclear (2/5). A more recent report from Guthrie, Ghiga and Wooding (2018) claim the evidence that peer review disincentivizes innovation research is suggestive, but not 'compelling'.</bibtext> </blist> <blist> <bibtext> This has led to some changes in the manner of presenting grant proposals (Philipps and Weißenborn 2019).</bibtext> </blist> <blist> <bibtext> This discussion is slightly oversimplified. Intuitively, a <emph>precise</emph> boundary is an unnecessarily high burden for tiered lotteries. Rather, we may have boundary that occupies a relatively small gray area. This means that tiered lotteries will introduce unreliable judgments, but these will only affect a small number of proposals.</bibtext> </blist> <blist> <bibtext> Specifically, Avin lists four criteria for evaluating a funding mechanism: efficiency, effectiveness, accountability, and fairness. The first three, to some extent, are contained within my discussions of cost and reliability.</bibtext> </blist> <blist> <bibtext> One advantage of partial lotteries, exploited by the Volkswagen Foundation, is that they can conceal who won grants via lottery (though this has caused some minor frustrations of scientists (Philipps 2021 : 108)).</bibtext> </blist> <blist> <bibtext> I would like to thank an anonymous reviewer for bringing this point to my attention.</bibtext> </blist> <blist> <bibtext> The RQ+ metric, developed by the International Development and Research Council, provides an exemplar that could be adapted to funding-by-lottery where stakeholders have a say in funding decisions.</bibtext> </blist> </ref> <ref id="AN0164246306-19"> <title> REFERENCES </title> <blist> <bibtext> Adam D. (2019) ' Science Funders Gamble on Grant Lotteries ', Nature, 575 : 574 – 5. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> AlShebli B., Rahwan T., Woon W. (2018) ' The Preeminence of Ethnic Diversity in Scientific Collaboration ', Nature Communications, 9 : 1 – 10. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Avin S. (2015) 'Funding Science by Lottery', in Mäki, U., Votsis, I., Ruphy, S. and Schurz, G. (eds). Recent Developments in the Philosophy of Science: EPSA13 Helsinki, pp. 111 – 26. Cham : Springer. Google Scholar Crossref Search ADS Google Preview WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Avin S. (2018) ' Policy Considerations for Random Allocation of Research Funds ', RT. A Journal on Research Policy and Evaluation, 6. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Avin S. (2019) ' Mavericks and Lotteries ', Studies in History and Philosophy of Science Part A, 76 : 13 – 23. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Ayoubi C., Pezzoni M., Visentin F. (2021) ' Does It Pay to Do Novel Science? The Selectivity Patterns in Science Funding ', Science and Public Policy, 48 : 635 – 48. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Baldwin M. (2018) ' Scientific Autonomy, Public Accountability, and the Rise of "Peer Review" in the Cold War United States ', ISIS, 109 : 538 – 58. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Barnett A. (2016) ' Funding by Lottery: Political Problems and Research Opportunities ', mBio, 7 : e01369 – 16. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Barnett A., Herbert D., Clarke P., Graves N. (2014) ' The Research Lottery: The Pressures on the Australian Grant System ', AQ-Australian Quarterly, 85 : 4 – 9. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Bartley W. (1990). Unfathomed Knowledge, Unmeasurable Wealth: On Universities and the Wealth of Nations. La Salle : Open Court Publishing. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Beattie A. (2020) ' Grant Lottery is Bureaucratic Short-Cut ', Nature, 577 : 472. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Bedeian A. (2004) ' Peer Review and the Social Construction of Knowledge in the Management Discipline ', Academy of Management Learning and Education, 3 : 198 – 216. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Bickerstaff K., Lorenzoni I., Jones M., Pidgeon N. (2010) ' Locating Scientific Citizenship: The Institutional Contexts and Cultures of Public Engagement ', Science, Technology, & Human Values, 35 : 474 – 500. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Bollen J., Crandall D., Junk D., Ding Y., Börner K. (2017) ' An Efficient System to Fund Science: From Proposal Review to Peer-to-Peer Distributions ', Scientometrics, 110 : 521 – 8. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Bornmann L. (2011) ' Scientific Peer Review ', Annual Review of Information Science and Technology, 45 : 197 – 245. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Bornmann L., Mutz R., Daniel H. (2010) ' A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants ', PLoS One, 5 : e14331. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Boyle C. (1998) ' Organizations Selecting People: How the Process Could Be Made Fairer by the Appropriate Use of Lotteries ', Journal of the Royal Statistical Society: Series D (The Statistician), 47 : 291 – 321. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Brezis E. (2007) ' Focal Randomisation: An Optimal Mechanism for the Evaluation of R&D Projects ', Science and Public Policy, 34 : 691 – 8. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Brooks H. (1978) ' The Problem of Research Priorities ', Daedalus, 107 ; 171 – 90. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Chubin D., Hackett E., Hackett E. (1990) Peerless Science: Peer Review and US Science Policy. New York : SUNY Press. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Cicchetti D. (1991) ' The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation ', Behavioral and Brain Sciences, 14 : 119 – 35. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Cicchetti D. (1997) ' Referees, Editors, and Publication Practices: Improving the Reliability and Usefulness of the Peer Review System ', Science and Engineering Ethics, 3 : 51 – 62. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Cole S., Cole J., Simon G. (1981) ' Chance and Consensus in Peer-Review ', Science, 214 : 881 – 6. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Critchley C., Nicol D. (2011) ' Understanding the Impact of Commercialization on Public Support for Scientific Research: Is It about the Funding Source or the Organization Conducting the Research? ', Public Understanding of Science, 20 : 347 – 66. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Csiszar A. (2016) ' Peer Review: Troubled from the Start ', Nature News, 532 : 306 – 8. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Darling E. (2015) ' Use of Double‐Blind Peer Review to Increase Author Diversity ', Conservation Biology, 29 : 297 – 9. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Day T. (2015) ' The Big Consequences of Small Biases: A Simulation of Peer Review ', Research Policy, 44 : 1266 – 70. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Dinges, M. (2005) The Austrian Science Fund: Ex Post Evaluation and Performance of FWF Funded Research Projects. Vienna Institute of Technology and Regional Policy.</bibtext> </blist> <blist> <bibtext> Dyson F. (1958 /1998) 'Innovation in Physics', in Hsu J. (ed.) JingShin Theoretical Physics Symposium in Honor of Professor Ta-You Wu, pp. 73 – 90. Singapore : World Scientific. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Elster J. (1989) Solomonic Judgements: Studies in the Limitation of Rationality. Cambridge : Cambridge University Press. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> England J. M. (1982). A Patron for Pure Science: The National Science Foundation's Formative Years, 1945– 57. Washington : National Science Foundation. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Fang F., Casadevall A. (2016) ' Research Funding: The Case for a Modified Lottery ', mBio, 7 : 1 – 7. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Feyerabend, P. (1975) Against Method. London: Verso.</bibtext> </blist> <blist> <bibtext> Fogelholm M., Leppinen S., Auvinen A., Raitanen J., Nuutinen A., Väänänen K. (2012) ' Panel Discussion Does Not Improve Reliability of Peer Review for Medical Research Grant Proposals ', Journal of Clinical Epidemiology, 65 : 47 – 52. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Gallo S., Carpenter A., Irwin D., McPartland C., Travis J., Reynders S., Thompson L., Glisson S. (2014) ' The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies ', PLoS One, 9 : e106474. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Gieryn T. (1983) ' Boundary-Work and the Demarcation of Science from Non-Science: Strains and Interests in Professional Ideologies of Scientists ', American Sociological Review, 48 : 781 – 95. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Gillies D. (2014) ' Selecting Applications for Funding: Why Random Choice Is Better than Peer Review ', RT. A Journal on Research Policy and Evaluation, 2 : 1 – 14. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Goldenberg M. (2021) Vaccine Hesitancy: Public Trust, Expertise, and the War on Science. Pittsburgh : University of Pittsburgh Press. Google Scholar Crossref Search ADS Google Preview WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Goodwin B. (2005) Justice by Lottery, 2nd edn. Exeter : Imprint Academic. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Graves N., Barnett A., Clarke P. (2011) ' Funding Grant Proposals for Scientific Research: Retrospective Analysis of Scores by Members of Grant Review Panel ', British Medical Journal, 343 : d4797. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Grayson L. (2002) Evidence Based Policy and the Quality of Evidence: Rethinking Peer Review. London : Centre for Evidence Based Policy and Practice (ESRC). Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Greenberg D. (1998) ' Chance and Grants ', The Lancet, 351 : 686. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Gross K., Bergstrom C. (2019) ' Contest Models Highlight Inherent Inefficiencies of Scientific Funding Competitions ', PLoS Biology, 17 : e3000065. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Gildenhuys P. (2020) ' Lotteries Make Science Fairer ', Journal of Responsible Innovation, 7 : S30 – 43. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Guthrie S., Ghiga I., Wooding S. (2018) ' What Do We Know about Grant Peer Review in the Health Sciences? ', F1000Research, 6 : 1335. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Guthrie S., Guerin B., Wu H., Ismail S., Wooding S. (2013) 'Alternatives to Peer Review in Research Project Funding ', RAND Europe. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Hall S., Wilcox A. (2007) ' The Fate of Epidemiologic Manuscripts: A Study of Papers Submitted to Epidemiology ', Epidemiology, 18 : 262 – 5. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Haustein S., Larivière V. (2015) 'The Use of Bibliometrics for Assessing Research: Possibilities, Limitations and Adverse Effects', in Welpe I., Wollersheim J., Ringelhan S., Osterloh M. (eds.). Incentives and Performance, pp. 121 – 139. Cham : Springer. Google Scholar Crossref Search ADS Google Preview WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Herbert, D. L., Barnett, A. G., Clarke, P., and Graves, N. (2013). ' On the Time Spent Preparing Grant Proposals: An Observational Study of Australian Researchers ', BMJ open, 3(5), e002800.</bibtext> </blist> <blist> <bibtext> Hofstra B., Kulkarni V., Galvez S., He B., Jurafsky D., McFarland D. (2020) ' The Diversity–Innovation Paradox in Science ', Proceedings of the National Academy of Sciences, 117 : 9284 – 91. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Hojat M., Gonnella J., Caelleigh A. (2003) ' Impartial Judgment by the "Gatekeepers" of Science: Fallibility and Accountability in the Peer Review Process ', Advances in Health Sciences Education, 8 : 75 – 96. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Horbach S. P. (2021) ' No Time for That Now! Qualitative Changes in Manuscript Peer Review during the Covid-19 Pandemic ', Research Evaluation, 30 : 231 – 9. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> HRC. (2014) HRC Explorer Grant Application Guidelines (EX215). 'Technical Report', RAND Europe.</bibtext> </blist> <blist> <bibtext> Huutoniemi K. (2012) ' Communicating and Compromising on Disciplinary Expertise in the Peer Review of Research Proposals ', Social Studies of Science, 42 : 897 – 2. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Ismail S., Farrands A., Wooding S. (2009). Evaluating Grant Peer Review in the Health Sciences – A Review of the Literature. Santa Monica : RAND Corporation. Google Scholar Crossref Search ADS Google Preview WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Jasanoff S. (2005). 'Technologies of Humility: Citizen Participation in Governing Science', in Bogner, A. (ed.). Wozu Experten ?, pp. 370 – 89. VS Verlag für Sozialwissenschaften. Google Scholar Crossref Search ADS Google Preview WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Ke Q., Ferrara E., Radicchi F., Flammini A. (2015) ' Defining and Identifying Sleeping Beauties in Science ', Proceedings of the National Academy of Sciences, 112 : 7426 – 31. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Kostoff R. (1995) ' Federal Research Impact Assessment: Axioms, Approaches, Applications ', Scientometrics, 34 : 163 – 206. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Kuhn T. (1962). The Structure of Scientific Revolutions. Chicago : University of Chicago Press. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Kupfersmid J. (1988) ' Improving What Is Published: A Model in Search of an Editor ', American Psychologist, 43 : 635 – 42. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Kummerfeld E., Zollman K. (2016) ' Conservatism and the Scientific State of Nature ', The British Journal for the Philosophy of Science, 67 : 1057 – 76. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Lakatos I. (1976) ' Understanding Toulmin ', Minerva, 14 : 126 – 43. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Langfeldt L. (2006) ' The Policy Challenges of Peer Review: Managing Bias, Conflict of Interests and Interdisciplinary Assessments ', Research Evaluation, 15 : 31 – 41. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Lauer M. S., Danthi N. S., Kaltman J., Wu C. (2015) ' Predicting Productivity Returns on Investment: Thirty Years of Peer Review, Grant Funding, and Publication of Highly Cited Papers at the National Heart, Lung, and Blood Institute ', Circulation Research, 117 : 239 – 43. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Lebel J., McLean R. (2020). 'Research Quality Plus: Another Way Is Possible', in McLean R., Tijssen R., Wallace M., Kraemer-Mbula E. (eds) Transforming Research Excellence: New Ideas from the Global South, pp. 248 – 58. South Africa : African Minds Publishers. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Lee C. (2015) ' Commensuration Bias in Peer Review ', Philosophy of Science, 82 : 1272 – 83. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Lee C., Grant S., Erosheva E. (2020) ' Alternative Grant Models Might Perpetuate Black–White Funding Gaps ', The Lancet, 396 : 955 – 6. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Li D., Agha L. (2015) ' Big Names or Big Ideas: Do Peer-Review Panels Select the Best Science Proposals? ', Science, 348 : 434 – 8. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Luukkonen T. (2012) ' Conservatism and risk-taking in peer review: emerging ERC practices ', Research Evaluation, 21 : 48 – 60. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Lynch W. T. (2020). Minority Report: Dissent and Diversity in Science. Rowman & Littlefield Publishers. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Martin B. (2000) ' Research Grants: Problems and Options ', Australian Universities' Review, 43 : 17 – 22. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Marsh H., Jayasinghe U., Bond N. (2008) ' Improving the Peer-Review Process for Grant Applications: Reliability, Validity, Bias, and Generalizability ', American Psychologist, 63 : 160 – 8. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Marsh H., Jayasinghe U., Bond N. (2011) ' Gender Differences in Peer Reviews of Grant Applications: A Substantive-Methodological Synergy in Support of the Null Hypothesis Model ', Journal of Informetrics, 5 : 167 – 80. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Mayo N., Brophy J., Goldberg M., Klein M., Miller S., Platt R., Ritchie J. (2006) ' Peering at Peer Review Revealed High Degree of Chance Associated with Funding of Grant Applications ', Journal of Clinical Epidemiology, 59 : 842 – 8. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> McDonald R., Cloft H., Kallmes D. (2009) ' Fate of Manuscripts Previously Rejected by the American Journal of Neuroradiology: A Follow-Up Analysis ', American Journal of Neuroradiology, 30 : 253 – 6. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Mervis J. (2014) ' Peering into Peer Review ', Science, 343 : 596 – 8. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Opthof T., Furstner F., van Geer M., Coronel R. (2000) ' Regrets or No Regrets? No Regrets! The Fate of Rejected Manuscripts ', Cardiovascular Research, 45 : 255 – 8. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Osterloh M., Frey B. (2020) ' How to Avoid Borrowed Plumes in Academia ', Research Policy, 49 : 103831. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Owen R. (1982) ' Reader Bias ', Journal of the American Medical Association, 247 : 2533 – 4. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Philipps A. (2021) ' Science Rules! A Qualitative Study of Scientists' Approaches to Grant Lottery ', Research Evaluation, 30 : 102 – 11. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Philipps A., Weißenborn L. (2019) ' Unconventional Ideas Conventionally Arranged: A Study of Grant Proposals for Exceptional Research ', Social Studies of Science, 49 : 884 – 97. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Pier E., Brauer M., Filut A., Kaatz A., Raclaw J., Nathan M., Ford C., Molly C. (2018) ' Low Agreement among Reviewers Evaluating the Same NIH Grant Applications ', Proceedings of the National Academy of Sciences of the United States of America, 115 : 2952 – 7. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Pirtle Z., Moore J. (2019) ' Where Does Innovation Come from?: Project Hindsight, TRACEs, and What Structured Case Studies Can Say about Innovation ', IEEE Technology and Society Magazine, 38 : 56 – 67. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Polanyi M. (1962) ' The Republic of Science: Its Political and Economic Theory ', Minerva, 38 : 1 – 21. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Reinhart M. (2009) ' Peer Review of Grant Applications in Biology and Medicine: Reliability, Fairness, and Validity ', Scientometrics, 81 : 789 – 809. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Reinhart M., Schendzielorz C. (2020) ' The Lottery in Babylon—On the Role of Chance in Scientific Success ', Journal of Responsible Innovation, 7 : S25 – 9. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Resch K., Ernst E., Garrow J. (2000) ' A Randomized Controlled Study of Reviewer Bias against an Unconventional Therapy ', Journal of the Royal Society of Medicine, 93 : 164 – 7. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Ritson S. (2016) ' Crackpots' and 'Active Researchers': The Controversy over Links between arXiv and the Scientific Blogosphere ', Social Studies of Science, 46 : 607 – 28. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Rothwell P., Martyn C. (2000) ' Reproducibility of Peer Review in Clinical Neuroscience: Is Agreement between Reviewers Any Greater than Would Be Expected by Chance Alone? ', Brain, 123 : 1964 – 9. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Roumbanis L. (2019) ' Peer Review or Lottery? A Critical Analysis of Two Different Forms of Decision-Making Mechanisms for Allocation of Research Grants ', Science, Technology, & Human Values, 44 : 994 – 1019. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Roumbanis L. (2021) ' Disagreement and Agonistic Chance in Peer Review ', Science, Technology, & Human Values, DOI: 01622439211026016. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Rowbottom D. (2013) ' Kuhn vs. Popper on Criticism and Dogmatism in Science, Part II: How to Strike the Balance ', Studies in History and Philosophy of Science Part A, 44 : 161 – 8. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Shaw J. (2021) ' Feyerabend's Well-Ordered Science: How an Anarchist Distributes Funds ', Synthese, 198 : 419 – 49. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Shaw J. (2022a) ' On the Very Idea of Pursuitworthiness ', Studies in History and Philosophy of Science, 91 : 103 – 12. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Shaw J. (2022b) ' Revisiting the Basic/Applied Science Distinction: The Significance of Urgent Science for Science Funding Policy ', Journal for General Philosophy of Science. https://link.springer.com/article/10.1007/s10838-021-09575-1. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Solovey M. (2020) Social Science for What?: Battles over Public Funding for the Other Sciences at the National Science Foundation. Cambridge : MIT Press. Google Scholar Crossref Search ADS Google Preview WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Stanford P. 2006. Exceeding Our Grasp: Science, History, and the Problem of Unconceived Alternatives. Oxford : Oxford University Press. Google Scholar Crossref Search ADS Google Preview WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Stanford P. (2019) ' Unconceived Alternatives and Conservatism in Science: The Impact of Professionalization ', Peer-Review, and Big Science. Synthese, 196 : 3915 – 32. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Stegehuis C., Litvak N., Waltman L. (2015) ' Predicting the Long-Term Citation Impact of Recent Publications ', Journal of Informetrics, 9 : 642 – 57. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Stone P. (2011). The Luck of the Draw: The Role of Lotteries in Decision Making. Oxford : Oxford University Press. Google Scholar Crossref Search ADS Google Preview WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Szell M., Sinatra R. (2015) ' Research Funding Goes to Rich Clubs ', Proceedings of the National Academy of Sciences, 112 : 14749 – 50. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Tilghman S., Bailey J., Bourguinon J.-P. (2021). ' Powering Discovery '. https://cca-reports.ca/reports/international-practices-for-funding-natural-science-and-engineering-research//. accessed 22 January 2022.</bibtext> </blist> <blist> <bibtext> Uhlmann E., Ebersole C., Chartier C., Errington T., Kidwell M., Lai C., McCarthy R., Riegelman A., Silberzahn R., Nosek B. (2019) ' Scientific Utopia III: Crowdsourcing Science ', Perspectives on Psychological Science, 14 : 711 – 33. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Van Bekkum J., Hilton S. (2014) ' UK Research Funding Bodies' Views towards Public Participation in Health-Related Research Decisions: An Exploratory Study ', BMC Health Services Research, 14 : 1 – 10. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Van Raan A. (2004) ' Sleeping Beauties in Science ', Scientometrics, 59 : 467 – 72. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> von Hippel T., von Hippel C. (2015) ' To Apply or Not to Apply: A Survey Analysis of Grant Writing Costs and Benefits ', PLoS One, 10 : e0118494. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Wang D., Song C., Barabási A. (2013) ' Quantifying Long-Term Scientific Impact ', Science, 342 : 127 – 32. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Weller A. (2002). Editorial Peer Review: Its Strengths and Weaknesses. Medford, NJ : Information Today, Inc. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Whitley R., Gläser J., eds (2007) The Changing Governance of the Sciences: The Advent of Research Evaluation Systems. Dordrecht : Springer. Google Scholar Crossref Search ADS Google Preview WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Wilholt T. (2010) ' Scientific Freedom: Its Grounds and Their Limitations ', Studies in History and Philosophy of Science Part A, 41 : 174 – 81. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Wilson J. (1978) ' Peer Review and Publication ', Journal of Clinical Investigation, 61 : 1697 – 701. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Wing J., Chi E. (2011) ' Reviewing Peer Review ', Communications of the ACM, 54 : 10 – 11. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Wood F., Wessely S. (2003). 'Peer Review of Grant Applications: A Systematic Review', in Godlee F., Jefferson T. (eds) Peer Review in Health Sciences, 2nd ed,, pp. 14 – 44. London : BMJ Books. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Woolston, C. (2015) 'What would happen if grant reviews were made public?', Nature, 517(7534): 247-247.</bibtext> </blist> <blist> <bibtext> Zollman K. (2012) ' Social Network Structure and the Achievement of Consensus ', Politics, Philosophy & Economics, 11 : 26 – 44. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Zuckerman H., Merton R. (1971) ' Patterns of Evaluation in Science: Institutionalisation, Structure and Functions of the Referee System ', Minerva, 9 : 66 – 100. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> </ref> <aug> <p>By Jamie Shaw</p> <p>Reported by Author</p> </aug> <nolink nlid="nl1" bibid="bib59" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib31" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib25" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib72" firstref="ref4"></nolink> <nolink nlid="nl5" bibid="bib114" firstref="ref6"></nolink> <nolink nlid="nl6" bibid="bib14" firstref="ref7"></nolink> <nolink nlid="nl7" bibid="bib103" firstref="ref8"></nolink> <nolink nlid="nl8" bibid="bib71" firstref="ref9"></nolink> <nolink nlid="nl9" bibid="bib46" firstref="ref10"></nolink> <nolink nlid="nl10" bibid="bib42" firstref="ref11"></nolink> <nolink nlid="nl11" bibid="bib17" firstref="ref12"></nolink> <nolink nlid="nl12" bibid="bib39" firstref="ref13"></nolink> <nolink nlid="nl13" bibid="bib37" firstref="ref14"></nolink> <nolink nlid="nl14" bibid="bib32" firstref="ref15"></nolink> <nolink nlid="nl15" bibid="bib78" firstref="ref17"></nolink> <nolink nlid="nl16" bibid="bib102" firstref="ref19"></nolink> <nolink nlid="nl17" bibid="bib52" firstref="ref20"></nolink> <nolink nlid="nl18" bibid="bib30" firstref="ref21"></nolink> <nolink nlid="nl19" bibid="bib44" firstref="ref23"></nolink> <nolink nlid="nl20" bibid="bib90" firstref="ref25"></nolink> <nolink nlid="nl21" bibid="bib36" firstref="ref31"></nolink> <nolink nlid="nl22" bibid="bib69" firstref="ref32"></nolink> <nolink nlid="nl23" bibid="bib96" firstref="ref33"></nolink> <nolink nlid="nl24" bibid="bib53" firstref="ref38"></nolink> <nolink nlid="nl25" bibid="bib18" firstref="ref43"></nolink> <nolink nlid="nl26" bibid="bib40" firstref="ref44"></nolink> <nolink nlid="nl27" bibid="bib49" firstref="ref50"></nolink> <nolink nlid="nl28" bibid="bib43" firstref="ref54"></nolink> <nolink nlid="nl29" bibid="bib23" firstref="ref56"></nolink> <nolink nlid="nl30" bibid="bib89" firstref="ref57"></nolink> <nolink nlid="nl31" bibid="bib94" firstref="ref59"></nolink> <nolink nlid="nl32" bibid="bib93" firstref="ref61"></nolink> <nolink nlid="nl33" bibid="bib15" firstref="ref62"></nolink> <nolink nlid="nl34" bibid="bib111" firstref="ref63"></nolink> <nolink nlid="nl35" bibid="bib77" firstref="ref64"></nolink> <nolink nlid="nl36" bibid="bib75" firstref="ref65"></nolink> <nolink nlid="nl37" bibid="bib48" firstref="ref66"></nolink> <nolink nlid="nl38" bibid="bib65" firstref="ref67"></nolink> <nolink nlid="nl39" bibid="bib47" firstref="ref68"></nolink> <nolink nlid="nl40" bibid="bib105" firstref="ref70"></nolink> <nolink nlid="nl41" bibid="bib57" firstref="ref71"></nolink> <nolink nlid="nl42" bibid="bib28" firstref="ref72"></nolink> <nolink nlid="nl43" bibid="bib10" firstref="ref73"></nolink> <nolink nlid="nl44" bibid="bib62" firstref="ref74"></nolink> <nolink nlid="nl45" bibid="bib11" firstref="ref75"></nolink> <nolink nlid="nl46" bibid="bib33" firstref="ref76"></nolink> <nolink nlid="nl47" bibid="bib97" firstref="ref77"></nolink> <nolink nlid="nl48" bibid="bib70" firstref="ref78"></nolink> <nolink nlid="nl49" bibid="bib35" firstref="ref79"></nolink> <nolink nlid="nl50" bibid="bib76" firstref="ref80"></nolink> <nolink nlid="nl51" bibid="bib64" firstref="ref81"></nolink> <nolink nlid="nl52" bibid="bib85" firstref="ref82"></nolink> <nolink nlid="nl53" bibid="bib12" firstref="ref83"></nolink> <nolink nlid="nl54" bibid="bib95" firstref="ref86"></nolink> <nolink nlid="nl55" bibid="bib13" firstref="ref87"></nolink> <nolink nlid="nl56" bibid="bib79" firstref="ref88"></nolink> <nolink nlid="nl57" bibid="bib51" firstref="ref90"></nolink> <nolink nlid="nl58" bibid="bib113" firstref="ref91"></nolink> <nolink nlid="nl59" bibid="bib55" firstref="ref92"></nolink> <nolink nlid="nl60" bibid="bib108" firstref="ref93"></nolink> <nolink nlid="nl61" bibid="bib63" firstref="ref94"></nolink> <nolink nlid="nl62" bibid="bib66" firstref="ref95"></nolink> <nolink nlid="nl63" bibid="bib88" firstref="ref97"></nolink> <nolink nlid="nl64" bibid="bib45" firstref="ref98"></nolink> <nolink nlid="nl65" bibid="bib26" firstref="ref99"></nolink> <nolink nlid="nl66" bibid="bib101" firstref="ref103"></nolink> <nolink nlid="nl67" bibid="bib50" firstref="ref104"></nolink> <nolink nlid="nl68" bibid="bib67" firstref="ref106"></nolink> <nolink nlid="nl69" bibid="bib73" firstref="ref107"></nolink> <nolink nlid="nl70" bibid="bib98" firstref="ref110"></nolink> <nolink nlid="nl71" bibid="bib87" firstref="ref115"></nolink> <nolink nlid="nl72" bibid="bib41" firstref="ref116"></nolink> <nolink nlid="nl73" bibid="bib54" firstref="ref117"></nolink> <nolink nlid="nl74" bibid="bib16" firstref="ref118"></nolink> <nolink nlid="nl75" bibid="bib92" firstref="ref122"></nolink> <nolink nlid="nl76" bibid="bib61" firstref="ref123"></nolink> <nolink nlid="nl77" bibid="bib60" firstref="ref130"></nolink> <nolink nlid="nl78" bibid="bib21" firstref="ref131"></nolink> <nolink nlid="nl79" bibid="bib22" firstref="ref132"></nolink> <nolink nlid="nl80" bibid="bib58" firstref="ref136"></nolink> <nolink nlid="nl81" bibid="bib74" firstref="ref137"></nolink> <nolink nlid="nl82" bibid="bib68" firstref="ref138"></nolink> <nolink nlid="nl83" bibid="bib27" firstref="ref143"></nolink> <nolink nlid="nl84" bibid="bib82" firstref="ref144"></nolink> <nolink nlid="nl85" bibid="bib19" firstref="ref145"></nolink> <nolink nlid="nl86" bibid="bib91" firstref="ref146"></nolink> <nolink nlid="nl87" bibid="bib20" firstref="ref148"></nolink> <nolink nlid="nl88" bibid="bib112" firstref="ref153"></nolink> <nolink nlid="nl89" bibid="bib24" firstref="ref154"></nolink> <nolink nlid="nl90" bibid="bib84" firstref="ref156"></nolink> <nolink nlid="nl91" bibid="bib56" firstref="ref158"></nolink> <nolink nlid="nl92" bibid="bib38" firstref="ref160"></nolink> <nolink nlid="nl93" bibid="bib86" firstref="ref161"></nolink> <nolink nlid="nl94" bibid="bib80" firstref="ref162"></nolink> <nolink nlid="nl95" bibid="bib110" firstref="ref169"></nolink> <nolink nlid="nl96" bibid="bib104" firstref="ref173"></nolink> <nolink nlid="nl97" bibid="bib29" firstref="ref176"></nolink> <nolink nlid="nl98" bibid="bib100" firstref="ref177"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1382459
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Peer Review in Funding-by-Lottery: A Systematic Overview and Expansion
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Shaw%2C+Jamie%22">Shaw, Jamie</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Research+Evaluation%22"><i>Research Evaluation</i></searchLink>. Jan 2023 32(1):86-100.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Oxford University Press. Great Clarendon Street, Oxford, OX2 6DP, UK. Tel: +44-1865-353907; Fax: +44-1865-353485; e-mail: jnls.cust.serv@oxfordjournals.org; Web site: http://applij.oxfordjournals.org/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 15
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2023
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Evaluative
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Peer+Evaluation%22">Peer Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Evaluation%22">Program Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Financial+Support%22">Financial Support</searchLink><br /><searchLink fieldCode="DE" term="%22Scientific+Research%22">Scientific Research</searchLink><br /><searchLink fieldCode="DE" term="%22Taxonomy%22">Taxonomy</searchLink><br /><searchLink fieldCode="DE" term="%22Holistic+Approach%22">Holistic Approach</searchLink><br /><searchLink fieldCode="DE" term="%22Trust+%28Psychology%29%22">Trust (Psychology)</searchLink><br /><searchLink fieldCode="DE" term="%22Social+Responsibility%22">Social Responsibility</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1093/reseval/rvac022
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0958-2029<br />1471-5449
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Despite the surging interest in introducing lottery mechanisms into decision-making procedures for science funding bodies, the discourse on funding-by-lottery remains underdeveloped and, at times, misleading. Funding-by-lottery is sometimes presented as if it were a single mechanism when, in reality, there are many funding-by-lottery mechanisms with important distinguishing features. Moreover, funding-by-lottery is sometimes portrayed as an alternative to traditional methods of peer review when peer review is still used within funding-by-lottery approaches. This obscures a proper analysis of the (hypothetical and actual) variants of funding-by-lottery and important differences amongst them. The goal of this article is to provide a preliminary taxonomy of funding-by-lottery variants and evaluate how the existing evidence on peer review might lend differentiated support for variants of funding-by-lottery. Moreover, I point to gaps in the literature on peer review that must be addressed in future research. I conclude by building off of the work of Avin in moving toward a more holistic evaluation of funding-by-lottery. Specifically, I consider implications funding-by-lottery variants may have regarding trust and social responsibility.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2023
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1382459
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1382459
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1093/reseval/rvac022
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 15
        StartPage: 86
    Subjects:
      – SubjectFull: Peer Evaluation
        Type: general
      – SubjectFull: Program Evaluation
        Type: general
      – SubjectFull: Financial Support
        Type: general
      – SubjectFull: Scientific Research
        Type: general
      – SubjectFull: Taxonomy
        Type: general
      – SubjectFull: Holistic Approach
        Type: general
      – SubjectFull: Trust (Psychology)
        Type: general
      – SubjectFull: Social Responsibility
        Type: general
    Titles:
      – TitleFull: Peer Review in Funding-by-Lottery: A Systematic Overview and Expansion
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Shaw, Jamie
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2023
          Identifiers:
            – Type: issn-print
              Value: 0958-2029
            – Type: issn-electronic
              Value: 1471-5449
          Numbering:
            – Type: volume
              Value: 32
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Research Evaluation
              Type: main
ResultId 1