<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE root>
<article xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ali="http://www.niso.org/schemas/ali/1.0/" article-type="research-article" dtd-version="1.2" xml:lang="en"><front><journal-meta><journal-id journal-id-type="publisher-id">Herald of the Russian Academy of Sciences</journal-id><journal-title-group><journal-title xml:lang="en">Herald of the Russian Academy of Sciences</journal-title><trans-title-group xml:lang="ru"><trans-title>Вестник Российской академии наук</trans-title></trans-title-group></journal-title-group><issn publication-format="print">0869-5873</issn><issn publication-format="electronic">3034-5200</issn><publisher><publisher-name xml:lang="en">The Russian Academy of Sciences</publisher-name></publisher></journal-meta><article-meta><article-id pub-id-type="publisher-id">659702</article-id><article-id pub-id-type="doi">10.31857/S0869587324010063</article-id><article-id pub-id-type="edn">HAXNUC</article-id><article-categories><subj-group subj-group-type="toc-heading"><subject>ОРГАНИЗАЦИЯ ИССЛЕДОВАТЕЛЬСКОЙ ДЕЯТЕЛЬНОСТИ</subject></subj-group><subj-group subj-group-type="article-type"><subject>Research Article</subject></subj-group></article-categories><title-group><article-title xml:lang="en">Two circuit assessments of the performance of scientific organizations in Russia: current state and development prospects from the point of view of international experience</article-title><trans-title-group xml:lang="ru"><trans-title>Два контура оценки результативности научных организаций в России: текущее состояние и перспективы развития с точки зрения международного опыта</trans-title></trans-title-group></title-group><contrib-group><contrib contrib-type="author"><name-alternatives><name xml:lang="en"><surname>Kosyakov</surname><given-names>D. V.</given-names></name><name xml:lang="ru"><surname>Косяков</surname><given-names>Денис Викторович</given-names></name></name-alternatives><address><country country="RU">Russian Federation</country></address><bio><p>заместитель заведующего лабораторией наукометрии и научных коммуникаций РИЭПП</p></bio><email>kosyakov@sciencepulse.ru</email><xref ref-type="aff" rid="aff1"/></contrib><contrib contrib-type="author"><name-alternatives><name xml:lang="en"><surname>Selivanova</surname><given-names>I. V.</given-names></name><name xml:lang="ru"><surname>Селиванова</surname><given-names>Ирина Вячеславовна</given-names></name></name-alternatives><address><country country="RU">Russian Federation</country></address><bio><p>кандидат технических наук, научный сотрудник РИЭПП</p></bio><email>i-seli@yandex.ru</email><xref ref-type="aff" rid="aff1"/></contrib><contrib contrib-type="author"><name-alternatives><name xml:lang="en"><surname>Guskov</surname><given-names>A. E.</given-names></name><name xml:lang="ru"><surname>Гуськов</surname><given-names>Андрей Евгеньевич</given-names></name></name-alternatives><address><country country="RU">Russian Federation</country></address><bio><p>кандидат технических наук, заведующий лабораторией наукометрии и научных коммуникаций РИЭПП</p></bio><email>guskov.andrey@gmail.com</email><xref ref-type="aff" rid="aff1"/></contrib></contrib-group><aff-alternatives id="aff1"><aff><institution xml:lang="en">Russian Research Institute of Economics, Politics and Law in the Scientific and Technical Sphere</institution></aff><aff><institution xml:lang="ru">Российский научно-исследовательский институт экономики, политики и права в научно-технической сфере</institution></aff></aff-alternatives><pub-date date-type="pub" iso-8601-date="2024-07-04" publication-format="electronic"><day>04</day><month>07</month><year>2024</year></pub-date><volume>94</volume><issue>1</issue><fpage>32</fpage><lpage>54</lpage><history><date date-type="received" iso-8601-date="2025-02-20"><day>20</day><month>02</month><year>2025</year></date></history><permissions><copyright-statement xml:lang="en">Copyright ©; 2024, Russian Academy of Sciences</copyright-statement><copyright-statement xml:lang="ru">Copyright ©; 2024, Российская академия наук</copyright-statement><copyright-year>2024</copyright-year><copyright-holder xml:lang="en">Russian Academy of Sciences</copyright-holder><copyright-holder xml:lang="ru">Российская академия наук</copyright-holder></permissions><self-uri xlink:href="https://journals.eco-vector.com/0869-5873/article/view/659702">https://journals.eco-vector.com/0869-5873/article/view/659702</self-uri><abstract xml:lang="en"><p>In Russia, in fact, two contours for assessing the scientific performance of organizations have developed: the expert one, which underlies the categorization of scientific organizations, and the quantitative one, which, according to the methodology of the comprehensive calculation of publication productivity, makes it possible to determine and monitor the implementation of the state assignment for fundamental scientific research. The article describes the history of the formation of these circuits, their advantages and disadvantages, as well as the prerequisites for further development. The British REF (Research Excellence Framework) examination system and the Norwegian quantitative assessment model are considered as model examples for improving both circuits. REF features include a differentiated assessment of the results, impact and environment of scientific units, which is organized by expert panels and groups. The features of the Norwegian model are an expert approach to the formation of a national white list of scientific journals, conferences and publishers, a methodology for calculating the publication indicator and an open database of publications of Norwegian scientists, from which this indicator is determined. The authors believe that these two examples can be considered as parametric assessment models, the adaptation of which taking into account national characteristics will make it possible to update and improve both contours of the assessment of scientific organizations in Russia. The importance of regular retrospective reflection on the experience, procedures and results of assessing organizations is emphasized, which is necessary for the systematic development of this system at the next stages.</p></abstract><trans-abstract xml:lang="ru"><p>В России фактически сложились два контура оценки научной результативности организаций: экспертный, лежащий в основе категорирования научных организаций, и количественный, который по методике Комплексного расчёта публикационной результативности позволяет определять и контролировать выполнение государственного задания на фундаментальные научные исследования. В статье описана история становления этих контуров, их достоинства и недостатки, а также предпосылки для дальнейшего развития. Модельными примерами для улучшения обоих контуров рассматриваются британская система экспертизы REF (Research Excellence Framework) и норвежская модель количественной оценки. Особенности REF состоят в дифференцированной оценке результатов, импакта и среды научных подразделений, которая организуется экспертными панелями и группами. Особенности норвежской модели — экспертный подход к формированию национального белого списка научных журналов, конференций и издателей, методика расчёта публикационного индикатора и открытая база данных публикаций норвежских учёных, по которой этот индикатор определяется. Авторы полагают, что эти два примера можно рассматривать как параметрические модели оценки, адаптация которых с учётом национальных особенностей позволит актуализировать и усовершенствовать оба контура оценки научных организаций в России. Акцентируется важность регулярного ретроспективного осмысления опыта, процедур и результатов оценки организаций, что необходимо для планомерного развития этой системы на следующих этапах.</p></trans-abstract><kwd-group xml:lang="en"><kwd>scientific productivity</kwd><kwd>performance assessment of scientific organizations</kwd><kwd>examination</kwd><kwd>integrated calculation of publication performance (CPPR)</kwd><kwd>categorization of organizations</kwd><kwd>scientometrics</kwd><kwd>Norwegian model</kwd><kwd>REF</kwd></kwd-group><kwd-group xml:lang="ru"><kwd>научная результативность</kwd><kwd>оценка результативности научных организаций</kwd><kwd>экспертиза</kwd><kwd>комплексный расчёт публикационной результативности (КБПР)</kwd><kwd>категорирование организаций</kwd><kwd>наукометрия</kwd><kwd>норвежская модель</kwd><kwd>REF</kwd></kwd-group><funding-group/></article-meta></front><body></body><back><ref-list><ref id="B1"><label>1.</label><mixed-citation>Gruening G. Origin and theoretical basis of new public management // International Public Management Journal. 2001. V. 4. № 1. P. 1–25. https://doi.org/10.1016/S1096-7494(01)00041-1</mixed-citation></ref><ref id="B2"><label>2.</label><mixed-citation>Hicks D. Performance-based university research funding systems // Research Policy. 2012. V. 41. № 2. P. 251–261. https://doi.org/10.1016/ j.respol.2011.09.007</mixed-citation></ref><ref id="B3"><label>3.</label><mixed-citation>Dougherty K. J., Natow R. S. Performance-based funding for higher education: how well does neoliberal theory capture neoliberal practice? // Higher Education. 2020. V. 80. № 3. P. 457–478. https://doi.org/10.1007/s10734-019-00491-4</mixed-citation></ref><ref id="B4"><label>4.</label><mixed-citation>Henkel M. The modernisation of research evaluation: The case of the UK // Higher Education. 1999. V. 38. № 1. P. 105–122. https://doi.org/10.1023/A:1003799013939</mixed-citation></ref><ref id="B5"><label>5.</label><mixed-citation>Mok K. H. Enhancing quality of higher education for world-class status: Approaches, strategies, and challenges for Hong Kong // Chinese Education and Society. 2014. V. 47. № 1. P. 44–64. https://doi.org/ 10.2753/CED1061-1932470103</mixed-citation></ref><ref id="B6"><label>6.</label><mixed-citation>Crowe S. F., Watt S. Excellence in Research in Australia 2010, 2012, and 2015: The Rising of the Curate’s Soufflé? // Australian Psychologist. 2017. V. 52. № 6. P. 503–513. https://doi.org/10.1111/ap.12248</mixed-citation></ref><ref id="B7"><label>7.</label><mixed-citation>Chatterjee B. et al. The spectacle of research assessment systems: insights from New Zealand and the United Kingdom // Accounting, Auditing and Accountability Journal. 2020. V. 33. № 6. P. 1219–1246. https://doi.org/10.1108/AAAJ-01-2019-3865</mixed-citation></ref><ref id="B8"><label>8.</label><mixed-citation>Abramo G., D’Angelo C. A. The VQR, Italy’s second national research assessment: Methodological failures and ranking distortions // Journal of the Association for Information Science and Technology. 2015. V. 66. № 11. P. 2202–2214. https://doi.org/10.1002/asi.23323</mixed-citation></ref><ref id="B9"><label>9.</label><mixed-citation>Luwel M. Performance-based Institutional Research Funding in Flanders, Belgium // Scholarly Assessment Reports. 2021. V. 3. № 1. P. 3. https://doi.org/10.29024/sar.29</mixed-citation></ref><ref id="B10"><label>10.</label><mixed-citation>Engels T. C.E., Guns R. The Flemish Performance-based Research Funding System: A Unique Variant of the Norwegian Model // Journal of Data and Information Science. 2018. V. 3. № 4. P. 45–60. https://doi.org/10.2478/jdis-2018-0020</mixed-citation></ref><ref id="B11"><label>11.</label><mixed-citation>Shu F., Liu S., Larivière V. China’s Research Evaluation Reform: What are the Consequences for Global Science? // Minerva. 2022. V. 60. № 3. P. 329–347. https://doi.org/10.1007/s11024-022-09468-7</mixed-citation></ref><ref id="B12"><label>12.</label><mixed-citation>Aagaard K., Bloch C., Schneider J. W. Impacts of performance-based research funding systems: The case of the Norwegian Publication Indicator // Research Evaluation. 2015. V. 24. № 2. P. 106–117. https://doi.org/10.1093/reseval/rvv003</mixed-citation></ref><ref id="B13"><label>13.</label><mixed-citation>Deutz D.B. et al. Quantitative quality: a study on how performance-based measures may change the publication patterns of Danish researchers // Scientometrics. 2021. V. 126. № 4. P. 3303–3320. https://doi.org/10.1007/s11192-021-03881-7</mixed-citation></ref><ref id="B14"><label>14.</label><mixed-citation>Mathies C., Kivistö J., Birnbaum M. Following the money? Performance-based funding and the changing publication patterns of Finnish academics // High Educ. 2020. V. 79. № 1. P. 21–37. https://doi.org/10.1007/s10734-019-00394-4</mixed-citation></ref><ref id="B15"><label>15.</label><mixed-citation>Hammarfelt B. Taking Comfort in Points: The Appeal of the Norwegian Model in Sweden // Journal of Data and Information Science. 2018. V. 3. № 4. P. 84–94. https://doi.org/10.2478/jdis-2018–0023</mixed-citation></ref><ref id="B16"><label>16.</label><mixed-citation>Постановление Правительства РФ от 8 апреля 2009 г. № 312 “Об оценке и о мониторинге результативности деятельности научных организаций, выполняющих научно-исследовательские, опытно-конструкторские и технологические работы гражданского назначения”. 2009. https://sciencemon.ru/documents/6 (дата обращения 05.10.2023).</mixed-citation></ref><ref id="B17"><label>17.</label><mixed-citation>Информационная справка к Совету при Президенте Российской Федерации по науке и образованию “Об оценке результативности деятельности научных организаций, подведомственных федеральным органам исполнительной власти и государственным академиям наук за 2010–2012 годы”. 2013. https://sciencemon.ru/documents/2 (дата обращения 05.10.2023).</mixed-citation></ref><ref id="B18"><label>18.</label><mixed-citation>Поручение Президента Российской Федерации от 30 апреля 2013 г. № Пр-1144 по итогам заседания Совета при Президенте Российской Федерации по науке и образованию. 2013. https://sciencemon.ru/documents/3 (дата обращения 05.10.2023).</mixed-citation></ref><ref id="B19"><label>19.</label><mixed-citation>Гуськов А. Е., Косяков Д. В., Селиванова И. В. Методика оценки результативности научных организаций // Вестник Российской академии наук. 2018. № 5. https://doi.org/10.7868/S0869587318050092</mixed-citation></ref><ref id="B20"><label>20.</label><mixed-citation>Kosyakov D., Guskov A. Research assessment and evaluation in Russian fundamental science // Procedia Computer Science. 2019. V. 146. P. 11–19. https://doi.org/10.1016/j.procs.2019.01.072</mixed-citation></ref><ref id="B21"><label>21.</label><mixed-citation>Kosyakov D., Guskov A. Reasons and consequences of changes in Russian research assessment policies // Scientometrics. 2022. V. 127. № 8. P. 4609–4630. https://doi.org/10.1007/s11192-022-04469-5</mixed-citation></ref><ref id="B22"><label>22.</label><mixed-citation>Guskov A. E., Kosyakov D. V., Selivanova I. V. Boosting research productivity in top Russian universities: the circumstances of breakthrough // Scientometrics. 2018. V. 117. № 2. P. 1053–1080. https://doi.org/10.1007/s11192-018-2890-8</mixed-citation></ref><ref id="B23"><label>23.</label><mixed-citation>Liu W., Hu G., Gu M. The probability of publishing in first-quartile journals // Scientometrics. 2016. V. 106. № 3. P. 1273–1276. https://doi.org/10.1007/s11192-015-1821-1</mixed-citation></ref><ref id="B24"><label>24.</label><mixed-citation>Kosyakov D. Analysis of the abnormal growth in the number of Russian publications in conference proceedings in Scopus // Scientififc and Technical Information. 2023. № 4. P. 13–24. https://doi.org/10.36535/0548-0019-2023-04-3</mixed-citation></ref><ref id="B25"><label>25.</label><mixed-citation>Phillimore A. J. University research performance indicators in practice: The University Grants Committee’s evaluation of British universities, 1985–86 // Research Policy. 1989. V. 18. № 5. P. 255–271. https://doi.org/10.1016/0048-7333(89)90053-X</mixed-citation></ref><ref id="B26"><label>26.</label><mixed-citation>Sizer J. The impacts of financial reductions on British universities: 1981–84 // Higher Education. 1987. V. 16. № 5. P. 557–580. https://doi.org/10.1007/BF00128422</mixed-citation></ref><ref id="B27"><label>27.</label><mixed-citation>Lee F. S., Pham X., Gu G. The UK Research Assessment Exercise and the narrowing of UK economics // Cambridge Journal of Economics. 2013. V. 37. № 4. P. 693–717. https://doi.org/10.1093/cje/bet031</mixed-citation></ref><ref id="B28"><label>28.</label><mixed-citation>Sizer J. In Search of Excellence —Performance Assessment in the United Kingdom // Higher Education Quarterly. 1988. V. 42. № 2. P. 152–161. https://doi.org/10.1111/j.1468–2273.1988.tb01811.x</mixed-citation></ref><ref id="B29"><label>29.</label><mixed-citation>Bekhradnia B. et al. Research Evaluation: Past, present and future. Higher Education Policy Institute (HEPI), 2022.</mixed-citation></ref><ref id="B30"><label>30.</label><mixed-citation>Curry S., Gadd E., Wilsdon J. Harnessing the Metric Tide: indicators, infrastructures and priorities for UK responsible research assessment. Research on Research Institute, 2022.</mixed-citation></ref><ref id="B31"><label>31.</label><mixed-citation>Smith S., Ward V., House A. ‘Impact’ in the proposals for the UK’s Research Excellence Framework: Shifting the boundaries of academic autonomy // Research Policy. 2011. V. 40. № 10. P. 1369–1379. https://doi.org/10.1016/j.respol.2011.05.026</mixed-citation></ref><ref id="B32"><label>32.</label><mixed-citation>Building on Success and Learning from Experience. An Independent Review of the Research Excellence Framework. UK Government. 2016. https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/541338/ind-16-9-ref-stern-review.pdf (дата обращения 30.01.2023).</mixed-citation></ref><ref id="B33"><label>33.</label><mixed-citation>Future Research Assessment Programme. https://www.ukri.org/about-us/research-england/research-excellence/future-research-assessment-programme-frap/ (дата обращения 30.01.2023).</mixed-citation></ref><ref id="B34"><label>34.</label><mixed-citation>Thelwall M. et al. Can REF output quality scores be assigned by AI? Experimental evidence. Wolverhampton, UK: Statistical Cybermetrics and Research Evaluation Group, University of Wolverhampton, 2022. P. 141.</mixed-citation></ref><ref id="B35"><label>35.</label><mixed-citation>Thelwall M. et al. Predicting article quality scores with machine learning: The U. K. Research Excellence Framework // Quantitative Science Studies. 2023. V. 4. № 2. P. 547–573. https://doi.org/10.1162/qss_a_00258</mixed-citation></ref><ref id="B36"><label>36.</label><mixed-citation>Singh Chawla D. AI system not yet ready to help peer reviewers assess research quality // Nature Index. 2022. https://doi.org/10.1038/d41586-022-04493-8</mixed-citation></ref><ref id="B37"><label>37.</label><mixed-citation>Review of research assessment. Report by Sir Gareth Roberts to the UK funding bodies. UK Government. 2003. P. 44. https://web.archive.org/web/20070720232304/http://www.rareview.ac.uk/reports/roberts.asp (дата обращения 30.01.2023).</mixed-citation></ref><ref id="B38"><label>38.</label><mixed-citation>Research Excellence Framework 2028: Initial decisions and issues for further consultation. Research England, 2023. https://www.ukri.org/publications/ref2028-initial-decisions-and-issues-for-further-consultation/ (дата обращения 30.01.2023).</mixed-citation></ref><ref id="B39"><label>39.</label><mixed-citation>FAQs on the REF 2021 Impact case study database. https://ref.ac.uk/guidance-on-results/impact-case-study-database-faqs/ (дата обращения 30.01.2023).</mixed-citation></ref><ref id="B40"><label>40.</label><mixed-citation>Pidd M., Broadbent J. Business and Management Studies in the 2014 Research Excellence Framework: Business and Management Studies in 2014 REF // Brit J Manage. 2015. V. 26. № 4. P. 569–581. https://doi.org/10.1111/1467–8551.12122</mixed-citation></ref><ref id="B41"><label>41.</label><mixed-citation>Sivertsen G. The Norwegian Model in Norway // Journal of Data and Information Science. 2018. V. 3. № 4. P. 3–19. https://doi.org/10.2478/jdis-2018–0017</mixed-citation></ref><ref id="B42"><label>42.</label><mixed-citation>A Bibliometric Model for Performance-based Budgeting of Research Institutions. Norwegian Association of Higher Education Institutions, 2004.</mixed-citation></ref><ref id="B43"><label>43.</label><mixed-citation>Norwegian Register for Scientific Journals, Series and Publishers. https://kanalregister.hkdir.no/publiseringskanaler/Forside (дата обращения 05.07.2023).</mixed-citation></ref><ref id="B44"><label>44.</label><mixed-citation>Costas Boletsis. Level 1 &amp; Level 2 publications — Norwegian Scientific Index. 2015. https://boletsis.net/level-1-level-2-publications-norwegian-scientific-index/ (дата обращения 05.07.2023).</mixed-citation></ref><ref id="B45"><label>45.</label><mixed-citation>Ahlgren P., Colliander C., Persson O. Field normalized citation rates, field normalized journal impact and Norwegian weights for allocation of university research funds // Scientometrics. 2012. V. 92. № 3. P. 767–780. https://doi.org/10.1007/s11192-012-0632-x</mixed-citation></ref><ref id="B46"><label>46.</label><mixed-citation>Cristin. https://www.cristin.no (дата обращения 05.07.2023).</mixed-citation></ref><ref id="B47"><label>47.</label><mixed-citation>Sivertsen G. The Norwegian Model in Norway // Journal of Data and Information Science. 2018. V. 3. № 4. P. 3–19. https://doi.org/10.2478/jdis-2018–0017</mixed-citation></ref><ref id="B48"><label>48.</label><mixed-citation>Schneider J. W. An Outline of the Bibliometric Indicator Used for Performance-Based Funding of Research Institutions in Norway // Eur Polit Sci. 2009. V. 8. № 3. P. 364–378. https://doi.org/10.1057/eps.2009.19</mixed-citation></ref><ref id="B49"><label>49.</label><mixed-citation>Ahlgren P., Colliander C., Persson O. Field normalized citation rates, field normalized journal impact and Norwegian weights for allocation of university research funds // Scientometrics. 2012. V. 92. № 3. P. 767–780. https://doi.org/10.1007/s11192-012-0632-x</mixed-citation></ref><ref id="B50"><label>50.</label><mixed-citation>Bloch C., Schneider J. W. Performance-based funding models and researcher behavior: An analysis of the influence of the Norwegian Publication Indicator at the individual level // Research Evaluation. 2016. P. rvv047. https://doi.org/10.1093/reseval/rvv047</mixed-citation></ref><ref id="B51"><label>51.</label><mixed-citation>Aagaard K. How incentives trickle down: Local use of a national bibliometric indicator system // Science and Public Policy. 2015. V. 42. № 5. P. 725–737. https://doi.org/10.1093/scipol/scu087</mixed-citation></ref><ref id="B52"><label>52.</label><mixed-citation>Skivenes M., Trygstad S. C. When whistle-blowing works: The Norwegian case // Human Relations. 2010. V. 63. № 7. P. 1071–1097. https://doi.org/10.1177/0018726709353954</mixed-citation></ref><ref id="B53"><label>53.</label><mixed-citation>Schneider J.W., Aagaard K., Bloch C.W. What happens when national research funding is linked to differentiated publication counts? A comparison of the Australian and Norwegian publication-based funding models // Research Evaluation. 2016. V. 25. № 3. P. 244–256. https://doi.org/10.1093/reseval/rvv036</mixed-citation></ref><ref id="B54"><label>54.</label><mixed-citation>Гуськов А. Е., Косяков Д. В. Национальный фракционный счёт и оценка научной результативности организаций // Научные и технические библиотеки. 2020. № 9. P. 15–42. https://doi.org/10.33186/1027-3689-2020-9-15-42</mixed-citation></ref></ref-list></back></article>
