SFeDu
of.211, Gorgoko St., 88 
Rostov-on-Don
Russia
344002
+7 (863) 250-59-54
This email address is being protected from spambots. You need JavaScript enabled to view it.
te@sfedu.ru

TERRA ECONOMICUS

  • Home
  • Issues
  • 2023
  • No 3
  • Russian experience with national rankings of academic journals: Mistakes, challenges, and prospects

Russian experience with national rankings of academic journals: Mistakes, challenges, and prospects


TERRA ECONOMICUS, , Vol. 21 (no. 3),

The article considers the excessive dependence of national Russian systems of scholarly periodicals on international scientometric databases. The problem has aggravated after the owners of the global citation indexes Web of Science and Scopus left the Russian market. We find out that national performance-based academic repositories show greater stability under the influence of external challenges. We analyze Russia’s experience in compiling national lists of academic journals, using a sample of publications on economics and related disciplines. We identify and classify the most common mistakes that were made during the introduction of journal metrics into the research performance evaluation system in Russia, and propose ways to correct them. Our findings reveal that the journal selection procedure has certain flaws resulting in the fact that some leading Russian journals, recognized at the international and national level, were not included in the “white lists”. We prove that a critical flaw in the ranking procedure of the journals included in national lists consists in the mixing of quartile metrics from different international databases; consequently, the logic of the distribution of publications that are similar in academic level and status was violated. Finally, we determine crucial qualitative criteria for evaluating academic journals and, relying on these criteria, propose three categories to rank publications on national lists. Our findings can be used in the state regulation of scientific activities of individual researchers and organizations.
Citation: Tretyakova O.V. (2023). Russian experience with national rankings of academic journals: Mistakes, challenges, and prospects. Terra Economicus 21(3), 102–121 (in Russian). DOI: 10.18522/2073-6606-2023-21-3-102-121


Keywords: science policy; research performance evaluation; quality indicators; international scientometric databases; Russian Journal Whitelist

JEL codes: A11, I23, I28

References:
  • Балацкий Е.В., Екимова Н.А., Третьякова О.В. (2021). Методы оценки качества научных экономических журналов. Journal of Institutional Studies 13(2), 27–52. [Balatsky, E., Ekimova, N., Tretyackova, O. (2021). Evaluation methods of scientific economic journals quality. Journal of Institutional Studies 13(2), 27–52 (in Russian)]. DOI: 10.17835/2076-6297.2021.13.2.027-052
  • Гайдин Б.Н. (2022). Российские научные журналы в новых геополитических условиях: сложности и перспективы развития. Управление наукой: теория и практика 4(3), 44–52. [Gaydin, B. (2022). Russian academic journals in new geopolitical conditions: Difficulties and prospects of development. Science Management: Theory and Practice 4(3), 44–52 (in Russian)]. DOI: 10.19181/smtp.2022.4.3.4
  • Губа К.С. (2022). Наукометрические показатели в оценке российских университетов: обзор исследований. Мир России 31(1), 49–73. [Guba, K. (2022). Scientometric indicators in the evaluation of Russian universities: A literature review. Universe of Russia (Mir Rossii) 31(1), 49–73 (in Russian)]. DOI: 10.17323/1811-038X-2022-31-1-49-73
  • Бородик К.А., Дикусар К.С., Богатов В.В. (2021). Тренды публикационной активности российских исследователей за период 2016–2020 гг. по данным международных баз научного цитирования Web of Science Core Collection и Scopus. Управление наукой и наукометрия 16(4), 571–595. [Borodik, K., Dikusar, K., Bogatov, V. (2021). Publication activity trends among Russian researchers in 2016–2020, Based on international scientific citation databases: Web of Science Core Collection and Scopus. Science Governance and Scientometrics 16(4), 571–595 (in Russian)]. DOI: 10.33873/2686-6706.2021.16-4.571-595
  • Дежина И.Г., Сорокин А.Н. (2022). Проект 5-100 в восприятии сотрудников университетов. Мир России 31(1), 74–90. [Dezhina, I., Sorokin, A. (2022). The Perception of the Project 5-100 by University Employees. Universe of Russia (Mir Rossii) 31(1), 74–90 (in Russian)]. DOI: 10.17323/1811-038X-2022-31-1-74-90
  • Кириллова О.В. (2022). Подводя своеобразные итоги десятилетия и рисуя планы. Научный редактор и издатель 7(1), 8–11. [Kirillova, O. (2022). Summing up the peculiar results of the decade and drawing plans... Science Editor and Publisher 7(1), 8–11 (in Russian)]. DOI: 10.24069/SEP-22-38
  • Мазов Н.А., Гуреев В.Н., Каленов Н.Е. (2018). Некоторые оценки списка журналов Russian Science Citation Index. Вестник Российской академии наук 88(4), 322–332. DOI: 10.7868/S0869587318040047 [Mazov, N., Gureev, V., Kalenov, N. (2018). Some assessments of the list of journals in the Russian Science Citation Index. Herald of the Russian Academy of Sciences 88(2), 133–141]. DOI: https://doi.org/10.1134/S1019331618020053
  • Максимов С.В. (2021). «Дорожная карта» развития конкуренции в сфере науки (теоретическая модель). Российское конкурентное право и экономика (3), 8–21. [Maksimov, S. (2021). “Roadmap” for the development of competition in the field of science (Theoretical model). Russian Competition Law and Economy (3), 8–21 (in Russian)]. DOI: 10.47361/2542-0259-2021-3-27-8-21
  • Паршин А.Н. (2018). Наука или библиометрия: кто кого? Вестник Российской академии наук 88(11), 982– 984. [Parshin, A. (2018). Science or bibliometry: Who will win? Herald of the Russian Academy of Sciences 88(11), 982–984 (in Russian)]. DOI: 10.31857/S086958730002330-1
  • Семенов Е.В. (2023). Национальная сеть научных журналов как система: проблемы до и после санкций. Мир России 32(3), 145–166. [Semenov, E. (2023). A national network of academic journals as a system: Problems before and after sanctions. Mir Rossii 32(3), 145–166 (in Russian)]. DOI: 10.17323/1811-038X-2023-32-3-145-166
  • Третьякова О.В. (2023). Национальные списки научных журналов: обзор мировой практики. Journal of Institutional Studies 15(3), в печати. [Tretyakova, O. (2023). National lists of academic journals: An overview of world practice. Journal of Institutional Studies 15(3), (in Russian), in press].
  • Третьякова О.В. (2020). Оценка журналов RSCI по экономическим наукам в контексте создания национального индексацитирования. Вестник Российскойакадемиинаук 90(4), 364–380. DOI: 10.31857/S0869587320040143 [Tretyakova, O. (2020). Assessing RSCI economics journals in the context of creating a national citation index. Herald of the Russian Academy of Sciences 90(2), 251–265]. DOI: 10.1134/S1019331620020173
  • Третьякова О.В. (2021). Российские экономические журналы в ESCI: ретроспектива и прогноз. Terra Economicus 19(4), 92–109. [Tretyakova, O. (2021). Russian economic journals in the ESCI: Retrospective overview and forecast. Terra Economicus 19(4), 92–109 (in Russian)]. DOI: 10.18522/2073-6606-2021-19-4-92-109
  • Третьякова О.В. (2022). Российские социологические журналы в международных базах данных: что необходимо учесть в новой системе оценки. Мир России 31(4), 100–121. [Tretyakova, O. (2022). Russian sociological journals in international scientometric databases: What should be taken into account in a new evaluation system. Universe of Russia (Mir Rossii) 31(4), 100–121 (in Russian)]. DOI: 10.17323/1811-038X- 2022-31-4-100-121
  • Тургель И.Д. (2022). Приоритеты трансформации редакционной политики научного журнала в условиях международных санкций. Научный редактор и издатель 7(1), 28–38. [Turgel, I. (2022). Transformation priorities in the editorial policy of a Russian scientific journal in the context of international sanctions. Science Editor and Publisher 7(1), 28–38 (in Russian)]. DOI: 10.24069/SEP-22-03
  • Черныш М.Ф. (2020). Реформа российской науки как институциональное конструирование. Управление наукой: теория и практика 2(2), 47–64. [Chernysh, M. (2020). Reform of a Russian science as an institutional constructing.Sciencemanagement:theoryandpractice2(2),47–64(inRussian)].DOI:10.19181/smtp.2020.2.2.2
  • Deutz, D., Drachen, T., Drongstrup, D. et al. (2021). Quantitative quality: A study on how performance-based measures may change the publication patterns of Danish researchers. Scientometrics 126(4), 3303–3320. DOI: 10.1007/s11192-021-03881-7
  • Good, B., Vermeulen, N., Tiefenthaler, B., Arnold, E. (2015). Counting quality? The Czech performance-based research funding system. Research Evaluation 24(2), 91–105. DOI: 10.1093/reseval/rvu035
  • Korytkowski, P., Kulczycki, E. (2019). Examining how country-level science policy shapes publication patterns: the case of Poland. Scientometrics 119(3), 1519–1543. DOI: 10.1007/s11192-019-03092-1
  • Kulczycki, E., Rozkosz, E. (2017). Does an expert-based evaluation allow us to go beyond the Impact Factor? Experiences from building a ranking of national journals in Poland. Scientometrics 111(1), 417–442. DOI: 10.1007/s11192-017-2261-x
  • Mouritzen, P., Opstrup, N. (2019). Performance Management at Universities: The Danish Bibliometric Research Indicator at Work. Palgrave Macmillan. DOI: 10.1007/978-3-030-21325-1
  • Pölönen, J., Guns, R., Kulczycki, E., Sivertsen, G., Engels, T. (2020). National lists of scholarly publication channels: An overview and recommendations for their construction and maintenance. Journal of Data and Information Science 6(1), 1–37. DOI: 10.2478/jdis-2021-0004
  • Pölönen, J., Pylvänäinen, E., Aspara, J., Puuska, H.-M., Rinne, R. (2021). Publication Forum 2010–2020: Self- evaluation report of the Finnish quality classification system of peer-reviewed publication channels. Helsinki: Federation of Finnish Learned Societies. https://julkaisufoorumi.fi/sites/default/files/2021-03/ Publication%20Forum%20self-evaluation%20report%202021_0.pdf
  • Pölönen, J., Auranen, O. (2022). Research performance and scholarly communication profile of competitive research funding: The case of Academy of Finland. Scientometrics 127(12), 7415–7433. DOI: 10.1007/s11192-022-04385-8
  • Sanz-Casado, E., De Filippo, D., Benavent, R. et al. (2021). Impact and visibility of Norwegian, Finnish and Spanish journals in the fields of humanities. Scientometrics 126(11), 9031–9049. DOI: 10.1007/s11192-021-04169-6
  • Shu, F., Liu, S., Larivière, V. (2022). China’s research evaluation reform: What are the consequences for global science? Minerva 60, 329–347. DOI: 10.1007/s11024-022-09468-7
  • Shu, F., Wang, X., Liu, S. et al. (2023). Global impact or national accessibility? A paradox in China’s science. Scientometrics 128(1), 263–277. DOI: 10.1007/s11192-022-04537-w
  • Sīle, L., Pölönen, J., Sivertsen, G., Guns, R., Engels, T. et al. (2018). Comprehensiveness of national bibliographic databases for social sciences and humanities: findings from a European survey. Research Evaluation 27(4), 310–322. DOI: 10.1093/reseval/rvy016
  • Sivertsen, G. (2010). A performance indicator based on complete data for the scientific publication output at research institutions. ISSI Newsletter 6(1), 22–28.
  • Sivertsen, G. (2016). Data integration in Scandinavia. Scientometrics 106(2), 849–855. DOI: 10.1007/s11192-015-1817-x
  • Sivertsen, G. (2018). The Norwegian model in Norway. Journal of Data and Information Science 3(4), 2–18. DOI: 10.2478/jdis-2018-0017
  • Schneider, J. (2009). An outline of the bibliometric indicator used for performance-based funding of research institutions in Norway. European Political Science 8(3), 364–378. DOI: 10.1057/eps.2009.19
  • Söderlind, J., Berg, L., Lind, J., Pulkkinen, K. (2019). National performance-based research funding systems: Constructing local perceptions of research? In: Pinheiro, R., Geschwind, L., Foss Hansen, H., Pulkkinen, K. (eds.) Reforms, Organizational Change and Performance in Higher Education. Palgrave Macmillan, Cham. DOI: 10.1007/978-3-030-11738-2_4
  • Vanecek, J. (2014). The effect of performance-based research funding on output of R&D results in the Czech Republic. Scientometrics 98(1), 657–681. DOI: 10.1007/s11192-013-1061-1
  • Zhang, L., Sivertsen, G. (2020). The new research assessment reform in China and its implementation. Scholarly Assessment Reports 2(1), 3. DOI: 10.29024/sar.15
  • Vȋiu, G., Păunescu, M. (2021). The lack of meaningful boundary differences between journal impact factor quartiles undermines their independent use in research evaluation. Scientometrics 126(2), 1495–1525. DOI: 10.1007/ s11192-020-03801-1
Publisher: Southern Federal University
Founder: Southern Federal University
ISSN: 2073-6606