Email this article Qualimetric assessment of research and development performance of a higher education institution
- Authors: Ivchenko E.V.1, Grigoriev S.G.1, Ovchinnikov D.V.1, Latypov I.F.1
-
Affiliations:
- Military Medical Academy of S.M. Kirov
- Issue: Vol 24, No 4 (2022)
- Pages: 719-725
- Section: Research paper
- URL: https://journals.eco-vector.com/1682-7392/article/view/111049
- DOI: https://doi.org/10.17816/brmma111049
- ID: 111049
Cite item
Open Access
Access granted
Subscription or Fee Access
Abstract
The study presents an original methodology for assessing the quality of scientific activity of the units of the Military Medical Academy named after S.M. Kirov. As a tool for its solution, we used qualimetric evaluation of all possible areas of scientific activity of departments and criteria that characterize these areas. Qualimetric methods of expert survey and weighting of performance indicators and descriptive statistics were used. Twenty-five experts took part in structuring the scientific performance of the academy. The resulting structure included six main sections, each described by a specific set of attributes. The attributes were both quantitative and qualitative with certain graduations and units of measurement, each of which was assigned a weighting coefficient. The values of the coefficients were expressed in fractions of 1, and the sum was 1. The consistency of the experts proved to be quite high, and the coefficient of concordance was 0.82 (p < 0.001). Based on the comprehensive assessment of each division of the academy, a scale score was calculated, which helped in determining the rating of a division. A sigma (using the mean square deviation) estimate was chosen as a scaling tool, which shows that 68% of all complex estimates of the units being evaluated are expected in the range from the arithmetic mean minus sigma to the arithmetic mean plus sigma. That is, units with the value of a complex indicator that falls within this interval are recognized as units with an average assessment of scientific activity. Approximately 16% of units whose value of the complex indicator will be greater than the calculated interval will make up a group of units with scientific activity above the average, and the same number of units whose value of the complex indicator will be less than the calculated interval will make up a group of units with scientific activity as below the average. In general, following the structuring and weighting coefficients, a calculation complex was developed to determine the scientific status of a unit, which was performed using an Excel spreadsheet and can be used, along with the evaluation of educational and methodological activities, in the general evaluation of units.
Full Text
INTRODUCTION
Different industries have a problem of assessing performance of departments. It is a complex management task to evaluate quality of products and performance of organizational structures, especially if there are no clear metrics. This includes scientific activities of scientific and educational organizations resulting mainly in development of various types of research products, which is ultimately a performance factor for this activity. As practice shows, evaluation of scientific activities of educational organizations mainly consists of publication- and citation-related activities which are scored by points [1–3].
There are no specific criteria for evaluating research products, and this predetermines the selection of an evaluation approach using both quantitative and qualitative indicators. The problem of combining and unifying these indicators can be solved by using qualimetric methods designed to evaluate any objects. In qualimetry, a measurement scale is a tool used for adequate comparison and calculation of individual parameters and indicators of various objects. In detail, suitable scales are described in papers on theory and methods of statistics [4–6].
Publications about monitoring of scientific activity of universities are focused on qualimetric technology based on a group Delphi method. The Delphi method shall be used for assessing the level of product quality or type of activity where it is impossible or very difficult to use objective methods for determining single or complex indicators [1–3, 7–9].
The S.M. Kirov Military Medical Academy pays great attention to evaluating scientific activity. However, some issues remain unresolved [9, 10].
The aim of our study is to develop a tool for complex assessment of the scientific activity status of the Military Medical Academy departments to determine their scientific rating.
MATERIALS AND METHODS
The aim of this study was to develop an original methodology for assessing quality of scientific activities of the Military Medical Academy, based on a qualimetric assessment of the most possible research areas of organizations and their characteristics. Therefore, the Military Medical Academy and scientific activity of its departments were an object and a subject of the study, respectively. The study used qualimetric methods of expert survey and weighing performance indicators, as well as methods of descriptive statistics such as calculating an arithmetic mean and standard deviation of a random variable (sigma). The expert concordance was assessed using a Kendall Concordance Coefficient.
RESULTS AND DISCUSSION
This study was based in some qualimetry principles. An indicator of any level of generalization, except for the lowest (initial) one, is predetermined by corresponding indicators of the previous hierarchical level. The lowest hierarchical level of indicators shall be considered as single indicators of the simplest properties defining quality. The higher hierarchical level consists of generalized quality indicators. An integral indicator is the quality indicator of the highest hierarchical level. All multidimensional property indicators shall be converted and unified or expressed in dimensionless units. Each indicator of a separate property must be adjusted by its weighting factor (significance). The quality of a whole object (such as a product or a process) is determined by the quality of its constituent parts. When quantifying quality, especially in terms of a complex indicator, it is unacceptable to use interdependent and, therefore, duplicate indicators of the same property [1, 6, 11, 12].
The study was carried out in 5 stages:
- Determining areas of scientific activities of scientific and educational departments as the main structural units of the Academy.
- Determining the list of indicators characterizing each area of scientific activity.
- Expert assessment of the weight of scientific activity indicators.
- Determining the scientific status of the department.
- Determining the scientific rating of the department.
The first 2 stages were the most difficult and time-consuming. They were completed by 25 experts from various Academy departments, who were engaged in scientific activities. Experts were asked to structure scientific activity as a subject of study, highlighting the main areas and indicators. The result was a complex three-dimensional structure, consisting of 11 areas, which included 316 indicators of varying degrees of detail. Work on this basic table revealed areas and indicators with similar content and, ultimately, defined the structure of scientific activity of departments. consisting of 6 areas such as R&D and military-historical (M&H) activities; training of academic staff; publication activity; inventive activity; event management activities; other. Each area was described by a set of specific indicators, the sum of which is 17 (Table).
Table. Sample calculation of the complex indicator of the scientific status of a unit
Таблица. Пример расчета комплексного показателя научного статуса подразделения
Area of activity | Weight of area | Seq No. | Indicators of activity area | Units | Indicator value | Weight of indicator | Calculation |
R&D and M&H | 0.3 | 1.1. | R&D included in a state assignment, as well as contractual ones, including clinical studies and trials as a principal contractor | Number of timely closed works | 2 | 0.178 | 1.42 |
1.2. | R&D as a contractor | Number of R&D completed | 3 | 0.083 | 0.99 | ||
1.3. | M&H | Number of new historical data introduced | 3 | 0.04 | 0.48 | ||
Training of academic staff | 0.25 | 2.1. | Effectiveness of scientific staff training | Number of trained scientific staff according to the plan for academic staff training, % of total academic staff of the department | 5 | 0.006 | 0.12 |
2.2. | Number of scientific staff trained for a fee | 2 | 0.090 | 0.72 | |||
2.3. | Staffing level of the organization for academic staff with a scientific degree | ≥60% — 1, and then + 0.1 for each percentage point | 1 | 0.074 | 0.30 | ||
2.4. | Presence of a registered scientific school | No school — 0; initial-level school, new and degrading — 1; school of high, top level and progressive — 2 | 1 | 0.044 | 0.18 | ||
2.5. | Management of MSSCT discussion group activity | Number of active members with results achieved | 12 | 0.036 | 1.74 | ||
Area of activity | Weight of area | Seq No. | Indicators of activity area | Units | Indicator value | Weight of indicator | Calculation |
Publication activity | 0.2 | 3.1. | Textbooks, monographs, manuals, educational and teaching aids published according to the plan of R&D and E&P activities | Number of typographical units | 10 | 0.090 | 3.60 |
3.2. | Scientific articles included in the international bibliographic databases Scopus and WoS | Number of articles | 3 | 0.051 | 0.62 | ||
3.3. | Scientific articles included in the list of VAK and RSCI | Number of articles | 2 | 0.039 | 0.31 | ||
3.4. | Scientific articles included in RSCI | Number of articles | 3 | 0.02 | 0.24 | ||
Inventive activity | 0.1 | 4.1. | Patents | Number of patents | 2 | 0.08 | 0.64 |
4.2. | Technical innovation proposals | Number of proposals | 0 | 0.02 | 0.00 | ||
Event management activity | 0.1 | 5.1. | Scientific events included in the plan of scientific events according to the plan of R&D | Number of events | 2 | 0.068 | 0.54 |
5.2. | Participation in third-party scientific events (with contributions only) | Number of events | 3 | 0.016 | 0.20 | ||
5.3. | Participation in a exhibition (with exhibit items only) | Number of events | 4 | 0.016 | 0.26 |
Note: E&P — editorial and production; M&H — military-historical activity; MSSCT — military scientific society of cadets and trainees; R&D — research and development; RSCI — Russian Science Citation Index; VAK — Higher Attestation Commission.
The next step was to evaluate the weight of each of the above characteristics using expert opinions. Experts engaged in scientific department activities at various management levels were invited. They included staff engaged in scientific activity management and academic staff training, deputy heads of departments and persons responsible for organizing research activity in departments. In total, 25 specialists determined the weighting factor for each of 17 indicators. The concordance of experts' opinions was assessed using the Kendall's concordance coefficient of 0.82 (p < 0.001), indicating the sufficiently high expert concordance. Based on the expert analysis, the average weighting factor was calculated for each indicator. All factors and coefficients were expressed in fractions of 1, and their sum was equal to 1. See the table for final results.
In this table, the Indicator Value column contains the real department data for the entire set of characteristics. The last column shows results of calculations i. e. the product of indicator values by the weighting factor, divided by the number of academic staff of the department and multiplied by 100. The total score for each indicator is a complex (integral) indicator of the scientific status of the department. In our study, it was 12.36.
Based on the comprehensive assessment of each Academy department, a scale is calculated to determine the rating of such a department. As a scaling tool, a sigmal (using standard deviation) score was chosen, which shows that 68% of all complex estimates of departments being evaluated are expected in the range from the arithmetic mean minus sigma to the arithmetic mean plus sigma (M ± δ) [13, 14]. This means that departments with the complex indicator falling within this range are recognized as departments with an average score of scientific activity. About 16% of departments with the complex indicator exceeding the calculated range will make up a group of departments with the above-average score of scientific activity. The same number of departments with the complex indicator below the calculated range will make up a group of departments with the below-average score of scientific activity.
CONCLUSION
In general, using qualimetric methods allowed to create a complex score for assessing the scientific activity status to determine the rating of departments and to be included in summarizing assessment of departments with assessing educational and methodological activities. The complex score for calculating the scientific status and rating of departments is implemented in an Excel spreadsheet and is currently being tested.
About the authors
Evgeniy V. Ivchenko
Military Medical Academy of S.M. Kirov
Email: latypov_ildar85@mail.ru
ORCID iD: 0000-0001-5582-1111
SPIN-code: 5228-1527
doctor of medical sciences, associate professor
Russian Federation, Saint PetersburgStepan G. Grigoriev
Military Medical Academy of S.M. Kirov
Email: latypov_ildar85@mail.ru
SPIN-code: 2391-4846
doctor of medical sciences, professor
Russian Federation, Saint PetersburgDmitriу V. Ovchinnikov
Military Medical Academy of S.M. Kirov
Email: latypov_ildar85@mail.ru
ORCID iD: 0000-0001-8408-5301
SPIN-code: 5437-3457
candidate of medical sciences, associate professor
Russian Federation, Saint PetersburgIldar F. Latypov
Military Medical Academy of S.M. Kirov
Author for correspondence.
Email: latypov_ildar85@mail.ru
ORCID iD: 0000-0003-4849-9261
SPIN-code: 2926-6607
candidate of medical sciences
Russian Federation, Saint PetersburgReferences
- Zubkova TM, Natochaya EN. Rating assessment of academic staff in a university based on an automated information system. Software and Systems. 2019;(3):525–533. (In Russ.). doi: 10.15827/0236-235X.127.525-533
- Laptev VV, Potemkin MN. Vnutrivuzovskie reitingi kafedr v sfere nauchnoi deyatel’nosti kak mekhanizm aktivizatsii nauchnoi raboty v universitete. Byulleten’ Uchenogo Soveta. 2006;(8(34)):3–20. (In Russ.).
- Sharshov IA. Otsenka effektivnosti nauchno-issledovatel’skoi deyatel’nosti v sovremennom universitete. Psychology-Pedagogical Journal Gaudeamus. 2014;(1(23)):38–44. (In Russ.).
- Azgaldov GG. Immeasurable or unmeasured. Product quality control. 2021;11:44–47. (In Russ.).
- Azgaldov GG, Kostin AV, Metrology and qualimetry: identification issues. The world of torments. 2021;2:69–73. (In Russ.).
- Pankova LA, Petrovsky AM, Schneiderman MV. Organizatsiya ekspertiz i analiz ekspertnoi informatsii. Moscow: Science; 1984. 120 р. (In Russ.).
- Cherepanov VS, Lyubimova OV. Kvalimetricheskii podkhod v obrazovanii. Education and Science Journal. 2009;63(6):36–43. (In Russ.).
- Sheshukova TG. Sergeeva NV. Formirovanie sistemy pokazatelei dlya otsenki effektivnosti nauchnoi deyatel’nosti natsional’nykh issledovatel’skikh universitetov. Economic Analysis: Theory and Practice. 2012;(4(259)):53–63. (In Russ.).
- Ivchenko EV, Ovchinnikov DV, Zubenko AI, etc. Istoricheskii ocherk nauchnogo otdela Voenno-meditsinskoi akademii. Saint Petersburg: VMA; 2016. 152 р. (In Russ.).
- Ovchinnikov DV. Scientific research of military medicine and training of scientific personnel in its interests (to the 90th anniversary of the department of organization of scientific work and training of scientific and pedagogical personnel of the Military Medical Academy). Izvestia of the Russian Military Medical Academy. 2021;40(3):5–12. (In Russ.). doi: 10.17816/rmmar76030
- Ekspertnye otsenki. Beshelev SD, Gurvich FG, editors. Moscow: Nauka; 1973. 157 р. (In Russ.).
- Fadeeva OM. Primenenie kvalimetrii kak teoreticheskoi bazy otsenki kachestva sistemy strategicheskogo upravleniya vysshim uchebnym zavedeniem. Sotsial’naya Politika i Sotsiologiya. 2013; (4-1(96)):24–35. (In Russ.).
- Korneshchuk NG, Rubin GSh, Abramova TV. Kvalimetriya kak instrument otsenki kachestva obrazovaniya. Standarty i Monitoring v Obrazovanii. 2006;(3):16–9. (In Russ.).
- Junkerov VI, Grigoriev SG, Rezvantsev MV. Matematiko-statisticheskaya obrabotka dannykh meditsinskikh issledovanii. Saint Petersburg: VMA; 2011. 318 р. (In Russ.).
Supplementary files
