A method for analyzing the trustworthiness of automated assessments

Cover Page

Cite item

Full Text

Open Access Open Access
Restricted Access Access granted
Restricted Access Subscription or Fee Access

Abstract

The study addresses the issues related to the evaluation of trustworthiness of assessments used in decision-making tasks. The relevance of evaluating the trustworthiness of assessment results within automated systems is demonstrated. А method for assessing the trustworthiness of automated assessments is proposed, based on the principle of trust transitivity. The validation of this method is described in the context of an automated system for assessing the functional state of software users. The applicability of the designed method for analysing the trustworthiness of assessments obtained in software systems that automate various aspects of human activity, requiring the assessment of objects and/or processes, also demonstrated.

Full Text

Restricted Access

About the authors

A. G. Davtyan

Moscow Institute of Physics and Technology

Author for correspondence.
Email: agvs@mail.ru

Cand. of Phys. And Math. Sc., Associate Professor

Russian Federation, Moscow

O. A. Shabalina

Volgograd State Technical University

Email: O.A.Shabalina@gmail.com

Cand. of Tech. Sc., Associate Professor

Russian Federation, Volgograd

A. A. Voronina

Volgograd State Technical University

Email: angelina.vaa@gmail.com

Postgraduate Student

Russian Federation, Volgograd

I. A. Molodtsova

Volgograd State Medical University

Email: irina.molodtsova@gmail.com

Cand. of Med. Sc., Associate Professor

Russian Federation, Volgograd

E. P. Khvastunova

Moscow University named after S. Yu. Witte

Email: elena_volga68@mail.ru

Cand. of Soc. Sc., Associate Professor

Russian Federation, Moscow

References

  1. Stojanovic A., Marič M. Decision-Making in Private and Professional Life, Proceedings of the 42nd International Conference on Organizational Science Development: Interdisciplinarity Counts, 2023, pp. 791—801, doi: 10.18690/um.fov.3.2023.59.
  2. Momdzhyan K. Kh. The decision-making process and its place in human activity, Elektronnoe nauchnoe izdanie "Al’manakh Prostranstvo i Vremya", 2015, no. 2 (in Russian).
  3. Kozioł-Nadolna K., Beyer K. Determinants of the decision-making process in organizations, Procedia Computer Science, 2021, vol. 92, pp. 2375—2384, doi: 10.1016/j.procs.2021.09.006.
  4. Diev V. S. The problem of choice and decision-making in an interdisciplinary context, Vestnik Tomskogo Gosudarstvennogo Universiteta. Filosofiya. Sotsiologiya. Politologiya, 2013, no. 2 (22) (in Russian).
  5. Serebryakova E. A., Barkalov S. A., Moiseev S. I. Evaluation of alternatives: new approaches to decision-making under certainty, Vestnik YuUrGU. Seriya: Komp’yuternye Tekhnologii, Upravlenie, Radioelektronika, 2024, no. 3 (in Russian).
  6. Suntsova Ya. S., Meshkov I. Yu. Phenomena of choice and decision-making in psychological science, Vestnik Udmurtskogo Universiteta, Seriya "Filosofiya. Psikhologiya. Pedagogika", 2022, no. 4 (in Russian).
  7. Sutuzhko V. V. Structure, principles and classification of the concept "Evaluation", Epistemology & Philosophy of Science, 2008, no. 4 (in Russian).
  8. Kotelnikov R. V., Martynyuk А. A. Mathematical assessment of the reliability of information about forest fires, Izvestiya Vysshikh Uchebnykh Zavedeniy. Lesnoy Zhurnal, 2023, no. 3 (393), pp. 21—34 (in Russian).
  9. Korosteleva O. N., Savinov G. V. Statistical analysis of expert assessments, Izvestiya Sankt-Peterburgskogo gosudarstvennogo ekonomicheskogo universiteta, 2023, no. 3-2 (141) (in Russian).
  10. Zatsarinnyy A. A., Ionenkov Yu. S. On the use of expert methods in assessing the efficiency and quality of information systems, Sistemy i sredstva inform., 2022, vol. 32, no. 2, pp. 47—57 (in Russian).
  11. Brown G. T. L. Editorial: Insights in assessment, testing, and applied measurement: 2022, Frontiers in Education, 2024, vol. 9, doi: 10.3389/feduc.2024.1488012.
  12. Mach K. J., Mastrandrea M. D., Freeman P. T., Field C. B. Unleashing expert judgment in assessment, Global Environmental Change, 2017, vol. 44, pp. 1—14.
  13. Serenkov P. S. Analysis of methods for researching the level of trust in expert assessments, Proceedings of the 8th International Scientific and Technical Conference, 2015, vol. 1, pp. 192—193 (in Russian).
  14. Volkov V. Yu., Volkova V. V. Improving the reliability of group assessment of expert opinions in the formation of an expert opinion, Izvestiya Tul’skogo Gosudarstvennogo Universiteta. Tekhnicheskie nauki, 2021, no. 9 (in Russian).
  15. Volkov V. Yu., Maslova N. V., Tsyganov N. N. Formulation of the problem of increasing the reliability of expert assessments in an intellectual control system, Izvestiya Tul’skogo gosudarstvennogo universiteta. Tekhnicheskie nauki, 2014, no. 2 (in Russian).
  16. Gostyunina V. A., Davidyuk N. V., Davidyuk V. V., Baitumenov А. Z. Procedure for assessing the degree of trust in an expert using the example of the task of age categorization of web content, Vestnik AGTU. Seriya: Upravlenie, vychislitel’naya tekhnika i informatika, 2019, no. 2 (in Russian).
  17. Mel’nik P. B., Ilyushchenko R. R. Reliability of expert assessments in the examination of scientific, technical and innovative projects in the information system of the federal register of experts, Innovatika i ekspertiza: nauchnye trudy, 2020, no. 1 (29), pp. 46—58 (in Russian).
  18. Sadchikov P. N., Soboleva V. V. Automated system for assessing the effectiveness of educational technologies, Inzhenerno-stroitel’nyy vestnik Prikaspiya, 2021, no. 4 (38) (in Russian).
  19. Lavrov E., Paderno P., Burkov E., Volosiuk A., Vu L. Expert assessment systems to support decision-making for sustainable development of complex technological and socioeconomic facilities, E3S Web of Conferences, 2020, vol. 166, art. 11002, doi: 10.1051/e3sconf/202016611002.
  20. Korytnev A. V., Semenov S. S. On the issue of the quality and reliability of expert assessments in determining the technical level of complex systems, Nadezhnost’, 2013, no. 4, pp. 90—109 (in Russian).
  21. Kachaykin E. I., Ivanov A. I., Bezyaev A. V., Perfilov K. A. Assessing the reliability of neural network automated examination of handwritten authorship, Voprosy kiberbezopasnosti, 2015, no. 2 (10) (in Russian).
  22. Rybakov А. Improving the quality and reliability of computer assessment of knowledge in automated learning control systems, Pedagogicheskie izmereniya, 2014, no. 2 (in Russian).
  23. Kohn S. C., de Visser E., Wiese E., Lee Y.-C., Shaw T. H. Measurement of Trust in Automation: А Narrative Review and Reference Guide, Frontiers in Psychology, 2021, vol. 12, doi: 10.3389/fpsyg.2021.604977.
  24. Voronina A. A., Shabalina O. A., Sadovnikova N. P., Gur’ev V. V. Models and methods for automated person’s functional status assessment, Matematicheskie metody v tekhnologiyah i tekhnike, 2025, no. 3, pp. 23—26 (in Russian).
  25. Davtyan A. G., Shabalina O. A., Berestneva O. G. Analysis of reliability of assessment based on the principle of trust transitivity, Matematicheskie metody v tekhnologiyah i tekhnike, 2025, no. 4, pp. 39—43 (in Russian).
  26. Voronina А., Guriev V., Moffat D. С., Molodtsova I. Models and Methods for Processing Heterogeneous Data for Assessing the State of a Human, Creativity in Intelligent Technologies and Data Science: 5th International Conference CIT&DS 2023, 2023, p. 488—499, doi: 10.1007/978-3-031-44615-3_34.Ris ina

Supplementary files

Supplementary Files
Action
1. JATS XML
Download
2. Fig. 1. Experimental process

Download (100KB)
Indexing metadata
3. Fig. 2. Graph of consistency measures for user FS ratings: a — for engagement ratings; b — for stress ratings

Download (58KB)
Indexing metadata
4. Fig. 3. Graphs of consistency measures for the FS ratings of six software users: a — for engagement ratings; b — for stress ratings

Download (159KB)
Indexing metadata
5. Fig. 4. Histograms of consistency measures for the FS ratings of all software users in the observation group: a — histogram of consistency measures for engagement ratings; b — histogram of consistency measures for stress ratings

Download (66KB)
Indexing metadata

Copyright (c) 2026 Informacionnye Tehnologii


СМИ зарегистрировано Федеральной службой по надзору в сфере связи, информационных технологий и массовых коммуникаций (Роскомнадзор).
Регистрационный номер и дата принятия решения о регистрации СМИ: серия ПИ № 77 - 15565 от 02 июня 2003 г.