Automation analysis X-ray of the spine to objectify the assessment of the severity of scoliotic deformity in idiopathic scoliosis: a preliminary report

Cover Page


Cite item

Full Text

Abstract

Background. A large number of studies have focused on automating the process of measuring the Cobb angle. Although there is no practical tool to assist doctors with estimating the severity of the curvature of the spine and determine the best suitable treatment type.

Aim. We aimed to examine the algorithms used for distinguishing vertebral column, vertebrae, and for building a tangent on the X-ray photographs. The superior algorithms should be implemented into the clinical practice as an instrument of automatic analysis of the spine X-rays in scoliosis patients.

Materials and methods. A total of 300 digital X-rays of the spine of children with idiopathic scoliosis were gathered. The X-rays were manually ruled by a radiologist to determine the Cobb angle. This data was included into the main dataset used for training and validating the neural network. In addition, the Sliding Window Method algorithm was implemented and compared with the machine learning algorithms, proving it to be vastly superior in the context of this research.

Results. This research can serve as the foundation for the future development of an automated system for analyzing spine X-rays. This system allows processing of a large amount of data for achieving >85% in training neural network to determine the Cobb angle.

Conclusions. This research is the first step toward the development of a modern innovative product that uses artificial intelligence for distinguishing the different portions of the spine on 2D X-ray images for building the lines required to determine the Cobb angle.

Full Text

Scoliotic deformities represent the most common orthopedic pathology in the childhood, with idiopathic scoliosis diagnosed in 80% of the children across the world. For the treatment and rehabilitation of patients with idiopathic scoliosis, the main approach adopted to confirm the diagnosis and to assess the prognosis and treatment outcomes is 2D radiography. Some other diagnostic methods are based on obtaining a 3D image, such as through CT or MRI, but they are more expensive and laborious. As a result, X-ray remains the leading approach for diagnosing scoliosis owing to its widespread availability and sufficient understanding. Historically, several methods have been so far used to assess the severity of the spine scoliotic deformity, namely those proposed by A.B. Ferguson, J.R. Cobb, G. Jentschura, and E.A. Abalmasova [1–4]. However, the Cobb angle is most commonly used as an objective indicator of scoliosis severity [5–7].

The main disadvantages of all of these abovementioned methods include the need to identify a “neutral” vertebra as a rough estimate as well as a high measurement error owing to the use of nonspecific instruments for plotting. It is believed that the analysis data of the same radiograph by several specialists may significantly differ [8]. The measurement error when using manual instruments reaches 5–7°, while it is no more than 3° when using digital methods under the stipulation that the same vertebrae are selected as “neutral”.

Literature review has indicated that since the past 10 years, a large number of studies have been conducted on digital software solutions for calculating the Cobb angle on radiographs; this increase in such studies can primarily be attributed to the development of digital radiography [9–13]. At the same time, the proposed solutions are actually either semi-automatic in situations where the doctor determines the initial and final vertebrae of the arches or are completely manual, albeit with the use of software that enables zooming in on an image and drawing tangents to the vertebral body projection more accurately [14]. There is a possibility of using smartphones with this software as well as the Microsoft PowerPoint program [15, 16] for measuring the grade of deformity, although these measurements are not conducted automatically and have the same disadvantages as that of desktop applications.

In the recent years, the potential possibility of using software based on neural network technologies and machine learning has been conducted to automatically determine the deformity severity. Thus, Pan et al. [13] analyzed chest radiographs to automatically isolate the spinal column with subsequent identification of individual vertebrae. An algorithm was also used to calculate the Cobb angle on detecting scoliotic deformity, which is a deformity with an angle of >3° between the most inclined endplates of different vertebrae. However, the program functioned correctly only when there was one deformity arch in the image. Horng et al. [12] noted the possibility of using neural networks to isolate the vertebral boundaries, provided that the spinal column was isolated using deterministic algorithms and the vertebrae were segmented on it [12]. The application of these algorithms requires a certain image structure (for example, the presence of a skull and cervical spine). These algorithms can be used to automatically measure the Cobb angle; however, this study only analyzed radiographs with scoliotic deformity of <20° and insufficient numbers of radiographs [35]. This small sample size makes it apparent the great difficulty in reproducing these results in other conditions. Thus, further profound studies and automation of the analysis of radiographs are warranted to determine the grade of the spine scoliotic deformity.

The present study aimed to investigate the algorithms for identifying the spinal column and vertebrae on an X-ray image and constructing tangent lines to the intervertebral discs for subsequent automated analysis of the radiographs of the spine of patients with idiopathic scoliosis in order to assess its severity.

Materials and methods

At the stage 1 of the X-ray image analysis, the vertebrae were marked; namely, the spinal column and vertebrae were isolated and tangents were drawn to the vertebrae. Based on the shape of the spinal column and the angles of the tangents to the vertebrae drawn, the doctor makes a diagnosis and creates a treatment plan. It should be noted that the tangents to the vertebrae on an X-ray image are usually not tangent in the mathematical word-sense, since a flat figure on an X-ray image represents a projection of a three-dimensional vertebra. Therefore, hereinafter, tangents indicate characteristic straight lines, which, in essence, represent projections of the tangent planes to the vertebra on a radiograph plane. For this study, we selected 300 digital radiographs of the spine of children and adolescents undergoing treatment at the Prosthetic and Orthopedic Center “Skoliologic.ru” for idiopathic scoliosis. To determine the grade of scoliosis with reference to the Cobb angle value, radiographs were drawn manually by a certified radiologist. The analyzed patients included 249 girls (83%) and 51 boys (17%). The age-wise distribution revealed 6 patients aged 1–3 years (2%), 45 aged 3–7 years (15%), 90 aged 7–12 years (30%), and 159 aged 12–18 years (53%). Based on the grade of scoliosis in accordance with the value of the Cobb angle, the radiographs distribution revealed 15% of the radiographs with grade I of scoliosis (angle up to 15–20º); 25% with grade II (angle up to 40º); 45% with grade III (angle up to 60º); and 15% with grade IV (angle >60°). In addition, the radiographs differed in the type and number of arches of scoliotic deformity (Fig. 1) and corresponded to the classification offered by Rigo et al., as recommended by the Society on Scoliosis Orthopedic and Rehabilitation Treatment for the conservative treatment of scoliosis [17].

 

Fig. 1. Types of arches of idiopathic scoliosis in accordance with the classification of Rigo et al., 2010: TP — transitional point that could be located between the thoracic arch and the lumbar or thoracolumbar one relative to the central sacrum line; TP on the central sacrum line indicate that its balanced, while that installed beyond the line specified indicates its imbalance. T — thoracic, L — lumbar, CSL — central sacrum line

 

From these radiographs, a neural network training dataset was created to isolate the spinal column and individual vertebrae. Thus, our task was to programmatically determine the line of the spinal column and individual vertebrae as well as to draw tangents to individual vertebrae to create an architecture consisting of 2 independent blocks. The block 1 highlighted the line of the spinal column and individual vertebrae and the block 2 drew tangents and calculated angles based on them. We employed 2 methodological approaches, namely a deterministic sliding window method and an algorithm based on a neural network.

Results

The deterministic sliding window method was applied on large radiographs of the spine, from the cervical to the lumbar spine. The difference in the arrangement (structure) of the images did not enable determination of the area containing the spinal column, which was based only on brightness. As a result, the areas presented by the algorithm were much wider than the spinal column and had to be redefined based on the available statistical information.

The next step represented isolation of areas on the spinal column containing the individual vertebra. This problem was resolved by regulating the brightness of the vertical projection of the spinal column. In our case, this algorithm did not allow obtaining high-quality outputs, as the images were often very blurry. In addition, in the case of a high-grade scoliosis, the projection on the vertical axis often turned into a short-length segment. This entire situation prevented determination of significant deviations in the brightness between the vertebra and the intervertebral disc. As a result, we used the curved windows to search for areas of a sharp decrease in brightness (intervertebral discs) and a sharp increase in brightness (projection of the vertebrae upper boundaries). The intersection of these areas with the line of the spinal column indicated the centers of the intervertebral discs or the vertebrae boundaries. Using the line of the spinal column and the borders of the vertebrae, the angles of the spine curvature (the Cobb angles) could be calculated. However, in most of these images, the deterministic sliding window method failed to yield acceptable outcomes.

The development of a neural network-based algorithm involved the formulation of a problem to be resolved by the algorithm, the choice of the neural network architecture, the assessment of the resources available for training, the elaboration of the loss function, the solution of the issue of data augmentation in order to artificially increase the dataset, as well as the formulation of data preparation criteria for the model.

The problem that the model resolves was to form the spinal column and to approximate the horizontal borders of the vertebrae to the extent possible to the straight lines tangent to the vertebra. This task belongs to the Semantic segmentation class or, in other words, we the points had to be divided into 2 classes: those belonging and not belonging to the spine. Numerous past publications support that the convolutional layers with pulling can perfectly cope with such tasks; therefore, convolutional networks were selected as the main ones. To implement this model, the neural networks (fully connected; Mask [R-CNN]; FCN [Fully Convolutional Networks]; ResNet with a wide output layer; U-Net; ParseNet) were analyzed in detail [18–22].

The final choice could not be based only on theoretical recommendations, since radiographs are characterized with certain specificity in relation to other graphic images, as they are black and white. In addition, the spine area is relatively smaller to the image area, and the spine is often difficult to distinguish from other elements of the image (for example, the ribs). On the radiograph, there may be extraneous images in the spine area, and the careless use of augmentation can lead to unacceptable image distortion. In particular, owing to spine radiograph specificity, additional training of previously trained networks could not be implemented. In this regard, it was decided to conduct basic testing for each potentially promising convolutional network, followed by evaluation of the results to make the final choice. Basic testing was performed on a small dataset (50 X-ray images); as a result, the estimated characteristics could be quickly obtained. The comparison was performed by the final value of the Accuracy parameter and by visual assessment of the results. The testing was conducted in parallel, and, obtaining the spine contours, even without isolating the vertebrae, was considered as an acceptable output at this stage.

The fully connected network caused huge problems considering that there were several layers of extremely large size. Even for images of 128 × 128 pixels in one layer, about 16,000 numbers were obtained. The test revealed a completely unacceptable output; therefore, the fully connected network was no longer considered. Mask (R-CNN) actually represented a combination of several networks working in the series. As in the case of a fully connected network, this solution required an extremely large dataset, which was discarded based on the results of preliminary testing. FCN network also caused issues not only due to the extremely long training period but also the fact that the test cases provided unsatisfactory outputs. ResNet produced the first promising results since the last few layers were replaced by fully connected networks. However, it did not function properly with a variety of shots. It seemed that this network was simply memorizing the training set and the results were good, if after training, only images from the training set were input. It was thus concluded that the use of ResNet to solve the issue was impractical. The U-Net network immediately showed a good result on a small dataset, and there was a systematic uniform increase in Accurancy from epoch to epoch. The values of this parameter were much higher than those of other networks. Further attempts to apply the ParseNet network did not provide positive results.

Therefore, according to the results of basic testing on images sized 128 × 128 pixels, the best result was presented by the U-Net network (Fig. 2), wherein cross entropy with the Adam optimization algorithm was used as a loss function.

 

Fig. 2. Neural network U-Net

 

The creation of an algorithm for constructing a tangent to the circled vertebra turned out to be a challenging task due to the fact that the tangent to the circled vertebra, as already discussed, is not a tangent in the mathematical sense of the word. It was therefore decided to construct a tangent line as a straight line at approximating the upper (lower) border of the vertebra.

Data preparation is one of the key issues in this task. Since a supervised machine-learning model was selected, a large amount of well-trained data was required. At the same time, the number of specialists preparing the data was required to be minimal, and their qualifications were supposed to be high, since the convolutional layers considers the characteristics in the vicinity of the point, mandating uniform criteria for the formation of the vertebral boundaries. Moreover, the size of the images presented to the model for training should provide distinguishable boundaries between the vertebrae. The lowest image resolution that provided an acceptable result was 256 × 256 pixels. Augmentation was used to increase the amount of data presented to the model for training. In this case, the parameters should be varied extremely carefully, considering that excessive rotation or compression/shear can lead to merging of the boundaries between the intervertebral discs, which in turn can render the solution of the problem impossible. There should be a minimum of 250 images to obtain an acceptable result. In addition, it is advisable to prepare image masks not with a clear outline of the vertebra, rather by “straightening” the upper and lower (“horizontal”) boundaries of its projections. This step greatly improves the performance of the deterministic tangent drawing algorithm.

As a result, the foundations of a computer system for the automation of radiograph analysis were created. Figure 3 presents the result of this neural network.

 

Fig. 3. An example of the result of the neural network operation

 

Figure 3 displays a spinal column with segmented vertebrae. The data prepared by the method described above ensured the straightening of the vertebral boundaries and, in most cases, the confident drawing of tangents to the vertebrae and the determination of the spinal deformity. To calculate the Cobb angle, it was necessary to supplement the algorithm for determining the vertebrae, the tangents to which determined the angle in question. For this purpose, the central line of the spinal column was determined and analyzed. Thus, the global parameter “Cobb angle” was calculated by considering the global characteristic “curvature of the spinal column” (Fig. 4).

 

Fig. 4. An example of the result of the program: a — the result of the program; b — the same image processed manually (digits in squares indicate the values of the Cobb angle of scoliotic deformity arches)

 

Training the neural network on a well-prepared dataset enabled accurate determination of the Cobb angle on >85% of the radiographs. In 15% of them, the results were considered unsatisfactory; 12% of these failures were associated with insufficient image clarity (such as dynamic smear and blurring of the contours of the vertebral bodies) and 3% were associated with a small value of scoliotic deformity (0–5°). With such magnitudes of scoliotic deformity at this stage, the program does not provide the correct values, warranting further performance improvement.

Discussion

The computerization of 2D radiographic images, unlike for 3D images, has not been developed so intensively, which explains the large number of errors in the interobserver or intraobserver conclusions on the quantitative assessment of the scoliosis grade. Therefore, the development of automated computer programs for measurements represents an important research topic for the clinical data objectivation in the future.

The experience of international specialists in the analysis automation of the spine 2D-radiographs suggest that the main challenges with this approach include improvement of the contrast ratio of the radiograph for more accurate identification of the spinal column and vertebrae as well as drawing tangents to the vertebrae, as also confirmed by the present study [9–14, 23, 24]. In order to resolve these problems, the expediency of using a machine-learning model based on a neural network has already been determined in conformance with the literature data.

Data preparation was selected as one of the key tasks, with considerable focus on creating a dataset of radiographs for training the neural network. If most of the research results were based on the analysis of 14–49 radiographs, then we trained the network on 300 digital radiographs of the spine of patients with idiopathic scoliosis, previously drawn by a radiologist [9–14, 25]. A comparable amount of research has been conducted by Wang et al. and Pan et al. [11, 13]. The use of a volumetric dataset for training a neural network ensured the coincidence of tangents drawn automatically with tangents constructed by an experienced specialist in >85% of the cases. Thus, in order to minimize the error in the quantitative assessment of scoliosis, large data samples and convolutional neural networks should be used to enable recognition of the grade of scoliotic deformity.

Conclusion

We evaluated an automated method for determining the scoliosis grade on a spinal radiograph. The possibility of using a machine-learning model based on a neural network for this process was thus confirmed. At the same time, the convolutional neural network U-Net was found to provide the best indicators for vertebral recognition. It should also be noted that only a large amount of well-prepared data (e.g., 300 X-ray images) enabled training of the neural network to achieve the correct determination of the Cobb angle for >85% of the radiographs. This system will be accompanied by a graphical interface where the doctor can download the image and obtain the result. Although the results obtained can already be considered satisfactory, the approach proposed should be developed primarily through additional training of the model and refinement and optimization of deterministic algorithms in order to create a modern domestic innovative product based on neural network technologies for recognizing 2D images of the spine and automatic plotting of Cobb angles in the future.

Additional information

Source of funding. The work was supported by the Innovation Promotion Fund grant 44GRCTS10-D5/56135.

Conflict of interests. The authors declare no conflict of interests.

Ethical statement. The pilot study was approved by the local ethics committee. Extract from Minutes No. 2 of the Ethics Committee of Skoliologic.ru as of 05/19/2020. The patients (their representatives) provided their consent for processing and publication of personal data.

Author contributions

G.A. Lein developed the methodology and research design as well as wrote the article.

N.S. Nechaeva prepared the data to create a dataset of spine radiographs.

G.M. Mamedova performed the literature analysis, model training, and participated in data preparation.

A.A. Smirnov analyzed the literature and developed deterministic algorithms for drawing the tangents.

M.M. Statsenko was involved in the justification of the choice of a neural network for the formation of a model and created a model training program.

All authors made significant contributions to the research and preparation of the article, and all authors read and approved the final version of the manuscript before its publication.

×

About the authors

Grigory A. Lein

Scoliologic.ru Limited Liability Company

Author for correspondence.
Email: Lein@scoliologic.ru
ORCID iD: 0000-0001-7904-8688

MD, traumatologist-orthopedist, PhD, General Director of Scoliologic.ru LLC

Russian Federation, Saint Petersburg

Natalia S. Nechaeva

Scoliologic.ru Limited Liability Company

Email: n.nechaeva@scoliologic.ru
ORCID iD: 0000-0003-3510-9164

MD, scientific worker, radiologist

Russian Federation, Saint Petersburg

Gulnar М. Mammadova

INPRIS Limited Liability Company

Email: mgm.gulnar@gmail.com
ORCID iD: 0000-0001-9738-9259

analyst

Russian Federation, Moscow

Andrey A. Smirnov

INPRIS Limited Liability Company

Email: smirnov.andrey.aleksandrovich@gmail.com
ORCID iD: 0000-0002-7062-5677

Analyst

Russian Federation, Moscow

Maxim M. Statsenko

Mail.ru Limited Liability Company

Email: maxstatsenko@gmail.com
ORCID iD: 0000-0002-6826-9116

head of the development team

Russian Federation, Moscow

References

  1. Ferguson AB. The study and treatment of scoliosis. South Med J. 1930;23(2):116-120.
  2. Сobb JR. Outline for the study of scoliosis. Instr Course Lect AAOS. 1948;5:261-275.
  3. Jentschura G. Zur pathogenese der säuglingsskoliose. Archiv für orthopädische und Unfall-Chirurgie, mit besonderer Berücksichtigung der Frakturenlehre und der orthopädisch-chirurgischen Technik. 1956;48(5):582-603.
  4. Абальмасова Е.А. Сколиоз в рентгеновском изображении и его измерение // Ортопедия и травматология. – 1964. – № 5. – С. 49–50. [Abalmasova EA. Skolioz v rentgenovskom izobrazhenii i ego izmerenie. Ortopediya i travmatologiya. 1964;(5):49-50. (In Russ.)]
  5. Тесаков Д.К., Тесакова Д.Д. Рентгенологические методики измерения дуг сколиотической деформации позвоночника во фронтальной плоскости и их сравнительный анализ // Проблемы здоровья и экологии. – 2007. – № 3. – С. 94–103. [Tesakov DK, Tesakova DD. Roetgenological methods of scoliotic spine deformity estimation in frontal plane and their comparative analysis. Problemy zdorov’ya i ekologii. 2007;(3):94-103. (In Russ.)]
  6. SOSORT. Методические рекомендации SOSORT 2011 г. Ортопедическое и реабилитационное лечение подросткового идиопатического сколиоза. 2011. [SOSORT. Metodicheskie rekomendatsii SOSORT 2011 g. Ortopedicheskoe i reabilitatsionnoe lechenie podrostkovogo idiopaticheskogo skolioza. 2011. (In Russ.)]
  7. Ньютон П.О., О’Браен М.Ф., Шаффлбаргер Г.Л., и др. Идиопатический сколиоз. Исследовательская группа Хармса: руководство по лечению. – М.: Лаборатория знаний, 2018. – 479 с. [Newton PO, O’Brien MF, Schafflebarger GL, et al. Idiopaticheskiy skolioz. Issledovatel’skaya gruppa Kharmsa: Rukovodstvo po lecheniyu. Moscow: Laboratoriya znaniy; 2018. 479 p. (In Russ.)]
  8. Wilson MS, Stockwell J, Leedy MG. Measurement of scoliosis by orthopedic surgeons and radiologists. Aviat Space Environ Med. 1983;54(1):69-71.
  9. Tanure MC, Pinheiro AP, Oliveira AS. Reliability assessment of Cobb angle measurements using manual and digital methods. Spine J. 2010;10(9):769-774. https://doi.org/10.1016/j.spinee.2010.02.020.
  10. Suwannarat P, Wattanapan P, Wiyanad A, et al. Reliability of novice physiotherapists for measuring Cobb angle using a digital method. Hong Kong Physiother J. 2017;37:34-38. https://doi.org/10.1016/ j.hkpj.2017.01.003.
  11. Wang J, Zhang J, Xu R, et al. Measurement of scoliosis Cobb angle by end vertebra tilt angle method. J Orthop Surg Res. 2018;13(1):223. https://doi.org/10.1186/s13018-018-0928-5.
  12. Horng MH, Kuok CP, Fu MJ, et al. Cobb angle measurement of spine from X-Ray images using convolutional neural network. Comput Math Methods Med. 2019;2019:6357171. https://doi.org/10.1155/ 2019/6357171.
  13. Pan Y, Chen Q, Chen T, et al. Evaluation of a computer-aided method for measuring the Cobb angle on chest X-rays. Eur Spine J. 2019;28(12):3035-3043. https://doi.org/10.1007/s00586-019-06115-w.
  14. Safari A, Parsaei H, Zamani A, Pourabbas B. A Semi-Automatic algorithm for estimating Cobb angle. J Biomed Phys Eng. 2019;9(3):317-326. https://doi.org/10.31661/jbpe.v9i3Jun.730.
  15. Qiao J, Liu Z, Xu L, et al. Reliability analysis of a smartphone-aided measurement method for the Cobb angle of scoliosis. J Spinal Disord Tech. 2012;25(4):E88-92. https://doi.org/10.1097/BSD.0b013e3182463964.
  16. Jones JK, Krow A, Hariharan S, Weekes L. Measuring angles on digitalized radiographic images using Microsoft PowerPoint. West Indian Med J. 2008;57(1):14-19.
  17. Rigo MD, Villagrasa M, Gallo D. A specific scoliosis classification correlating with brace treatment: Description and reliability. Scoliosis. 2010;5(1):1. https://doi.org/10.1186/1748-7161-5-1.
  18. He К, Gkioxari G, Dollár P, Girshick R. Mask R-CNN. 2017. arXiv: 1703.06870.
  19. Long J, Shelhamer E, Darrell T. Fully Convolutional Networks for Semantic Segmentation. 2014. arXiv: 1411.4038.
  20. He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition. 2015. arXiv: 1512.03385.
  21. Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. 2015. arXiv: 1505.04597.
  22. Liu W, Rabinovich A, Berg AC. ParseNet: Looking Wider to See Better. 2015. arXiv: 1506.04579.
  23. Mukherjee J, Kundu R, Chakrabarti A. Variability of Cobb angle measurement from digital X-ray image based on different de-noising techniques. Int J Biomed Eng Technol. 2014;16(2):113. https://doi.org/10.1504/ijbet. 2014.065656.
  24. Okashi OA, Du H, Al-Assam H. Automatic spine curvature estimation from X-ray images of a mouse model. Comput Methods Programs Biomed. 2017;140:175-184. https://doi.org/10.1016/j.cmpb.2016.12.010.
  25. Pinheiro AP, Coelho JC, Veiga ACP, Vrtovec T. A computerized method for evaluating scoliotic deformities using elliptical pattern recognition in X-ray spine images. Comput Methods Programs Biomed. 2018;161:85-92. ttps://doi.org/10.1016/j.cmpb.2018.04.015.

Supplementary files

Supplementary Files
Action
1. Fig. 1. Types of arches of idiopathic scoliosis in accordance with the classification of Rigo et al., 2010: TP — transitional point that could be located between the thoracic arch and the lumbar or thoracolumbar one relative to the central sacrum line; TP on the central sacrum line indicate that its balanced, while that installed beyond the line specified indicates its imbalance. T — thoracic, L — lumbar, CSL — central sacrum line

Download (492KB)
Indexing metadata
2. Fig. 2. Neural network U-Net

Download (173KB)
Indexing metadata
3. Fig. 3. An example of the result of the neural network operation

Download (84KB)
Indexing metadata
4. Fig. 4. An example of the result of the program: a — the result of the program; b — the same image processed manually (digits in squares indicate the values of the Cobb angle of scoliotic deformity arches)

Download (266KB)
Indexing metadata

Copyright (c) 2020 Lein G.A., Nechaeva N.S., Mammadova G.М., Smirnov A.A., Statsenko M.M.

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.
СМИ зарегистрировано Федеральной службой по надзору в сфере связи, информационных технологий и массовых коммуникаций (Роскомнадзор).
Регистрационный номер и дата принятия решения о регистрации СМИ: серия ПИ № ФС77-54261 от 24 мая 2013 г.


This website uses cookies

You consent to our cookies if you continue to use our website.

About Cookies