3D-Oculography: New Method of Determination of Human Gaze Point Position in Space

Full Text

LIST OF ABBREVIATIONS

CS — coordinate system

GP — gaze point

rel. units—relative units

INTRODUCTION

Oculography methods of have been known for quite a long time [1, 2] and are used to solve a number of diagnostic tasks determination of pronounced nystagmus, of characteristic changes in human oculomotor reactions under changing conditions of observation. There are known attempts to use oculographic signals to control various devices used by patients who have lost the ability to move [3, 4]. A possibility to evaluate the angular position of the eyeball in the horizontal and vertical planes [5] has led to implementation of the so-called eye-trackers (a device following eye movement), which are widely used to obtain information about gaze point (GP) movement trajectory, to determine the areas of increased attention within the observer’s field of vision [6]. Besides, of interest is determining the position of such area in three-dimensional space, as well as the dynamics of its movement. It can be argued that having determined the trajectory of movement of this area, it is possible to characterize the trajectory of movement of an object which is picked out by the observer’s cognitive system against a certain background. This system also follows movement of the object activating muscles moving the eyeballs. Thus, the method of obtaining information about position of GP in space is of interest both to physiologists and, in future, to practicing physicians.

The aim of this study to experimentally prove the functionality of the developed method of 3D-oculography permitting determination of the gaze point position in space.

MATERIALS AND METHODS

The idea underlying the developed 3D-oculography method is based on the principles of stereoscopic vision, when the image of an object is formed by two organs of vision (formally, by even a greater number, as in some insects). The distance between the centers of eye pupils of a human observer (b_OU), called the base, determines the achievable resolution of positions of the object relative to the geometric center of the human ‘system of observation’ let us call it that, using the concepts familiar for optics specialists. Figure 1a presents a scheme of observation of an object moving along a circle of radius R with the center having coordinates x_c, y_c in the rectangular coordinate system (CS) XOZ, which we will call the coordinate system (CS) of observation.

 

Fig. 1. Geometry of the task to determine the coordinates of gaze point when observing an object moving along a circle of radius R in the XOZ plane (a), and of the coordinates of this point in the left Cartesian coordinate system (b). Notes: Positions of pupil centers of the observer’s eyes are designated with points OS and OD the trajectory of the moving object Ω is shown by a dotted line (b).

 

Assume that the positions of the centers of the pupils of the left and right eyes of a human observer are determined by the coordinates (x_S, 0) and (x_D, 0) of points O_S and O_D, respectively, in this CS. The coordinates x_Ù and z_Ù of the intersection point of the optical axes of the left and right eyes, oriented by the oculomotor muscles to observe the object Ù , are determined by solving the system of equations:

$\left\{\begin{array}{l}{z}_{\Omega }=(x_{\Omega }-x_S)\cdot tg\left(90-{\alpha }_S\right)\\ z_{\Omega }=(x_{\Omega }-x_D)\cdot tg\left(90-{\alpha }_D\right)\end{array}\right.$ (1),

where á_S and á_D are angles that determine positions of the optical axes of the left and right eyes relative to OZ axis of CS of observation.

A general case of observing an object is shown in Figure 1b. To determine the coordinates of an object Ù in the three-dimensional XYOZ coordinate system, besides coordinates x_Ù and z_Ù, it is necessary to calculate the coordinate y_Ù. The calculation is possible knowing the values of x_θ, z_θ and the elevation angle ϑ in the plane θ, where the optical axes of the left and right eye lie.

To simplify the analysis, we will assume that the XOZ, plane passing through the centers of the eye pupils and the center of the observed object is horizontal, i. e. ϑ = 0. Then at the desired point of intersection of the optical axes of the left and right eyes the correlation following from (1) must be satisfied:

_{$\left(x_{\Omega }-x_S\right)\cdot tg\left(90-{\alpha }_S\right)=\left(x_{\Omega }-x_D\right)\cdot tg\left(90-{\alpha }_D\right)$.}

Considering this, |x_S| = |x_D| = b_OU /2 , we obtain a new form of record:

$\left(x_{\Omega }+b_{OU}/2\right)\cdot ctg\left({\alpha }_S\right)=\left(x_{\Omega }+b_{OU}/2\right)\cdot ctg\left({\alpha }_D\right)$,

where

$x_{\Omega }\cdot ctg\left({\alpha }_S\right)-x_{\Omega }\cdot ctg\left({\alpha }_D\right)=-\left(b_{OU}/2\right)\cdot ctg\left({\alpha }_D\right)-\left(b_{OU}/2\right)\cdot ctg\left({\alpha }_S\right)$.

and, thus, coordinates of the gaze point in XOZ plane can be represented by a system of equations

$\left\{\begin{array}{l}{x}_{\Omega }=\frac{-\left(b_{OU}/2\right)\cdot \left(ctg\left({\alpha }_D\right)+ctg\left({\alpha }_S\right)\right)}{ctg\left({\alpha }_S\right)+ctg\left({\alpha }_D\right)}\\ z_{\Omega }=\left(x_{\Omega }+b_{OU}/2\right)\cdot ctg\left({\alpha }_S\right)\end{array}\right.$ (2).

In this case, the following correlations between the values of angles á_D, á_S, â_D and â_S: are satisfied: at á_S > 0 and á_D < 0, â_S > 0 and â_R < 0; are realized, at á_S > 0 and á_D > 0 we obtain â_S > 0 and â_D > 0; at á_S < 0 and á_D < 0 we have â_S < 0 and â_D < 0.

The base of the visual system b_OU in an adult is on average 60 mm–61 mm this is an objective parameter obtained in direct measurements. The assessment of distance resolution turns out to be subjective direct measurements are impossible, since it is impossible to obtain the final result of the functioning of only the visual system the observer’s cognitive system also participates in making the final decision. However, based on the results of numerous observations, it can be considered that at a small (≈ 1 m) distance to an observed small-sized object located within the plane of symmetry of the visual system at eye level, a 10 mm resolution is achievable. At an average distance (≈ 5 m) it is about 20 cm, and at a long distance (15 m or more) it is about 0.5 m. It follows that the average absolute error in estimating the angle of deviation of the optical axis of each eye to achieve such estimates should be 0.01°–0.02° (Table 1). Äá is defined as the difference of the angles á₂ and á₁ corresponding to the observation of the vertices of a segment of length ÄR, oriented along the ОZ, axis, with the nearest located at a distance R from the observer.

 

Table 1. Assessment of Implemented Angular Resolution when Observing Object at Distance R from Human Observer’s Face

Distance R, m	Resolution ΔR, mm	á1,°	á2,°	Äá,°	Average Δα,°
1	10	1.718	1.701	0.017	0.014
5	200	0.343	0.331	0.012
15	500	0.115	0.111	0.014

 

It is obvious that finding a solution to (2) based on recorded analog oculographic signals (the difference in electric potentials of the skin surface at the points of attachment of the electrodes), proportional to the angles of deviation of the optical axes of the left and right eyes from the normal lines, is possible only with the digital presentation of these signals after the analog-to-digital conversion.

Assuming that oculographic signal is recorded with gaze direction changing within ±15°, the required angular resolution will be provided by using 12-digit analog-to-digital converter permitting resolution of 212 = 4096 levels of input analog signal. Within the discussed 30-degree range of variation of á_D and á_S the resolution will make about 0.007°. This permits to realize some ‘margin’ of the input signal in the amplitude, which provides that there are no restrictions of this signal, with the required angular resolution.

Judging by the data of F Simini, et al. (2011) [7], correction of the oculogram readings should not be done, since eye movements within such angular limits are characterized by a linear dependence of potential difference on the angle of eyeball rotation.

Registration of oculographic signals in three pairs of points on the surface of the face skin of an observer (Figure 2) provides information sufficient to calculate the 3D coordinates of the GP in the left Cartesian coordinate system. With an error about 0.01 m, the coordinates of an object Ω can be determined in a space region of 0.5 × 0.5 × 0.5 m located at a distance of 1 m from the face of the person observing this object.

Signals from 1–2 and 2–3 pairs of electrodes were proposed to be used to determine coordinates y_θ and z_θ of GP in è, plane passing through the centers of pupils and the point of object Ù, and signal from 4–5 pair of electrodes to determine ϑ angle of deviation of this plane from the horizontal one and to recalculate the obtained GP coordinates to coordinates in the Cartesian coordinate system XOYZ (Figure 1b).

This method of successive determination of GP coordinates is conditioned by the principles of functioning of human visual system, as well as by minimization of errors occurring when solving a system of type (1) consisting of three coordinates: x_Ù, y_Ù and z_Ù.

The additional electrode 6 is necessary to organize compensation for interference induced on a human body (this solution is used rather frequently in one- or two-channel oculographs [8] and its implementation differs only by the method of generating the compensation signal). The practice of research works carried out with developed oculographs of various modifications [9] showed that an effective solution providing a low level of interference and convenient operation of the device is the use of a radio channel for transmitting the recorded oculograms readings to the processing computer, and of autonomous power supply battery or accumulator. Such configuration of the oculograph permits satisfactory mobility of the observer and elimination of interference inevitably present when using a power supply connected to the mains. To reduce the impact of low-frequency fields of industrial electric circuits on the obtained results, electrodes are connected to the electronic circuit by segments of shielded cables of limited length (maximum 1 m).The functional diagram of a 3D oculograph (Figure 2), in addition to signal amplifiers, contains correcting circuits, each of which, in a certain range, permits to compensate for changes in the transition resistance between the conductive material of the electrode and the observer’s skin. To note, this phenomenon has the greatest negative impact on the reproducibility of measurement results changes in the amounts of sweat and fat on the skin, changes in the pressing force of the electrodes, probable electrochemical reactions on the surface of the electrodes affect the value of the recorded potentials and prevent the achievement of a stable balance of average values and amplitudes of the signals on output of amplification paths of the oculograph. Besides, the elasticity of the skin, which depends on the physiological state, health and age of the observer, also influences the result. This problem has been repeatedly noted by many researchers, and its final solution has not yet been found, even when electrodes are manufactured of such stable materials as gold [10].

 

Fig. 2. Functional diagram of 3D-oculograph.

 

The subject of the research was a human observing an object moving along a known trajectory, and simultaneously recorded oculomotor reactions of the left and right eyes used to restore the trajectory of movement of the gaze point. An employee of Bauman Moscow State Technical University, a man aged 22 years, voluntarily took part in the experimental studies to demonstrate the method (an Informed Consent form was signed). He denied the presence of any pathology of the visual system.

At a meeting of March 24, 2022 (Protocol No. 1), the Ethics Committee of Bauman Moscow State Technical University approved conduction of the mentioned studies in accordance with the presented methodology and informed consent obtained from the participant.

The realizations were modeled using Intel Visual Fortran Compiler 11 software in accordance with the modern methods of implementing classic algorithms.

Besides mathematical modeling, experimental studies were conducted with the main task to establish a principal possibility of reconstructing the trajectory of GP movement especially in the presence of inevitably recorded interference.

The characteristics achieved by a 3D-oculograph were studied by recording the oculographic signals in the process of observation of a moving object a load 20 mm in diameter moving in space on a suspension 2.1 m long in gravity field.

The rotation of the load at speed V (Figure 1a) was initiated by deflecting it from the equilibrium point by 0.125 m in the direction of the OZ axis, followed by imparting an impulse in the direction opposite to the OX axis. The parameters of the trajectory of the load were determined based on the results of 10 launches it was found to be an ellipse with semi-axes of 0.12 m and 0.16 m.

RESULTS

Using numerical modeling, implementations of oculographic signals were formed corresponding to the movement of GP along a circle of R = 0.2 m (Figure 3). These are enharmonic periodic signals, which is clearly seen when compared with the implementation of a harmonic signal presented in the same figure. With sufficient amplitude resolution of the device in record of U_S(t) and U_D(t) signals, information about realization of á_S(t) and á_D(t), angles can be extracted with moderate error, after applying (2) to them x_Ù(t) and y_Ù(t), dependencies are formed describing GP trajectory. Thus, with voltage resolution of signal readings 0.001 V, the resolution in the angle of deviation of the eyeball will be no worse than 0.01°, which is consistent with the data presented in Table 1.

 

Fig. 3. Model trajectory of movement of gaze point (a) and results of modeling of signals (b): oculograms of the left US(t) and right UD(t) eyes; harmonic signal Ucos(t) = A cos(ωt). Note: rel. un. — relative units.

 

Since the recorded oculographic signals are additive mixtures of useful signals and noise, and, besides, these signals may have non-random shifts in the value of the recorded voltage, the shape of the calculated trajectory may notably differ from the initial one, and, with a significant noise level, the reconstruction of the trajectory can appear impossible. The performed numerical modeling of the reconstructing the GP movement trajectory shows that the maximal noise dispersion value ó²_N can be considered a value numerically equal to 1% of signal amplitude. With lower noise dispersion values, distortions of the reconstructed trajectory are noted, however, its general form is preserved. Of the same order of magnitude are permissible deviations in the shift of signal levels and unbalance of signal amplification coefficients in a pair (Figure 4).

 

Fig. 4. Characteristic distortions of the reconstructed trajectory of gaze point with different distortions of the recorded signals: the presence of additive noise with dispersion σ2 N (а), shift of one of signals in amplitude by DU (b); variations of transmission coefficient KD of the amplification path recording these signals (c); value; величина Umax = max {|US|,|UD|}.

 

The presence of non-random shift of signals level leads to distortion of the reconstructed trajectory; however, in most practical cases it is insignificant.

The series of recorded oculographic signals (each series contained at least 5 implantations of these signals registered in observation of four-five rotations of the load by the subject) were characterized by a qualitative coincidence.

One of the recorded implementations of oculographic signals is shown in Figure 5a. The figure demonstrated that phase difference of U_S(t) and U_D(t) signals periodically change. In general, this fact contradicts the previously considered model. In our opinion, the source of this phenomenon is the peculiarities of cognitive processing of the recorded images by the brain, on the basis of which results the parameters of eyeball movements are corrected. This statement is illustrated in Figure 6 that shows a moving object observed by the left and right eyes, changes in positions of visual fields of these eyes in time, and changes in oculograms signals U_S(t) and U_D(t). characteristic of these situations. Considering the fact that the average period between registrations of separate images by a human cognitive system is 35 ms–40 ms, it is reasonable that integration of readings of U_S(t) and U_D(t) signals be performed within this interval before application of (2).

 

Fig. 5. An oculogram recorded during observation of a moving object (a) and results of reconstruction of trajectories of gaze point movement (b–f) on A … F time intervals, respectively. The line designates parts of trajectories that do not significantly correspond to the geometry of the object trajectory (dotted line). The beginning of the trajectory of each part is marked with arrow, positions of the centers of eye pupils are shown by dots on ОX axes of the coordinate system. Note: rel. un. — relative units.

 

.

Fig. 6. To occurrence of phase shifts of oculogram signals: positions of visual fields of the left and right eyes at different time points(a–c); characteristic levels of oculographic signals in the neighborhood of these time points (d–f). Boundaries of visual fields and oculograms of the left and right eyes are of the same color. Note: rel. un. — relative units.

 

Taking into account what was said above regarding the slow changes in the values of the constant components of the signals U_S(t) and U_D(t), in the process of reconstructing the trajectory of the observed object, the observation interval was decomposed into five-time intervals A ... F, the duration of which approximately corresponded to the time of one rotation of the load. After this, for each interval, the values of permanent shift of ÄU_D, signal were iteratively determined at which the number of incorrectly reconstructed vertices of the broken N_NC the reconstructed trajectory of the load would be minimal. A vertex was considered reconstructed incorrectly, if z_Ù coordinate calculated by (2) appeared negative, or this vertex was located at a large distance from O point of the CS (the threshold value was assumed 5 m). For all intervals except interval D, constant shift values were found at which N_NC = 0.

For D interval, N_NC = 40. For each interval, numbers of vertices N_M of the reconstructed trajectory, and numbers of N_E vertices, whose positions do not significantly correspond to the movement trajectory of the observed object, are indicated.

DISCUSSION

Reconstructions of trajectories of movement of the object performed for separate time intervals A … F, are shown in Figures 5 b–f. Reconstructed trajectories in general agree with elliptical trajectory of object marked by dashed line. There is agreement both in terms of sequence of passage through the trajectory points, and in terms of correspondence of its geometric parameters to dimensions of the area of reconstructed readings. The greatest discrepancies are observed in sections of the trajectory with a high probability of observing the load with almost parallel lines of sight of the left and right eyes. It should be noted that the number of N_E vertices in such areas is about 10% of the total number of reconstructed vertices N_M. Another possible reason for such decisions and distortion of the geometry of the reconstructed trajectory is a fairly high level of additive noise.

Reconstruction of trajectory readings in the è plane is the most critical part of the procedure for reconstructing the GP trajectory in space both in terms of the requirement to ensure the accuracy of calculations of (2), and in terms of the significant influence of the noise component of the signal on the result obtained. To achieve better results, it may be necessary to perform experimental studies in a chamber shielded from external electromagnetic fields. Nevertheless, the results permit to conclude that in principle it is possible to reconstruct the trajectory of GP based on the analysis of a set of oculographic signals.

In subsequent studies, useful information about potentials of the new methods can be obtained in 3D-oculography conducted in large groups of participants and comparing the results obtained in healthy individuals and individuals with various diseases of the visual system.

Speaking about the prospects of 3D-oculography method, its practical application also seems justified in cases when an observer, because of his age or condition, cannot adequately describe the observed event, for example, movement of a point light source. In these cases, record of unconditioned oculomotor reactions or their absence may be indicative of certain pathologies. For example, absence of tracking eye movements may speak for pathology of the optic nerve and may become a reason for conducting a more complex examinations to identify both the cause and degree of development of the pathology.

Other situations where an objective analysis of the position of the GP and the dynamics of its movement is a source of information about the observed event, include studying the effectiveness of advertisements, of the peculiarities of communication between a lecturer and listeners in a lecture hall, determination of an observer’s resistance to a complex of light stimuli, etc. [11–14].

Analysis of the effectiveness of 3D oculography in various situations remains to be analyzed by specialists. The results obtained in our study indicate the informative value of 3D oculography and reproducibility of its results. When creating systems for recording these signals, the main attention should be paid to minimizing the noise level, since its increase leads to the most significant errors in determining trajectory points.

CONCLUSION

The principles of 3D-oculography implemented by determining the spatial coordinates of gaze point of an observer by recording oculographic signals on the surface of his face are theoretically substantiated.

Problems faced by researchers when implementing such measurement methods, and ways of their solution, are described. The boundaries of successful application of the method on exposure to interference of different nature have been determined.

The experimental part of the work demonstrated the functionality of the developed 3D-oculography method and the possibility of reconstructing parameters of the gaze point trajectory in space with multi-channel recording of oculographic signals.

ADDITIONALLY

Funding. This study was not supported by any external sources of funding.

Conflict of interests. The authors declare no conflicts of interests.

Acknowledgment. The authors would like to thank M. V. Zuyeva (Dr. Sci. (Biol.), Professor), I. V. Tsapenko (Cand. Sci. (Biol.)), V. I. Kotelin (MD, Cand. Sci. (Med.)) from Helmholtz Moscow Research Institute of Eye Diseases for their help in results and practical use perspectives discussions.

Contribution of the authors: P. V. Arakcheyev — oculographic signals parameters automatic correction units circuitry and software design; V. L. Bezdelov — 3D-oculograph's quantized and discretized signals microprocessor processing units development, experimental oculographic signals samples registration; E. V. Buryi — general task setting and method development for the ‘point of gaze’ 3D-coordinates determining; D. A. Semerenko — wireless data transmission channel design, experimental results processing and visualization software design; A. L. Shlemenkov — a sample unit of portable 3D-oculograph with an independent galvanic power supply design, experimental data acquisition and processing, whiting the text. The authors confirm the correspondence of their authorship to the ICMJE International Criteria. All authors made a substantial contribution to the conception of the work, acquisition, analysis, interpretation of data for the work, drafting and revising the work, final approval of the version to be published and agree to be accountable for all aspects of the work.

Финансирование. Авторы заявляют об отсутствии внешнего финансирования при проведении исследования.

Конфликт интересов. Авторы заявляют об отсутствии конфликта интересов.

Благодарность. Авторы считают своим приятным долгом поблагодарить сотрудников Национального медицинского исследовательского центра глазных болезней имени Гельмгольца — д.б.н., профессора М. В. Зуеву, к.б.н. И. В. Цапенко, к.м.н. В. И. Котелина за обсуждения полученных результатов и перспектив их практического применения.

Вклад авторов: Аракчеев П. В. — разработка схемотехнических и программных решений узлов автоматической коррекции параметров окулографических сигналов; Безделов В. Л. — разработка узлов микропроцессорной обработки квантованных и дискретизированных сигналов 3D-окулографа, регистрация экспериментальных реализаций окулографических сигналов; Бурый Е. В. — постановка задачи исследования и создание метода определения 3D координат «точки взора»; Семеренко Д. А. — разработка радиоканала передачи данных, создание программного обеспечения обработки и визуализации результатов экспериментов; Шлеменков А. Л. — создание образца портативного 3D-окулографа с независимым гальваническим источником питания, получение и обработка экспериментальных данных, написание текста. Авторы подтверждают соответствие своего авторства международным критериям ICMJE (все авторы внесли существенный вклад в разработку концепции и подготовку статьи, прочли и одобрили финальную версию перед публикацией).

About the authors

Pavel V. Arakcheyev

Bauman Moscow State Technical University

Email: arpv2002@inbox.ru
ORCID iD: 0000-0002-7818-9427
SPIN-code: 7796-8362

Cand. Sci. (Engineering)

Russian Federation, Moscow

Valeriy L. Bezdelov

Bauman Moscow State Technical University

Email: balery_wr@inbox.ru
ORCID iD: 0000-0001-8761-8388
SPIN-code: 1336-6719
Russian Federation, Moscow

Evgeniy V. Buryi

Bauman Moscow State Technical University

Author for correspondence.
Email: buryi@bmstu.ru
ORCID iD: 0000-0001-8954-5434
SPIN-code: 4587-5149

Dr. Sci. (Engineering), Professor

Russian Federation, Moscow

Denis A. Semerenko

Bauman Moscow State Technical University

Email: ddd183@mail.ru
ORCID iD: 0000-0001-5822-5120
SPIN-code: 6290-8469

Cand. Sci. (Physics and Mathematics)

Russian Federation, Moscow

Anton L. Shlemenkov

Bauman Moscow State Technical University

Email: anthony99@mail.ru
ORCID iD: 0000-0001-9318-0888
SPIN-code: 4747-8558
Russian Federation, Moscow

References

Supplementary files