Algorithm for correcting camera pan and tilt on aircraft based on recorded video

全文:

Introduction

The main method to determine the position of an aircraft in space is to use satellite navigation systems [1]. Currently their use to conduct trajectory measurements on the territory of airfields is impossible due to a number of reasons [2]. Inertial navigation systems, currently used instead of satellite ones, accumulate significant errors during long flights [3]. In addition, carrying out this type of measurement is necessary to install additional equipment directly on the aircraft, which is not always possible for legal or technical reasons, for example, when using small-sized unmanned aerial vehicles with limited payload [4].

One of the options to solve this problem is the use of optical systems [5–7], such as cine-theodolites [8], to calculate the position of the aircraft based on azimuth and elevation angles from two measuring points [9] (two-point direction finding method, shown in Fig. 1).

 

Рис. 1. Суть угломерного или пеленгационного способа определения координат

Fig. 1. Direction finding model

 

However, accurately tracking the movement of an aircraft is practically impossible. When manually tracking a target, the tracking result is mainly under the impact of operator errors. For example, an operator cannot perfectly keep a fast moving object in the center of the frame due to the relatively high human reaction time. Also, the operator may make mistakes due to human factor.

Using computer vision algorithms simplifies the task somewhat, but does not solve the problem completely. Brightness-contrast algorithms of computer vision are extremely sensitive to weather conditions and extraneous objects in the frame [10]. More advanced algorithms based on machine learning can solve this problem quite effectively, but they require more computing power and a large training sample for each individual aircraft model to be tracked. Therefore, the development of such algorithms to conduct small-scale tests is economically unprofitable.

In addition, even with an error-free computer vision algorithm, it is fundamentally impossible to solve the problem of the pan-tilt mechanism having a limited rotation speed. Moreover, using the maximum rotation speed cannot be always realized, even if the pan-tilt mechanism allows it. The camera installed on it can be quite heavy [11], and if the speed changes abruptly, the pan-tilt mechanism can fail.

A solution could be a frame-by-frame post-flight processing algorithm, where the operator, not limited by time frames and physical characteristics of the measuring complex, can indicate the position of the aircraft with pixel accuracy [12] and specify the angles of rotation and tilt of the camera relative to the center of the frame [13].

Algorithm description

The essence of post-flight processing is as follows.

A post-flight processing operator reviews the recorded flight video footage. The operator moves to the frame (Fig. 2) corresponding to the required moment in time, points the cursor at a specific point on the aircraft, along which it was decided to conduct trajectory measurements, and by pressing the mouse button starts the operation of the algorithm, which will correct the aircraft sight angles.

 

Рис. 2. Примерная схема кадра при послеполётной обработке

Fig. 2. Frame structure during post-flight processing

 

Since the operator's monitor resolution, frame display area size and aspect ratio may change during the post-flight processing, unlike the original video, it is necessary to take this fact into account and not be tied to the pixel resolution, but to work relative to the horizontal and vertical viewing angles.

The APIs that provide the current position of the mouse cursor return it relative to the upper-left corner of the display area [14]. But since the target sighting angles for each frame are set relative to its center, it is necessary to transform the coordinates of the mouse cursor. The position of the mouse cursor relative to the center of the frame can be calculated using the following formula:

$\begin{array}{c}{x}_{center}=x_{topleft}-\frac{w}{2},\\ y_{center}=\frac{h}{2}-y_{topleft},\end{array}$

where $x_{topleft}$ and $y_{topleft}$ – the position of the mouse pointer relative to the upper left corner of the frame; $w$ and h – the total number of pixels in the frame in width and height respectively.

Now we need to move from a linear representation in pixels to an angular representation in degrees or radians. To do this, we divide the frame into quadrants. We will take the axes denoting the width and height of the frame to be equal to one angle of the solution horizontally and vertically, respectively. Consequently, points along these axes will have coordinates in the range from –0.5 to +0.5. Therefore, we eliminate the need to know the resolution of the original video to calculate the pixel value in angular measure. The coefficients of linear displacement of the target relative to the frame center can be calculated using the following formula:

$\begin{array}{c}{w}_{coeff}=\frac{x_{center}}{w}=\frac{x_{topleft}}{w}-\frac{1}{2},\\ h_{coeff}=\frac{y_{center}}{h}=\frac{1}{2}-\frac{y_{topleft}}{h}.\end{array}$

The angular displacement of the target relative to the center of the frame can be obtained by multiplying the lens aperture angles by the corresponding coefficients obtained earlier:

$\begin{array}{c}{a}_{offset}=h_{FOV}\times w_{coeff},\\ e_{offset}=v_{FOV}\times h_{coeff},\end{array}$

where $h_{FOV}$ and $v_{FOV}$ – horizontal and vertical lens aperture angles, respectively.

Now, to obtain the more accurate angles of pan and tilt of the camera on the aircraft, it is necessary to add the obtained angular offset to the angles of the frame center.

$\begin{array}{c}{a}_{t\arg et}=a_{center}+a_{offset},\\ e_{t\arg et}=e_{center}+e_{offset}.\end{array}$

Conclusion

The considered algorithm can be used when conducting trajectory measurements to improve the accuracy [15] of aircraft position calculations. Figure 3 presents an example of a post-flight processing program that uses this algorithm. Moreover, this algorithm can be used in systems with fixed wide-angle cameras as the main method to determine the viewing angles of an aircraft. Such a system can be used to measure the radiation patterns of aircraft onboard antennas. However, when using wide-angle cameras, lens distortion is necessarily taken into account.

 

Рис. 3. Послеполётная обработка траекторных измерений МС-21

Fig. 3. Post-flight processing of Yakovlev MC-21

 

The main disadvantage of this algorithm is the need to manually specify the position of an aircraft for each frame, which makes its use for long flights labor-intensive. A way to solve this problem may be to sample only key frames at a certain time interval (for example, 2 times per second) depending on the requirements indicated in the technical specifications. Alternatively, more resource-intensive computer vision algorithms can also be used to automate the frame-by-frame processing process. They cannot be used in real time, but are suitable when there are no time constraints on processing a single frame, since we are talking about post-flight processing of the recorded video.

The main advantage of this algorithm is the ability to obtain the highest possible accuracy when conducting trajectory measurements with cine-theodolites, since frame-by-frame processing makes it possible to consistently obtain angles of the same point on the aircraft.

作者简介

Danil Nikiforov

Reshetnev Siberian State University of Science and Technology

Email: nikiforov-danil1997@yandex.ru
ORCID iD: 0009-0004-8238-3427

Postgraduate student

俄罗斯联邦, 31, Krasnoyarskii rabochii prospekt, Krasnoyarsk, 660037

Sergey Efimov

Reshetnev Siberian State University of Science and Technology

编辑信件的主要联系方式.
Email: efimov@bk.ru
ORCID iD: 0000-0002-4506-3510

Cand. Sc., assistant professor, department of informational and control systems

俄罗斯联邦, 31, Krasnoyarskii rabochii prospekt, Krasnoyarsk, 660037

参考

补充文件