Interpreting and processing side-scan sonar data with the objective of further automation

Full Text

Introduction

A side-scan sonar (SSS) is an effective means of underwater remote sensing, providing a high degree of visualization of the benthic surface of various reservoirs. It is significantly superior to optical means. This device is used to conduct a wide range of hydrological studies, from geomorphological to archaeological ones, as well as to perform applied tasks in the field of hydrography, hydraulic design and construction, search and fixation of underwater objects. The development of SSS technologies and the production of compact and at the same time affordable models have significantly expanded its operational characteristics in terms of increasing the number of users and expanding the conditions for its use. The miniaturization of SSSs has made it possible to include them as one of the components of amateur echo sounders available on the market of recreational devices: they can be successfully used even by one operator on board a small vessel (special works are devoted to the issue of using this SSS subtype [1–4]).

In the scientific literature, the problem of developing software for SSS systems has received significantly less attention than their hardware. For example, the fundamental work of the British scientist Phillip Blondel [5] is almost entirely devoted to the physical features of the operation and application of SSSs. It is explained by the fact that it is the physical parameters of the device (the higher the frequency, the more detailed the visualization) that are responsible for the quality of data. In addition, for the correct interpretation of images, it is necessary to take into account the propagation of sound waves in water, the reflective abilities of benthic objects and other hydroacoustic phenomena. Thus, a trained operator knows, for example, that dark areas of the image are softer, dispersed surfaces, and light areas, on the contrary, are hard and dense, giving a stronger reflective signal. Therefore, equipment users are often focused on improving the SSS hardware, rather than its software. It should be noted that the software installed by default fully meets the requirements of the majority of users of this device: searching for objects and studying the area are carried out mainly in situ, which is quite acceptable in small water areas. Proprietary programs (Scanline Starfish, Reefmaster, Humminbird, etc.) quite satisfy the user with the capabilities of viewing images in real time, recording, as well as built-in post-processing functions.

At the same time, it should be pointed out that there is no effective and accessible software on the market that allows for the automated identification of detected underwater objects, their classification and cataloging, as well as the automated calculation of telemetric data coming from the device. The solution to these issues is the focus of this paper.

The problem of visualizing SSS sonograms is considered in [2–6], but in these works the main attention is paid not so much to the issues of creating new software, but to using existing one. Thus, visualization of field material was carried out according to the method proposed in [3–5] for sonograms obtained by recreational SSSs.

Due to its physical characteristics, the data obtained by SSSs cannot be parsed within the framework of existing pattern recognition algorithms for optical images. The operating principle of SSSs, as well as other acoustic imaging devices, is to process reflected sound rays from the surface of objects (Fig. 1). While at nadir, the transducer studies a thin acoustic beam directed towards the bottom of the reservoir, then receives the reflected echo. The data processing unit processes the received signal and displays it as an image on the operator’s monitor. At the same time, the physical identity of this principle of working with ultrasonic medical devices, as well as non-destructive testing devices, should not create a misleading impression about the possibility of using their data processing methods in hydro-acoustics. This is due, first of all, to completely different operating conditions of this technology. The above devices are used in direct contact with the object being studied, while SSSs can be operated in ranges from one meter to several thousand meters. The features and nature of the objects being studied within the framework of diagnostics are relatively known, while at the bottom of reservoirs there can be a variety of sometimes unpredictable objects of both natural and anthropogenic origin. Finally, the amount of data generated by SSSs is significant.

 

Рис. 1. Принцип формирование ГБО изображения и положение трансдьюсера (x_f, y_f, z_f): ф – угол атаки; θ – угол рыскания. Сост. по [7]

Fig. 1. Principles of SSS image formation. The position of transducer is denoted by (x_f, y_f, z_f): its pitch angle by ф, its yaw angle by θ. Based on [7]

 

The problem of visualizing SSS data

SSS systems make it possible to obtain an image of the aquatic environment by converting the amplitude values of the own acoustic signal reflected from objects into successive rows of pixels that make up the image of the bottom of the reservoir. Thus, this system, measuring the amplitude of the signal, converts the values into the tone of pixels of the future image. Hard and dense objects reflect more sonar signal than soft and loose objects. Therefore, based on the tone or color of the pixel, one can make assumptions about the underlying object. There are other factors that influence the tonality of pixels in the final image: characteristics of the water body itself (water composition, its density, temperature), scanning parameters - scanning range (scanning bandwidth) and frequency of the emitted sound signal, survey route, the speed of movement of the transducer and other sources, the occurrence and influence of which is not always possible to foresee and prevent, for example, different speeds of water flows on the surface and under water, thermoclines, meteorological conditions (precipitation, atmospheric pressure) and other factors.

Considering that the stability of the hydrosphere depends on a set of fixed factors of both natural and anthropogenic origin [8], the accumulation of information about the state of the bottom of water bodies is the most important task not only for developing a strategy for the economic exploitation of water resources, but also for creating geographical information systems using SSS data. Thus, the interpretation of SSS images by a human operator is based on a combination of knowledge of factors and their causes that influence the operation of the device, personal experience, as well as parameters and settings of the equipment.

Let us consider, as an example, a fragment of a survey of several sections of the Yenisei River in the upper and lower reaches (depths 3–15 m). During the work, a Starfish 990F SSS (manufactured by Tritech) was used, operating at a constant frequency of 1 MHz and intended for work at depths of up to 30 m. Figure 2 shows underwater objects, as well as the characteristics of the water space.

Рис. 2. Фрагмент отснятого прохода ГБО (ширина прохода 50 м) района поиска с элементами снимка

Fig. 2. A fragment of the SSS scanline (overall breadth of 50 m) showing the main elements of the image

 

There is no shoreline in the image because the passage was made more than 25 m from the shore. The left half of the image shows the clear geometric shape of a reinforced concrete hydraulic structure, with an acoustic beam passing through the structure, which is about 5 m thick. The image is clear and bright. This is ensured by the uniformity of the passage (tack), as well as the presence of a rocky and pebble bottom, which provides the strongest return signal. Large boulders and fragments of building materials are clearly visible at the bottom. In the middle of the image, traces of the sound wave reflected from the mirror of the reservoir are clearly visible. This effect occurs when the signal is reflected twice, first from the bottom of the reservoir, then from the mirror, which is due to the choice of acute angles of the signal pitch at shallow depths of the reservoir. As depth values increase, the data in the image becomes less clear. Although SSSs do not determine depth, it can be calculated from the width of the shadow area in the center of the image, representing the water column below the transducer. There may be fish, floating debris, and various suspended matter. Thus, it is possible to make a rough assessment of the level of water contamination with large particles. Changes in depths along the entire survey route are significant - both small areas with a depth of up to 2 m and large depressions with a depth of more than 8 m are observed. For the most part, the bottom material is homogeneous, presumably fine stone. At the very edge of the water, the stone formations increase in size.

Based on the above, it is clear that with a small area of water at shallow depths (up to 10 m), as well as with the ability to build a correct, uniform trajectory of movement, one can obtain images of high quality, amenable to simple interpretation, and not requiring automation. However, due to their physical characteristics, SSS sonograms are subject to distortion in any case, be it distortions in the intensity of the SSS signal (caused by deviations in the ideal linear relationship between image intensity and signal backscatter) or geometric distortions (caused by inconsistencies between the relative location of properties in the image and the true position of the object on the bottom) [7].

Geometric distortions of benthic objects

One of the key problems is the geometric distortion of underwater objects. The SSS image is a monochrome digital image with return acoustic signals reflected from benthic objects applied to it. For a mathematical description of this transformation, it is necessary to introduce a three-dimensional rectangular coordinate system (x, y, z) of the benthic surface and a two-dimensional rectangular coordinate system (m, n) on the sonogram. The main problem of image formation is the instability of the transducer position, which can change the direction of movement in different planes. Changes in the speed of transducer movement, sudden deepening or ascent, fluctuations in pitch and yaw angles lead to significant distortion of the image. So, in Fig. 3, curvature of all objects to the right side of the image is observed, which is caused by a sharp change in the movement of the transducer tack. The man-made structure on the left side is distorted and has acquired a typical “twisted” shape.

 

Рис. 3. Пример геометрических искажений донных объектов

Fig. 3. An example of SSS geometric distortion caused by irregularity of vessel movement

 

One of the most effective ways to solve the problem of geometric distortion is a set of methods based on a combination of the least squares method (extended and recursive identifier) and the use of an effective recursive filter, for example, the Kalman filter, presented in [7; 9]. The main merit of the developers of this approach is the ability to perform automatic image correction without additional navigation or field data. The proposed method is suitable for images with high resolution (frequencies 100 KHz and higher), which is fully consistent with our examples. This approach does not require navigation information and does not rely on image correction by slant range determination.

Let us consider one of the mathematical models of geometric distortions proposed in [7]. Let us imagine the absolute position of the benthic surface points (x_s [m, n], y_s [m, n]) as a function of the values of the transducer position parameters relative to fixed coordinates (x, y, z) (see Fig. 1). If the values of the measured parameters could be obtained directly from sensors installed on the transducer, it would be enough to substitute them into a known set of equations to obtain the coordinates of the benthic points, and then correct geometric distortions to obtain the correct image. Unfortunately, for the reasons stated above, it is not possible to directly reference underwater objects to the GNSS system. To assess changes in the positioning parameters of the towed transducer, it is necessary to extract some values of geometric distortions from the sonogram.

Assessing the geometric distortion of images requires making a number of assumptions about it. The basic assumption to obtain the size of the geometric distortion is that the 2D backscattering function is an arbitrary process with an isotropic autocorrelation function. The assumption about the constancy of the backscattering function may turn out to be erroneous in the presence of significant diversity in the benthic surface topography and its geomorphology. However, this technique can well be used to correct hydroacoustic images if the surface under study is first divided into homogeneous areas according to their morphology. In this case, a certain degree of constancy can be assumed. At the same time, the assumption of isotropy is valid in the absence of a systematic direction of benthic objects (direction of current, geology, prevailing direction of waves, etc.).

Thus, assuming that the backscattering function is constant and its autocorrelation function is isotropic, the degree of geometric distortion of a hydroacoustic image can be calculated by measuring changes in the shape of an autocorrelation sequence sample of small sections of the image. Estimation of local geometric transducers can be performed for two images using the method presented in [10]. Due to the fact that the medium under study is not isotropic, the parameters will be determined at the local level. After performing the analysis of SSS images, automatic compensation of the data acquisition speed occurs. Thus, the speed factor (the main distortion factor) is no longer needed to perform running scanning. In the case of using this method, the geometric transformation of a certain object in the scalar factor is equal to zero at a zero rotation angle for each identified object.

Image adjustments can be made during surveys. Thus, the real distance to the benthic object from the transducer can be calculated using a simple formula

$x=\sqrt{y^2-h^2},$

where x is the actual distance to the object; y is the distance indicated on the SSS image; h is the transducer height at nadir. Although this problem is more relevant for towed transducers, it should also be taken into account when operating fixed devices, since each point on the sonogram is conditionally referenced to a geographic coordinate system, calculated depending on the distance of the point from the onboard GNSS receiver.

When processing sonograms, the acoustic shadow area can be removed using graphic editing programs, thus connecting the visible areas on both sides of the vessel into one image. In this case, the width of the shadow along one side represents the conditional distance from the bottom point of the transducer to the bottom of the reservoir.

The choice of the trajectory of a hydrographic vessel is also important. There are several standard methods for covering a given water area, but the meander type performed by a sequence of reciprocal parallel tacks is suitable for SSSs [11].

The results of image correction based on the methods presented above can be seen in Fig. 4.

 

Рис. 4. Результаты коррекции изображения с ГБО

Fig. 4. Results of a corrected SSS image

 

Construction of an SSS mosaic and recognition of acoustic images

A sonogram and its description are not the final product of hydroacoustic study. A sonogram can be used as the main source when compiling maps of water bodies or as an additional source of information in the case of studying a specific object located in the water column or lying at the bottom of a reservoir. An example of constructing a SSS mosaic for images is presented in works [2–4; 6], however, they were performed using recreational echo sounders, for which commercial mapping software exists. Starfish Scanline does not have the function of constructing a route track, as well as its further plotting on the map. In [12], we referenced the SSS sonogram to the Landsat-8 satellite image (Fig. 5). The work was performed in the Quantum GIS software package.

 

Рис. 5. Постоение ГБО мозаики, наложенной на космический снимок Landsat-8

Fig. 5. SSS mosaic transferred to a Landsat-8 satellite image

 

The sonar sonogram is a file with a LogDoc extension, a standard file format for StarFish, which is visualized by the instrument manufacturer's Starfish Scanline software. The software offers several ways to extract data - directly the finished sonogram and the so-called “raw data”, which is a table with parameters entered into it. We used the function to extract the finished sonogram. Thus, the sonogram file was divided into 43 uniform fragments, representing the details of a large mosaic. For each element, the coordinates of 15 points were recorded in the table. It is necessary for carrying out the procedure of georeferencing each element; in addition, this number of points is sufficient to transform the display of sonograms by a second-order polynomial, which will increase the accuracy of the work. Thus, georeferencing gave the data a natural location in space for each sonogram, not in the form of a “straight line”, but along the trajectory of the watercraft with filming equipment with all the turns along the route during research work. As the georeferencing proceeds, the elements are joined to each other one by one, making up a single mosaic.

The problem when carrying out georeferencing was the so-called “corner fragments” (fragments located in areas of the vessel). Special attention was paid to them. Since it is not always possible to shoot on straight lines, sharp bends are visible in the images, which can lead to severe distortion of the sonogram, which, in turn, negatively affects the clarity of the image and the referencing accuracy. In our work there is a fragment of a sharp turn (Fig. 6). One can see how much the shape of the fragment has changed after spatial referencing and transformation of the image. It is also necessary to note that a “tear” has formed on the outer corner of the sonogram - this is the lack of data in this place due to a sharp turn [13].

 

Рис. 6. Изменение угловой сонограммы после географической привязки и трансформирования полиномом второго порядка

Fig. 6. Correction of an angular sonogram after georeferencing and transformation by a second-degree polynomial

 

After completing the mosaic, we add a layer of satellite imagery from Landsat-8. Combining the image and sonar data allows one to accurately determine the position of objects in the water column and at the bottom of the reservoir relative to the shoreline and, in general, for correct visual perception (Fig. 7).

 

Рис. 7. Построенная ГБО мозаика с точечным слоем обнаруженных объектов и явлений

Fig. 7. A SSS mosaic with a point layer showing detected objects and artifacts

 

The discovered phenomena and objects can be divided into several groups. The objects themselves stand out against the background of a relatively flat surface of the river bottom, regarding which it can be assumed that they are fragments of a woody nature, since they have a characteristic elongated rectangular shape, and their volume is determined by the falling shadow. It was also noted that these objects have average reflected sound signal values (20–30 dB). The second group consists of areas with distortions (including geometric ones) and fading of the reflected sound signal, which ultimately leads to data loss. Such areas must be determined by knowing the coordinates and locations of such “dark spots” on the map in order to re-explore this area. Note that this problem has so far been solved only partially due to a significant error in referencing satellite coordinates to the sonogram. In areas where the trajectory deviates from a given meander, geometric distortion (stretching) of the raster image occurs, which subsequently affects the work on object recognition. Taking into account the location of these sections, it will also be possible to avoid distortions or minimize the turning radius.

All identified objects were presented on the map (Fig. 7) by creating and overlaying a new layer that stored information about the location, the order of the sonogram (a mosaic fragment), as well as a brief descriptive characteristic. This data is located in the attribute table of the layer.

This method of presenting information made it possible to detect some characteristics of objects that were not so obvious in the original form of the SSS data. For example, at the junction of two sonograms, a vertical object was discovered with a high degree of sound reflection and a characteristic elongated sound shadow. This may indicate a large object. Also, in the vicinity of the object, other sources of high reflection were discovered, having a characteristic rectangular shape and their own shadow (Fig. 8–9).

 

Рис. 8. Фрагмент построенной ГБО мозаики с обнаруженным вытянутым вертикально залегающим объектом на стыке мозаики

Fig. 8. A SSS mosaic fragment showing distorted (elongated) object at mosaic join

 

Рис. 9. Фрагмент построенной ГБО мозаики с обнаруженным вытянутым горизонтально залегающим объектом

 Fig. 9. A SSS mosaic fragment showing a horizontal object distorted by stretching

 

The problem of automating the process of processing and interpreting SSS data

It should be noted that, despite the fact that the issue of creating a system that allows excluding the human operator from the process of processing SSS data was considered by a number of researchers, including [5; 14–15], they achieved very modest results. Thus, in work [15], by constructing a complex neural network, it was possible, according to the authors, to create a system that makes it possible to automatically identify boulders (Fig. 10).

 

Рис. 10. Результаты работы системы по автоматизированному распознаванию валунов (выделенные объекты) [15]

Fig. 10. The results of the work of a system for automatic identification of boulders (seen as highlighted shapes) [15]

 

As it can be seen from the image, the results are very modest and can hardly claim further development. It should be noted that objects such as large stones and other rocks can be quite easily recognized by the strength of the return signal (more than 30 dB), and not by their graphical appearance, which in conditions of geometric distortion can be very deceptive. Thus, the use of various methods of information processing and its automation, including methods such as machine learning algorithms [10], can be considered effective (the impossibility of fully implementing high-precision determination of final objects, as well as the limitations of time determination models) only on condition that the problems of object distortion will be taken into account, as well as taking into account acoustic parameters. Attempts to apply these methods to SSS images in the optical data recognition paradigm cannot be of practical significance.

In this sense, work [16] compares favorably, in which the authors describe the process of classification of SSS images, and also identify problems such as the inapplicability of most extractor programs, since they are created for optical images.

The recognition and classification process for SSS images traditionally contains three main steps:

– data preprocessing (grey tone correction);

– feature extraction (image segmentation);

– classification (performed by a human operator on the basis of segmentation).

The authors of [16] proposed to solve the problem of automated recognition of underwater objects using an algorithm built on the basis of a neural network with spatial pyramidal convolution and referencing to network databases of hydroacoustic images. In the proposed method, five different neural networks were used in data preprocessing for object recognition. Then, spatial pyramidal convolution and network SSS databases were introduced into this system. Then a comparative analysis of the results obtained by the networks before and after the inclusion of additional components was carried out. This approach is in many ways identical to the principle of n-version programming, according to which the optimal software or individual component is selected based on a voting algorithm. Note that the data preprocessing within the proposed method was limited to improving image quality.

Conclusion

Taking into account the above, we can conclude that, despite a fairly large number of studies in the field of recognition, correction and automated interpretation of underwater objects visualized using SSS, currently there is no effective way to implement this process. In this regard, we were the first to propose combining methods for correcting geometric distortions of underwater objects with modern methods of information processing proposed in recent scientific research. In our opinion, image correction using effective recursive filters, as well as a set of mathematical methods presented in [7; 9], is promising. The use of more accurate satellite imagery data, as well as correction of the motion trajectory (meander) using satellite coordinates, makes it possible to correct distortions in the planes of movement of the transducer. As post-processing methods, both image correction using the proposed methods and image fragmentation turned out to be effective, which provided the possibility of detailed elaboration of each fragment and implementation of georeferencing at 15 points with subsequent transformation by a second-order polynomial.

It can be affirmed that mathematical methods can quite effectively solve problems of geometric distortion of images obtained from SSSs. This area of research has promise in terms of creating new methods for image correction, as well as transferring them to solving similar problems in the field of earth remote sensing (ERS).

Based on all of the above, we can conclude that creating a list of detected objects using SSSs is an important step in processing the received data. However, such data analysis today cannot be made without participation of a human operator, despite the availability of experimental automation methods. Human participation makes it possible to rationally conduct repeated observations (if necessary), to further use data for more easy determining objects and phenomena, identifying patterns or general characteristics of the area under study.

Automated SSS image recognition systems must include all of the above elements of image distortion, and also rely on the physical features of the images themselves, obtained by processing acoustic rather than optical signals. This issue needs to be paid attention to in the further development of this topic.

About the authors

Alexander E. Goncharov

Reshetnev Siberian State University of Science and Technology

Email: goncharovae@sibsau.ru

Associate Professor

Russian Federation, 31, Krasnoyarskii Rabochii prospekt, Krasnoyarsk, 660037

Ekaterina A. Goncharova

Reshetnev Siberian State University of Science and Technology

Author for correspondence.
Email: goncharovae@sibsau.ru

Graduate Student

Russian Federation, 31, Krasnoyarskii Rabochii prospekt, Krasnoyarsk, 660037

References

Supplementary files