Cosmic ResearchCosmic ResearchКосмические исследования0023-42063034-5502The Russian Academy of Sciences68344610.31857/S0023420625020072GNCBWVArticlesСтатьиResearch ArticleAutonomous spacecraft control in the solar gravitational lens' focus via reinforcement learningАвтономное управление космическим аппаратом в области фокуса гравитационной линзы Солнца на основе методов машинного обучения с подкреплениемShirobokovM. G.ШиробоковМ. Г.
Russian Federation
shirobokov@keldysh.ruKorneevK. R.КорнеевК. Р.
Russian Federation
shirobokov@keldysh.ruPerepukhovD. G.ПерепуховД. Г.
Russian Federation
shirobokov@keldysh.ruKeldysh Institute of Applied MathematicsИнститут прикладной математики им. М.В. Келдыша РАН1504202563220422009062025Copyright ©; 2025, Russian Academy of SciencesCopyright ©; 2025, Российская академия наук2025Russian Academy of SciencesРоссийская академия наукhttps://journals.eco-vector.com/0023-4206/article/view/683446

The problem of autonomous control of the translational motion of the spacecraft in the vicinity of the focus of the gravitational lens of the Sun is formulated. The problem is solved by a reinforcement machine learning method using contemporary stochastic numerical methods. The costs of the characteristic velocity for targeting the focal line of a remote extended source, the final accuracy of targeting and the quality of the control function are investigated. The results of the study are given for various forms of state and observation: 1) position and velocity, 2) noisy position and velocity, 3) image of the Einstein ring. The efficiency of control strategies when using recurrent layers and fully connected layers with an input in the form of a measurement stack is compared. The training of control models accounting for execution errors of maneuvers is also being explored.

Формулируется задача автономного управления поступательным движением космического аппарата в окрестности фокуса гравитационной линзы Солнца. Поставленная задача решается методом машинного обучения с подкреплением с использованием современных стохастических численных методов. Исследуются затраты характеристической скорости на нацеливание на фокусную линию удаленного протяженного источника, финальная точность нацеливания и качество работы функции управления. Результаты исследования приводятся для различных форм состояния и наблюдения: 1) положение и скорость; 2) зашумленные положение и скорость; 3) изображение кольца Эйнштейна. Сравнивается эффективность работы стратегий управления при использовании рекуррентных слоев и полносвязных слоев с входом в виде стека измерений. Также рассматривается обучение моделей управления с учетом ошибок исполнения маневров.

Российский научный фондRussian Science Foundation22-71-00051Brandt P.C., Provornikova E.A., Cocoros A. et al. Interstellar Probe: Humanity’s exploration of the Galaxy Begins // Acta Astronautica. 2022. V. 199. P. 364–373. https://doi.org/10.1016/j.actaastro.2022.07.011Einstein A. The Field Equations of Gravitation // Preussische Akademie der Wissenschaften, Sitzungsberichte, (Math. Phys.). Berlin, 1915. P. 844–847.Eddington A.S. Space, time and gravitation: An outline of the general relativity theory. Cambridge University Press, 1920.Фок В.А. Теория пространства, времени и тяготения. Москва: Физматгиз, 1955.Turyshev S.G., Toth V.T. Resolved imaging of exoplanets with the solar gravitational lens // Monthly Notices of the Royal Astronomical Society. 2022. V. 515. Iss. 4. P. 6122–6132. https://doi.org/10.1093/mnras/stac2130Turyshev S.G. Wave-theoretical description of the solar gravitational lens // Physical Review. 2017. V. 95. Iss. 8. Art. ID. 084041. https://doi.org/10.1103/PhysRevD.95.084041Turyshev S.G., Toth V.T. Wave-optical treatment of the shadow cast by a large gravitating sphere // Physical Review. 2018. V. 98. Iss. 10. Art. ID. 104015. https://doi.org/10.1103/PhysRevD.98.104015Turyshev S.G., Toth V.T. Optical properties of the solar gravitational lens in the presence of the solar corona // European Physical J. Plus. 2019. V. 134. Art. ID. 63. https://doi.org/10.1140/epjp/i2019-12426-4Turyshev S.G., Toth V.T. Image formation for extended sources with the solar gravitational lens // Physical Review. 2020. V. 102. Iss. 2. Art. ID. 024038. https://doi.org/10.1103/PhysRevD.102.024038Toth V.T., Turyshev S.G. Image recovery with the solar gravitational lens // Physical Review. 2021. V. 103. Iss. 12. Art. ID. 124038. https://doi.org/10.1103/PhysRevD.103.124038Willems P.A. Photometric Limits on the High Resolution Imaging of Exoplanets Using the Solar Gravity Lens // Acta Astronautica. 2018. V. 152. P. 408–414. https://doi.org/10.1016/j.actaastro.2018.08.013Turyshev S.G., Shao M., Alkalai L. et al. Direct Multipixel Imaging and Spectroscopy of an exoplanet with a Solar Gravity Lens Mission // Final Report. NASA Innovative Advanced Concepts (NIAC). Phase I. 2018. https://arxiv.org/abs/1802.08421Turyshev S.G., Shao M., Toth V.T. et al. Direct Multipixel Imaging and Spectroscopy of an Exoplanet with a Solar Gravity Lens Mission // Final Report. NASA Innovative Advanced Concepts (NIAC). Phase II. 2020. https://arxiv.org/abs/2002.11871Саттон Р.С., Барто Э.Г. Обучение с подкреплением. Москва: Бином. Лаборатория знаний, 2017.Bertsekas D.P. Reinforcement learning and optimal control. Belmont: Athena Scientific, 2019.Kamalapurkar R., Walters P., Rosenfeld J. et al. Reinforcement Learning for Optimal Feedback Control. A Lyapunov-Based Approach. Cham: Springer, 2018.Shirobokov M., Trofimov S., Ovchinnikov M. Survey of machine learning techniques in spacecraft control design // Acta Astronautica. 2021. V. 186. P. 87–97. https://doi.org/10.1016/j.actaastro.2021.05.018Gaudet B., Linares R., Furfaro R. Terminal adaptive guidance via reinforcement meta-learning: Applications to autonomous asteroid close-proximity operations // Acta Astronautica. 2020. V. 171. P. 1–13. https://doi.org/10.1016/j.actaastro.2020.02.036Gaudet B., Linares R., Furfaro R. Adaptive guidance and integrated navigation with reinforcement meta-learning // Acta Astronautica. 2020. V. 169. P. 180–190. https://doi.org/10.1016/j.actaastro.2020.01.007Scorsoglio A., D’Ambrosio A., Ghilardi L. et al. Image-based deep reinforcement meta-learning for autonomous lunar landing // J. Spacecraft and Rockets. 2022. V. 59. Iss. 1. P. 153–165. https://doi.org/10.2514/1.A35072Gaudet B., Linares R., Furfaro R. Six degree-of-freedom body-fixed hovering over unmapped asteroids via LIDAR altimetry and reinforcement meta-learning // Acta Astronautica. 2020. V. 172. P. 90–99. https://doi.org/10.1016/j.actaastro.2020.03.026Широбоков М.Г. Методика построения управления космическими аппаратами с использованием методов обучения с подкреплением // Косм. исслед. 2024. Т. 62. № 5. С. 498–515. https://doi.org/10.31857/S0023420624050082Lefor A.T., Futamase T., Akhlaghi M. A systematic review of strong gravitational lens modeling software // New Astronomy Reviews. 2013. V. 57. Iss. 1–2. P. 1–13. https://doi.org/10.1016/j.newar.2013.05.001Oguri M. The Mass Distribution of SDSS J1004+4112 Revisited // Public. Astronomical Society of Japan. 2010. V. 62. Iss. 4. P. 1017–1024. https://doi.org/10.1093/pasj/62.4.1017Silver D., Lever G., Heess N. et al. Deterministic policy gradient algorithms // Proc. 31st Intern. Conf. Machine Learning. Beijing, China. 2014. V. 32. Iss. 1. P. 387–395. http://proceedings.mlr.press/v32/silver14.htmlMnih V., Badia A.P., Mirza M. et al. Asynchronous Methods for Deep Reinforcement Learning // Proc. 33rd Intern. Conf. Machine Learning. New York, USA. 2016. V. 48. P. 1928–1937. https://proceedings.mlr.press/v48/mniha16.htmlSchulman J., Wolski F., Dhariwal P. et al. Proximal Policy Optimization Algorithms // arXiv preprint. 2017. Art. ID. 1707.06347. https://arxiv.org/abs/1707.06347Moriarty D.E., Schultz A. C., Grefenstette J.J. Evolutionary algorithms for reinforcement learning // J. Artificial Intelligence Research. 1999. V. 11. P. 241–276.Sehgal A., La H., Louis S. et al. Deep reinforcement learning using genetic algorithm for parameter optimization // Proc. Third IEEE International Conference on Robotic Computing (IRC). 2019. P. 596–601. https://doi.org/10.1109/IRC.2019.00121Hochreiter S., Schmidhuber J. Long short-term memory // Neural computation. 1997. V. 9. Iss. 8. P. 1735–1780.Hoeffding W. Probability inequalities for sums of bounded random variables // J. American Statistical Association. 1963. V. 58. Iss. 301. P. 13–30. https://doi.org/10.1080/01621459.1963.10500830