Development of intellectual information system for solving typical tasks of text data analysis, processing and synthesis

We consider the architecture of an intelligent information system (hereinafter — IIS), which contains the necessary components for solving typical tasks of analysis, processing and synthesis of text data, as well as means for configuring such components and monitoring their operation. In this case, the main task was to create a modular structure of the system, allowing horizontal scaling by flexible customization of the components and the sequence of their work as part of a single automated data processing pipeline. To describe the system structure we used the "C4" model of three levels: Context, Containers, Components. The 4th level "Code" is not used in this article, as it represents, designing a solution for module level abstractions. As an example for the design we have chosen an information system for collecting data on literary works, processing unstructured texts of such documents, loading the information into the analytical storage and analytical constructions based on the available data. To test the design solutions, a service catalog for data extraction, transformation and loading pipelines, a catalog of interaction contracts between services and a project structure for a containerization and multi-container management tool — Docker Compose — were developed. Apache Airflow and its REST API for integration with the IIS configuration component was used as an orchestration solution. To configure and optimize the system settings, a diagram of components of the module of own development in Python using the Django framework is given, as well as a description of the logic of its components. In conclusion, the advantage of the used approach over the implementation of ready-made software products with a set of necessary functions is described.