- obtaining data

- reliable preservation of arrays

- data processing

- integration, API

Every year we are increasingly immersed in the digital age. Data warehouses, which have existed on the market for many years, began to rebuild to meet new requests, forming entire data factories. 

Relatively simple systems such as storage and data lakes are gradually being replaced by more complex ecosystems containing a large number of technological solutions. These solutions cover the need for high availability of large volumes of highly structured data, their cost-effective storage and protection from unauthorized access. 

To solve such problems, complex software and hardware systems are built by combining a large number of different components. The means of providing information to end users are also being transformed. Instead of simple reports, there is a huge class of different services that allow the user to solve a large number of tasks related to obtaining benefits from data: either through the direct implementation of information, or indirectly from the optimization of related processes. There are various services, "data sandboxes", in which users can test their proposals, analyze how effective and applicable they are. 

Complex of solutions is called a “data factory”, this is one of the modern tools for digital business transformation. The need for a data factory for enterprises arises when: 

-   there is a complication, an increase in the amount of data 

-   there is an understanding that more use can be made from the data 

-   there is a need to approach data responsibly to ensure greater efficiency in the use of corporate information. 

 Implementation methodology 

The implementation process of a data factory cannot be simple, these are always large projects designed for a long time. As a rule, large companies that want to get comprehensive benefits are included in these projects. Initially, such cases were quite risky due to the long implementation period and impressive funding. By carefully assessing all the risks our teams have developed two concepts for the implementation of such projects.  

- The first concept is a classic approach, when the platform is fully implemented under the IT strategy of the customer company, all the benefits of its implementation and development potential are realized.

- The second approach is to mitigate risk is the interactive approach, implementation through MVP. On the technical side, the platform consists of many interconnected components, each of which covers a certain functional and technical area, which allows the components to be implemented as separate components. From a business point of view, small business blocks are distinguished, on which R&D testing of various theories from the implementation of this platform can be carried out. Moving step by step, every 4-6 months demonstrating what benefits the introduction of certain components brings. Thus, the customer can quickly evaluate the benefits at the stage of project testing. 

Data management 

The data management process appeared a long time ago and has already been implemented in many large companies. However, it is worth highlighting data management in relation to the data factory. 

Data Quality. Responsibility for the quality of data is important. If earlier these were technical data checks (format, convergence equations, etc.), now these are checks with the following questions “If I receive incorrect data in these fields, what it means for my company?”, “If I don’t form some report, what will it affect? How much money will I lose because of this?”, “Is this data important to me or did I just store it for some related operations?”. 

Data Catalog.  A a data factory with a catalog provides a convenient way to control and manage data glossary.  

Data architecture, a physical and business model, a set of tables, relationships between them and their compliance with the company's business processes. 

Summary