Using metadata to resolve big data problems - Zakharova, O.V. (PROBLEMS IN PROGRAMMING, 2019)
Today, the volumes of data used by application systems are growing exponentially and have reached such sizes that they cannot be processed by traditional systems. So the term "Big data" appeared. The main problems of such data sets are associated, first of all, not only with their volumes, but also with the variety and complexity of the information they contain. Thus, along with the growth of data volumes and the number of big data initiatives, the metadata become the most important priority for the success of large data projects. Enterprises understand that the full use of the operational potential of machine learning, in-depth learning and artificial intellect requires the unprocessed data was supplemented with metadata. Therefore, the purpose of this work is to analyze the effect of metadata to solving the big data problems, determine the main categories of data to be annotated by metadata, and the main types of metadata used for this. Today, metadata is a means of classifying, organizing, and characterizing data or its contents. Depending on the role they play in solving big data problems, NISO identifies four main types of metadata: administrative, descriptive, structural, and markup languages. Different types of metadata can be used in a certain way to effectively solve problems of management, search, data integration, etc. A separate issue is the way of their creation/automatic generation, since the manual creation of metadata is a laborious process, and their volume is often several times larger than the volume of the data itself.