Проблема автоматичної класифікації зображень за використання інтелектуальної системи прийняття рішень на основі графа знань і точного аналізу зображень
In order to prevent the illegal export of paintings abroad, a museum examination using various methods for studying a work of art is carried out. At the same time, an analysis is also made of historical, art history, financial and other information and documents confirming the painting’s authenticit...
Saved in:
| Date: | 2022 |
|---|---|
| Main Authors: | , , , |
| Format: | Article |
| Language: | English |
| Published: |
The National Technical University of Ukraine "Igor Sikorsky Kyiv Polytechnic Institute"
2022
|
| Subjects: | |
| Online Access: | https://journal.iasa.kpi.ua/article/view/275082 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Journal Title: | System research and information technologies |
| Download file: |  |
Institution
System research and information technologies| _version_ | 1867334432613990400 |
|---|---|
| author | Martynenko, Andrii Tevyashev, Andriy Kulishova, Nonna Moroz, Boris |
| author_facet | Martynenko, Andrii Tevyashev, Andriy Kulishova, Nonna Moroz, Boris |
| author_institution_txt_mv | [
{
"author": "Andrii Martynenko",
"institution": "Dnipro University of Technology, Dnipro"
},
{
"author": "Andriy Tevyashev",
"institution": "Kharkiv National University of Radio Electronics, Kharkiv"
},
{
"author": "Nonna Kulishova",
"institution": "Kharkiv National University of Radio Electronics, Kharkiv"
},
{
"author": "Boris Moroz",
"institution": "Dnipro University of Technology, Dnipro"
}
] |
| author_sort | Martynenko, Andrii |
| baseUrl_str | http://journal.iasa.kpi.ua/oai |
| collection | OJS |
| datestamp_date | 2023-05-21T20:04:38Z |
| description | In order to prevent the illegal export of paintings abroad, a museum examination using various methods for studying a work of art is carried out. At the same time, an analysis is also made of historical, art history, financial and other information and documents confirming the painting’s authenticity — provenance. Automation of such examination is hampered by the need to take into account numerical values of visual features, quality indicators, and verbal descriptions from provenance. In this paper, we consider the problem of automatic multi-task classification of paintings for museum expertise. A system architecture is proposed that checks provenance, implements a fine-grained image analysis (FGIA) of visual image features, and automatically classifies a painting by authorship, genre, and time of creation. Provenance is contained in a knowledge graph; for its vectorization, it is proposed to use a graph2vec type encoder with an attention mechanism. Fine-grained image analysis is proposed to be performed using searching discriminative regions (SDR) and learning discriminative regions (LDR) allocated by convolutional neural networks. To train the classifier, a generalized loss function is proposed. A data set is also proposed, including provenance and images of paintings by European and Ukrainian artists. |
| doi_str_mv | 10.20535/SRIT.2308-8893.2022.4.05 |
| first_indexed | 2025-07-17T10:28:05Z |
| format | Article |
| fulltext |
A.A. Martynenko, A.D. Tevyashev, N.Ye. Kulishova, B.I. Moroz, 2022
58 ISSN 1681–6048 System Research & Information Technologies, 2022, № 4
UDC 303.732.4 + 004.67 + 004.8
DOI: 10.20535/SRIT.2308-8893.2022.4.05
THE PROBLEM OF AUTOMATIC CLASSIFICATION
OF PICTURES USING AN INTELLIGENT DECISION-MAKING
SYSTEM BASED ON THE KNOWLEDGE GRAPH AND
FINE-GRAINED IMAGE ANALYSIS
A.A. MARTYNENKO, A.D. TEVYASHEV, N.Ye. KULISHOVA, B.I. MOROZ
Abstract. In order to prevent the illegal export of paintings abroad, a museum
examination using various methods for studying a work of art is carried out. At the
same time, an analysis is also made of historical, art history, financial and other in-
formation and documents confirming the painting’s authenticity — provenance.
Automation of such examination is hampered by the need to take into account nu-
merical values of visual features, quality indicators, and verbal descriptions from
provenance. In this paper, we consider the problem of automatic multi-task classifi-
cation of paintings for museum expertise. A system architecture is proposed that
checks provenance, implements a fine-grained image analysis (FGIA) of visual
image features, and automatically classifies a painting by authorship, genre, and
time of creation. Provenance is contained in a knowledge graph; for its vectoriza-
tion, it is proposed to use a graph2vec type encoder with an attention mechanism.
Fine-grained image analysis is proposed to be performed using searching discrimi-
native regions (SDR) and learning discriminative regions (LDR) allocated by convo-
lutional neural networks. To train the classifier, a generalized loss function is proposed.
A data set is also proposed, including provenance and images of paintings by Euro-
pean and Ukrainian artists.
Keywords: automatic multi-task classification, knowledge graph, attention mecha-
nism, fine-grained image analysis, museum expertise, paintings, convolutional neu-
ral networks.
INTRODUCTION
The problem of art objects illegal export continues to be relevant, since they are a
means of accumulating value. In Ukraine, normative documents that regulate the
procedure for customs control and examination of cultural property have been
adopted [1–6]. These documents establish and approve the procedure according to
which it is possible to export values abroad, for which the Authority for Control
over the Movement of Cultural Property and the Protection of the Cultural Heri-
tage of Ukraine issued a certificate for the right to export. At the same time, basis
for such a certificate is customs and museum expertise. The customs examination,
first, aims to establish painting age, since, according to mentioned regulatory
documents, antiques prohibited for export, include items over 100 years old.
During the museum examination, the authenticity and authorship of painting is
established, which, of course, also serves the purpose of dating the work of art.
A wide range of approaches used to establish paintings authenticity, include
forensic, technological, attributive and other methods. They involve various forms
of research, for example, study of artist’s fingerprints, signatures, seals, lists of
invoices confirming the painting sale, reproductions in books and catalogs, a
description of the history of painting creation and ownership to the present
The problem of automatic classification of pictures using an intelligent decision-making system …
Системні дослідження та інформаційні технології, 2022, № 4 59
moment (provenance). Experimental studies also include researching of paintings
using microscopy, fluoroscopy, macrophotography, spectroscopy, etc.
Conducting such a comprehensive study takes a lot of time, requires the participa-
tion of dozens of highly qualified art historians, chemists, digital technology specialists.
Under conditions of a customs check, it is impossible to implement such an
examination. Therefore, for prompt decision-making on a work of art exporting
possibility, an intelligent decision-making system was proposed [7–9]. It provides
for automatic identification of a painting and the establishment of its authenticity
and value based on a photo. However, such an operational check is one part of the pro-
posed two-stage procedure — it can only prevent the export of suspicious art values, but
cannot replace a full museum expertize, which is the basis for a permit certificate.
It is possible to speed up an export permit by automating a full museum
examination, which, from the point of view of machine learning, can be repre-
sented as a classification task. Research in this direction has been going on for
many years, and in recent years, there has been a certain breakthrough associated
with the use of deep networks, in particular, Convolutional Neural Networks
(CNN). They are distinguished by ability to automatically generate vectors of
non-obvious features, especially in image processing tasks, provide higher classi-
fication accuracy compared to other machine learning methods, and have high
speed. Many works demonstrate that the application of CNN to automatic paintings
classification gives positive results [10–14].
However, as noted, an important component of museum expertise is the
study of painting’s provenance, which is usually presented as textual descriptions
of changing size. For CNN that analyze a painting image, provenance turns out to
be useless. On the other hand, deep networks that have proved to be highly effective
in word processing tasks, such as LSTM (Long Short-Term Memory), do not allow
tracking signs of a correlation nature in two-dimensional signals – paintings photos.
Thus, the purpose of this work is to develop the architecture of an intelligent
decision-making system based on deep networks for automatic classification of
paintings, taking into account their provenance.
ANALYSIS OF AREAS OF RESEARCH AND STATEMENT OF THE PROBLEM
Deep networks are currently a common tool for solving a variety of data analysis
problems: searching for objects in images and videos, automatic translation,
handwriting recognition, processing streaming information. There are also exam-
ples of deep architectures use for solving various tasks of preserving cultural heri-
tage in general [15–16], and paintings in particular. Thus, convolutional neural
networks have been used to automatic paintings classification by author and artis-
tic genre [10, 12, 14, 17, 18]. The initial data was digital paintings images, based
on analysis of which CNN generates a response about painting authorship with
high accuracy. At the same time, the classification attributes are formed automati-
cally by the input layers of the CNN, and form internal descriptions – embed-
dings, “understandable” network parameters in numerical form. It is important to
note that specially created datasets are used to train such networks, including tens
of thousands of paintings images [19, 20]. However, the list of artists whose
paintings are included in such sets is quite narrow and is limited to three to four
dozen world-famous masters who worked during the 15th–20th centuries. In the
course of training on such datasets, CNN studies the features of artists’ writing
A.A. Martynenko, A.D. Tevyashev, N.Ye. Kulishova, B.I. Moroz
ISSN 1681–6048 System Research & Information Technologies, 2022, № 4 60
and is able to quite accurately distinguish between paintings of different styles,
but of the same master, or vice versa, of different authors, but of the same genre.
The works of artists included in the typical datasets used to train CNN are
world masterpieces, their location is known, they are well protected. Therefore,
the probability of their presentation for export from Ukraine is extremely small.
However, there are a large number of paintings by lesser-known masters in the
country that should be banned from export due to their significant value. These
pictures were not used to train deep networks, so there is no guarantee that such a
network will confirm the authorship with sufficient accuracy. An unequivocal
help in this situation can be such an architecture that will allow using information
about paintings provenance. In particular, in [19] it is proposed to use the
Knowledge Graph for a provenance branched formalized description in the form
of a graph structure available for further implementation using deep networks.
On the other hand, Ukrainian artists, whose works are of value and may be
banned for export, worked for a much narrower period of time – during the
17th–20th centuries (those created no later than 1920 can be recognized as
antiques). Due to historical circumstances, these paintings do not differ in genre
and stylistic diversity, so it is possible that a network trained to distinguish
between Renaissance and abstract art will not be able to accurately distinguish
between landscapes painted in the style of 19th century classicism and early 20th
century realism. The emerging field of machine intelligence Fine-Grained Image
Analysis (FGIA) develops methods for analyzing sub-categories of images in a
single meta-category [21, 22]. These methods are focused on finding more subtle
and little noticeable, but significant (from authorship point of view) differences
between images, which allow us to single out stable subclasses within the same
class of objects.
In this paper, we propose to apply the Knowledge Graph to formalize
provenance and use it as an attribute when categorizing paintings using Fine-
Grained Image classification implemented in deep learning architecture.
METHODOLOGY
Representation of provenance
Information about painting creation history, its sale to past and current owners is
an undoubted and weighty confirmation of authenticity, along with such charac-
teristics as features of strokes, coloring, chemical composition of paints, primers,
canvas and stretcher wood. This information can be quite scattered, since docu-
ments confirming it can be stored in various institutions, by different persons, or
even be lost. Therefore, provenance data does not have a standardized format, and
is represented most often by field’s text entries such as [23]:
– author’s name;
– artist’s life years;
– picture name;
– picture creation date;
– technique (oil on canvas, oil on wood, watercolor, etc.);
– current location;
– URL link with a digital photo of the painting;
– form (painting, sculpture);
– type (portrait, still life, etc.);
The problem of automatic classification of pictures using an intelligent decision-making system …
Системні дослідження та інформаційні технології, 2022, № 4 61
– school (French, Dutch, etc.);
– era (years of the artist’s work).
Examples of such records [23]:
TOULOUSE-LAUTREC, Henri de, “(b. 1864, Albi, d. 1901, Château
Malromé, Langon)”, Countess Adèle de Toulouse-Lautrec in the Salon of
Malromé Château, 1887, “Oil on canvas, 5445 cm”, “Musée Toulouse-Lautrec,
Albi”, https://www.wga.hu/html/t/toulouse/2/1misc02.html, painting, portrait,
French, 1851–1900.
UNKNOWN MASTER, German, (active 1490s in Nuremberg), Portrait of a
Man, 1491, “Oil on linden panel, 3720 cm”, “Metropolitan Museum of Art, New
York”, https://www.wga.hu/html/m/master/zunk_ge/zunk_ge4a/portrman.html,
painting, portrait, German, 1451–1500.
MONET, Claude, “(b. 1840, Paris, d. 1926, Giverny)”, Monet’s Garden at
Argenteuil, 1873, “Oil on canvas, 6182 cm”, Private collection, https://www.
wga.hu/html/m/monet/03/argent08.html, painting, landscape, French, 1851–1900.
In Ukraine, work is underway to catalog museum collections and draw up
Scientifically Unified Passports. Although it is far from complete, it is being
carried out in accordance with international experience (ICOM requirements,
UNESCO Model export sertificate for cultural objects, etc.) [24].
A record about a painting can be represented as a graph, which in [19] is
called the Knowledge Graph and displays elements of description of painting and
the relationship between these elements. Taking into account Ukraine conditions,
the structure is proposed to be modified (Fig. 1).
Graph embeddings are extracted from model using an encoder – a pre-trained
CNN that implements the node2vec transformation [25] and solves the task of
classification a picture by provenance attributes represented as a graph model.
At present, networks such as ResNet50, ResNet101, ResNet152 demonstrate
the highest classification accuracy in such problems [26, 27]. It is proposed to
train network using a loss function [20]:
2
2Pr ),( jjjjovenance upupL ,
Fig. 1. Nodes and edges of a knowledge graph modeling metadata about a picture
A.A. Martynenko, A.D. Tevyashev, N.Ye. Kulishova, B.I. Moroz
ISSN 1681–6048 System Research & Information Technologies, 2022, № 4 62
where jp — predicted embedding; ju — ground truth context embedding; j —
expertize object (painting).
Using Fine-Grained Image Analysis to paintings classification
Even experienced art historians sometimes make mistakes when determining the
authorship or dating of works painted in the same style in a short time period. In
addition to results of chemical, spectroscopic, and X-ray studies, the FGIA ap-
proach can help in solving this problem, which makes it possible to use informa-
tion about difference in fine details of objects belonging to the same class. The
main difficulty in approach implementing is preservation of information about
regional features when network learns from hundreds of thousands of sample im-
ages. The attention mechanism allows finding the most significant regional fea-
tures in images, and save information about them, despite large size of training
datasets.
In [28], it is proposed to form special regions that store information about
individual objects features belonging to subclasses – Searching and Learning Dis-
criminative Regions (SDR, LDR).
Just as global features are extracted from images using a CNN and images
are assigned to classes based on the mapping of these feature vectors, in
discriminative regions, the deep network generates feature vectors within
individual parts of images that are in some sense similar to each other. An
example of such tasks is distinction between aircraft by type, birds by subspecies
within the same family, and so on. The resulting vector includes both global
features inherent in class images, and private (partial) inherent in subclass images.
Searching Discriminative Regions (SDR) are designed to search and locate
particular features in an image. The scheme of search areas formation is shown in
Fig. 2 [28].
At the heart of this system block operation is an attention-based search
mechanism. The convolutional neural network here provides for search and selec-
tion of all possible features in image, and only thanks to the attention function
does it become possible to search for heterogeneous characteristic features
nLLL ,...,, 21 in one image, and use them as references for searching pictures in
other images.
Fig. 2. Scheme of searching discriminative regions formation
The problem of automatic classification of pictures using an intelligent decision-making system …
Системні дослідження та інформаційні технології, 2022, № 4 63
The network is trained to minimizing the objective function that describes
the angular measure ycos of differences between actual categories logits (fea-
tures) and their values predicted by network:
C
yjj
yj
y
arc
ss
s
L
,1
))(cos(exp))(cos(exp
))(cos(exp
log ,
where C is the number of categories to classify. In this problem, it is equal to
number of compared paintings in dataset.
When locating searching regions on image, it is necessary to minimize losses
associated with excluded zones characteristics:
))((
1
idd
n
i
arcdropDrop emCLL
, (1)
where
idem — image features generated by network that do not go beyond the
network (embeddings) nidi ,...,1 , associated with input picture; dC — is a classi-
fier that maps embeddings into classification objects (logits) categories features.
After searching regions finding, we need to compose their descriptions, tak-
ing into account their possible appearance in other dataset images. For this pur-
pose, learning discriminative regions (LDR) are formed on basis of SDR, in ac-
cordance with scheme of Fig. 3.
By analogy with (1), loss functions are determined for training of convolu-
tional networks that form embeddings fdr ememememem
m
,,...,,...,, 61 :
,))((
;))((
;))((
intint
1
ffarcJoJo
n
i
llarcLocalLocal
rrarcRawRaw
emCLL
emCLL
emCLL
where fdr ememememem
m
,,...,,...,, 61 – embeddings associated with individual
SDRs that are highlighted on input image; flr CCC ,, – classifiers that map em-
bedding features into category features (logits).
Fig. 3. Approach to LDR formation
A.A. Martynenko, A.D. Tevyashev, N.Ye. Kulishova, B.I. Moroz
ISSN 1681–6048 System Research & Information Technologies, 2022, № 4 64
Local regional features (embeddings) are combined in Fusion module, form-
ing a generalized vector fem :
),...,,...,,( 61 mdrf ememememFusionem .
Merge can be implemented by concatenation or convolution. In the second
case, a 1D convolutional layer with H channels will need to be added to the network
architecture, then dimension of generalized embedding vector will be ( 2)n H .
Architecture of an automatic system for classifying paintings using
Knowledge Graph and Fine-Grained Image Analysis
To solve the paintings classification problem, taking into account provenance in
vector representation, and with possibility of distinguishing features of artists of
the same genre, one time period, a system is proposed that is built using
knowledge representation in the form of a graph structure, where feature extrac-
tion on images of paintings is performed using SDR and LDR .
The system general architecture is shown in Fig. 4. The main idea is to learn
convolutional network to project metadata about picture and its fine-grained features
into classification objects space. The solution is carried out in a multi-task mode due
to concatenation of visual feature embeddings from original image, provenance em-
beddings from encoder, and fine-grained feature embeddings from SDR and LDR.
The system loss function in multitasking classification mode is defined as
N
j
rrrrarcjRawRaw emCemCLL
1
))(()1(
N
j
n
i
ilarcjLocalLocal
N
j
ffarcjJoJo emCLemCL
1 11
intint ))(())((
N
j
jjovenance
N
j
iffarcjDropDrop upL
N
emCL
1
Pr
1
),(
1
))(( ,
where are error weights of system modules, i are hyperparameters that ta-
keinto account individual tasks contribution to classification result.
C
re
at
io
n
P
er
io
d
G
en
re
St
yl
e
A
rt
is
t
Fig. 4. Architecture of an automatic system for classifying paintings using Knowledge
Graph and Fine-Grained Image Analysis
The problem of automatic classification of pictures using an intelligent decision-making system …
Системні дослідження та інформаційні технології, 2022, № 4 65
Datasets for solving the automatic paintings classification problem
Since provenance in this task is an integral part of initial information array, it is
necessary to select data for system learning in an appropriate way. Many world-
famous museums include metadata in verbal descriptions form when digitizing
paintings. There are no detailed lists of all documents that verify the entire picture
sale history in such descriptions, but even brief information about the time, place
of creation, style, genre, school, etc. will increase accuracy of multitasking classi-
fication.
In this paper, it is proposed to use datasets [23, 29] that are freely available.
They contain images of paintings by world masters who worked in 15th–20th
centuries, in various techniques, styles and genres. In addition, these datasets con-
tain brief information related to provenance.
To apply developed system in Ukraine, it is obviously necessary to supple-
ment these sets with images of paintings by Ukrainian artists, for example, from
the National Art Museum of Ukraine funds [30]. Metadata about these paintings
and artists can be collected both on museum portal and on Wikipedia.
CONCLUSIONS
The paper considers the problem of paintings automatic classification using an
intelligent decision-making system based on a knowledge graph and Fine-Grained
Image Analysis. A solution is proposed in the form of a classifier based on convo-
lutional neural networks with attention model, operating in a multitasking mode.
The architecture of system that performs visual features automatic detection
and analysis of Fine-Grained features from picture image, provenance vector for-
mation and picture identification by author, style, genre and time of creation has
been developed.
To organize classifier learning, it is proposed to use the loss function based
on the angular mismatch between intranet representations of classification objects.
It is proposed to select data for system training and validation from open ac-
cess datasets, which contain both images of paintings and metadata with descrip-
tions of provenance. In addition, it is proposed to use resources of Ukrainian mu-
seums to update the system in Ukraine.
REFERENCES
1. “Constitution of Ukraine dated June 28, 1996 No. 254k/96-ВР,” Bulletin of the
Verkhovna Rada of Ukraine, 1996.
2. “Law of Ukraine “Fundamentals of the Legislation of Ukraine on Culture” dated
February 14, 1992 N 2117-XII,” Bulletin of the Verkhovna Rada of Ukraine, 1992.
3. “Law of Ukraine “On export, import and return of cultural values” of September 21,
1999 N 1068-XIV,” Bulletin of the Verkhovna Rada of Ukraine, 1999.
4. Resolution of the CMU of June 20, 2000 No. 983 “On the approval of the Regulation
on the State Service of Control over the Movement of Cultural Values across the
State Border of Ukraine”.
5. Letter of the State Customs Service dated 30.12.99 N 09/1619 “Regarding customs
control over the movement of cultural values”.
6. Order of the Ministry of Culture and Tourism of Ukraine dated 11.15.2002 N 647
“On Approval of the List of State Institutions, Cultural Institutions, and Other Or-
ganizations Granted the Right to Conduct State Expertise of Cultural Values”.
A.A. Martynenko, A.D. Tevyashev, N.Ye. Kulishova, B.I. Moroz
ISSN 1681–6048 System Research & Information Technologies, 2022, № 4 66
7. A. Martynenko, V. Moroz, and I. Nulina, “An intelligent decision support system
for cultural property identification,” Computer-IntegratedTechnologies: Education,
Science, Production, 39, pp. 78–82, 2020. Available: http://cit-journal.
com.ua/index.php/cit/article/view/126
8. A.A. Martynenko, A.D. Tevyashev, N.Ye. Kulishova, and B.I. Moroz, «System
analysis of the problem of establishing the authenticity and authority of painting
works,” System Research and Information Technologies, no. 1, pp. 1–16, 2022.
9. A.A. Martynenko, A.D. Tevyashev, N.Ye. Kulishova, B.I. Moroz, and A.S. Ser-
gienko, “Automatic classification of paintings by year of creation,” Radio Electron-
ics, Computer Science, Control, no. 2(61), pp. 80–89, 2022.
10. D. Pancaroglu, Artist, style and year classification using face recognition and clus-
tering with convolutional neural networks; Eds.: D.C. Wyld et al., COMIT, SIPO,
AISCA, MLIOB, BDHI, 2020, CS & IT, CSCP 2020, pp. 41–54. doi:
10.5121/csit.2020.101604.
11. N. Banar, M. Sabatelli, P.Geurts, W. Daelemans, and M. Kestemont, “Transfer
Learning with Style Transfer between the Photorealistic and Artistic Domain,” Elec-
tronic Imaging, Computer Vision and Image Analysis of Art, pp. 41-1–41-9, 2021.
Available: doi.org/10.2352/ISSN.2470-1173.2021.14.CVAA-041
12. E. Cetinic, T. Lipic, and S. Grgic, “Fine-tuning Convolutional Neural Networks for
fine art classification,” Expert Systems with Applications, vol. 114, pр. 107–118,
2018. Available: doi.org/10.1016/j.eswa.2018.07.026
13. T. Chen and J. Yang, “A Novel Multi-Feature Fusion Method in Merging Informa-
tion of Heterogenous-View Data for Oil Painting Image Feature Extraction and Rec-
ognition,” Front. Neurorobot., 12, 2021. Available: doi.org/10.3389/fnbot.2021.709043
14. W. Zhao, D. Zhou, X. Qiu, W. Jiang, “Compare the performance of the models in art
classification,” PLoS ONE, 16(3), 2021. Available: doi.org/10.1371/journal.pone.0248414
15. Data Analytics for Cultural Heritage: Current Trends and Concepts; Eds.: A. Belhi,
A. Bouras, A. Kh. Al-Ali, A.H. Sadka. Springer, 2021, 279 p.
16. M. Fiorucci, M. Khoroshiltseva, M. Pontil, A. Traviglia, A. Del Bue, and S. James,
“Machine learning for cultural heritage: A survey”, Pattern Recognition Letters,
vol. 133, pp.102–108, 2020.
17. M. Kelek, N. Calik, and T. Yildirim, “Painter Classification Over the Novel Art
Painting Data Set via The Latest Deep Neural Networks,” Procedia Computer Sci-
ence, 154, pp. 369–376, 2019. doi: 10.1016/j.procs.2019.06.053.
18. M. Sabatelli, M. Kestemont, W. Daelemans, P. Geurts, “Deep transfer learning for
art classification problems,” Proceedings of the European Conference on Computer
Vision (ECCV), 2018. Available: https://doi.org/10.1007/978-3-030-11012-3_48
19. G. Castellano, G. Sansaro, and G. Vessio, “Integrating Contextual Knowledge to Visual
Features for Fine Art Classification,” arXiv:2105.15028v2 [cs.CV], 28 Sep 2021.
20. N. Garcia, B. Renoust, and Y. Nakashima, “ContextNet: Representation and explora-
tion for painting classification and retrieval in context,” International Journal of
Multimedia Information Retrieval, 9(1), pp.17–30, 2020.
21. X.-S. Wei, J.-H. Luo, J. Wu, and Z.-H. Zhou, “Selective convolutional descriptor
aggregation for fine-grained image retrieval,” IEEE Trans. Image Process., vol. 26,
no. 6, pp. 2868–2881, 2017.
22. X. Zeng, Y. Zhang, X. Wang, K. Chen, D. Li, and W. Yang, “Fine-grained image
retrieval via piecewise cross entropy loss,” Image and Vision Computing, vol. 93,
p.103820, 2020.
23. Open Access Images - National Gallery of Art. Available: https://www.nga.gov/
open-access-images.html
24. O.L. Kalashnikova, “Legal consequences of imperfect description and accounting of
museum collections. International law and the law of the European Union,” Actual
problems of domestic jurisprudence, no. 2, vol. 2, 2018. Available:
http://apnl.dnu.in.ua/2_2018/tom_2/32.pdf
The problem of automatic classification of pictures using an intelligent decision-making system …
Системні дослідження та інформаційні технології, 2022, № 4 67
25. A. Grover and J.Leskovec, “node2vec: Scalable Feature Learning for Networks,”
ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
(KDD), 2016.
26. Deep Residual Networks (ResNet, ResNet50) – 2022 Guide. Available:
https://viso.ai/deep-learning/resnet-residual-neural-network/
27. RESNET50. Available: https://pytorch.org/vision/main/models/generated/torchvision
.models.resnet50.html#resnet50
28. K. Sun and J. Zhu, “Searching and Learning Discriminative Regions for Fine-
Grained Image Retrieval and Classification,” IEICE TRANS. INF. & SYST.,
vol. e105–d, no.1, pp. 141–147, 2022.
29. Best artworks of all time. Available: https://www.kaggle.com/ikarus777/best-
artworks-of-all-time/tasks
30. National Art Museum of Ukraine. Available: https://namumuseum.business.site/
Received 06.11.2022
INFORMATIONON THE ARTICLE
Andrii A. Martynenko, ORCID: 0000-0002-5033-4696, Dnipro University of Technol-
ogy, Ukraine, e-mail: martynenko.andrey.a@gmail.com
Andriy D. Tevyashev, ORCID: 0000-0002-2846-7089, Kharkiv National University of
Radio Electronics, e-mail: andrew.teviashev@nure.ua
Nonna Ye. Kulishova, ORCID: 0000-0001-7921-3110, Kharkiv National University of
Radio Electronics, e-mail: nonna.kulishova@nure.ua
Boris I. Moroz, ORCID: 0000-0003-1290-5694, Dnipro University of Technology,
Ukraine, e-mail: moroz.boris.1948@gmail.com
ПРОБЛЕМА АВТОМАТИЧНОЇ КЛАСИФІКАЦІЇ ЗОБРАЖЕНЬ ЗА ВИКОРИ-
СТАННЯ ІНТЕЛЕКТУАЛЬНОЇ СИСТЕМИ ПРИЙНЯТТЯ РІШЕНЬ НА
ОСНОВІ ГРАФА ЗНАНЬ І ТОЧНОГО АНАЛІЗУ ЗОБРАЖЕНЬ /
А.А. Мартиненко, А.Д. Тевяшев, Н.Є. Кулішова, Б.І. Мороз
Анотація. Для запобігання незаконному вивезенню картин за кордон прово-
диться музейна експертиза з використанням різних методів дослідження твору
мистецтва, зокрема аналіз історичних, мистецтвознавчих, фінансових та інших
відомостей і документів, що підтверджують справжність картин – провенансу.
Автоматизація такої експертизи ускладнюється необхідністю враховувати
числові значення візуальних ознак, показників якості та словесні описи з про-
венансу. Розглянуто завдання автоматичної багатозадачної класифікації кар-
тин під час музейної експертизи. Запропоновано архітектуру системи, яка
перевіряє провенанс, реалізує детальний аналіз (FGIA) візуальних ознак зоб-
раження та виконує автоматичну класифікацію картини за авторством, жанром
та часом створення. Провенанс міститься у графі знань, для векторизації якого
запропоновано використовувати енкодер типу graph2vec з механізмом уваги, а
детальний аналіз пропонується виконувати за допомогою пошукових
відмітних регіонів (SDR) та навчальних відмітних регіонів (LDR), що
виділяються згортковими нейронними мережами. Для навчання класифікатора
запропоновано узагальнену функцію втрат, а також набір даних, що включає
провенанс та зображення картин європейських та українських художників.
Ключові слова: автоматична багатозадачна класифікація, граф знань, меха-
нізм уваги, дрібнодетальний аналіз зображень, музейна експертиза, твори жи-
вопису, згорткові нейронні мережі.
|
| id | journaliasakpiua-article-275082 |
| institution | System research and information technologies |
| keywords_txt_mv | keywords |
| language | English |
| last_indexed | 2025-07-17T10:28:05Z |
| publishDate | 2022 |
| publisher | The National Technical University of Ukraine "Igor Sikorsky Kyiv Polytechnic Institute" |
| record_format | ojs |
| resource_txt_mv | journaliasakpiua/78/bcc77957d91769379cdf6f1cdde9ab78.pdf |
| spelling | journaliasakpiua-article-2750822023-05-21T20:04:38Z The problem of automatic classification of pictures using an intelligent decision-making system based on the knowledge graph and fine-grained image analysis Проблема автоматичної класифікації зображень за використання інтелектуальної системи прийняття рішень на основі графа знань і точного аналізу зображень Martynenko, Andrii Tevyashev, Andriy Kulishova, Nonna Moroz, Boris автоматична багатозадачна класифікація граф знань механізм уваги дрібнодетальний аналіз зображень музейна експертиза твори живопису згорткові нейронні мережі automatic multi-task classification knowledge graph attention mechanism fine-grained image analysis museum expertise paintings convolutional neural networks In order to prevent the illegal export of paintings abroad, a museum examination using various methods for studying a work of art is carried out. At the same time, an analysis is also made of historical, art history, financial and other information and documents confirming the painting’s authenticity — provenance. Automation of such examination is hampered by the need to take into account numerical values of visual features, quality indicators, and verbal descriptions from provenance. In this paper, we consider the problem of automatic multi-task classification of paintings for museum expertise. A system architecture is proposed that checks provenance, implements a fine-grained image analysis (FGIA) of visual image features, and automatically classifies a painting by authorship, genre, and time of creation. Provenance is contained in a knowledge graph; for its vectorization, it is proposed to use a graph2vec type encoder with an attention mechanism. Fine-grained image analysis is proposed to be performed using searching discriminative regions (SDR) and learning discriminative regions (LDR) allocated by convolutional neural networks. To train the classifier, a generalized loss function is proposed. A data set is also proposed, including provenance and images of paintings by European and Ukrainian artists. Для запобігання незаконному вивезенню картин за кордон проводиться музейна експертиза з використанням різних методів дослідження твору мистецтва, зокрема аналіз історичних, мистецтвознавчих, фінансових та інших відомостей і документів, що підтверджують справжність картин – провенансу. Автоматизація такої експертизи ускладнюється необхідністю враховувати числові значення візуальних ознак, показників якості та словесні описи з провенансу. Розглянуто завдання автоматичної багатозадачної класифікації картин під час музейної експертизи. Запропоновано архітектуру системи, яка перевіряє провенанс, реалізує детальний аналіз (FGIA) візуальних ознак зображення та виконує автоматичну класифікацію картини за авторством, жанром та часом створення. Провенанс міститься у графі знань, для векторизації якого запропоновано використовувати енкодер типу graph2vec з механізмом уваги, а детальний аналіз пропонується виконувати за допомогою пошукових відмітних регіонів (SDR) та навчальних відмітних регіонів (LDR), що виділяються згортковими нейронними мережами. Для навчання класифікатора запропоновано узагальнену функцію втрат, а також набір даних, що включає провенанс та зображення картин європейських та українських художників. The National Technical University of Ukraine "Igor Sikorsky Kyiv Polytechnic Institute" 2022-12-27 Article Article application/pdf https://journal.iasa.kpi.ua/article/view/275082 10.20535/SRIT.2308-8893.2022.4.05 System research and information technologies; No. 4 (2022); 58-67 Системные исследования и информационные технологии; № 4 (2022); 58-67 Системні дослідження та інформаційні технології; № 4 (2022); 58-67 2308-8893 1681-6048 en https://journal.iasa.kpi.ua/article/view/275082/270205 |
| spellingShingle | автоматична багатозадачна класифікація граф знань механізм уваги дрібнодетальний аналіз зображень музейна експертиза твори живопису згорткові нейронні мережі Martynenko, Andrii Tevyashev, Andriy Kulishova, Nonna Moroz, Boris Проблема автоматичної класифікації зображень за використання інтелектуальної системи прийняття рішень на основі графа знань і точного аналізу зображень |
| title | Проблема автоматичної класифікації зображень за використання інтелектуальної системи прийняття рішень на основі графа знань і точного аналізу зображень |
| title_alt | The problem of automatic classification of pictures using an intelligent decision-making system based on the knowledge graph and fine-grained image analysis |
| title_full | Проблема автоматичної класифікації зображень за використання інтелектуальної системи прийняття рішень на основі графа знань і точного аналізу зображень |
| title_fullStr | Проблема автоматичної класифікації зображень за використання інтелектуальної системи прийняття рішень на основі графа знань і точного аналізу зображень |
| title_full_unstemmed | Проблема автоматичної класифікації зображень за використання інтелектуальної системи прийняття рішень на основі графа знань і точного аналізу зображень |
| title_short | Проблема автоматичної класифікації зображень за використання інтелектуальної системи прийняття рішень на основі графа знань і точного аналізу зображень |
| title_sort | проблема автоматичної класифікації зображень за використання інтелектуальної системи прийняття рішень на основі графа знань і точного аналізу зображень |
| topic | автоматична багатозадачна класифікація граф знань механізм уваги дрібнодетальний аналіз зображень музейна експертиза твори живопису згорткові нейронні мережі |
| topic_facet | автоматична багатозадачна класифікація граф знань механізм уваги дрібнодетальний аналіз зображень музейна експертиза твори живопису згорткові нейронні мережі automatic multi-task classification knowledge graph attention mechanism fine-grained image analysis museum expertise paintings convolutional neural networks |
| url | https://journal.iasa.kpi.ua/article/view/275082 |
| work_keys_str_mv | AT martynenkoandrii theproblemofautomaticclassificationofpicturesusinganintelligentdecisionmakingsystembasedontheknowledgegraphandfinegrainedimageanalysis AT tevyashevandriy theproblemofautomaticclassificationofpicturesusinganintelligentdecisionmakingsystembasedontheknowledgegraphandfinegrainedimageanalysis AT kulishovanonna theproblemofautomaticclassificationofpicturesusinganintelligentdecisionmakingsystembasedontheknowledgegraphandfinegrainedimageanalysis AT morozboris theproblemofautomaticclassificationofpicturesusinganintelligentdecisionmakingsystembasedontheknowledgegraphandfinegrainedimageanalysis AT martynenkoandrii problemaavtomatičnoíklasifíkacíízobraženʹzavikoristannâíntelektualʹnoísistemiprijnâttâríšenʹnaosnovígrafaznanʹítočnogoanalízuzobraženʹ AT tevyashevandriy problemaavtomatičnoíklasifíkacíízobraženʹzavikoristannâíntelektualʹnoísistemiprijnâttâríšenʹnaosnovígrafaznanʹítočnogoanalízuzobraženʹ AT kulishovanonna problemaavtomatičnoíklasifíkacíízobraženʹzavikoristannâíntelektualʹnoísistemiprijnâttâríšenʹnaosnovígrafaznanʹítočnogoanalízuzobraženʹ AT morozboris problemaavtomatičnoíklasifíkacíízobraženʹzavikoristannâíntelektualʹnoísistemiprijnâttâríšenʹnaosnovígrafaznanʹítočnogoanalízuzobraženʹ AT martynenkoandrii problemofautomaticclassificationofpicturesusinganintelligentdecisionmakingsystembasedontheknowledgegraphandfinegrainedimageanalysis AT tevyashevandriy problemofautomaticclassificationofpicturesusinganintelligentdecisionmakingsystembasedontheknowledgegraphandfinegrainedimageanalysis AT kulishovanonna problemofautomaticclassificationofpicturesusinganintelligentdecisionmakingsystembasedontheknowledgegraphandfinegrainedimageanalysis AT morozboris problemofautomaticclassificationofpicturesusinganintelligentdecisionmakingsystembasedontheknowledgegraphandfinegrainedimageanalysis |