Software framework for satellite spatial resolution enhancement
Remote sensing provides many crucial data today. Thankfully to the ease of access, global coverage and short revisit time intervals it became possible to retrieve global Earth’s land coverage data effortlessly. This data can provide useful information of the Earth’s land cover current state to make...
Збережено в:
| Дата: | 2024 |
|---|---|
| Автори: | , , |
| Формат: | Стаття |
| Мова: | English |
| Опубліковано: |
PROBLEMS IN PROGRAMMING
2024
|
| Теми: | |
| Онлайн доступ: | https://pp.isofts.kiev.ua/index.php/ojs1/article/view/633 |
| Теги: |
Додати тег
Немає тегів, Будьте першим, хто поставить тег для цього запису!
|
| Назва журналу: | Problems in programming |
| Завантажити файл: |  |
Репозитарії
Problems in programming| id |
pp_isofts_kiev_ua-article-633 |
|---|---|
| record_format |
ojs |
| resource_txt_mv |
ppisoftskievua/a1/c3bc851f87e168926d1f99bc850421a1.pdf |
| spelling |
pp_isofts_kiev_ua-article-6332025-02-15T12:01:17Z Software framework for satellite spatial resolution enhancement Програмна основа підвищення просторової розрізненості супутникових зображень Stankevich, S.A. Shklyar, S.V. Lysenko, A.R. framework; remote sensing; spatial resolution; super-resolution; modulation transfer function; subpixel shift UDC 681.3 основа (фреймворк); дистанційне зондування; просторова розрізненість; надрозрізненість; функція передачі модуляції; субпіксельне зміщення УДК 681.3 Remote sensing provides many crucial data today. Thankfully to the ease of access, global coverage and short revisit time intervals it became possible to retrieve global Earth’s land coverage data effortlessly. This data can provide useful information of the Earth’s land cover current state to make necessary assessments, forecasts, and other tasks that can be in handy for humanity, governments or even farmers. One of the main characteristics of image data quality is its spatial resolution. Thus, spatial resolution enhancement is a relevant topic nowadays. In this article a generalized software framework for satellite spatial resolution enhancement is presented. Due to sensitivity to the satellite data distortion, the applied method considers fusion of several low-resolution images into a single super-resolved one. The proposed framework takes into account satellite data specificity, that is given in a corresponding section. The framework was described to be capable to operate with radar and optical data. For the radar data a corresponding module, that ensures applicability of the super-resolution approach, is given. The framework was implemented using, mainly, C/C++ programming language and tested on a series of real satellite images. The result was evaluated using the modulation transfer function (MTF) approach and has shown an increasement in 135.91% for threefold scale optical images spatial resolution enhancement and 30.93% for the twofold scale radar spatial resolution enhancement. Despite the given representability of the test image set, the presented approach can be beneficial for the tasks that may have a need of the satellite data with higher spatial resolution. The paper concludes with overview of the authors implementation of the given framework and highlighting its drawbacks with suggestions for improvement.Prombles in programming 2024; 2-3: 163-172 Наразі дистанційне зондування надає багато важливих даних. Завдяки вільному доступу, глобальному покриттю та коротким інтервалам повторного знімання, стає можливим легке отримання глобальних даних щодо Земної поверхні. Ці дані є носіями корисної інформації про стан Земної поверхні для необхідних оцінок, прогнозів та інших задач, що можуть стати у нагоді людству, урядам та навіть фермерам. Одна з якісних характеристик зображень є їх просторова розрізненість. Підвищення просторової розрізненості є актуальною задачею. У цій статті представлено загальну програмну основу (фреймворк) для підвищення просторової розрізненості супутникових зображень. Враховуючи чутливість супутникових даних до будь-яких спотворень, робочим методом обрано поєднання кількох зображень низької просторової розрізненості в єдине зображення з підвищеною просторовою розрізненістю. Запропонована основа враховує специфіку супутникових даних, що представлена у відповідному розділі. Наведена основа здатна обробляти як оптичні, так і радарні дані завдяки відповідному модулю, що забезпечує придатність методу надрозрізненості до радарних даних. Основу реалізовано, переважно, за допомогою мови програмування C/C++. Тестування проводилось на вибірці реальних супутникових даних, а оцінка результату проводилась завдяки підходу з використанням функції передачі модуляції (ФПМ). Для оптичних зображень покращення показало 135.91%, де було задіяно трикратне збільшення розмірів зображення, а для радару 30.93%, де було задіяне двократне збільшення розмірів зображення. Незважаючи на низьку репрезентативність результатів тестового набору, описаний підхід може бути корисним для багатьох задач, які мають нестачу в даних з високою просторовою розрізненістю. Стаття завершується оглядом авторської реалізації описаної програмної основи з зазначенням їх недоліків та пропозиціями щодо їх виправлення.Prombles in programming 2024; 2-3: 163-172 PROBLEMS IN PROGRAMMING ПРОБЛЕМЫ ПРОГРАММИРОВАНИЯ ПРОБЛЕМИ ПРОГРАМУВАННЯ 2024-12-17 Article Article application/pdf https://pp.isofts.kiev.ua/index.php/ojs1/article/view/633 10.15407/pp2024.02-03.163 PROBLEMS IN PROGRAMMING; No 2-3 (2024); 163-172 ПРОБЛЕМЫ ПРОГРАММИРОВАНИЯ; No 2-3 (2024); 163-172 ПРОБЛЕМИ ПРОГРАМУВАННЯ; No 2-3 (2024); 163-172 1727-4907 10.15407/pp2024.02-03 en https://pp.isofts.kiev.ua/index.php/ojs1/article/view/633/685 Copyright (c) 2024 PROBLEMS IN PROGRAMMING |
| institution |
Problems in programming |
| baseUrl_str |
https://pp.isofts.kiev.ua/index.php/ojs1/oai |
| datestamp_date |
2025-02-15T12:01:17Z |
| collection |
OJS |
| language |
English |
| topic |
framework remote sensing spatial resolution super-resolution modulation transfer function subpixel shift UDC 681.3 |
| spellingShingle |
framework remote sensing spatial resolution super-resolution modulation transfer function subpixel shift UDC 681.3 Stankevich, S.A. Shklyar, S.V. Lysenko, A.R. Software framework for satellite spatial resolution enhancement |
| topic_facet |
framework remote sensing spatial resolution super-resolution modulation transfer function subpixel shift UDC 681.3 основа (фреймворк) дистанційне зондування просторова розрізненість надрозрізненість функція передачі модуляції субпіксельне зміщення УДК 681.3 |
| format |
Article |
| author |
Stankevich, S.A. Shklyar, S.V. Lysenko, A.R. |
| author_facet |
Stankevich, S.A. Shklyar, S.V. Lysenko, A.R. |
| author_sort |
Stankevich, S.A. |
| title |
Software framework for satellite spatial resolution enhancement |
| title_short |
Software framework for satellite spatial resolution enhancement |
| title_full |
Software framework for satellite spatial resolution enhancement |
| title_fullStr |
Software framework for satellite spatial resolution enhancement |
| title_full_unstemmed |
Software framework for satellite spatial resolution enhancement |
| title_sort |
software framework for satellite spatial resolution enhancement |
| title_alt |
Програмна основа підвищення просторової розрізненості супутникових зображень |
| description |
Remote sensing provides many crucial data today. Thankfully to the ease of access, global coverage and short revisit time intervals it became possible to retrieve global Earth’s land coverage data effortlessly. This data can provide useful information of the Earth’s land cover current state to make necessary assessments, forecasts, and other tasks that can be in handy for humanity, governments or even farmers. One of the main characteristics of image data quality is its spatial resolution. Thus, spatial resolution enhancement is a relevant topic nowadays. In this article a generalized software framework for satellite spatial resolution enhancement is presented. Due to sensitivity to the satellite data distortion, the applied method considers fusion of several low-resolution images into a single super-resolved one. The proposed framework takes into account satellite data specificity, that is given in a corresponding section. The framework was described to be capable to operate with radar and optical data. For the radar data a corresponding module, that ensures applicability of the super-resolution approach, is given. The framework was implemented using, mainly, C/C++ programming language and tested on a series of real satellite images. The result was evaluated using the modulation transfer function (MTF) approach and has shown an increasement in 135.91% for threefold scale optical images spatial resolution enhancement and 30.93% for the twofold scale radar spatial resolution enhancement. Despite the given representability of the test image set, the presented approach can be beneficial for the tasks that may have a need of the satellite data with higher spatial resolution. The paper concludes with overview of the authors implementation of the given framework and highlighting its drawbacks with suggestions for improvement.Prombles in programming 2024; 2-3: 163-172 |
| publisher |
PROBLEMS IN PROGRAMMING |
| publishDate |
2024 |
| url |
https://pp.isofts.kiev.ua/index.php/ojs1/article/view/633 |
| work_keys_str_mv |
AT stankevichsa softwareframeworkforsatellitespatialresolutionenhancement AT shklyarsv softwareframeworkforsatellitespatialresolutionenhancement AT lysenkoar softwareframeworkforsatellitespatialresolutionenhancement AT stankevichsa programnaosnovapídviŝennâprostorovoírozríznenostísuputnikovihzobraženʹ AT shklyarsv programnaosnovapídviŝennâprostorovoírozríznenostísuputnikovihzobraženʹ AT lysenkoar programnaosnovapídviŝennâprostorovoírozríznenostísuputnikovihzobraženʹ |
| first_indexed |
2025-07-17T09:36:48Z |
| last_indexed |
2025-07-17T09:36:48Z |
| _version_ |
1850412926589992960 |
| fulltext |
163
Прикладне програмне забезпечення
УДК 681.3 http://doi.org/10.15407/pp2024.02-03.163
S.A. Stankevich, S.V. Shklyar, A.R. Lysenko
SOFTWARE FRAMEWORK FOR SATELLITE SPATIAL
RESOLUTION ENHANCEMENT
Remote sensing provides many crucial data today. Thankfully to the ease of access, global coverage and short
revisit time intervals it became possible to retrieve global Earth’s land coverage data effortlessly. This data can
provide useful information of the Earth’s land cover current state to make necessary assessments, forecasts, and
other tasks that can be in handy for humanity, governments or even farmers. One of the main characteristics of
image data quality is its spatial resolution. Thus, spatial resolution enhancement is a relevant topic nowadays.
In this article a generalized software framework for satellite spatial resolution enhancement is presented. Due to
sensitivity to the satellite data distortion, the applied method considers fusion of several low-resolution images
into a single super-resolved one. The proposed framework takes into account satellite data specificity, that is giv-
en in a corresponding section. The framework was described to be capable to operate with radar and optical data.
For the radar data a corresponding module, that ensures applicability of the super-resolution approach, is given.
The framework was implemented using, mainly, C/C++ programming language and tested on a series of real
satellite images. The result was evaluated using the modulation transfer function (MTF) approach and has shown
an increasement in 135.91% for threefold scale optical images spatial resolution enhancement and 30.93% for
the twofold scale radar spatial resolution enhancement. Despite the given representability of the test image set,
the presented approach can be beneficial for the tasks that may have a need of the satellite data with higher spa-
tial resolution. The paper concludes with overview of the authors implementation of the given framework and
highlighting its drawbacks with suggestions for improvement.
Key words: framework, remote sensing, spatial resolution, super-resolution, modulation transfer function,
subpixel shift
С.А. Станкевич, С.В. Шкляр, А.Р. Лисенко
ПРОГРАМНА ОСНОВА ПІДВИЩЕННЯ ПРОСТОРОВОЇ
РОЗРІЗНЕНОСТІ СУПУТНИКОВИХ ЗОБРАЖЕНЬ
Наразі дистанційне зондування надає багато важливих даних. Завдяки вільному доступу, глобальному
покриттю та коротким інтервалам повторного знімання, стає можливим легке отримання глобальних даних
щодо Земної поверхні. Ці дані є носіями корисної інформації про стан Земної поверхні для необхідних
оцінок, прогнозів та інших задач, що можуть стати у нагоді людству, урядам та навіть фермерам. Одна з
якісних характеристик зображень є їх просторова розрізненість. Підвищення просторової розрізненості є
актуальною задачею. У цій статті представлено загальну програмну основу (фреймворк) для підвищення
просторової розрізненості супутникових зображень. Враховуючи чутливість супутникових даних до
будь-яких спотворень, робочим методом обрано поєднання кількох зображень низької просторової
розрізненості в єдине зображення з підвищеною просторовою розрізненістю. Запропонована основа
враховує специфіку супутникових даних, що представлена у відповідному розділі. Наведена основа здатна
обробляти як оптичні, так і радарні дані завдяки відповідному модулю, що забезпечує придатність методу
надрозрізненості до радарних даних. Основу реалізовано, переважно, за допомогою мови програмування
C/C++. Тестування проводилось на вибірці реальних супутникових даних, а оцінка результату проводи-
лась завдяки підходу з використанням функції передачі модуляції (ФПМ). Для оптичних зображень покра-
щення показало 135.91%, де було задіяно трикратне збільшення розмірів зображення, а для радару 30.93%,
де було задіяне двократне збільшення розмірів зображення. Незважаючи на низьку репрезентативність
результатів тестового набору, описаний підхід може бути корисним для багатьох задач, які мають нестачу
в даних з високою просторовою розрізненістю. Стаття завершується оглядом авторської реалізації
описаної програмної основи з зазначенням їх недоліків та пропозиціями щодо їх виправлення.
Ключові слова: основа (фреймворк), дистанційне зондування, просторова розрізненість,
надрозрізненість, функція передачі модуляції, субпіксельне зміщення.
© С.А. Станкевич, С.В. Шкляр, А.Р. Лисенко, 2024
ISSN 1727-4907. Проблеми програмування. 2024. №2-3
164
Прикладне програмне забезпечення
Introduction
Nowadays, information is a necessity
for scientific breakthroughs, development of
new methods and technologies and many other
problem solving. Information age brought us
a plethora of means and methods for informa-
tion collection, processing and application. At
present, people are equipped and live among
so many modern, interconnected via internet,
gadgets and tools that improve our quality of
life, relieve our routine burden or help to solve
monotonous tasks, that the digital information
flow became overwhelming.
Storing and processing such informa-
tion brought many new possibilities. As a re-
sult, new technologies have arisen and some
stagnated ones were significantly improved.
For example, a tremendous information ca-
pacity led to the emergence of the well-known
Big Data (Curry et al., 2021) and brought
enormous processing capabilities to the Neu-
ral Networks, comparing to their advent in
somewhat 1950’s (Goodfellow, Bengio and
Courville, 2016), resulting in a rapid develop-
ment leap.
Growing demands in digital processing
powers require an appropriate nature resource
management. Thus, a vast amount of sus-
tainable development tasks became relevant,
which goal is to maintain enough diverse re-
sources for humanity future, for example, wa-
ter resources monitoring (Lock et al., 2023),
crop state monitoring (Nguyen et al., 2020),
yield forecasting (Szabó et al., 2021), forest
fire prevention (Hu, Ban and Nascetti, 2021),
desertification assessment (Lubskyi et al.,
2023) and so on.
Resolving these tasks require avail-
ability of Earth’s land cover data in the terms
of global coverage, i.e., satellite’s visible,
short infrared and near infrared wave bands.
Different satellite sensors provide data with
different quality, namely, spatial resolution:
one sensor can provide data with spatial res-
olution of 30 meters, while other provide data
with spatial resolution of 5 meters (Brown et
al., 2005). The variance in spatial resolution
is mainly caused by dissimilarity in different
orbit heights, on which those sensors operate,
as well as different manufacturing technology.
One may assume that if we have sensors that
provide high spatial resolution data, than there
is no need in low spatial resolution ones. Alas,
high spatial resolution sensor has significantly
lower land coverage during its sensing period,
as well as sensing periods are much longer be-
cause of their low orbit disposition. It may be
thought that it might be appropriate to combine
low-resolution data with global coverage with
high spatial resolution data. Indeed, many tasks
are solved this way. But it may be difficult to
find low-resolution and high-resolution data
within a relatively short time shift interval.
Thus, it is appropriate to develop su-
per-resolution methods for low-resolution data
spatial resolution enhancement. These meth-
ods require an adequate software framework
which will combine main principal ideas for
satellite image super-resolution.
Data characteristics
Before we delve into the framework
itself, it may be expedient to examine source
data characteristics, that are directly tied to the
nature of remote sensing process and to the
principal of super-resolution methods.
Mainly, super-resolution methods are
based on the idea that data enhancement re-
quires directly high-resolution data or high-res-
olution can be combined from several distinct
low-resolution data sources (Fathi, Hadhoud
and El-Khamy, 2012). Fusing several low-res-
olution images into enhanced one can bring
new information to the result. On the contrary,
although filtering (Assia Kourgli and Youcef
Oukil, 2013) and neuron-network-based meth-
ods (Lu et al., 2019) can give remarkable re-
sults, they cannot present any new data in the
result without involvement of additional data
sources. As such, source data requirements
become obvious: input data must contain at
least two different images and the imaging
scene must remain mostly unchanged, i.e.,
there should be a little to none moving objects,
so data in each point could be treated as con-
stant. In satellite imaging such moving objects
that can affect the result’s quality are, usually,
165
Прикладне програмне забезпечення
clouds. To preserve the constancy of the imag-
ing scene the time shift between each satellite
image pair should be short enough to not cause
representation of different objects within same
subscene. To be able to provide any new in-
formation to each other, any image pair should
have some unique subpixel (less than a pixel
size) shift along any or all axes. Otherwise,
fusing images with same data won’t enhance
spatial resolution of the result.
As for the imaging sensors, to be short,
there are mainly two general types: the optical
and the radar ones. The optical can give us a
good RGB representation of the image, but are
limited: they are affected by cloudiness (Zhou
et al., 2022) and require to find corresponding
images with short time delay between them.
The radar, on the contrary, is uninfluenced
by clouds due to its physical nature – much
larger wavelength. And, usually, radar sens-
ing includes consecutive land cover sensing
in two or more different polarizations with-
in a very small (1 second or less) time delay.
Thus, there is no need for image pair search.
However, because such image pairs are tak-
en in different polarization, they do not repre-
sent common physical property and cannot be
processed in the same manner. So, radar data
requires additional processing to convert data
from different polarizations into some unified
representation.
Framework
The spatial resolution framework in-
corporates main ideas for super-resolved im-
age restoration – fusing several low-resolution
images into single super-resolved one. The aim
is to increase target image spatial resolution;
thus, some considerations must be carried.
Firstly, image noisiness is intertwined with
its spatial resolution, so noise must be sup-
pressed. Secondly, because each image pair
must have some subpixel shift, the framework
should take them into account, for example,
by evaluating them. Not to lose generality, the
framework may provide means to process op-
tical or radar data. And last, but not least, the
framework must enhance the spatial resolution
of the result. The schematic presentation of the
framework is shown in figure 1.
As you can see, firstly the satellite data
is fetched from the corresponding data provid-
er. Before it can be processed by the super-res-
olution system it must be preprocessed for the
needs of the specific tasks that stand before the
scientist. After that, preprocessed data flows to
the super-resolution system, which has the fol-
lowing structure:
Input/output system – it may be
graphical user interface (GUI), console appli-
cation or any other form of interaction between
the user and the system;
Fig. 1. Framework for satellite spatial resolution enhancement
166
Прикладне програмне забезпечення
Convert to common property module
– a module that simply converts data, that rep-
resent different physical properties, into a com-
mon one. For example, as for the radar, conver-
sion of multi-polarized data can be converted
into common physical property – land surface
dielectric permittivity via one of the radar back-
scattering models, for example, Oh model (Oh,
Kamal Sarabandi and Ulaby, 1992).
Optimal image subset selection mod-
ule – a module that selects such a subset from
an input image set, that leads to the result with
the best quality according to the super-resolu-
tion model that super-resolution system imple-
ments. As for the described framework, this
module’s source is converted into common
physical property images. Remark: if opti-
cal images within same optical bands (wave-
length) are used – there are no need in con-
version; thus, converted into common property
images are simply the input ones.
Image shift evaluation model – as
the name implies, evaluates the shift between
each source image pair. Later, these shifts are
used as arguments for the enhancement pro-
cess, noise suppression, as well as they may
be the arguments for the optimal image subset
selection (in terms of best geometrical align-
ment for the resulted image). Remark: if the
source images have shifts greater than a single
pixel size, this module should implement inte-
ger-pixel cropping of all input set of images to
the common region of interest.
Noise suppression module – evalu-
ates the noise between converted into common
physical property (if needed) images in respect
to their pairwise subpixel shifts.
Enhancement module – with regard
to pairwise subpixel shifts takes noise-sup-
pressed images in order to rebuild the super-re-
solved image with enhanced spatial resolution.
Remark: besides the noise-suppressed images
the enhancement procedure may take into ac-
count also the form of that suppression.
Implementation
The framework was implemented using
Python (for conversion into a common physi-
cal property) and C/C++ (for everything else)
programming languages with the help of such
external free open-access libraries as: NumPy,
Pillow – for Python; OpenCV, Eigen, ImGUI –
for C/C++. Numpy was used for general data
processing, while Pillow was used as an input/
output library. OpenCV handled input/output
as well as most part of the image processing.
Eigen was used to solve linear equations in the
super-resolution model and ImGUI was used
for the GUI.
The input/output system was devel-
oped in the form of a GUI. A demonstration
of the Super-resolution system GUI is shown
in figure 2.
Fig. 2. Satellite image spatial resolution en-
hancement GUI
The convert to common property
module was implemented using Python lan-
guage. It is in a state of a working prototype
that converts multi-polarized radar data into
the common physical land surface property
– dielectric permittivity – via Oh radar back-
scattering model. It is very time-consuming,
because, as for now, it directly goes over all
possible discrete values of two parameters,
namely surface roughness and dielectric per-
mittivity, that can theoretically satisfy equality
in both (vertical and horizontal) backscattering
polarizations. Thus, as for now, it is a stand-
alone module.
The image shift evaluation module
uses the Young algorithm (Young, Driggers
and Jacobs, 2008) to evaluate pairwise image
shifts displacements. To increase its speed, we
choose the 1/8 discretization, for interval of
along abscissa and ordinate axes ensuring that
precision lost is insignificant.
The optimal image subset selection
module, as for now, is presented in a form of
finding such a subset from an input image set,
that is most informative having least inter-
cover.
The noise suppression module
evaluates the mean noise matrix, from the
input ones, with respect to their subpixel shifts.
It is used to correct the result in the
enhancement process.
The enhancement module is
implemented according to the mathematic
model described in our previous work
(Stankevich et al., 2020). Keeping it short and
simple, it inverses the downscale procedure in
order to rebuild the enhanced image.
Simplistic model for a twofold scale spatial
resolution enhancement is shown below:
𝑌𝑌(𝑦𝑦, 𝑥𝑥) = 𝐺𝐺(∆𝑦𝑦, ∆𝑥𝑥) ⊗ 𝑋𝑋(𝑦𝑦, 𝑥𝑥).
The shifts are given by:
−0.5 ≤ ∆𝑦𝑦 ≤ 0.5, −0.5 ≤ ∆𝑥𝑥 ≤ 0.5.
The general convolution matrix of a
super-resolution transform 𝐺𝐺(∆𝑦𝑦, ∆𝑥𝑥):
𝐺𝐺(∆𝑦𝑦, ∆𝑥𝑥) = (
(0.5 − ∆𝑦𝑦)(0.5 − ∆𝑥𝑥) (0.5 − ∆𝑦𝑦) (0.5 − ∆𝑦𝑦)(0.5 + ∆𝑥𝑥)
(0.5 − ∆𝑥𝑥) 1 (0.5 + ∆𝑥𝑥)
(0.5 + ∆𝑦𝑦)(0.5 − ∆𝑥𝑥) (0.5 + ∆𝑦𝑦) (0.5 + ∆𝑦𝑦)(0.5 + ∆𝑥𝑥)
).
The frequency domain transfer
function is given by:
𝑇𝑇(𝜂𝜂, 𝜉𝜉) = ( (0.5 − ∆𝑦𝑦)𝑒𝑒−2𝜋𝜋𝜋𝜋𝜋𝜋 𝑚𝑚⁄ +
+1 + (0.5 − ∆𝑦𝑦)𝑒𝑒2𝜋𝜋𝜋𝜋𝜋𝜋 𝑚𝑚⁄ ) ×
× ( (0.5 − ∆𝑥𝑥)𝑒𝑒−2𝜋𝜋𝜋𝜋𝜋𝜋 𝑛𝑛⁄ +
+1 + (0.5 − ∆𝑥𝑥)𝑒𝑒2𝜋𝜋𝜋𝜋𝜋𝜋 𝑛𝑛⁄ ).
And the super-resolution model itself
is:
4�̂�𝑌𝑘𝑘 = 𝑇𝑇𝑘𝑘(𝜂𝜂, 𝜉𝜉)�̂�𝑋(𝜂𝜂, 𝜉𝜉) + 𝑇𝑇𝑘𝑘(𝜂𝜂 ± 𝑚𝑚, 𝜉𝜉) ×
× �̂�𝑋(𝜂𝜂 ± 𝑚𝑚, 𝜉𝜉) + 𝑇𝑇𝑘𝑘(𝜂𝜂, 𝜉𝜉 ± 𝑛𝑛)�̂�𝑋(𝜂𝜂, 𝜉𝜉 ± 𝑛𝑛) +
+𝑇𝑇𝑘𝑘(𝜂𝜂 ± 𝑚𝑚, 𝜉𝜉 ± 𝑛𝑛) × �̂�𝑋(𝜂𝜂 ± 𝑚𝑚, 𝜉𝜉 ± 𝑛𝑛) +
+4�̂�𝐸(𝜂𝜂, 𝜉𝜉),
where (∙)̂ – is a Fourier transform
operator and 𝐸𝐸 – is a mean noise matrix. Later,
described model was enhanced to the threefold
scale spatial resolution enhancement
(Stankevich et al., 2023).
In order to increase image processing
speed all pairwise shifts are being evaluated on
a part of the region of interest with 500 × 500
pixel size (if source image is bigger than that),
as well as image processing was carried out in
frequency domain through image’s Fourier
transform. That was needed to significantly
improve speed of images convolution. The
enhancement process was organized in a
window-processing manner to preserve
memory and allow to enhance sets with large
images (5000 × 5000 pixel size and bigger).
Results
The testing was conducted using
graphics workstation equipped with a 16-core
4.2 GHz central processing unit (CPU) on a
Sentinel-1 and Jilin-1 images. Sentinel-1
represents the radar imagery while Jilin-1
represents the optics. The source data is shown
in figure 3.
167
Прикладне програмне забезпечення
The optimal image subset selection
module, as for now, is presented in a form of
finding such a subset from an input image set,
that is most informative having least inter-
cover.
The noise suppression module
evaluates the mean noise matrix, from the
input ones, with respect to their subpixel shifts.
It is used to correct the result in the
enhancement process.
The enhancement module is
implemented according to the mathematic
model described in our previous work
(Stankevich et al., 2020). Keeping it short and
simple, it inverses the downscale procedure in
order to rebuild the enhanced image.
Simplistic model for a twofold scale spatial
resolution enhancement is shown below:
𝑌𝑌(𝑦𝑦, 𝑥𝑥) = 𝐺𝐺(∆𝑦𝑦, ∆𝑥𝑥) ⊗ 𝑋𝑋(𝑦𝑦, 𝑥𝑥).
The shifts are given by:
−0.5 ≤ ∆𝑦𝑦 ≤ 0.5, −0.5 ≤ ∆𝑥𝑥 ≤ 0.5.
The general convolution matrix of a
super-resolution transform 𝐺𝐺(∆𝑦𝑦, ∆𝑥𝑥):
𝐺𝐺(∆𝑦𝑦, ∆𝑥𝑥) = (
(0.5 − ∆𝑦𝑦)(0.5 − ∆𝑥𝑥) (0.5 − ∆𝑦𝑦) (0.5 − ∆𝑦𝑦)(0.5 + ∆𝑥𝑥)
(0.5 − ∆𝑥𝑥) 1 (0.5 + ∆𝑥𝑥)
(0.5 + ∆𝑦𝑦)(0.5 − ∆𝑥𝑥) (0.5 + ∆𝑦𝑦) (0.5 + ∆𝑦𝑦)(0.5 + ∆𝑥𝑥)
).
The frequency domain transfer
function is given by:
𝑇𝑇(𝜂𝜂, 𝜉𝜉) = ( (0.5 − ∆𝑦𝑦)𝑒𝑒−2𝜋𝜋𝜋𝜋𝜋𝜋 𝑚𝑚⁄ +
+1 + (0.5 − ∆𝑦𝑦)𝑒𝑒2𝜋𝜋𝜋𝜋𝜋𝜋 𝑚𝑚⁄ ) ×
× ( (0.5 − ∆𝑥𝑥)𝑒𝑒−2𝜋𝜋𝜋𝜋𝜋𝜋 𝑛𝑛⁄ +
+1 + (0.5 − ∆𝑥𝑥)𝑒𝑒2𝜋𝜋𝜋𝜋𝜋𝜋 𝑛𝑛⁄ ).
And the super-resolution model itself
is:
4�̂�𝑌𝑘𝑘 = 𝑇𝑇𝑘𝑘(𝜂𝜂, 𝜉𝜉)�̂�𝑋(𝜂𝜂, 𝜉𝜉) + 𝑇𝑇𝑘𝑘(𝜂𝜂 ± 𝑚𝑚, 𝜉𝜉) ×
× �̂�𝑋(𝜂𝜂 ± 𝑚𝑚, 𝜉𝜉) + 𝑇𝑇𝑘𝑘(𝜂𝜂, 𝜉𝜉 ± 𝑛𝑛)�̂�𝑋(𝜂𝜂, 𝜉𝜉 ± 𝑛𝑛) +
+𝑇𝑇𝑘𝑘(𝜂𝜂 ± 𝑚𝑚, 𝜉𝜉 ± 𝑛𝑛) × �̂�𝑋(𝜂𝜂 ± 𝑚𝑚, 𝜉𝜉 ± 𝑛𝑛) +
+4�̂�𝐸(𝜂𝜂, 𝜉𝜉),
where (∙)̂ – is a Fourier transform
operator and 𝐸𝐸 – is a mean noise matrix. Later,
described model was enhanced to the threefold
scale spatial resolution enhancement
(Stankevich et al., 2023).
In order to increase image processing
speed all pairwise shifts are being evaluated on
a part of the region of interest with 500 × 500
pixel size (if source image is bigger than that),
as well as image processing was carried out in
frequency domain through image’s Fourier
transform. That was needed to significantly
improve speed of images convolution. The
enhancement process was organized in a
window-processing manner to preserve
memory and allow to enhance sets with large
images (5000 × 5000 pixel size and bigger).
Results
The testing was conducted using
graphics workstation equipped with a 16-core
4.2 GHz central processing unit (CPU) on a
Sentinel-1 and Jilin-1 images. Sentinel-1
represents the radar imagery while Jilin-1
represents the optics. The source data is shown
in figure 3.
168
Прикладне програмне забезпечення
Fig. 3. Jilin-1 source images (3005×3007 pixels)
a b c
d e f
Fig. 4. Sentinel-1 (vertical polarization – b, horizontal polarization – c) and Sentinel-2 (a)
(optical representation) images of a land parcel near Zhytomyr city, Ukraine (600×1000 pixels)
a b c
The radar source images are shown in figure 4.
Optical images do not need any conversion
into a common physical property, because they
already represent one. However, radar images
need such processing; thus, they were con-
verted into dielectric permittivity as shown in
figure 5.
169
Прикладне програмне забезпечення
Fig. 5. Sentinel-1 radar data conversion into a common physical property – land surface dielectric
permittivity (a – vertical polarization, b – horizontal polarization)
a b
a b
c d
Optimal image subset selection is not de-
scribed here, because source data consist only of
6 images for optics and just 2 images for radar.
The enhancement result is given by
figure 6 for optical and 7 for radar. Note: optical
images were threefold scale enhanced, while
radar images were twofold scale enhanced, due
to the lack of sufficient images quantity for the
threefold enhancement.
Fig. 6. Jilin-1 source (a) and enhanced (b) image with their corresponding zoomed fragments
(c – source, d – enhanced)
170
Прикладне програмне забезпечення
As for the radar, maximal memory con-
sumption was 596.1 MB and processing time
was 5 seconds.
In order to evaluate how much spatial
resolution was enhanced a specially developed
module was used. The main principal in this
module is to use modulation transfer function
(MTF) and its threshold value to find a spatial
resolution value that corresponds to the point,
where two objects become indistinguishable.
The example of spatial resolution evaluation is
shown in figure 8.
During optical enhancement procedure
maximal memory consumption that was re-
corded is 6104.8 MB and processing time was
7 minutes and 4 seconds.
Fig. 7. Sentinel-1 source (vertical polarization – a) and enhanced (b) image
with their corresponding zoomed fragments (c – source, d – enhanced)
a b
c d
Fig. 8. Spatial resolution evaluation
171
Прикладне програмне забезпечення
Here you can see the gaussian approx-
imated edge spread function (ESF) of the Ji-
lin-1 (a) source image. Spatial resolution en-
hancement percentage is given in table 1.
So, for optical images with threefold
scale enhancement the spatial resolution en-
hancement was 135.91%, while the twofold
scale enlargement for the radar images was
30.93%. It is worth to note, that processing
time will grow linearly with source data size,
so adequate technique to process large datasets
can be handful.
Table 1
Source images spatial resolution
Figure
number
Spatial resolution,
lines per mm
Enhanced,
%
6 (b) 5.627 -
3 (a-f,
mean) 4.425 135.91%
7 (b) 4.596 -
5 (a-b,
mean) 3.009 30.93%
Conclusion
In this paper a general software frame-
work for satellite spatial resolution enhance-
ment was presented. Its testing was carried
out using dedicated implemented software for
image spatial resolution enhancement. The
result has shown 30.93% enhancement of the
testing radar image using twofold scale image
enhancement, while optical gained 135.91%
spatial resolution enhancement.
The future works will be aimed to elim-
inate existing drawbacks. As it is, the optimal
image subset selection module is realized in
a form of optimizing the source image subset
geometrical informativity cover and might
have to be considered to be developed in a form
signal-to-noise (SNR) ratio maximization. The
image shift evaluation module evaluates the
shift between image pair and presents it as a
single value. But, due to the geometric distor-
tions between image pairs, it may be beneficial
to compute shift for every pixel or for some
pixel-window for additional precision. How-
ever, it may lead to significant computation
burden and processing time increasement. The
enhancement procedure has a very high memo-
ry usage, thus, it may be adequate to reorganize
mathematical model in a way that will allow
small pixel-window enhancement, instead of
keeping whole image in memory.
While the test dataset representability
is some of a concern, nonetheless the result can
be useful for many tasks that can benefit from
high resolution data having in disposal data
with a lack in spatial resolution.
References
1. Curry, E., Metzger, A., Zillner, S., Pazzaglia,
J.-C. and García RoblesA. (2021). The ele-
ments of big data value : foundations of the
research and innovation ecosystem. Cham:
Springer.
2. Goodfellow, I., Bengio, Y. and Courville, A.
(2016). Deep Learning. Cambridge, Massa-
chusetts: The Mit Press.
3. Lock, M., Saintilan, N., Iris van Duren and
Skidmore, A.K. (2023). Monitoring Coastal
Water Body Health with Sentinel-2 MSI Im-
agery. Remote Sensing, 15(7), pp.1734–1734.
doi:https://doi.org/10.3390/rs15071734.
4. Nguyen, M., Baez-Villanueva, O., Bui, D.,
Nguyen, P. and Ribbe, L. (2020). Harmoniza-
tion of Landsat and Sentinel 2 for Crop Moni-
toring in Drought Prone Areas: Case Studies of
Ninh Thuan (Vietnam) and Bekaa (Lebanon).
Remote Sensing, 12(2), p.281. doi:https://doi.
org/10.3390/rs12020281.
5. Szabó, A., Odunayo David Adeniyi, János
Tamás and Nagy, A. (2021). Assessment of a
Yield Prediction Method Based on Time Se-
ries Landsat 8 Data. Acta horticulturae et re-
giotecturae, 24(s1), pp.12–15. doi:https://doi.
org/10.2478/ahr-2021-0003.
6. Hu, X., Ban, Y. and Nascetti, A. (2021). Senti-
nel-2 MSI data for active fire detection in major
fire-prone biomes: A multi-criteria approach.
International Journal of Applied Earth Ob-
servation and Geoinformation, 101, p.102347.
doi:https://doi.org/10.1016/j.jag.2021.102347.
7. Mykola Lubskyi, Tetiana Orlenko, Iryna Pie-
stova, Artem Andreiev and Lysenko, A. (2023).
Evaluation of indicators for desertification risk
assessment of Oleshky sands desertification
based on Landsat data time series. Ukraïnsʹkij
žurnal distancìjnogo zonduvannâ Zemlì,
172
Прикладне програмне забезпечення
УДК 004.896 http://doi.org/10.15407/pp2024.02-03.173
Т. М. Романенко
РОЗРОБЛЕННЯ АЛГОРИТМІВ АВТОМАТИЧНОЇ
РОЗМІТКИ ГІПОКСИЧНИХ ПРОБ ЗА ОДНОКАНАЛЬНОЮ
ЕЛЕКТРОКАРДІОГРАМОЮ: МОДЕЛЬНИЙ ЕКСПЕРИМЕНТ
Наслідки пандемії Covid-19 вимагають нових підходів до реабілітації людини, зокрема, нових моніто-
рингових технологій, що мали б можливості оцінки функціонального стану та рівня тренованості сер-
цево-судинної та дихальної систем. Гіпоксичні проби дають змогу оцінити толерантність людини до
гіпоксії та, як наслідок, визначити рівень тренованості на даний час. Такими пробами є проба Штанге,
у якій пропонують затримувати дихання на вдиху, і проба Генчі, у якій затримка дихання виконується
на видиху. Інформативною є тривалість можливої затримки дихання. Існують методи прямого вимірю-
вання респіраторного сигналу та його опосередкованого контролю. Вони потребують залучення спеці-
альних пристроїв, особливих умов застосування, часто знерухомлення пацієнтів, тому використову-
ються переважно в умовах стаціонару. Вплив дихання призводить до змін у електрокардіограмі люди-
ни, що дає можливість отримати реконструкцію респіраторного сигналу. Реєстрація електрокардіогра-
ми є наразі рутинною процедурою і може бути виконана як в умовах стаціонару, амбулаторії, так і
вдома у пацієнта завдяки наявності великого спектру сучасних мобільних електрокардіографів, одно- і
багатоканальних. Існують певні алгоритми для отримання кардіореспіраторної інформації, які викорис-
товують різні елементи сигналу електрокардіограми. Але усі вони не застосовуються у реальному часі.
Ця доповідь присвячена модельному експерименту щодо пошуку оптимального алгоритму автоматиза-
ції гіпоксичних проб за рахунок обробки одноканальної електрокардіограми у реальному часі. Завдяки
розробленій нами програмі керування диханням під час виконання гіпоксичної проби («Гармонія»)
отримуємо момент початку затримки дихання. Оскільки період затримки дихання у гіпоксичній пробі
має на електрокардіограмі характерні ознаки, які суттєво відрізняються від інших фаз дихання, таких
як вдих, видих, спокійне дихання, момент закінчення затримки знаходимо автоматично. Це дозволяє
автоматизувати проведення гіпоксичних проб.
Ключові слова: гіпоксичні проби, електрокардіограма, реконструкція респіраторного сигналу.
T. Romanenko
DEVELOPING ALGORITHMS FOR AUTOMATIC
HYPOXING TEST FHASING FROM THE SINGLE-CHANNEL
ELECTROCARDIOGRAMS: A MODEL EXSPERIMENT
As an outcome of the COVID-19 pandemic, there is a need to seek new approaches to patients’ rehabilitation,
in particular, the novel monitoring technologies enabling the assessment of the functionality and fitness of the
cardiovascular and respiratory systems. Hypoxic tests allow for estimating a person’s tolerability to hypoxic
conditions and, eventually, for making conclusions about their fitness. Among these tests are the Stange test
for which the breath is held after inhaling, and the Genchi test involving holding the breath after exhaling. The
important information is the duration of the breath hold. There are methods of direct respiratory signal meas-
urement and indirect ones to control it. They require the use of specialized equipment and specific conditions,
often including the need for patient immobilization, therefore, are usually performed in hospitals. Breathing af-
fects the electrocardiogram, which can be used to reconstruct the respiratory signal. Electrocardiogram regis-
tration is now a routine procedure performed in hospitals, outpatient clinics, and, due to various options for
modern portable single- and multichannel electrocardiographs, even at home by the patients themselves. There
are several types of algorithms for obtaining the cardiorespiratory information that rely on different elements
of the electrocardiogram signal but they are not suitable for real-time application. This report describes the
model experiment developing the optimal algorithm of hypoxic test automatization with the electrocardiogram
processing in real-time conditions. We have developed the software called “Harmony” for breathing control
during hypoxic test which suggests the starting moment for breath hold. Since the period of breath hold during
hypoxic test has specific characteristics on the electrocardiogram that are substantially different from other
breathing phases, such as inhaling, exhaling, and calm breathing, the moment of finishing the breath hold can
be determined automatically. This allows us to automate hypoxic tests.
Key words: hypoxic tests, electrocardiogram, reconstruct the respiratory signal.
10(1), pp.17–28. doi:https://doi.org/10.36023/
ujrs.2023.10.1.229. [in Ukrainian]
8. Brown, C.W., Connor, L.N., Lillibridge, J.L.,
Nalli, N.R. and Legeckis, R.V. (2007). An
Introduction to Satellite Sensors, Observa-
tions and Techniques. Remote sensing and
digital image processing (Print), pp.21–50.
doi:https://doi.org/10.1007/978-1-4020-3100-
7_2.
9. Fathi, Hadhoud, M.M. and El-Khamy, S.E.
(2012). Image Super-Resolution and Applica-
tions. CRC Press.
10. Assia Kourgli and Youcef Oukil (2013). Very
High Resolution Satellite Images Filtering.
doi:https://doi.org/10.1109/bwcca.2013.81.
11. Lu, T., Wang, J., Zhang, Y., Wang, Z. and Jiang,
J. (2019). Satellite Image Super-Resolution via
Multi-Scale Residual Deep Neural Network.
Remote Sensing, 11(13), p.1588. doi:https://
doi.org/10.3390/rs11131588.
12. Zhou, J., Luo, X., Rong, W. and Xu, H. (2022).
Cloud Removal for Optical Remote Sensing
Imagery Using Distortion Coding Network
Combined with Compound Loss Functions.
Remote sensing, 14(14), pp.3452–3452.
doi:https://doi.org/10.3390/rs14143452.
13. Oh, Y.D., Kamal Sarabandi and Ulaby, F.T.
(1992). An empirical model and an inver-
sion technique for radar scattering from bare
soil surfaces. IEEE Transactions on Geosci-
ence and Remote Sensing, 30(2), pp.370–381.
doi:https://doi.org/10.1109/36.134086.
14. S. Susan Young, Driggers, R.G. and Eddie
Lynn Jacobs (2008). Signal Processing and
Performance Analysis for Imaging Systems.
Artech House.
15. Stankevich, S., Popov, M., Shklyar, S.,
Sukhanov, K., Andreiev, A., Lysenko, A.,
Kun, X., Shixiang, C., Yupan, S., Xing, Z.
and Boya, S. (2020). Subpixel-shifted sat-
ellite images superresolution: software im-
plementation. WSEAS TRANSACTIONS on
COMPUTERS, 19, pp.31–37. doi:https://doi.
org/10.37394/23205.2020.19.5.
16. Stankevich, S.А., Popov, M., Shklyar, S., Ly-
senko, A., Artem Andreiev, Xing, K., Cao, S.
and Tao, R. (2023). Satellite Imagery Super-
resolution Based on Optimal Frame Accu-
mulation. Springer proceedings in physics,
pp.395–412. doi:https://doi.org/10.1007/978-
981-99-4098-1_35.
Одержано: 09.04.2024
Внутрішня рецензія отримана: 19.04.2024
Зовнішня рецензія отримана: 24.04.2024
About authors:
Sergey Stankevich,
head of the department,
doctor of science,
https://orcid.org/0000-0002-0889-5764
Sergiy Shklyar,
senior researcher,
doctor of science,
https://orcid.org/0000-0001-8726-7936
Artur Lysenko,
junior researcher,
doctor of philosophy,
https://orcid.org/0000-0003-2923-8648
Place of work:
State Institution
“Scientific Centre for Aerospace Research
of the Earth of the Institute of Geological
Sciences of the NAS of Ukraine”,
Email: casre@casre.kiev.ua
|