Parallel Computations with Large-scale Air Pollution Models
Large-scale mathematical models are very powerful tools in the efforts to provide more information and more detailed information about the pollution levels, especially about pollution levels which exceed certain critical values.. However, the model used must satisfy at least two conditions: (i) it m...
Gespeichert in:
| Datum: | 2015 |
|---|---|
| Hauptverfasser: | , , |
| Format: | Artikel |
| Sprache: | Ukrainian |
| Veröffentlicht: |
PROBLEMS IN PROGRAMMING
2015
|
| Schlagworte: | |
| Online Zugang: | https://pp.isofts.kiev.ua/index.php/ojs1/article/view/19 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| Назва журналу: | Problems in programming |
| Завантажити файл: |  |
Institution
Problems in programming| id |
pp_isofts_kiev_ua-article-19 |
|---|---|
| record_format |
ojs |
| resource_txt_mv |
ppisoftskievua/cd/0a1b1abf0a0697722f22118981dd50cd.pdf |
| spelling |
pp_isofts_kiev_ua-article-192018-10-09T10:24:50Z Parallel Computations with Large-scale Air Pollution Models Параллельные вычисления на основе крупномасштабных моделей загрязнения атмосферы Паралельні обчислення на основі великомасштабних моделей забруднення атмосфери Dimov, I. Farago, I. Zlatev, Z. UDC 51.681.3 УДК 51.681.3 УДК 51.681.3 Large-scale mathematical models are very powerful tools in the efforts to provide more information and more detailed information about the pollution levels, especially about pollution levels which exceed certain critical values.. However, the model used must satisfy at least two conditions: (i) it must be verified that the model results are reliable and (ii) it should be possible to carry out different study by using the model. It is clear that comprehensive studies about relationships between different input parameters and the model results can only be carried out (a) if the numerical methods used in the model are sufficiently fast and (b) if the code runs efficiently on the available high-speed computers. Some results obtained recently by a new unified version of the Danish Eulerian Model will be presented in this paper. Крупномасштабные математические модели – очень мощный инструмент для получения более детальной информации относительно уровней загрязнений. Однако модель должна удовлетво- рить по крайней мере двум условиям: (i) результаты моделирования должны быть надежными и (ii) должна существовать возможность уточнения и изучения характеристик модели. Всестороннее изучение отношений между различными параметрами входа и результатами моделирования может быть выполнено, если (a) численные методы, используемые в модели, достаточно быстры и (b) программное обеспечение на доступных быстродействующих компь- ютерах достаточно эффективно. Представлены результаты параллельной реализации моделирования загрязнения атмосферы, полученные в новой объединенной версии датской Эйлеровой модели. Великомасштабні математичні моделі – дуже потужний інструмент для одержання більш детальної інформації щодо рівнів забруднень. Проте використовувана модель повинна задовольнити принаймні двом умовам: (i) результати моделювання повинні бути надійними і (ii) повинна існувати можливість уточнення і вивчення різноманітних характеристик моделей. Всебічне вивчення відношень між різноманітними параметрами входу і результатами моделювання може бути виконане, якщо (a) чисельні методи, використовувані в моделі, достатньо швидкі та (b) програмне забезпечення на доступних швидкодіючих комп'ютерах достатньо ефективне. Подані результати рівнобіжної реалізації моделювання забруднення атмосфери, отримані в новій об'єднаній версії датської Ейлерової моделі. PROBLEMS IN PROGRAMMING ПРОБЛЕМЫ ПРОГРАММИРОВАНИЯ ПРОБЛЕМИ ПРОГРАМУВАННЯ 2015-07-01 Article Article application/pdf https://pp.isofts.kiev.ua/index.php/ojs1/article/view/19 PROBLEMS IN PROGRAMMING; No 3 (2003) ПРОБЛЕМЫ ПРОГРАММИРОВАНИЯ; No 3 (2003) ПРОБЛЕМИ ПРОГРАМУВАННЯ; No 3 (2003) 1727-4907 uk https://pp.isofts.kiev.ua/index.php/ojs1/article/view/19/23 Copyright (c) 2015 ПРОБЛЕМИ ПРОГРАМУВАННЯ |
| institution |
Problems in programming |
| baseUrl_str |
https://pp.isofts.kiev.ua/index.php/ojs1/oai |
| datestamp_date |
2018-10-09T10:24:50Z |
| collection |
OJS |
| language |
Ukrainian |
| topic |
UDC 51.681.3 |
| spellingShingle |
UDC 51.681.3 Dimov, I. Farago, I. Zlatev, Z. Parallel Computations with Large-scale Air Pollution Models |
| topic_facet |
UDC 51.681.3 УДК 51.681.3 УДК 51.681.3 |
| format |
Article |
| author |
Dimov, I. Farago, I. Zlatev, Z. |
| author_facet |
Dimov, I. Farago, I. Zlatev, Z. |
| author_sort |
Dimov, I. |
| title |
Parallel Computations with Large-scale Air Pollution Models |
| title_short |
Parallel Computations with Large-scale Air Pollution Models |
| title_full |
Parallel Computations with Large-scale Air Pollution Models |
| title_fullStr |
Parallel Computations with Large-scale Air Pollution Models |
| title_full_unstemmed |
Parallel Computations with Large-scale Air Pollution Models |
| title_sort |
parallel computations with large-scale air pollution models |
| title_alt |
Параллельные вычисления на основе крупномасштабных моделей загрязнения атмосферы Паралельні обчислення на основі великомасштабних моделей забруднення атмосфери |
| description |
Large-scale mathematical models are very powerful tools in the efforts to provide more information and more detailed information about the pollution levels, especially about pollution levels which exceed certain critical values.. However, the model used must satisfy at least two conditions: (i) it must be verified that the model results are reliable and (ii) it should be possible to carry out different study by using the model. It is clear that comprehensive studies about relationships between different input parameters and the model results can only be carried out (a) if the numerical methods used in the model are sufficiently fast and (b) if the code runs efficiently on the available high-speed computers. Some results obtained recently by a new unified version of the Danish Eulerian Model will be presented in this paper. |
| publisher |
PROBLEMS IN PROGRAMMING |
| publishDate |
2015 |
| url |
https://pp.isofts.kiev.ua/index.php/ojs1/article/view/19 |
| work_keys_str_mv |
AT dimovi parallelcomputationswithlargescaleairpollutionmodels AT faragoi parallelcomputationswithlargescaleairpollutionmodels AT zlatevz parallelcomputationswithlargescaleairpollutionmodels AT dimovi parallelʹnyevyčisleniânaosnovekrupnomasštabnyhmodelejzagrâzneniâatmosfery AT faragoi parallelʹnyevyčisleniânaosnovekrupnomasštabnyhmodelejzagrâzneniâatmosfery AT zlatevz parallelʹnyevyčisleniânaosnovekrupnomasštabnyhmodelejzagrâzneniâatmosfery AT dimovi paralelʹníobčislennânaosnovívelikomasštabnihmodelejzabrudnennâatmosferi AT faragoi paralelʹníobčislennânaosnovívelikomasštabnihmodelejzabrudnennâatmosferi AT zlatevz paralelʹníobčislennânaosnovívelikomasštabnihmodelejzabrudnennâatmosferi |
| first_indexed |
2025-07-17T09:59:26Z |
| last_indexed |
2025-07-17T09:59:26Z |
| _version_ |
1850411598497185792 |
| fulltext |
Параллельное программирование
© I. Dimov, I. Farago, Z. Zlatev, 2003
44 ISSN 1727-4907. Проблемы программирования. 2003. № 3
UDC 51:681.3
I. Dimov, I. Farago, Z. Zlatev
PARALLEL COMPUTATIONS WITH LARGE-SCALE AIR
POLLUTION MODELS
Large-scale mathematical models are very powerful tools in the efforts to provide more
information and more detailed information about the pollution levels, especially about pol-
lution levels which exceed certain critical values.. However, the model used must satisfy at
least two conditions: (i) it must be verified that the model results are reliable and (ii) it
should be possible to carry out different study by using the model. It is clear that compre-
hensive studies about relationships between different input parameters and the model re-
sults can only be carried out (a) if the numerical methods used in the model are suffi-
ciently fast and (b) if the code runs efficiently on the available high-speed computers.
Some results obtained recently by a new unified version of the Danish Eulerian Model will
be presented in this paper.
1. Need for a Unified Air Pollution
Model
It will be possible to apply an air
pollution model in many different studies
only if it is very flexible. The work on the
development of the Danish Eulerian
Model (DEM) has been initiated in the
beginning of the 80’s. Only two species
were studied in the first version of the
model. The space domain was rather
large (the whole of Europe), but the dis-
cretization was very crude; the space do-
main was divided by use of a 32x32 grid,
which means, roughly speaking, that 150
km x 150 km grid-squares were used.
Both a two-dimensional version and a
three-dimensional version of this model
were developed. In both versions the
chemical scheme was very simple. There
was only one linear chemical reaction.
The two-dimensional version of this
model is fully described in Zlatev [27],
while the three-dimensional version was
discussed in Zlatev et al. [30]. These first
two versions were gradually upgraded to
much more advanced versions: (i) ver-
sions containing more chemical species,
(ii) versions discretized on more refined
grids and (iii) versions utilizing more ad-
vanced numerical algorithms (see Zlatev
[28] and Zlatev et al. [29]).
Recently, the different versions of
DEM were united in a common model.
Several options can easily be specified
when the new model is applied in differ-
ent scientific studies:
• the model can be used both as
a 2-D model and a 3-D model,
• different horizontal resolutions
can be specified (50 km x 50 km, 16.67
km x 16.67 km and 10 km x 10 km cells
can be used in the horizontal planes),
• different chemical schemes can
be selected (a scheme with 35 species
and a scheme with 56 species being
available at present),
• an algorithm for dividing the
large arrays into chunks can be applied
in an attempt to exploit better the cache
memory of the computer used.
The new model is called UNI-
DEM.
2. Mathematical Description of UNI-DEM
UNI-DEM is described mathemati-
cally, see [28], by a system of partial dif-
ferential equations (PDEs). The number
of equations in this system is equal to the
number of chemical species which are
treated by the model. Five physical and
chemical processes are represented in the
model with different mathematical opera-
tors. These processes are: (i) horizontal
advection (horizontal transport due to the
wind), (ii) horizontal diffusion, (iii) chem-
istry + emissions, (iv) deposition and (v)
vertical exchange (vertical advection +
vertical diffusion).
The system of PDEs by which
these five processes are described in
UNI-DEM is of the following type:
Параллельное программирование
45
( )
( ) ( )
( )
( ) ( )
( ) [ ]
1 2
1 2
, ,...,
, , , ,
, , , 0, , 1, 2,..., ,
ii i
z
i i i
x
i
y i q
i i i i
wcc cK
t z z z
uc vc cK
x y x x
cK Q c c c
y y
E x y z t c
x y z t T i q
κ κ
∂∂ ∂∂ = − + − ∂ ∂ ∂ ∂
∂ ∂ ∂∂ − − + + ∂ ∂ ∂ ∂
∂∂
+ + + ∂ ∂
+ − +
∈Ω ∈ =
(1)
where (i) Ω is the space domain on
which the system of PDEs (1) is defined
(in the 2-D options of UNI-DEM Ω is a
4800 km x 4800 km square, while a
4800 km x 4800 km x 3 km parallelepi-
ped is the space domain Ω in the 3-D
options of UNI-DEM), (ii) [ ]T,0 is the
time-interval (UNI-DEM is normally run
over a time-period covering meteorologi-
cal data for a given year) and (iii) q is
the number of chemical species involved
in the model.
The unknown functions ic in (1)
are concentrations of the involved
chemical species. The quantities u, v
and w are wind velocities along the co-
ordinate axes. xK , yK and zK are diffu-
sion coefficients. The chemical reactions
are described by the non-linear func-
tions iQ . The emissions sources in the
space domain Ω of the model are rep-
resented by the functions iE . Finally,
i1κ and i2κ are deposition coefficients
(for dry deposition and wet deposition
respectively).
The system of PDEs (1) has to be
considered with appropriate initial and
boundary conditions. The choice of such
conditions is beyond the scope of this
paper (some details can be found in
Zlatev [28]).
Different air pollution models can
be obtained from the generic form (1) of
the system of PDEs by using different
mechanisms in the description of the un-
derlying physical and chemical processes
(More details can be found in Ebel [11]
and Zlatev [28]).
3. Applying Splitting in UNI-DEM
It is difficult to treat directly the
system of PDEs (1) by which the UNI-
DEM is described mathematically. There-
fore, different kinds of operator splitting
are used in the large-scale air pollution
models. Any splitting procedure leads to
sub-models, which can be treated by dif-
ferent numerical methods. It is also easier
to choose and use standard packages for
some of the sub-models. Thus, efficient
methods can be selected for each sub-
model, which leads to better overall per-
formance when the whole model is han-
dled computationally.
An attempt to simplify the splitting
procedure used in [28] is made in UNI-
DEM. The big model is split in three sub-
models by using this procedure:
• a sub-model containing the ver-
tical exchange operator,
( )(1)(1) (1)
,
1,2,..., ,
ii i
z
wcc cK
t z z z
i q
∂ ∂ ∂∂
= − + ∂ ∂ ∂ ∂
=
(2)
• a sub-model containing the
horizontal advection and the horizontal
diffusion operators,
( ) ( )(2) (2)(2)
(2) (2)
,
1, 2,..., ,
i ii
i i
x y
uc vcc
t x y
c cK K
x x y y
i q
∂ ∂∂
= − − +
∂ ∂ ∂
∂ ∂∂ ∂
+ + ∂ ∂ ∂ ∂
=
(3)
• a sub-model containing the
chemistry, the emissions and the deposi-
tion operators,
( )
( ) ( )
(3)
(3) (3) (3)
1 2
(3)
1 2
, ,...,
, , , ,
1, 2,..., .
i
i q
i i i i
c Q c c c
t
E x y z t c
i q
κ κ
∂
= +
∂
+ − +
=
(4)
In the actual computations the
three sub-models are treated successively
at a sequence of small time sub-intervals
of length τ of [ ]T,0 . The three sub-
models, (2)-(4), are connected by using
appropriate initial conditions (the cou-
Параллельное программирование
46
pling of the sub-models obtained after
applying some splitting procedure are
discussed in Zlatev [28]).
The first of these three sub-models
is not used when any of the 2-D options
is specified.
The particular splitting procedure
proposed here, see (2)—(4), is based on
ideas discussed in Marchuk [19] and
McRae et al. [20]. Other splitting proce-
dures may also be devised. A very popu-
lar splitting procedure is that proposed
independently in 1968 by Marchuk [18]
and Strang [25].
The splitting procedure is in gen-
eral producing some errors. The order of
the splitting procedure described by (2)-
(4) is one, i.e. ( )τO . The splitting proce-
dure proposed in Marchuk [17] and
Strang [25] is of second order, ( )2τO . At
the same time, it should also be men-
tioned that the latter splitting procedure
is considerably more expensive computa-
tionally.
If some assumptions are imposed,
then the errors due to the splitting pro-
cedure may vanish. Different cases, in
which such assumptions are imposed and,
as a result of this, the errors due to the
splitting procedure disappear, are dis-
cussed in Dimov et al. [5] and Lanser and
Verwer [17].
4. Numerical Methods Used in UNI-DEM
The sub-models are to be treated
by appropriate numerical methods. The
particular numerical methods that are im-
plemented in UNI-DEM are listed below.
• The finite elements discussed in
Pepper et al. [23] are used to discretize
the spatial derivatives in the vertical ex-
change sub-model (2). The semi-discre-
tized model is handled by the ϑ -method
(see Lambert [15] or Samarskii [24]).
• The finite elements from Pepper
et al. [24] are used at present also in the
discretization of the spatial derivatives in
the horizontal advection-diffusion sub-
model (3). Many other methods were
tried: psedospectral algorithm, finite dif-
ferences, methods based on the Bott’s al-
gorithm, semi-Lagrangian algorithm, etc.
The semi-discretized sub-model is treated
by using a set of predictor-corrector (PC)
schemes with several different correctors.
The major idea is to vary the PC schemes
in an attempt both to preserve the stabil-
ity of the computations and to keep the
same time-stepsize (Zlatev [28]).
• The chemistry-emission-deposi-
tion sub-model (4) is treated by using an
improved QSSA (Quasi-Steady-State-
Approximation) algorithm. The original
QSSA algorithm, several classical time-
integration methods for stiff systems of
ODEs and a partitioning algorithm have
also been tried, see Alexandrov et al. [1].
5. Need for high-speed computers
The spatial discretization of UNI-
DEM leads to huge systems of ordinary
differential equations (especially for the
fine resolution options and for the 3-D
options). This is illustrated in Table 1 and
Table 2, where the numbers of equations
and time-steps in the advection part for
the option with 35 chemical species are
given. The numbers of time-steps for the
3-D options of UNI-DEM are the same as
the numbers of time-steps for the corre-
sponding 2-D options (which are given in
Table 2).
Table 1. Numbers of equations when six
options of UNI-DEM are used
Spatial resolution
(horizontal)
2-D option
(1-layer)
3-D option
(10 layers)
96x96 grid 322 560 3 225 600
288x288 grid 2 903 040 29 030 400
480x480 grid 8 064 000 80 640 000
Table 2. Numbers of time-steps when
UNI-DEM is used in a one-year run
Spatial resolution
(horizontal)
2-D option
(1-layer)
96x96 grid 35 520
288x288 grid 106 560
480x480 grid 213 120
In many studies the model has to
be run over many years (in, for example,
climatic studies) and/or with many hun-
dreds of different scenarios (when, for
example, the influence of the biogenic
emissions on high pollution levels is
studied).
Параллельное программирование
47
This short discussion explains why
comprehensive air pollution studies can
successfully be carried out only on mod-
ern high-speed computers and when fur-
thermore the code optimized for runs on
such computers, which is by far not an
easy task.
Moreover, it must be emphasized
that the computers which are available
for us at present are not sufficiently fast
for all options of UNI-DEM. This fact is
clearly seen when the results in Table 4
are studied.
6. Organization of the parallel
computations
It is very important to apply stan-
dard parallelization tools. This is crucial
when the code has to be moved from one
computer to another (which is often
needed in practice).
If a shared memory computer is
available, OpenMP [26] can successfully
be implemented. It is quite straightfor-
ward to use this tool. However one
should be careful in the efforts to solve
the following task: to identify as large as
possible well balanced parallel tasks.
While the programming by using
OpenMP is relatively simple, the solution
of the latter tasks might be difficult.
If the computer is with distributed
memory, then one can apply MPI [14]
(the Message Passing Interface). The im-
plementation of MPI is a much more
complicated task than the implementa-
tion of OpenMP. On the other side, the
use of MPI leads to dividing the data be-
tween the processors, which implies the
use of shorter arrays. This very often re-
sults automatically in a better utilization
of the different cache memories of the
computer used.
Very often MPI can be used even
on shared memory machines and, more-
over, may give good results on such
computers, because as mentioned above
the caches are utilized in a more effi-
cient way.
6.1. Identification of the parallel
tasks when OpenMP is to be used. The
first of the two actions is to identify the
parallel tasks in the different sub-models
(obtained as a result of applying the
splitting procedure used).
Parallel Tasks in the Vertical Ex-
change Sub-Model. In this sub-model,
the vertical exchange of each pollutant
along each vertical grid-line can be con-
sidered as a parallel task. The number of
parallel tasks is equal to the product of
the number of chemical species and the
number of vertical grid-lines. Since the
number of chemical species in UNI-DEM
is in fact 32, the numbers of
parallel tasks are 96x96x32=294912,
288x288x32=2654208 and 480x490x32=
=7372800 for the 3-D options discretized
on 96x96, 288x288 and 480x480 grids re-
spectively. The tasks are small. Therefore,
it is convenient to group several grid-
points as a bigger task in an attempt to
improve the performance.
Parallel Tasks in the Horizontal
Advection-Diffusion Sub-model. In the
horizontal advection-diffusion sub-model,
the transport of each pollutant can be
considered as a parallel task. The number
of species in the UNI-DEM is 35 (see
Section 1). However, three of them are
linear combinations of other species.
Therefore, the number of independent
species is 32. The number of parallel
tasks in the 2-D options is equal to the
number of independent species. This
means that the loading balance is not
perfect when the number of processors is
not a divisor of 32 for the 2-D options. If
the 3-D options are used, then the
number of parallel tasks become equal to
the product of the number of species and
the number of layers, i.e. this number is
now 320. Again the loading balance will
not be perfect when the number of
processors used is not a divisor of 320.
Parallel Tasks in the Chemistry-
Emission-Deposition Sub-model. In the
chemical module, the chemical reactions
at a given grid-point can be considered
as parallel tasks. This means that the
number of parallel tasks is very large
and, moreover, the loading balance is
perfect. The numbers of parallel tasks are
9216, 82944 and 230400 for the 2-D op-
tions discretized on 96x96, 288x288 and
480x480 grids respectively. The numbers
Параллельное программирование
48
of tasks will be increased by a factor of
10 when the corresponding 3-D options
are used. However, in all cases the tasks
are small. Therefore it is convenient to
group several grid-points as a bigger
task. The performance can be improved
considerably by a suitable choice of the
groups of grid-points. The particular
choices of grouping of the small tasks are
discussed in detail in Owczarz and Zlatev
[21], [22].
6.2. Implementation of MPI
options. The approach used when MPI
tools are to be implemented is based on
dividing the space domain of the model
into p sub-domains, where p is the num-
ber of processors which are to be used in
the run. Two specific modules are needed
in the MPI versions:
• a pre-processing module and
• a post-processing module.
Pre-processing Module. The first of
these modules performs a pre-processing
procedure. In this procedure the data are
divided into several parts (according to
the nodes that are to be applied) and
sent to the assigned for the job nodes. In
this way each processor will work on its
own data during the whole run.
Post-processing Module. The sec-
ond module performs a post-processing
procedure. This means that every node
prepares during the run its own output
files. When the run is finished, then the
post-processing module collects the
output data on one of the nodes and
prepares them for future applications
(visualizations, animations, etc.).
Benefits of Using the Additional
Modules. By the preparation of the two
additional modules, one avoids some
excessive communications during the
run. However, it should also be stressed
that not all communications are avoided.
The use of the pre-processor and the
post-processor is in fact equivalent to the
application of a kind of domain decom-
position. Therefore, it is clear that some
communications are needed along the
inner boundaries of the domains.
Such communications are to be carried
out only once per step and only a few
data are to be communicated. Thus, the
actual communications that are to be
carried out during the computational
process are very cheap.
6.3. Combination of MPI and
OpenMP options. On some computer
architectures, like IBM SMP, it is
efficient to use a combination of MPI and
OpenMP options. These computers
consist of several nodes. Every node
contains several processors which share
the same memory. Every mode has its
own local memory. Therefore, it is natural
to use OpenMP within every node and
MPI across the nodes. Some results from
runs on such an architecture with a
combination of OpenMP and MPI options
are presented in Owczarz and Zlatev [22].
7. Parallel runs on SUN computers
The results presented in this sec-
tion were obtained by running six options
of UNI-DEM on 8 processors of SUN
computers. The code was also run on
many other parallel computers: SGI Ori-
gin 2000, IBM SMP, Cray T3E, etc. (see
Georgiev and Zlatev [15] and Owczarz
and Zlatev [21], [22]).
7.1. Information about the com-
puters used. SUN shared memory com-
puters were used in all runs. These com-
puters are located at the High Perform-
ance Centre at the Technical University
of Denmark (DTU). The characteristics of
the computers used are listed in Table 3.
More information about these computers
can be found in http://www.hpc.dtu.dk.
7.2. Results obtained by the MPI
options and different spatial resolutions.
Both 2-D and 3-D options discretized on
all grids mentioned above (a 96x96 grid,
a 288x288 grid and a 480x480 grid) were
used in the experiments. Some results ob-
tained by using these computers are
given in Table 4. MPI programming is
used in the runs the results of which are
shown in Table 4.
The speed-ups were linear (and for
the large problems even super-linear).
This means that the computing times will
be increased with a factor approximately
equal to eight if the jobs are run on one
processor. The largest job will be com-
pleted in more than 286 days if the com-
Параллельное программирование
49
puter is continuously running in so many
days (i.e. no rebooting, no upgrading and
no crashes in at least 286 days). On the
other hand, if 32 processors are used
when the most time-consuming job is run
(the 3-D option on a 480x480 grid), then
the computing time is 192.08 CPU hours
(a speed-up of 4.46 when compared with
the corresponding run on 8 processors).
Table 4. Computing times in CPU hours
when 8 processors are used
It is clearly seen that if the refined
options (especially the 3-D refined op-
tions) are used, then more powerful proc-
essors and more processors are needed
when runs over long-time periods are to
be carried out and/or when results from
many scenarios are to be investigated.
7.3. OpenMP options versus MPI
options. It is more profitable to use MPI
than OpenMP when the UNI-DEM code
is run on SUN computers. Some results,
which illustrate this statement, are given
in Table 5. Both the OpenMP option and
the MPI option were run over a time-
interval consisting of 1000 steps and the
computing times given in Table 5 are in
seconds.
It is seen that the results obtained
by the MPI version are better than those
obtained by the OpenMP option. This
deserves some explanation remarks. In
the MPI option the largest arrays are di-
vided to p sub-domains (where p is the
number of processors) containing arrays
Table 5. Computing times in seconds
when 8 processors are used to run both
the OpenMp option and the MPI option
over a time-interval of 1000 steps
Physical processes
(and total time)
OpenMP
option
MPI
option
Advection-diffusion 2618 457
Chemistry-emission-
deposition
1339 688
Total computing
time
4011 1281
which are p times smaller. This leads
automatically to a reduction of the cache
misses and, thus, improves the perform-
ance. A general framework and theory to
overcome shared memory bottlenecks
and to increase performance by meanse
of enhanced schemes of data exchanges
in multilevel memory systems is pre-
sented in [8—10]. In Table 5 we have
p=8. If we increase the number of proc-
essors, then the MPI option will perform
even better, because the arrays in the
sub-domains become smaller. To illus-
trate that such a statement is true, we
performed the same runs (as those in
Table 5) on 32 processors. The results
are shown in Table 6. The comparison of
the speed-ups (which are given in
brackets in Table 6) shows that the effi-
ciency of the parallelization is greater
for the MPI option.
7.4. Remark about the use of com-
putational grids. Using computational
grids might resolve the difficulties arising
when the refined options are to be ap-
plied in comprehensive air pollution stud-
ies (as the studies listed in the next sec-
tion). If advanced modules for treatment
of particles and/or if meteorological fore-
cast models are attached to UNI-DEM in
Spatial resolution
(horizontal)
2-D option
(1-layer)
3-D option
(10 layers)
96x96 grid 1.41 12.39
288x288 grid 19.56 152.72
480x480 grid 98.72 856.75
Table 3. The computers available at the SUN grid at the Danish Centre for
Scientific Computing
Computer Type Power RAM Processors
Bohr SUN Fire 6800 UltraSparc-III 750 MHz 48 GB 24
Erlang SUN Fire 6800 UltraSparc-III 750 MHz 48 GB 24
Hald SUN Fire 12K UltraSparc-III 750 MHz 144 GB 48
Euler SUN Fire 6800 UltraSparc-III 750 MHz 24 GB 24
Hilbert SUN Fire 6800 UltraSparc-III 750 MHz 36 GB 24
Newton SUN Fire 15K UltraSparc-III 900 MHz 404 GB 72
Параллельное программирование
50
an attempt to improve the reliability of
the results, then the computational work
is increased very considerably. Therefore,
also here computational grids might be
very useful.
Table 6. Computing times in seconds
when 32 processors are used to run both
the OpenMp option and the MPI option
over a time-interval of 1000 steps. The
speed-ups (related to the corresponding
results obtained in the runs on 8 proces-
sors) are given in brackets
8. Comprehensive Studies with UNI-DEM
UNI-DEM has been used in many
comprehensive studies related to air pol-
lution levels in Europe and/or in differ-
ent specified areas in Europe (as, for ex-
ample, Ukraine, Denmark, etc.). The
most important of these studies are
listed below.
• Studying the impact of future
climate changes on the pollution levels in
Europe, Dimov et al. [6].
• Studying the influence of the
biogenic emissions on high ozone levels
in Europe, Geernaert and Zlatev [11].
• Economical estimates of losses
of crops that are caused by high ozone
levels were found in Dimov et al. [7] and
Zlatev et al. [31]. Until now results for
Denmark and Bulgaria have been ob-
tained. However, the same study can be
performed for any other country in
Europe.
• Long-term variations of the pol-
lution levels in Europe, Ambelas Skjoth
et al. [3], Bastrup-Birk et al. [4], Havasi
and Zlatev [15], Zlatev et al. [30].
• Studying the relationship be-
tween the human-made emissions and
the pollution levels in Europe, Ambelas
Skjøth et al. [3], Bastrup-Birk et al. [4],
Dimov et al. [7], Zlatev et al. [30].
• Studying the frequency of ap-
pearance of pollution levels that exceed
the established by EU critical levels [2]
(and, thus, might lead to damages on
plants, animals and human health),
Geernaert and Zlatev [11], Havasi and
Zlatev [15].
• Studying the contribution of
emission sources from any given area in
Physical processes
(and total time)
OpenMP
option
MPI
option
Advection-diffusion 1001 (2.62) 120
(3.81)
Chemistry-emission-
deposition
607 (2.21) 171
(4.02)
Total computing
time
1771 (2.26) 348
(3.68)
Figure 1. Distribution of the nitrogen dioxide concentrations in Europe
(the left-hand-side plot) and in Ukraine (the right-hand-side plot)
Параллельное программирование
51
Europe to its neighbouring areas (for ex-
ample, studying the contribution from the
European emission sources outside Den-
mark to the pollution levels in Denmark),
Ambelas Skjoth et al. [3], Dimov et al.
[7], Havasi and Zlatev [15], Zlatev [28]
and Zlatev et al. [30].
9. Some Results from the Studies
The distribution of nitrogen diox-
ide in Europe is shown, as an example, in
the left-hand-side plot of Figure 1. From
this plot it is seen that the levels of the
2NO concentrations in Ukraine are con-
siderably lower than the levels in the
most polluted parts of Europe.
The corresponding distribution in
Ukraine is given in the right-hand-side
plot of Figure 1 (note that the scales in the
two plots of Figure 1 are different). Some
more details about the levels of the 2NO
concentrations in Ukraine can be seen
from this plot. One can see that the levels
of the 2NO concentrations are rather high
in the area around Kiev (this should be
expected around such a big city).
NH3 + NH4 CONCENTRATIONS
IN THE PERIOD FROM 1989 TO 1998
CHANGES (RELATIVE TO 1989) IN PERCENT
TANGE CALC:
KELDSNOR CALC:
ANHOLT CALC:
The whole of Denmark:
OBS:
OBS:
OBS:
1989 1990 1991 1992 1993 1994 1995 1996 1997 1998
YEAR
50
70
90
110
C
O
N
C
E
N
T
R
A
T
IO
N
R
A
T
IO
S
Figure 2. Demonstration of the decreasing
trend for the Danish ammonia-ammonium
concentrations
Some trends of the Danish ammo-
nia-ammonium levels in the ten-year pe-
riod 1989—1998 are shown in Fig. 2. The
reduction of the ammonia-ammonium
concentrations is typical not only for
Denmark but also for many other coun-
tries in Europe in the same time period.
Similar reductions of the concentration
levels can be seen also for other species.
The reductions of the concentrations are
due to reductions of the emissions.
Acknowledgements
All runs presented in this paper
were performed on the computers of the
Danish Centre for Scientific Computing
(DCSC) under grants: CPU-1002-27 and
CPU-1101-17. The authors should like to
thank DCSC very much for giving us ac-
cess to several parallel computers.
The work of the first author was
supported by the European Commission
under project "Centre of Excellence
BIS-21" (contract number ICA1-CT-2000-
70016) and by the Ministry of Education
and Science of Bulgaria under grand
I-811.
The work of the second author
was supported partly by the National
Scientific Research Foundation (OTKA)
N. T043765.
1. Numerical integration of chemical ODE prob-
lems arising in air pollution models / V. Ale-
xandrov, A. Sameh, Y. Siddique and Z. Zlatev
// Environmental Modeling and Assessment.
– 1997. – 2. – P. 365—377.
2. Directive 2002/3/EC of the European Parlia-
ment relating to ozone in ambient air // Offi-
cial J. of the European Communities. – 2002.
– L67. – P. 14—30.
3. Studying variations of pollution levels in a
given region of Europe during a long time-
period / C. Ambelas Skjoth, A. Bastrup-Birk,
J. Brandt, Z. Zlatev // Systems Analysis Model-
ling Simulation. – 2000. – 37. – P. 297—311.
4. Studying cumulative ozone exposures in
Europe during a seven-year period / A. Bas-
trup-Birk, J. Brandt, I. Uria, Z. Zlatev // Jour-
nal of Geophysical Research. – 1997. – 102.
– P. 23917—23935.
5. Commutativity of the operators in splitting
methods for air pollution models / I. Dimov, I.
Farago, Á. Havasi, Z. Zlatev // Annal. Univ.
Sci. Budapest. – 2001. – 44. – P. 129—150.
6. Dimov I., Geernaert G., Zlatev Z. Influence of
future climate changes in Europe on ozone
critical levels // Nordic Meteorological Meet-
ing 23, 2002 (H. E. Joergensen, ed.). –
http://www.dams.dk/nmm2002/proceedings.htm.
7. Economical estimations of the losses of crops
due to high ozone levels / I. Dimov, Tz. Os-
tromsky, I. Tzvetanov and Z. Zlatev // Agri-
cultural Economics and Management. –
1999. – 5. – P. 48—52.
Параллельное программирование
52
8. Doroshenko A. Mathematical Models and Methods
for Organization of High Performance Parallel Com-
putation. — Kiev: Naukova dumka, 2000. — 177 p.
9. Doroshenko A., Godlevsky A.B., Hluchy L. Models
and Parallel Programming Abstractions for Synchro-
nization and Communication // Computers and Artifi-
cial Intelligence. — 1999. — 18, N 4. — P. 361–381.
10. Doroshenko A. Modeling synchronization and com-
munication abstraction for dynamical paralelization //
HPCN’97: Proc. Int. Conf. High-Performance Com-
puting and Networking, Vienn, Austria, Apr.1997. —
Lect. Not. in Comput. Sci. — 1997. — Vol. 1225. —
P. 752–761.
11. Ebel A. Chemical Transfer and Transport
Modelling // Transport and Chemical Trans-
formation of Pollutants in the Troposphere /
Eds. P. Borrell, P.M. Borrell. – Berlin:
Springer, 2000. – P. 85—128.
12. Geernaert G., Zlatev Z. Testing the influence
of the biogenic emissions on high ozone lev-
els in Denmark and in Europe // Proceedings
from the EUROTRAC-2 Symposium 2002 /
Eds. P. M. Midgley and M. Reuther, Weiker-
sheim: Margraf Verlag. – 2002.
13. Georgiev K., Zlatev Z. Parallel sparse matrix
algorithms for air pollution models // Parallel
and Distributed Computing Practices. –
1999. – 2. – P. 429—442.
14. Gropp W., Lusk E., Skjellum A. Using MPI:
portable programming with the message pass-
ing interface. – Cambridge, Massachusetts:
MIT Press, 1994.
15. Havasi Á., Zlatev Z. Trends in Hungarian air
pollution levels on a long time-scale // At-
mospheric Environment. – 2002. – 36. –
P. 4145—4156.
16. Lambert J. D. Numerical methods for ordinary
differential equations. – New York: Willey,
1991.
17. Lanser D., Verwer J.G. Analysis of operator
splitting in advection-diffusion-reaction prob-
lems in air pollution modeling // J. of Com-
putational and Applied Mathematics. – 1999.
– 111. – P. 201—216.
18. G.I. Marchuk: Some application of splitting-up
methods to the solution of mathematical
physics problems. Applik. Mat., Vol. 13
(1968), 46—75.
19. Marchuk G.I. Mathematical modelling for the
problem of the environment // Studies in
Mathematics and Applications: North-Hol-
land, Amsterdam, 1985. – 16.
20. McRae G.J., Goodin W.R., Seinfeld J.H. Nu-
merical solution of the atmospheric diffusion
equations for chemically reacting flows // J.
of Computational Physics. – 1982. – 45. –
P. 1—42.
21. Owczarz W., Zlatev Z. Running a large air pol-
lution model on an IBM SMP computer // In-
tern. J. of Computer Research. – 2001. – 10.
– P. – 321—330.
22. Owczarz W., Zlatev Z. Parallel matrix compu-
tations in air pollution modeling // Parallel
Computing. – 2002. – 28. – P. – 355—368.
23. Pepper D.W., Kern C.D., Long P.E. Modelling
the dispersion of atmospheric pollution using
cubic splines and chapeau functions // At-
mospheric Environment. – 1979. – 13. –
P. 223—237.
24. Samarskii A.A. Theory of difference schemes.
– Moscow: Nauka, 1977.
25. Strang G. On the construction and comparison
of difference schemes // SIAM J. on Numeri-
cal Analysis. – 1968. – 5. – P. 506—517.
26. WEB-site for OpenMP tools, http://www.-
openmp.org, 1999.
27. Zlatev Z. Mathematical model for studying
the sulphur pollution over Europe // J. of
Computational and Applied Mathematics. –
1985. – 12. P. 651—666.
28. Zlatev Z. Computer treatment of large air pol-
lution models. – Dordrecht-Boston-London:
Kluwer Academic Publishers, 1995.
29. Zlatev Z. Parallel algorithms for the chemical
part of large air pollution models. Mathemati-
cal Modelling. – 2001. – 13. – P. 22—32.
30. Zlatev Z., Berkowicz R., Prahm L.P. Three di-
mensional advection-diffusion modelling for
regional scale. Atmospheric Environment. –
1983. – 17. – P. 491—499.
31. Calculating losses of crops in Denmark
caused by high ozone levels / Z. Zlatev, I.
Dimov, Tz. Ostromsky, G. Geernaert, I. Tzve-
tanov, A. Bastrup-Birk // Environmental
Modelling and Assessment. – 2001. – 6. –
P. 35—55.
Date received 18.08.03
About authors
Dr.Sc. Ivan Dimov
director of Central Laboratory for Parallel
Processing
Bulgarian Academy of Sciences, Sofia, Bulgaria
E-mail: ivdimov@bas.bg
C.Sc. Istvan Farago
ass. professor of Department of Applied
Analysis
Eotvos Lorand University, Budapest, Hungary
E-mail: faragois@cs.elte.hu
Dr.Sc. Zlatko Zlatev
senior researacher of National Environ-
mental Research Laboratory
Roskilde, Denmark
E-mail: ZZ@DMU.dk
|