Узагальнена реалізація процедури методу Дельфі та її застосування до нечітких даних
This paper aims to propose mathematical construction of iterative consensus reaching procedure for expert opinions, based on classic Delphi method concept. This procedure can be applied in mathematical support for qualitative analysis within various types of foresight studies that require expert exa...
Gespeichert in:
| Datum: | 2019 |
|---|---|
| 1. Verfasser: | |
| Format: | Artikel |
| Sprache: | Englisch |
| Veröffentlicht: |
The National Technical University of Ukraine "Igor Sikorsky Kyiv Polytechnic Institute"
2019
|
| Online Zugang: | https://journal.iasa.kpi.ua/article/view/165664 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| Назва журналу: | System research and information technologies |
| Завантажити файл: |  |
Institution
System research and information technologies| _version_ | 1867334361007783936 |
|---|---|
| author | Dzugaev, A. A. |
| author_facet | Dzugaev, A. A. |
| author_institution_txt_mv | [
{
"author": "A. A. Dzugaev",
"institution": null
}
] |
| author_sort | Dzugaev, A. A. |
| baseUrl_str | http://journal.iasa.kpi.ua/oai |
| collection | OJS |
| datestamp_date | 2019-04-26T15:35:46Z |
| description | This paper aims to propose mathematical construction of iterative consensus reaching procedure for expert opinions, based on classic Delphi method concept. This procedure can be applied in mathematical support for qualitative analysis within various types of foresight studies that require expert examinations. The construction of procedure is supported with methodical recommendations and application example based on fuzzy data. |
| first_indexed | 2025-07-17T10:25:02Z |
| format | Article |
| fulltext |
© A.A. Dzugaev, 2005
132 ISSN 1681–6048 System Research & Information Technologies, 2005, № 3
UDC 519.711
THE GENERAL IMPLEMENTATION OF DELPHI METHOD
PROCEDURE WITH APPLICATION ON FUZZY DATA
A.A. DZUGAEV
This paper aims to propose mathematical construction of iterative consensus reach-
ing procedure for expert opinions, based on classic Delphi method concept. This
procedure can be applied in mathematical support for qualitative analysis within
various types of foresight studies that require expert examinations. The construction
of procedure is supported with methodical recommendations and application exam-
ple based on fuzzy data.
The problems of scenarios creation for evolution processes in complex systems
require thorough estimation of various properties for functional elements of sys-
tems being investigated. Expert panels and examinations relied on expert knowl-
edge are known to be the popular and reliable tools for such estimations. At the
same time, the problems of reaching consensus between experts and formation of
consenting expert opinions are still greatly important, while these opinions are
used as background for construction of alternate scenarios and determination of
conditions for their implementations.
Solutions of foresight problems for industrial and economical systems often
demand estimations for such complicated characteristics as competitiveness of
production, economical effectiveness of enterprise, availability and feasibility of
innovations [1]. All mentioned estimations are performed in conditions of incom-
pleteness and uncertainty of incoming information and are remarkable for pres-
ence of some cryptic parameters, which cannot be measured immediately [2].
This situation leads to growing actuality of expert estimations, based upon knowl-
edge, experience and insight of a human being specialist in his domain.
As a rule, no expert can give exact estimation in conditions of uncertainty, so
his opinion is subjective and is characterized by certain degree of assurance. Ow-
ing to this, one of the most important characteristics of expert estimation proce-
dures is the need for iterative modification and refinement of expert opinions, tak-
ing account of the earlier responses and feedback. These iterative refinements, if
properly organized, may also solve the problem of reaching consensus within a
group of experts, using technique known as Delphi.
It is necessary to mention, that in spite of longstanding experience and nu-
merous applications of Delphi method [3, 4], a common holistic approach to Del-
phi examination procedure and mathematical support still does not exist. Most of
previously described examples of Delphi applications either miss the description
of data representation and processing technique, or this description is given in
narrow data domain of concrete application. However, a lot of statistical data and
qualitative information can be obtained from Delphi surveys, and data processing
procedure developed for one type of examination will probably not satisfy the
demands of another without considerable modification.
The general implementation of Delphi method procedure with application on fuzzy data
Системні дослідження та інформаційні технології, 2005, № 3 133
Research objectives. The Delphi method implementation for foresight prob-
lems requires that its mathematical support should be developed on the level of
abstraction enough high to fit the whole variety of possible Delphi examinations
no matter what problems or data domains they correspond to. Another objective
for creation of this common Delphi procedure is to ensure high flexibility of data
analysis algorithms that will be reached through independent multidimensional
processing of Delphi surveys. This approach seems to be especially effective for
investigation of unique, new and lately unknown objects of research, which pa-
rameters and organization are to be found. At the same time, for recently known
objects of research, regarding to which experts has gained some experience, mul-
tidimensional analysis will allow reconsidering available knowledge in a new
fashion and, finally, enlarging it.
Let us now consider the formal target setting, specified in [2].
Target setting. Let us consider a set (group) { }KkeE k ,1| == of experts
ke , Kk ,1= that were asked to answer a set (survey) { }JjqQ j ,1| == of ques-
tions jq , Jj ,1= . Every expert Eek ∈ is giving the answer to survey question
Qq j ∈ as some opinion kjq~ , Kk ,1= , Jj ,1= . Let us emphasize, that any opin-
ion kjq~ from subset { }KkqQ kjj ,1|~~
== of expert answers to survey question jq
is composed by expert ke independently from other experts. We will also not
specify here the nature of opinion jkj Qq ~~ ∈ . Requirements to opinion subsets jQ~ ,
Jj ,1= , namely existence of metric and quality functional on them, perhaps own
for any Jj ,1= , will be stated later.
It is necessary after some iteration (rounds) of examination and analysis of
expert opinions (Fig. 1) to meet the specified consensus criterion and find for any
survey question Qq j ∈ a group of experts, whose opinions on this question are
forming consenting clusters. In every cluster, then, a resulting opinion should be
selected at the end of every round that will represent an agreed decision for corre-
sponding group of experts. Another important, yet not formal goal of examination
is to retrieve comments from experts, containing reasoning and argumentations
for their opinions and perform the subsequent anonymous exchange with this in-
formation, which will allow experts to refresh their knowledge and modify their
opinions.
The final agreed opinions of the last round will represent the alternate an-
swers of expert groups to each question of survey and may be used as background
material for other methods of qualitative analysis [5], such as the Analytic Hierar-
chy Process, scenario creation and others.
Method of solution. Let us construct the analysis procedure for one round of
examination. This procedure aims to analyze opinions kjq~ from all Kk ,1= ex-
perts to all Jj ,1= survey questions on each examination round. As we have
mentioned above, all questions are formulated and answered independently, there-
fore expert opinions on each question will also be processed independently. Of
A.A. Dzugaev
ISSN 1681–6048 System Research & Information Technologies, 2005, № 3 134
course, the idea of independent processing sets up some claims on the structure of
questions [6]. First, all questions have to be independent and there must not be
any ambiguity. Second, there must not be any conditional statements, which make
the primary question dependent on the fulfillment of a series of conditions. Ques-
tions where this occurs should be split into two or more separate questions.
Prior to opinion processing, we have to create all necessary tools of analysis.
As will be shown below, there are just two things we need: the metric and the
quality functional.
Introducing metric. To find out, how much expert opinions differ from
each other, it is necessary to introduce metric on every opinion subset
{ }KkqQ kjj ,1|~~
== , and we are able to use different metrics for different
Jj ,1= . The metric ℜ→× jjj QQ ~~:ρ allows for any pair of opinions ∈kjij qq ~,~
jQ~∈ , Kji ,1, = to define the measure of their distinction as distance
Start new round of
examination;
retrieve expert opinions
Select clusters
of consenting opinions
For each cluster, find
weights of opinions Proposed experts to
modify their opinions,
taking account of the
earlier responses For each cluster, find
median opinion and
trusted set of opinions
as results of round
Apply consensus criterion
to each cluster of
expert opinions
[consensus not reached]
Lay cluster opinions and
results of round open to
experts' inspection
[next round available]
Lay cluster opinions and
final results open to
public inspection
[rounds expired]
[consensus
reached]
Fig. 1. The Delphi procedure flow diagram
The general implementation of Delphi method procedure with application on fuzzy data
Системні дослідження та інформаційні технології, 2005, № 3 135
( )kjijjikj qqr ~,~ρ= . The choice of metric is a crucial question that has to take into
account the properties of data that represents expert opinions. In such a way we
shift the features of data domain from method implementation to implementation
of metric.
Introducing quality functional. Assigning weights to opinions aims to en-
sure their iterative refinement therefore requires mechanism of opinion quality
determination. Opinions of highest quality should be considered more thoroughly,
while the less quality opinions should be excluded from consideration, or at least
their impact on analysis result should be reduced. We can create this mechanism
by introducing the quality functional on every opinion subset { }KkqQ kjj ,1|~~
== ,
and again, we are able to use different functional for different Jj ,1= . The func-
tional [ ]1;0~: →jj Qω allows defining the weight of any expert opinion
jkj Qq ~~ ∈ , Kj ,1= as ( )kjjkj qw ~ω= , the weight of opinion is as high as its qual-
ity is considered to be. As you can see, we have again shifted the influence of data
domain from method implementation to implementation of quality functional.
The choice of quality functional has a strong impact on examination results, there-
fore it is better to rely on analytical characteristics of expert opinions, rather than
on self-estimation of quality by experts. These self-estimations are mainly subjec-
tive and depend on personalities, their modesty and self-confidence.
Now, as we have all necessary tools, let us describe the method. Analysis
procedure consists of the following five steps:
1. Opinions clustering. The aim of clustering is to prepare background for
success of further analysis by means of dividing every opinion subset
{ }KkqQ kjj ,1|~~
== into groups (clusters) of consenting opinions jjl QC ~
⊆ ,
jLl ,1= , 1≥jL . Clusters may be composed in various ways using metric jρ or,
perhaps, some other properties of expert opinions. It is suggested, however, that
in every cluster opinions should be enough close to each other, so that experts will
be able to reach consensus among further iterations and, finally, agree upon one
opinion. There probably will be some outlying opinions, and thorough argumenta-
tion will be required from their authors to share it with other experts. Every clus-
ter will be analyzed with the object of median determination.
2. Median determination. Consider cluster jlC containing N expert opin-
ions jiq~ , jji QqNi ~~,1 ∈=∀ .
Median jlM is defined as expert opinion from cluster jlC least distant from
the other opinions of jlC in metric jρ (1).
( )⎟⎟
⎠
⎞
⎜
⎜
⎝
⎛
== ∑
==
∗
N
i
jpjij
Np
jpjl qqqM
1,1
~,~minarg~ ρ . (1)
Having built a symmetric matrix of distances { }ipjjl r=D Npi ,1, = , where
( )jpjijipj qqr ~,~ρ= for opinions jlji Cq ∈~ we are able to find median as opinion
with minimal row sum.
A.A. Dzugaev
ISSN 1681–6048 System Research & Information Technologies, 2005, № 3 136
For each examination round median represents the result of examination for
group of experts that formed cluster jlC on survey question jq . The fact that
examination result is represented by median, instead of average opinion that may
be calculated in some possible ways, is of fundamental importance. First of all,
this ensures commonality of method, since median determination requires nothing
but metric, while calculation of average opinion will probably require some sup-
plementary tools, probably dependent on the nature of data that represents opin-
ions. Secondly, in general case there is never enough information for any unbi-
ased averaging calculations on expert opinions, while median is always a
reasoned opinion of some concrete expert.
3. Weights assignment. Before we build trusted set of opinions on cluster
jlC we should take into consideration quality of expert opinions. It is desirable to
include in trusted set opinions with high quality, that is, with high weights, rather
than with low weights and quality, which impact on examination result we are
determined to reduce. Using the quality functional jω for opinions jlji Cq ∈~ we
can calculate the weight vector { }jijl w=W Ni ,1= ( )jijji qw ~ω= . As long as
trusted set is built around the median, we may add to distance
( )jlijjiMj Mqr ,~ρ= from opinion ijq~ to median jlM a weight-determined cor-
rection factor, in order to increase distance from low quality opinions to median
that will decrease their chances to get into trusted set. For example, to double the
distance from median to opinion with weight 0 (the lowest quality) and to keep
unchanged the distance to opinion with weight 1 (highest quality), we need a cor-
rection factor equal ( )jiw−2 , Ni ,1= , Jj ,1= .
4. Building trusted set. The trusted set jlT has to contain some fixed part
of opinions from cluster jlC least distant from its median jlM . Opinions from
trusted set are considered «consenting», while the rest opinions are «extreme». On
the first round trusted set is built using a priori consent index S , that sets the ratio
of consenting opinions to total amount of opinions in cluster. Thus, when
( ) SNT jl ≤card (card(T ) means cardinality of the set, i.e. number of items in T),
opinion jlji Cq ∈~ , Ni ,1= is added to trusted set jlT , if
( )( )( )jijljij
TCq
ji wMqq −=
∈
2,~minarg~
/~
ρ . (2)
where ( )jiw−2 is weight-determined correction factor considered above. After
trusted set is built, we define it's radius jlR as distance from median jlM to the
most distant opinion in trusted set, considering correction factor ( )jiw−2 .
( )( )( )jijljij
Tq
jl wMqR −=
∈
2,~max~ ρ . (3)
On the next rounds of examination jlR is fixed to its value 1
jlR on the first
round and the principle is another: opinion jlji Cq ∈~ , is added to trusted set, if
The general implementation of Delphi method procedure with application on fuzzy data
Системні дослідження та інформаційні технології, 2005, № 3 137
( )( ) 12,~
jljijljij RwMq ≤−ρ (4)
relatively the new found median.
5. Applying consensus criterion. On the first round of examination the
trusted set radius 1
jlR is used for consensus criterion within each cluster jlC . On
the next rounds, consensus criterion is consent index S . Let us regard expert
opinions in cluster jlC as convergent, if cluster remains stable from round to
round in the course of examination and for any round n holds nn SS ≤−1 . If the
ratio S is decreasing, then expert opinions within given cluster are divergent and
no consensus can be reached. If it is possible to pass another round, then diver-
gent cluster will probably be divided between some other clusters. A signal to
stop examination for cluster jlC is when consent index S exceeds some pre-
scribed threshold ∗S , or it is just impossible to start another round for organiza-
tional reasons. With all this going on, the median of last round is taken as final
agreed answer to survey question considered.
On the every new round of examination experts are acquainted with results
of the previous round, including the fact of hitting or not hitting the trusted set
with their opinions. Those experts, whose opinions are «extreme», that is cluster-
ing outliers or just out of trusted set, are asked to modify their opinions consider-
ing the judgment of the rest experts. Authors insisting on extreme opinions must
share with others their arguments and reasoning, as feedback from them may be
very important. Those who hit the trusted set may also reconsider their opinions
under the influence of their colleagues. After new opinions are retrieved, median
and trusted set are determined anew for all clusters on all survey questions. Re-
sults of examination and feedback are passed to experts anonymously, without
concrete identification of authors. This aims to exclude obtrusion of opinions and
influence of some authoritative personalities and may ensure far better conver-
gence of opinions than any artificial methods and models.
Apparently, the global convergence of opinions never aims to be the primary
outcome of Delphi examination. Variety of alternate opinions, estimations and
views is highly appreciated in foresight and futures studies. Not only the opinions,
but often reasoning, feedback and arguments given within experts' communica-
tion are important deliverables of the process. Another outcome of Delphi is the
multiplication of expert knowledge, that is reached through experts cooperation
and networking, when sharing views and feedback helps all stakeholders to learn
more about the subject of studies.
Before we go on to application example for technique developed above, let
us again accentuate, that mathematical methods and models, whatever sophisti-
cated they could be, can not separately ensure the successful implementation of
Delphi method. That is why especially the psychological process in connection
with anonymous communication, together with reach feedback and argumentation
has to be stressed.
A.A. Dzugaev
ISSN 1681–6048 System Research & Information Technologies, 2005, № 3 138
Case study. Application on fuzzy data. Consider the application of Delphi
method to expert opinions represented with fuzzy estimations. Let six experts
( 6=K ) estimate the risk of failure for some innovation activity. The segmented
fuzzy estimation is built by expert through single-valued mapping the segments of
risk value scale on segments of risk level scale (see Table), the scale segments
having both quantitative and qualitative description.
Consequently, the opinion kq~ of expert ke on the risk q of given innova-
tion activity is represented by fuzzy value kµ , built by means of interpolation on
discrete set of points chosen inside the segments checked by expert (see
Figs 2–7). The fuzzy values are regarded here as a class of continuous functions
[ ] [ ]1;01;0: →µ , ( ) [ ]1;0Cx ∈µ , that allows to introduce on them a standard 1L
metric ( ) ( ) ( )∫ −=
1
0
2121, dxxx µµµµρ .
The quality functional for these fuzzy values will be stated as
( ) ( )µµρµω ,1−= , where µ is a model for some kind of «perfect» estimation,
representing Gaussian density function normalized to value area of fuzzy
function ( )xµ [7]. The model function mean (5) and dispersion (5a) are defined
as follows:
[ ]
( )xa
x
µ
1,0
maxarg
∈
= (5)
[ ]
( )xa
x
µσ
1,0
minarg
3
1
∈
−= (5a)
In such a case, the quality of opinion represented as fuzzy estimation is as high, as
less fuzzy value function is distant from its model function in 1L metric.
Consider now the sample expert opinions shown on Figs. 2–7. As there are
just six opinions, they all will be included in a single cluster C , in which we have
to choose median M and trusted set T .
qualitative
description lowest lower low medium high higher highest
highest [0,90; 1,00]
higher checked [0,75; 0,90]
high checked checked checked [0,60; 0,75]
medium [0,40; 0,60]
low checked checked [0,25; 0,40]
lower checked [0,10; 0,25]
lowest [0,00; 0,10]
[0,0; 0,1] [0,1; 0,25] [0,25; 0,4] [0,4; 0,6] [0,6; 0,75] [0,75; 0,9] [0,9; 1,0]
quantitative
description
ris
k
le
ve
l
risk value
Risk estimation for innovation activity
The general implementation of Delphi method procedure with application on fuzzy data
Системні дослідження та інформаційні технології, 2005, № 3 139
Expert 1 answer
Fig. 2. Opinion of the first expert (trusted)
Expert 2 answer Rusulting (median)
Fig. 3. Opinion of the second expert (median)
Expert 3 answer
Fig. 4. Opinion of the third expert (not trusted)
A.A. Dzugaev
ISSN 1681–6048 System Research & Information Technologies, 2005, № 3 140
Expert 4 answer
Fig. 5. Opinion of the fourth expert (not trusted)
Expert 5 answer
Fig. 6. Opinion of the fifth expert (not trusted)
Expert 6 answer
Fig. 7. Opinion of the sixth expert (trusted)
The general implementation of Delphi method procedure with application on fuzzy data
Системні дослідження та інформаційні технології, 2005, № 3 141
With the help of 1L metric, the symmetric distance matrix ( ){ }ki µµρ ,=D
Kki ,1, = for cluster C is calculated as
The vector of row sums for matrix D equals
The median M by definition must have the least row sum, therefore the me-
dian in cluster C is the opinion of second expert, 2
~qM = .
The vector ( ){ }Mqk
M ,~ρ=D , Kk ,1= of distances between cluster C
opinions and median in terms of 1L metric equals
As the result of weights assignment with the above defined quality func-
tional ω we get the weights vector ( ){ }kµω=W , Kk ,1= , equal
Adding the weight-determined correction factor ( )( )kµω−2 , Kk ,1= into
distances vector MD we are transforming it to
With these weighted distances the trusted set T is calculated as the half
( 5,0=S ) of opinions in cluster C less distant from median. In this case, together
with median, the opinions of the first and sixth experts are hitting trusted set.
Consequently, the resulting opinion within considered six experts is the opinion
of second expert 2
~qM = (Fig. 3), and trusted set contains opinions of first, sec-
ond and sixth experts, { }621
~,~,~ qqqT = . The consensus criterion for the next round
of examination will be the trusted set radius 1287,0=R .
Fig. 8 shows the optimistic and pessimistic risk estimations built on opinion
of second expert as median (Fig. 3). As we can see, well, the innovation is
unlikely to fail.
Conclusion. The general implementation of Delphi method procedure pro-
posed here is based upon the classic principles of experts examination, that were
generalized and comprehended based upon the modern requirements to apparatus
of qualitative analysis. The mathematical method for analysis of expert opinions
is implemented on the higher level of abstraction, than required by any type of
experts' examination that allows applying Delphi technique on any metric space
whatever type of information it represents. Thanks to this approach both the
0,0000 0,0556 0,1678 0,1878 0,1453 0,0791
0,0556 0,0000 0,1666 0,1622 0,1166 0,1072
0,1678 0,1666 0,0000 0,1956 0,1081 0,2431
0,1878 0,1622 0,1956 0,0000 0,2144 0,2181
0,1453 0,1166 0,1081 0,2144 0,0000 0,1969
0,0791 0,1072 0,2431 0,2181 0,1969 0,0000
0,6356 0,6081 0,8813 0,9781 0,7813 0,8444
0,0556 0,0000 0,1666 0,1622 0,1166 0,1072
0,7688 0,8239 0,7150 0,8392 0,7305 0,7998
0,0685 0,0000 0,2140 0,1883 0,1480 0,1287
A.A. Dzugaev
ISSN 1681–6048 System Research & Information Technologies, 2005, № 3 142
mathematical method and its software implementation may be successfully ap-
plied to solve various problems of qualitative analysis, foresight and other fields
of application of expert systems.
REFERENCES
1. Zgurovsky M.Z. The Scenario Analysis Platform as Methodological Base of the Na-
tional Foresight Program of Ukraine. // System Research and Information Tech-
nologies. — 2003. — № 1. — P. 7–25.
2. Pankratova N.D. Software for Technology Foresight Problems with Reference to the
Industry. // System Research and Information Technologies. — 2003. — № 1. —
P. 26–33.
3. Linstone H.A., Turoff M. (eds.) The Delphi Method: Techniques and Applications.
Addison-Wesley, Reading, Mass., 1975. — 570 p.
4. Brockhaus W.L., Mickelsen J.F. An analysis of prior Delphi applications and some
observations on its future applicability. — Tech. Forecast and Soc. Change,
1977. — Ch.10. — P. 103–110.
5. Loveridge D. Foresight and Delphi Processes as Information Sources for Scenario
Planning. Paper to the IIR Conference on «Scenario Planning» London, Novem-
ber, 1996. — 11 p. Available from http://www.les.man.ac.uk/PREST/People/
Staff/ Denis_Loveridge.html
6. Rauch W. The Decision Delphi. — Tech. Forecast and Soc. Change, 1979. —
Ch.15. — P. 157–170.
7. Dzugaev A.O. Fuzzy Estimations as Expert Opinions in Qualitative Analysis // Sixth
International Conference on Applied Mathematics and Computer Science. Na-
tional University of Lvov, Ukraine. — 2003. — P. 105–109.
Received 05.02.2004
From the Editorial Board: The article corresponds completely to submitted manu-
script.
Pessimistic forecast Optimistic forecast
Fig. 8. The optimistic and pessimistic risk estimations
|
| id | journaliasakpiua-article-165664 |
| institution | System research and information technologies |
| keywords_txt_mv | keywords |
| language | English |
| last_indexed | 2025-07-17T10:25:02Z |
| publishDate | 2019 |
| publisher | The National Technical University of Ukraine "Igor Sikorsky Kyiv Polytechnic Institute" |
| record_format | ojs |
| resource_txt_mv | journaliasakpiua/8b/a370225bb05a306585073c040e5d258b.pdf |
| spelling | journaliasakpiua-article-1656642019-04-26T15:35:46Z The general implementation of Delphi method procedure with application on fuzzy data Обобщенная реализация процедуры метода Дельфи и ее применение к нечетким данным Узагальнена реалізація процедури методу Дельфі та її застосування до нечітких даних Dzugaev, A. A. This paper aims to propose mathematical construction of iterative consensus reaching procedure for expert opinions, based on classic Delphi method concept. This procedure can be applied in mathematical support for qualitative analysis within various types of foresight studies that require expert examinations. The construction of procedure is supported with methodical recommendations and application example based on fuzzy data. Предлагается построение математического аппарата для итеративной процедуры экспертного оценивания, основанной на классической концепции метода Дельфи. Описанная методика может быть применена в математическом обеспечении задач качественного анализа в технологическом предвидении, требующих экспертного оценивания. Даны методические рекомендации и приведен пример применения процедуры экспертного оценивания. Запропоновано побудову математичного апарата для ітеративної процедури експертного оцінювання, заснованої на класичній концепції методу Дельфі. Описана методика може бути застосована у математичному забезпеченні задач якісного аналізу в технологічному передбаченні, що вимагають експертного оцінювання. Подано методичні вказівки та наведено приклад застосування процедури експертного оцінювання. The National Technical University of Ukraine "Igor Sikorsky Kyiv Polytechnic Institute" 2019-04-26 Article Article application/pdf https://journal.iasa.kpi.ua/article/view/165664 System research and information technologies; No. 3 (2005); 132-142 Системные исследования и информационные технологии; № 3 (2005); 132-142 Системні дослідження та інформаційні технології; № 3 (2005); 132-142 2308-8893 1681-6048 en https://journal.iasa.kpi.ua/article/view/165664/164902 Copyright (c) 2021 System research and information technologies |
| spellingShingle | Dzugaev, A. A. Узагальнена реалізація процедури методу Дельфі та її застосування до нечітких даних |
| title | Узагальнена реалізація процедури методу Дельфі та її застосування до нечітких даних |
| title_alt | The general implementation of Delphi method procedure with application on fuzzy data Обобщенная реализация процедуры метода Дельфи и ее применение к нечетким данным |
| title_full | Узагальнена реалізація процедури методу Дельфі та її застосування до нечітких даних |
| title_fullStr | Узагальнена реалізація процедури методу Дельфі та її застосування до нечітких даних |
| title_full_unstemmed | Узагальнена реалізація процедури методу Дельфі та її застосування до нечітких даних |
| title_short | Узагальнена реалізація процедури методу Дельфі та її застосування до нечітких даних |
| title_sort | узагальнена реалізація процедури методу дельфі та її застосування до нечітких даних |
| url | https://journal.iasa.kpi.ua/article/view/165664 |
| work_keys_str_mv | AT dzugaevaa thegeneralimplementationofdelphimethodprocedurewithapplicationonfuzzydata AT dzugaevaa obobŝennaârealizaciâprocedurymetodadelʹfiieeprimenenieknečetkimdannym AT dzugaevaa uzagalʹnenarealízacíâprocedurimetodudelʹfítaíízastosuvannâdonečítkihdanih AT dzugaevaa generalimplementationofdelphimethodprocedurewithapplicationonfuzzydata |