Operator pencils of the second order and linear fractional relations

The notions of a pencil of the second order and a linear fractional relation (LFR) are defined in spaces of linear bounded operators acting between Banach spaces. It is shown that these notions are closely connected with various theoretical and applied problems and have diverse applications. A numbe...

Повний опис

Збережено в:
Бібліографічні деталі
Дата:2006
Автори: Khatskevich, V., Karelin, I., Zelenko, L.
Формат: Стаття
Мова:English
Опубліковано: Інститут прикладної математики і механіки НАН України 2006
Назва видання:Український математичний вісник
Онлайн доступ:http://dspace.nbuv.gov.ua/handle/123456789/124564
Теги: Додати тег
Немає тегів, Будьте першим, хто поставить тег для цього запису!
Назва журналу:Digital Library of Periodicals of National Academy of Sciences of Ukraine
Цитувати:Operator pencils of the second order and linear fractional relations / V. Khatskevich, I. Karelin, L. Zelenko // Український математичний вісник. — 2006. — Т. 3, № 4. — С. 467-503. — Бібліогр.: 87 назв. — англ.

Репозитарії

Digital Library of Periodicals of National Academy of Sciences of Ukraine
id irk-123456789-124564
record_format dspace
spelling irk-123456789-1245642017-09-30T03:03:42Z Operator pencils of the second order and linear fractional relations Khatskevich, V. Karelin, I. Zelenko, L. The notions of a pencil of the second order and a linear fractional relation (LFR) are defined in spaces of linear bounded operators acting between Banach spaces. It is shown that these notions are closely connected with various theoretical and applied problems and have diverse applications. A number of the open problems, both for pencils and LFR, are posed in this paper. Some of the above problems are solved and applied to the study of dichotomic behavior of dynamical systems. 2006 Article Operator pencils of the second order and linear fractional relations / V. Khatskevich, I. Karelin, L. Zelenko // Український математичний вісник. — 2006. — Т. 3, № 4. — С. 467-503. — Бібліогр.: 87 назв. — англ. 1810-3200 2000 MSC. 47B50, 32H99, 93C15, 37D99. http://dspace.nbuv.gov.ua/handle/123456789/124564 en Український математичний вісник Інститут прикладної математики і механіки НАН України
institution Digital Library of Periodicals of National Academy of Sciences of Ukraine
collection DSpace DC
language English
description The notions of a pencil of the second order and a linear fractional relation (LFR) are defined in spaces of linear bounded operators acting between Banach spaces. It is shown that these notions are closely connected with various theoretical and applied problems and have diverse applications. A number of the open problems, both for pencils and LFR, are posed in this paper. Some of the above problems are solved and applied to the study of dichotomic behavior of dynamical systems.
format Article
author Khatskevich, V.
Karelin, I.
Zelenko, L.
spellingShingle Khatskevich, V.
Karelin, I.
Zelenko, L.
Operator pencils of the second order and linear fractional relations
Український математичний вісник
author_facet Khatskevich, V.
Karelin, I.
Zelenko, L.
author_sort Khatskevich, V.
title Operator pencils of the second order and linear fractional relations
title_short Operator pencils of the second order and linear fractional relations
title_full Operator pencils of the second order and linear fractional relations
title_fullStr Operator pencils of the second order and linear fractional relations
title_full_unstemmed Operator pencils of the second order and linear fractional relations
title_sort operator pencils of the second order and linear fractional relations
publisher Інститут прикладної математики і механіки НАН України
publishDate 2006
url http://dspace.nbuv.gov.ua/handle/123456789/124564
citation_txt Operator pencils of the second order and linear fractional relations / V. Khatskevich, I. Karelin, L. Zelenko // Український математичний вісник. — 2006. — Т. 3, № 4. — С. 467-503. — Бібліогр.: 87 назв. — англ.
series Український математичний вісник
work_keys_str_mv AT khatskevichv operatorpencilsofthesecondorderandlinearfractionalrelations
AT karelini operatorpencilsofthesecondorderandlinearfractionalrelations
AT zelenkol operatorpencilsofthesecondorderandlinearfractionalrelations
first_indexed 2025-07-09T01:38:42Z
last_indexed 2025-07-09T01:38:42Z
_version_ 1837131497142747136
fulltext Український математичний вiсник Том 3 (2006), № 4, 467 – 503 Operator pencils of the second order and linear fractional relations Victor Khatskevich, Irina I. Karelin, Leonid Zelenko (Presented by M. M. Malamud) Abstract. The notions of a pencil of the second order and a linear frac- tional relation (LFR) are defined in spaces of linear bounded operators acting between Banach spaces. It is shown that these notions are closely connected with various theoretical and applied problems and have di- verse applications. A number of the open problems, both for pencils and LFR, are posed in this paper. Some of the above problems are solved and applied to the study of dichotomic behavior of dynamical systems. 2000 MSC. 47B50, 32H99, 93C15, 37D99. Key words and phrases. Indefinite metric, plus-operator, invariant subspace, fixed point, linear fractional relation, operator pencil, dynam- ical system with continuous and discrete time, dichotomy of solutions to dynamical systems. 1. Basic definitions, history of the subject 1.1. Introduction It is well known that publication of unsolved problems accompanied by a discussion is a specific sort of mathematical literature which has been of great importance (see, for example, the review of the book “Arnold’s Problems”, Bulletin of the AMS, vol. 43, No 1 (2006), p. 101). Following this line, in the first part of the paper we formulate 15 problems in the topic of operator pencils of the second order and operator linear fractional relations. In its second part we show, how the the solutions of some of these problems can be applied to the study of dynamical systems with continuous and discrete time. Herewith we establish for such systems a so called dichotomy compatible with the signature. These results are natural continuation and development of the corresponding results of the recent work [46] and also of the papers [49,51–54]. Received 16.10.2006 ISSN 1810 – 3200. c© Iнститут математики НАН України 468 Operator pencils of the second order... The purpose of this paper is to study two relatively new notions, those of second order operator pencils whose domains are operator spaces (see Section 1.2) and of linear fractional relations (LFR) (see Section 1.3). These objects are closely related (see below Section 2.3). We are going to study these notions in their interaction, because even at this, rather early stage of the study of these notions, it is already clear that the study of these notions in their interaction leads to remarkable insights into the structure of each of them. Linear fractional relations are generalizations of linear fractional trans- formations (LFT), which are defined by the same formula as linear frac- tional functions: W (Z) = (AZ +B)(CZ +D)−1, but now A,B,C,D,Z, and W are operators acting on infinite-dimension- al spaces. Linear fractional transformations were studied in depth by M. G. Krein and Yu. L. Shmul’yan [83] and were used to solve important problems in the theory of spaces with an indefinite metric (spaces with an indefinite metric were introduced and the relevant problems were posed by Sobolev and Pontryagin in their study of a very concrete applied problem, see Section 1.4 for more history of this direction of research). Later LFT were used to solve problems on Abel–Schröder type func- tional equations, the Koenigs embedding problem (see [11–13]), dichoto- my of non-autonomous differential equations in a Hilbert space (see [49, 51–53] for more information). Applicability of LFT to these problems is restricted by the possible non-invertibility of the values of the operator pencil CZ +D. It is quite difficult (in the infinite-dimensional setting) to extend do- mains of LFT by means of the analytic continuation or other function- theoretic methods. In this connection we extend the domain of an LFT just rewriting the defining formula for it as a relation: AZ +B = W (CZ +D), (1.1) and regarding the set of all W satisfying (1.1) as the image of Z. (Formal definitions and the notation we use can be found in Section 1.3.) The paper is naturally divided into three parts. In the first part we give basic definitions of the operator LFR and the operator pencil of the second order, and tell about history of the topic. In the second part we pose a number of questions, concerning LFR and pencils, and we answer a part of these questions. In particular, we describe the domain and the image of LFR as the set of non- positivity of the corresponding pencil P . We use this fact to find conditions supplying V. Khatskevich, I. Karelin, L. Zelenko 469 to domFT and imFT with non-emptiness, convexity and compactness in the weak operator topology (WOT). The final part of the paper is devoted to applications of the above results to dynamical systems with continuous and discrete time. We establish dichotomous behaviour of trajectories of such systems under some natural conditions. 1.2. Operator pencils which domains are operator spaces Usually (see, e.g. [72]) an operator pencil is defined as an operator- valued function of the form A(λ) = A0 +λA1 +λ2A2 + · · ·+λnAn, where λ is a complex variable and A0, A1, . . . , An are given linear operators acting on a Hilbert space. In this paper operator pencils are understood as polynomial operator-valued functions defined on some subsets of spaces of operators. In particular, by an operator pencil of the second order we mean an operator-valued function of the form P (X,Y ) = PA,B,C,D(X,Y ) = A+XB + CY +XDY, (1.2) where X ∈ L(X ,Y), Y ∈ L(U ,V), A ∈ L(U ,Y), B ∈ L(U ,X ), C ∈ L(V,Y), D ∈ L(V,X ), X ,Y,U ,V are Banach spaces, and by L(X ,Y) we denote the space of all bounded linear operators from X to Y. In this formula A,B,C, and D are fixed, and P is considered as a function on L(X ,Y) × L(U ,V). The formula for P (X,Y ) can be written in a matrix form: P (X,Y ) = ( I X ) ( A C B D )( I Y ) . 1.3. Linear fractional relations In our discussion of LFR it will be convenient to use the following notation. Let Ei, E ′ i, i = 1, 2, be Banach spaces. The space of bounded linear operators acting from Ej to E′ i is denoted by L(Ej , E ′ i). Let T be a 2 × 2 operator matrix of the form T = ( T11 T12 T21 T22 ) , (1.3) where Tij ∈ L(Ej , E ′ i), i, j = 1, 2. For each such matrix we introduce three set-valued maps. All these maps will be called (with some abuse of terminology) linear fractional relations (LFR). 470 Operator pencils of the second order... Definition 1.1. The set-valued (multi-valued) map GT from L(E1, E2) into the set of closed affine subspaces of L(E′ 1, E ′ 2) is defined by GT (X) = {Y ∈ L(E′ 1, E ′ 2) : T21 + T22X = Y (T11 + T12X)}. (1.4) Definition 1.2. The LFR FT is the set-valued mapping of the closed unit ball K+ of L(E1, E2) into the set of subsets of the unit ball K′ + of L(E′ 1, E ′ 2) defined by FT (K) = {K ′ ∈ K′ + : T21 + T22K = K ′(T11 + T12K)}. (1.5) Definition 1.3. The LFR HT is the set-valued mapping of K− (the unit ball of L(E2, E1)) onto sets of operators in K′ − (the unit ball of L(E′ 2, E ′ 1)) defined by HT (Q) = {Q′ : T11Q+ T12 = Q′(T21Q+ T22)} (1.6) Definition 1.4. A linear fractional relation GT is said to be defined at X if GT (X) 6= ∅. The set of all X ∈ L(E1, E2) at which GT is defined will be denoted by dom(GT ). The same terminology and notation will be used for FT and HT . One of the main reason of our interest to these concepts is their re- lations with the theory of operators in spaces with an indefinite metric (see Section 2.1 for discussion of these relations). 1.4. History of the topic At the beginning the theory of linear operators in spaces with an indefinite metric concentrated around two classes of operators: symme- tries and isometries. In a space with J -metric (H is a Hilbert space, J = J ∗ = J −1, [x, y] = (J x, y), x, y ∈ H, (·, ·) is the standard inner product on H,H = H1 ⊕H2, Hi = PiH, i = 1, 2, where P1 = I+J 2 , P2 = I−J 2 ) they are called J -symmetries and J -isometries. The pioneering papers [80, 85] are devoted to description of maximal invariant semidefinite subspaces (non-positive and non-negative) of a J selfadjoint operator A and the structure of the spectra of the restrictions of A to such subspaces. Let κ = min{dimH1,dimH2}. The paper [85] is devoted to the case κ = 1, the paper [80] is devoted to the case κ < ∞. Now the spaces with finite κ are called Pontryagin spaces and denoted ∏ κ. V. Khatskevich, I. Karelin, L. Zelenko 471 Soon afterwards I. S. Iokhvidov [22] developed a theory of Cayley- Newmann linear fractional transformations and established connections between the classes of J -symmetries and J -isometries, later these re- sults were generalized to classes of J -dissipative and J -nonexpanding operators, and even for plus-operators. It is worth mentioning that the papers [80] and [85] are examples of a delicate work “by hand”. Their authors prove the main results using deep, complicated, lengthy, mostly algebraic, computations (especially [80]). In 1950 M. G. Krein [58] started to use “modern technologies” to study invariant subspaces and spectra of J -unitary operators. He used fixed point theorems originated in work of Bohl, Brouwer, Schauder, and Tikhonov. Later he developed this method in [59]. The papers [61, 62], and [63] laid the foundation of the theory of linear fractional transforma- tions. This theory also serves as a basis for the whole theory of linear operators in J -spaces. Now such spaces are called Krein spaces. Methods developed in [61, 62], and [63] motivated many other works continuing this line of research (see, e.g., [4, 5, 21,23,24,32], and [33].) As an instructive example of usefulness of methods of linear fractional transformations we discuss how effective were they in applications to the well-known Phillips problem. In [77–79] he studied systems of PDEs in Hilbert spaces and stated [79] the following conjecture: Any invariant with respect to a commutative group G of J-unitary operators dual pair of subspaces can be extended to an invariant dual pair of maximal subspaces. (Let us recall that a pair of subspaces L1, L2 of a Krein space H is called a dual pair, if L1 is nonnegative, L2 is non- positive and they are orthogonal each other with respect to the indefinite metric [x, y] = (Jx, y)). R. Phillips proved this statement under an additional restriction that G is uniformly bounded. On the other hand, respectively to the general case of a group G he wrote: “We have been able to verify this conjecture for finite dimensional H, ...however a proof for infinite dimensional H has thus far eluded us”. He obtained the result using a complicated algebraic argument in- volving spaces of maximal ideals, etc. Only after publication of the papers [61, 62], and [63], and using the methods developed in these papers, the results of Phillips and Naimark [75] were extended to the general case, see [4, 21,32], and [33]. Later the theory of linear fractional transformations was extended both for the general case of operators — matrices of these transforma- tions — acting in Banach indefinite spaces, see [42, 43, 82] and for the general case of linear fractional mappings, even unbounded, but for finite dimensional Krein spaces, see [11] and [12]. 472 Operator pencils of the second order... Simultaneously the theory found further applications. C. Cowen [11] proved that in diverse areas of Complex Analysis, for example, in the Koenigs’ problem on embedding of a discrete iteration semigroup into a continuous one-parameter semigroup, in Abel–Schroeder equations linear fractional functions form a representative class of holomorphic functions (in many cases the validity of a statement for linear fractional functions implies its validity for any holomorphic function). Later C. Cowen and B. MacCluer [13] started to extend the results of [11] to the case of many variables. Results of [11,12], and [13] inspired a series of papers in which linear fractional transformations were applied to the study of composition oper- ators on Hilbert and Banach function spaces (such as Hardy and Bergman spaces), see [2], to study of generators of non-linear one-parameter semi- groups, see [86] and [39], to general Abel–Schroeder equations [2, 40–43], to Koenigs’ embedding problem, see [40] and [41]. The study of the image of a linear fractional transformation, in par- ticular, their convexity and compactness in the weak operator topol- ogy made it possible to use linear fractional mappings to the study of the dychotomy of non-autonomous differential equations in a Hilbert space [36,49,51–53]. On the other hand, the expansion of the areas of applicability of the results motivated the study of objects that are more general than linear fractional mappings. We mean linear fractional relations between operators acting in Hilbert and Banach spaces. In terms of indefinite metrics the purpose is to study classes of operators that are more general than bi-strict plus operators, that induce holomorphic linear fractional mappings. We mean all strict plus operators, or even more generally, all operators of the form T = ( T11 T12 T21 T22 ) defining a linear fractional relation FT of the form (1.5). Study of such linear fractional relations was undertaken in [25, 33, 35, 36, 38], (see also [37,76]). Another important problem which is connected with our considera- tions is the Koenigs embedding problem. Let D be a domain in a com- plex Banach space X. We denote by Hol(D,X) the set of holomorphic mappings on D with values in X and by Hol(D) the semigroup (with respect to composition) of holomorphic self-mappings of D. The follow- ing problem is classical: given a mapping ϕ ∈ Hol(D), find a continuous family {F (t)} ⊂ Hol(D), t ≥ 0 with F (1) = ϕ, F (0) = I (where I is the identity mapping on D), and satisfying the semigroup property V. Khatskevich, I. Karelin, L. Zelenko 473 F (s+ t) = F (s) ◦ F (t), s, t ≥ 0. For more than one hundred years this problem has been considered by many mathematicians in the framework of different theories (see, for example, [10–12,18,20,30,55,69]). The answer depends on what one requires precisely and is known to be “yes” in some cases, “no” in others. As far as we know the problem is still of interest, even for the one-dimensional case. In 1884 G. Koenigs showed how the problem may be solved (locally) by using the solution of Schröder’s equation f ◦ ϕ = λf. (1.7) Namely, if z0 ∈ D is a fixed point of ϕ and |ϕ′(z0)| 6= 0, 1, then (1.7) with λ = ϕ′(z0) has a unique solution f ∈ Hol(D,C) normalized by the condition f ′(z0) = 1. This function f is called the Koenigs function of ϕ.) Hence (locally) ϕn = f−1 ◦ (λnf). This expression then serves as a definition of ϕn when n is not a natural number. Now consider the more general case of the so-called Abel–Schröder equation. Let ∆ be the open unit disk in the complex plane C. The equation f ◦ ϕ = ψ ◦ f, (1.8) where ϕ and ψ, which belong to Hol(∆,C) are given, is called [11] the Abel–Schröder equation. In the particular case where ϕ ∈ Hol(∆) fixes 0, that is ϕ(0) = 0, and ψ = ϕ′(0) = λ, (1.8) becomes Schröder’s equation (1.7) with the constraint ϕ(0) = 0, and in the case where ψ ◦ f = f + 1, (1.8) becomes Abel equation (see [11]). In 1981 C. C. Cowen [11] showed that in order to solve Schröder’s equation it is sufficient to solve it in the particular case where ϕ = ϕA is a linear fractional mapping. Using this fact he gave in [11] the general solution of the Schröder equation. These observations lead us to the following idea: To look for suffi- cient and necessary geometric and analytic conditions for ϕ ∈ Hol(D) to be embeddable in a one-parameter semigroup in terms of its Koenigs function f . It turns out that this approach can work not only for the one- dimensional case, but also for multi-dimensional and infinite-dimensional cases whenever we know how to find explicitly the solution to Schröder’s equation. The latter problem was solved for fractional-linear mappings (see [41, 43, 45]). On the other hand even in the one-dimensional case ϕ is embeddable if and only if the corresponding Schröder’s equation has so-called spiral like solution [41]. The problem of existence of such a solution is very difficult and no efficient approaches to it are known, it restricts the applicability of this approach. 474 Operator pencils of the second order... We suggest another approach. For fractional linear mappings the Koenigs problem admits the following equivalent reformulation. Problem. Let A be a plus-operator. (a) Under what conditions are all fractional powers At defined? (b) When is At a plus-operator for each t ≥ 0? An answer to question (a) is usually given in terms of existence of logA. One can get an answer to the question (b) studying the set {X : P (X) ≤ 0}, where P (X) is a self-adjoint operator pencil of the second order. 2. LFR and operator pencils 2.1. Relations between LFR and the theory of operators in spaces with an indefinite metric There is a natural correspondence between LFR FT and plus-opera- tors (see [6]). Let us recall the definition of plus-operators. We introduce on E = E1 ⊕ E2 and E′ = E′ 1 ⊕ E′ 2 indefinite structures in the follow- ing way. Let P1, P2 be the canonical projections corresponding to the decomposition E = E1 ⊕E2 and P ′ 1, P ′ 2 be the canonical projections cor- responding to the decomposition E = E′ 1 ⊕ E′ 2. We define the sets P+ and P ′ + of non-negative vectors in E and E′ by P+ = {x ∈ E : ‖P1x‖ ≥ ‖P2x‖} and P ′ + = {x ∈ E′ : ‖P ′ 1x‖ ≥ ‖P ′ 2x‖}, respectively. The sets of non-positive vectors in E and E′ are defined by P− = {x ∈ E : ‖P1x‖ ≤ ‖P2x‖} and P ′ − = {x ∈ E′ : ‖P ′ 1x‖ ≤ ‖P ′ 2x‖}. Definition 2.1. A linear continuous operator T : E → E′ is called a plus-operator if TP+ ⊂ P ′ +, and is called a minus-operator if TP− ⊂ P ′ −. Definition 2.2. Denote by K+ and K′ + the closed unit ball of L(E1, E2) and L(E′ 1, E ′ 2), respectively, and by K− and K′ − the closed unit ball of L(E2, E1) and L(E′ 2, E ′ 1), respectively. These sets are the sets of angular operators of maximal non-negative and maximal non-positive subspaces, respectively. The mentioned above correspondence between LFR and plus-opera- tors can be described in the following way: An operator T : E → E′ is a plus-operator if and only if the LFR FT (see (1.5)) is defined at each point of K+. Using projections P ∗ 1 , P ∗ 2 , (P ′ 1) ∗, and (P ′ 2) ∗ we introduce an indefinite structure on E∗ and (E′)∗. For each LFR there are two dual objects: V. Khatskevich, I. Karelin, L. Zelenko 475 1) FT ∗ , where FT ∗ is induced by the matrix T ∗ and maps contractions from L(E∗ 1 , E ∗ 2) onto sets of contractions in L((E′ 1) ∗, (E′ 2) ∗). 2) HT (see Definition 1.3) Properties of FT correspond to plus-properties of T , whereas proper- ties of HT correspond to its minus-properties. Observe that each of these relations can be empty, in the sense that their domains are empty. We shall use the following terminology. If an LFR FT is univalent (that is, the set FT (K) contains exactly one element K ′ for each K ∈ domFT ), we call it a quasi linear fractional mapping (QLFM). If T11 + T12K is invertible for each K ∈ domFT , we call it a linear fractional mapping (LFM). If an LFR FT is defined everywhere in K+ (that is, T is a plus- operator), we call FT a plus-LFR. If, in addition, E = E′, and hence K+ = K′ +, and FT is a plus-QLFM, we shall call it also a quasi linear fractional transformation (QLFT) of the ball K+. If, in addition, T11 + T12K is invertible for all K ∈ K+, we call FT a linear fractional transformation (LFT) of the ball K+. It is natural to consider two groups of problems: 1) global properties of pencils and LFR in the space L(E1, E2); 2) local properties of pencils and LFR acting between the unit balls K+ and K′ +. The first group is connected with the problems of the optimal control theory, in particular, with operator Riccati equation (see, e.g., [1, 56, 57, 64]), with the problems of extension of J-nonnegative definite operators up to J-selfadjoint operators (see, e.g. [70, 71]) and with the generalized Liouville theorem as well. The second group of problems is closely connected with various prob- lems of the operator theory in a space with indefinite metric, with Köenig’s embedding problem, as well as the problem of extension of the invari- ant pairs of dual subspaces up to the maximal subspaces. This group of problems is also connected with applications to dynamical systems ( [11–13,40–43,45–47,49,51–53]). We consider in this paper pencils and relations between the unit balls. Since LFR is the divisibility relation between two linear pencils it is natural to discuss linear pencils. We are also interested in quadratic pencils and selfadjoint quadratic pencils. 476 Operator pencils of the second order... 2.2. Linear pencils and general properties of LFR 2.2.1. Basic results Let FT be a an LFR corresponding to the operator T = (Tij) 2 i,j=1 : E → E′. The following is true (see [38,44,82]). 1. If coker(T11 + T12K) = {0}, K ∈ dom(FT ), then FT is a QLFM. 2. If FT is a plus-QLFM, then FT ∗ is a plus-LFM. 3. If E,E′ are reflexive, FT is a plus-QLFM and ker(T11 + T12K) = {0}, then FT ∗ is a plus-QLFM. 4. If E,E′ are reflexive, FT is a plus-LFR, then FT ∗ is a plus-LFM. So we see that properties of LFR are closely related to the behavior of linear pencils obtained from (1.2) when D = 0 = B. 2.2.2. General problems Taking in account statements 1–4 on the first stage of study of linear operator pencils it is natural to ask the following questions for pencils P (K) = A+ CK, where A and C are fixed, K ∈ K+. Problem 2.1. A criterion for kerP (K) = {0}. Problem 2.2. A criterion for cokerP (K) = {0}. Problem 2.3. A criterion for invertibility of P (K). Since the invertibility of A is a necessary condition for invertibility of P (K) on K+, Problem 2.3 is equivalent to the following problem. Problem 2.3′. A criterion for invertibility of P̃ (K) = I + C ′K. Criteria for 2.1 and 2.2 in terms of inequalities ‖Cx‖ ≤ ‖Ax‖ and ‖C∗y‖ ≤ ‖A∗y‖ and in terms of embeddings of images of unit balls were found in [50] (see Theorems 5, 14, 16, 17, 19). In particular, it would be interesting to find criteria for divisibility of operators in Banach spaces generalizing the Hilbert space case criteria found by Yu. Shmul’yan [83]. The well-known criterion of divisibility of operators in a Hilbert space can be stated in the following way: A = KB if and only if A∗A ≤ kB∗B, where k is a positive real number satisfying ‖K‖ ≤ k. Corollary 9 in [50] is a generalization of this result to the Banach space case. In further study of LFR, it is natural to study the following questions. Problem 2.4. Criteria for dim cokerP (K) = const. Problem 2.5. Criteria for stability of indP (K) on K+. V. Khatskevich, I. Karelin, L. Zelenko 477 2.3. Quadratic Pencils and fixed points of LFR 2.3.1. Definitions In this section we consider the quadratic pencil P (X) = A+BX +XC +XDX (2.1) which is obtained from (1.2) when Y = X. Let FT be an LFR and let K+ be its fixed points, that is, K+ ∈ FT (K+). This condition can be written as T21 + T22K+ = K+(T11 + T12K+). (2.2) Opening the brackets in (2.2) and moving all terms to the left-hand side, we get the equation P (K+) = 0, (2.3) where P (K+) is a pencil of the form (2.1) with A = T21, B = −T11, C = T22, D = −T12. Hence (2.3) is an equation for fixed points of LFR. Analogous equation takes place for a fixed point K− of an LFR HT . These fixed points are, respectively, angular operators of invariant maxi- mal non-negative L+ and maximal non-positive L− subspaces of an op- erator T , that is, L+ = (P1 +K+)E1, L− = (P2 +K−)E2, TL+ ⊂ L+, TL− ⊂ L−. The matrix quadratic equation A+BX +XC +XDX = 0 (2.4) and for the Hilbert space case its symmetric version (known as algebraic Riccati equation) A+BX +XB∗ +XDX = 0 (2.5) where D ≥ 0 and A = A∗ also arise in another kind of problems dealing with connections between the set of solutions of the equations (2.4) or (2.5) and factorizations of certain matrix functions of the Popov type associated with (2.4) or (2.5). (See, e.g., [27–29, 68]). The connection between the problem of determining solutions of the matrix quadratic equation and the problem of describing the factorizations of a matrix function is provided by generalized Bezoutians. The notion of generalized Bezoutians based on representations of the functions in realized form was introduced and developed in some papers (see, e.g., [17, 19, 27, 67]), as a generalization of the coefficient Bezoutian for quadruple of matrix polynomials introduced in [3]. 478 Operator pencils of the second order... One of the results in this regard ( [27]) establishes the one-to-one correspondence between the set of solutions of (2.4) and the set of fac- torizations of the rational matrix function of the Popov type associated with (2.4) GC,K(λ) = I − [ K Φ ] ( λI − [ −C − ΨK 0 A+NK B −NΦ ])−1 [ Ψ N ] . (2.6) where K and N are arbitrary feedback matrices of appropriate sizes, and D = ΨΦ is a rank decomposition of the matrix D. Such correspondence is provided by the generalized Bezoutian introduced and studied in [27]. It is worth to mention the polynomial approach, where the role of the rational matrix function (2.6) plays a matrix polynomial of the certain type (see, e.g., [28, 68]). One of the advantages of such approach is that it provides more insight into the structure of hermitian solutions of the homogeneous algebraic Riccati equation BX +XB∗ +XDX = 0. (2.7) 2.3.2. Problems The problem of finding maximal invariant semi-definite subspaces of a linear operator is one of the central problems in the theory of operators in spaces with indefinite metric. In this section we pose the following problems: Problem 2.6. Find criteria for existence of zeros of a quadratic pencil. Problem 2.7. Creation of methods of finding zeros of quadratic pencils. Problem 2.8. The description of a geometric and topological structure of zero sets of quadratic pencils. Problem 2.9. To study the structure of hermitian solutions of the non- homogeneous algebraic Riccati equation (2.5). There is an extensive literature devoted to Problems 2.6 and 2.8 stated in terms of fixed points of LFT and, sometimes, LFR. See [6] and refer- ences therein. Additional information on Problem 2.8 can be found in [8] and [48]. The mentioned papers employed methods of two types: meth- ods of the theory of spaces with an indefinite metric and holomorphic analysis. In particular, it was shown in [48] that the set of fixed points of an LFT is a smooth manifold (let us recall that an LFT is holomorphic on K+). The corresponding purposes are: V. Khatskevich, I. Karelin, L. Zelenko 479 (i) To find new methods for proving the existence of fixed points of LFT, based on the study of zeros of operator pencils. (ii) To develop new methods for description of zeros of an operator pencil. (iii) To describe the geometric and topological structure of the set of fixed points of general LFR in terms of zeros of the pencil P (K+). 2.4. Selfadjoint quadratic pencils and geometrical and topological structure of the image and the domain of LFR, the Hilbert space case 2.4.1. Description of the image and the domain of LFR in terms of pencils There is a simple, popular and useful theory of linear operator equa- tions of the form AX = B or XC = B which describes the properties of the sets of solutions X ∈ L(H1,H2) and, in particular, the conditions for solvability (existence of suchX). On the other hand, many problems from different areas (extensions of operators [71], indefinite metric spaces [63], linear fractional relations [49], operator functional equations [11], control theory and systems theory [14, 26, 64–66], and others) lead to quadratic equations and inequalities of the forms: X∗AX = C, (2.8) X∗AX ≤ C, (2.9) and, more generally, X∗AX +B∗X +X∗B + C = 0, (2.10) X∗AX +B∗X +X∗B + C ≤ 0. (2.11) The equation (2.10) is very close to the equation XAX +B∗X +X∗B+ C = 0 which self-adjoint A and C which is usually called a continu- ous algebraic Riccati equation. It is clear that a self-adjoint solution of (2.10) is also a solution of the corresponding continuous algebraic Riccati equation. Since self-adjoint solutions are the most important in systems theory, our study can be of interest for infinite-dimensional systems the- ory. In control theory and systems theory there are many results stating that under certain assumptions the continuous algebraic Riccati equa- tion has a non-negative solution (see, e.g. [66, Theorem 2.2.1], [14, Theo- rem 6.2.7]). Many other results on continuous algebraic Riccati equations 480 Operator pencils of the second order... can be found in [14,26], and [65]. Another object studied in systems the- ory is an inequality of the form XAX + B∗X +X∗B + C ≤ 0 (A ≥ 0), called a Riccati inequality, see [15], [65, Section 9.1], [81]. The mentioned sources contain results on conditions of solvability of Riccati inequality and comparison of the sets of self-adjoint solutions of the continuous al- gebraic Riccati equation and the corresponding Riccati inequality. Scalar versions of the inequality (2.11) for an infinite dimensional Hilbert space were studied in [87]. This survey of the literature shows that informa- tion about properties of sets M(A,B,C) can be of interest from different points of view. Our interest in this topic was motivated by the fact that such sets appear in the study of linear fractional relations (LFR), and we consider the present paper as a continuation of [38], in which we started systematic study of properties of LFR. Namely, in terms of M(A,B,C) one can express the domains of LFR and the images of balls under LFR. Relatively few results are known on topological and geometric proper- ties of sets of solutions of the inequalities (2.9) and (2.11). We will denote by E(A,C), N(A,C) and M(A,B,C) the sets of all solutions for (2.8), (2.9) and (2.11), respectively. It is known, that M(A,B,C) = L(H1,H2) if and only if A,C are non-positive and B = (−A)1/2T (−C)1/2 for some operator T ∈ L(H1,H2) with ‖T‖ ≤ 1. If the operators A,C are in- vertible, this condition is equivalent to C − B∗A−1B ≤ 0. It is the so-called generalized Sylvester criterion (see [9], [60, pp. 732–733], and [84]). If A is invertible and positive, then the sets E(A,C) and N(A,C) are non-void only for C ≥ 0 and in this case E(A,C) = A−1/2UC1/2, N(A,C) = A−1/2RC1/2 where U is the set of all isometric operators from CH1 into H2, and R is the unit ball of L(CH1,H2). Hence N(A,C) is convex and weak operator topology-compact. If A is non-negative but not invertible, then N(A,C) is still convex (see, for example, [49]), and hence weak operator topology-closed. The analysis of M(A,B,C), and even of N(A,C) for an arbitrary self- adjoint operator A, are much more complicated. Our purpose is to find conditions under which the sets N(A,C) and M(A,B,C) are non-empty, closed, or convex, and to characterize their interior points. In this section we consider pencils of the form P (X) = A+B∗X∗ +XB +XDX∗, (2.12) where E = H, E′ = H′ are Hilbert spaces, A = A∗, D = D∗. This pencil is obtained from (1.2) when Y = X∗ and C = B∗. Let FT be an LFR, that is, FT (K+) = {K ′ + : T21 + T22K+ = K ′ +(T11 + T12K+)}, (2.13) K+ ∈ domFT , K ′ + ∈ imFT . V. Khatskevich, I. Karelin, L. Zelenko 481 Let us describe domFT and imFT in terms of the pencil (2.12). The equality from (2.13), combined with the Shmul’yan divisibility criterion [83], implies (T21 + T22K+)∗(T21 + T22K+) ≤ (T11 + T12K+)∗(T11 + T12K+). (2.14) Opening brackets in (2.14) and moving everything to the left-hand side we get P (K+) ≤ 0, (2.15) where P is a pencil of the form (2.12) with X = K∗ +, D = T ∗ 22T22 − T ∗ 12T12, B ∗ = T ∗ 21T22 − T11T12, A = T ∗ 21T21 − T ∗ 11T11. Hence domFT = Md := {K+ ∈ K+ : P (K+) ≤ 0}, where P is a pencil of the form (2.12) with coefficients described after (2.15) and X = K∗ +. Let us rewrite the equality from (2.13) in the form T21 −K ′ +T11 = (K+T12 − T22)K, from where, as before, we get (T21 −K ′ +T11)(T21 −K ′ +T11) ∗ ≤ (K+T12 − T22)(K+T12 − T22) ∗. This implies imFT = MIm := {K ′ + ∈ K+ : P (K ′ +) ≤ 0}, (2.16) where P is a pencil of the form (2.12) with X = K ′ + and A = T21T ∗ 21 − T22T ∗ 22, B = −T21T ∗ 11 + T22T ∗ 12, D = T11T ∗ 11 − T12T ∗ 12. Let us find the connections between the relations FT ∗ and HT which are dual to FT . Theorem 2.1. −domHT = (imFT ∗)∗, − imHT = (domFT ∗)∗. In the case of invertible T we have Theorem 2.2. Let T be an invertible operator. If FT is a plus-QLFM, then FT is a plus-LFM, HT−1 is a minus-LFM, FT−1 and HT are LFM and the following equalities hold; FT−1 = (FT )−1, HT−1 = (HT )−1. 482 Operator pencils of the second order... Corollary 2.1. Let FT be a plus-LFR and let T be invertible. If at least one of the relations FT and HT−1 is univalent (that is, is a QLFM), then FT and FT ∗ are plus-LFM, HT−1 is a minus-LFM, FT−1 and HT are LFM, and the following conditions are satisfied: imHT−1 = −(imFT ∗)∗, imFT = −(imH(T−1)∗) ∗. Proof of Theorem 2.1. By means of criterion of divisibility of Shmul’yan we obtain from (1.6) that for any Q ∈ HT the following inequality takes place: (T11Q+ T12) ∗(T11Q+ T12) ≤ (T12Q+ T22) ∗(T21Q+ T22). (2.17) Setting all the terms of (2.17) in the left-hand side we have Q∗(T ∗ 11T11 − T ∗ 21T21)Q+Q∗(T ∗ 11T12 − T ∗ 21T22) + (T ∗ 12T11 − T ∗ 22T21)Q+ T ∗ 12T12 − T ∗ 22T22 ≤ 0. (2.18) Using (2.16) rewrite imFT ∗ in the form imFT ∗ = {K ′ + ∈ K+ : (K ′ +)(T ∗ 11T11 − T ∗ 21T21)((K ′ +)∗ +K ′ +)(T ∗ 21T22 − T ∗ 11T12) + (T ∗ 22T21 − T ∗ 12T11)(K ′ +)∗ + T ∗ 12T12 − T ∗ 22T22 ≤ 0}. (2.19) We see that (2.18) and (2.19) coincide for Q = −(K ′ +)∗, and the first equality of Theorem 2.1 is proved. The other equalities can be proved in just the same way. In this section we pose the following problems. Problem 2.10. Find criteria for semi-definitness of the pencil P on the whole space. Problem 2.11. Find criteria for semi-definitness of the pencil P on the ball K+. Problem 2.12. Find conditions under which the set M := {X : P (X) ≤ 0} is non-empty. Problem 2.13. Find condition under which the intersection of M and K+ is non-empty. Problem 2.14. Find conditions under which M is convex. Problem 2.15. Find conditions under which M is closed in the weak operator topology. V. Khatskevich, I. Karelin, L. Zelenko 483 2.4.2. Non-emptiness, convexity and compactness of the image and the domain of LFR Now let us study conditions for compactness in the weak operator topology and for convexity of the sets imFT and domHT . From now on, we set D = T11T ∗ 11 − T12T ∗ 12. Also we will use two notions of conjugation for a given linear bounded operator T : T ∗ with respect to usual scalar product (·, ·) and T c with respect to indefinite metric [·, ·]. Denote by γ∞ the set of all compact operators. Theorem 2.3. Suppose that T ∈ L(H) and D = R + C, where R ≥ 0 and C ∈ γ∞. Then imFT is compact in the weak operator topology on the space L(H1,H2). Proof. Let X ∈ K+. We use T21 + T22K+ = K ′ +(T11 + T12K+) (2.20) to rewrite the condition “X ∈ imFT ” as follows: “there exists an op- erator K+ ∈ K+ such that (XT12 − T22)K+ = T21 −XT11.” The latter means (e.g., see [83]) that (XT12−T22)(XT12−T22) ∗ ≥ (T21−XT11)(T21− XT11) ∗. Thus, it suffices to prove the closedness in the weak operator topology of the set {X} of all operators X ∈ K+ satisfying the inequality X(R+ C)X∗ +B∗X∗ +XB +A ≤ 0, (2.21) that is P (X) ≤ 0, where P (X) is a pencil (2.12) with D = R+C, where R, C, B, A ∈ L(H), R ≥ 0, C ∈ γ∞, and A = A∗. We rewrite (2.21) as (XRX∗ x, x) ≤ ((−XCX∗ +B∗X∗ +XB +A)x, x), (2.22) where x ∈ H. Since the operator C is compact, the function on the right-hand side of (2.22) is continuous in the weak operator topology. We transform the left-hand side as follows: (XRX∗x, x) = ‖R′X∗x‖2, where R′ is the nonnegative root of R. Let Xn weakly −→ X0 (here {Xn} is a sequence or a net). Then, by the well-known property of the weak limit, we have ‖R′X∗ 0x‖ ≤ limn ‖R ′X∗ nx‖. Passing in (2.22) to the lower limit (which coincides with the limit on the right-hand side of this inequality), we hence see that inequality (2.22) also holds for the operator X0. Therefore, the set {X} is closed in the weak operator topology, and the proof of the theorem is complete. It is easy to show that, for an arbitrary self-adjoint operator D, the condition “D = R + C, where R ≥ 0 and C ∈ γ∞”, is equivalent to 484 Operator pencils of the second order... the following condition: “D∈γ∞, where D = D+ + D− is the spectral resolution of the operator D (here D− = ∫ 0 −∞ t dEt and D+ = ∫∞ 0 t dEt, where Et is the partition of unity corresponding to D)”. Corollary 2.2. Suppose that T ∈ L(H) and T12 ∈ γ∞. Then imFT is compact in the weak operator topology. Corollary 2.3. Suppose that T ∈ L(H) and T21 ∈ γ∞. Then imF ∗ T is compact in the weak operator topology. Proof. It follows from the relation T c = JT ∗J that (T c)c 12 = −T ∗ 21 ∈ γ∞, and hence imF ∗ T is compact in the weak operator topology. Thus, by Theorem 2.1, we readily see that the set domHT is compact in the weak operator topology. The following assertion is a generalization of Theorems 1.2 ( [49]) and 2.3 ([36]). Theorem 2.4. Suppose that T ∈ L(H) and D ≥ 0. Then the set domHT is compact in the weak operator topology on the space L(H2,H1) and convex. Proof. The compactness in the weak operator topology of domHT follows from Theorems 2.1 and 2.3 of the present paper, and the convexity of this set in the case C = 0 follows from Lemma 1.1 [49]. A closely related condition ensures that the set domHT is also non- empty. Namely, we have the following assertion. Theorem 2.5. Let T ∈ L(H). The set domHT is nonempty precisely if there exists an operator K ∈ K1 such that D +K∗S + S∗K +K∗EK ≥ 0, (2.23) where S = T22T ∗ 12 − T21T ∗ 11, E = T21T ∗ 21 − T22T ∗ 22. Proof. It follows from Theorem 2.1 that the nonemptiness of domHT is equivalent to the nonemptiness of imFT c , i.e., to the existence of an operator K ∈ K+ such that T c(P1 + K)H1 ⊂ P+. We rewrite the last condition as ( ((T c)11 + (T c)12K)x, ((T c)11 + (T c)12K)x ) ≥ ( ((T c)21 + (T c)22K)x, ((T c)21 + (T c)22K)x ) for all x ∈ H. (T c = JT ∗J, J = ( crI 0 0 −I ) .) V. Khatskevich, I. Karelin, L. Zelenko 485 Transforming this inequality and taking into account the fact that (T c)11 = T ∗ 11, (T c)12 = −T ∗ 21, (T c)21 = −T ∗ 12, and (T c)22 = T ∗ 22, we obtain inequality (2.23). Thus, the inequalityD ≥ 0 means that inequality (2.23) holds already for K = 0. For the case in which D ≥ 0 and A is a plus-operator, the nonempti- ness of the set domHT was proved in [36] (see Theorem 2.3). Now let A be a plus-operator. In this case, the set domHT is compact in the weak operator topology and convex ([36, Theorem 2.1]). Conditions for compactness in the weak operator topology and convexity of the set imFT were studied in several papers (e.g., see [36,49,53]). The following assertion is a generalization of Theorems 1.2 ([49]) and 2.3 ([36]), which is somewhat different than Theorem 2.4. Theorem 2.6. Suppose that T is a plus-operator, the operator D = T11T ∗ 11 − T12T ∗ 12 is semidefinite (i.e., D ≥ 0 or D ≤ 0). Then the set imFT is compact in the weak operator topology and convex. In this case, if D ≤ 0, then the operator T annihilates a subspace (P+ +Q)H+, where Q ∈ K+, and imFT = FT (Q) = K+. Moreover, the block-matrix of the operator A with respect to the canonical decomposition of the space H has the form ( −T12Q T12 −ST12Q ST12 ) , where S ∈ K+. Proof. The case D ≥ 0 was studied in Theorem 2.4. Let D ≤ 0; this means that T cH1 ⊂ P−, that is T cH1 is the set of nonpositive vectors [6]. There exists a maximal nonnegative subspace L+ which is J-orthogonal to T cH1 [6]; we have [TL+, H1] = [L+, T cH1] = 0. This means that TL+ ⊆ H2, and hence T is a plus-operator, TL+ = {0}. We have obtained imFT = FT (Q) = K+, where Q is an angular operator of the subspace L+. Now we describe the block-matrix of a plus-operator T such that D ≤ 0. Since T11 + T12Q = 0, we have T11 = −T12Q. Next, a plus- operator annihilating a nonzero nonnegative vector is not strict [82]. Thus, we have T21 = ST11 and T22 = ST12, where S ∈ K+ [6], which completes the proof of the theorem. We note that the inequality D ≥ 0 holds for any operator T ∈ L(H) such that T ∗ is a plus-operator. This remark, Theorem 2.1 ([36]) and Theorem 2.6 imply the following assertion. 486 Operator pencils of the second order... Proposition 2.1. Let T be a bilateral plus-operator, that is both T and T ∗ are plus-operators. Then the sets imFT and domHT are nonempty, compact in the weak operator topology and convex. This proposition is a generalization of Corollary 2.4 ([36]). In conclusion of Section 2.4 we would like to mention that the Riccati equations P (X) = 0 are used in the control theory (see, for example [64]). On the other hand, solutions of Problems 2.13–2.15 will provide a criterion for non-emptiness, convexity, and compactness in the weak operator topology of the image and the domain of LFR (see (2.15) and (2.16)). In a special case of LFT FT convexity and compactness of imFT were established in [49]. The same paper contains the result showing that this property characterize Hilbert spaces in the following sense: if E is a Banach space and in L(E) the image of each LFT is convex, then E is isometric to a Hilbert space. The paper [35] contains an example of a unitary, with respect to the usual inner product on H, plus-operator T for which HT (hence, by Theorem 2.1, also FT ∗) is empty. Examples of plus-LFR with non-convex and non-compact images can be found in [44]. 3. Applications to dynamical systems In this section we consider the expected significance of results on Problems 2.13–2.15 for the theory of dynamical systems. 3.1. The case of continuos time Consider a uniformly well-posed problem x′ = A(t)x x(t0) = x0 (3.1) in a Hilbert space H, where the operators A(t) are self-adjoint with the common dense domain D. Uniform well-posedness implies the existence of the evolution operator U(t) : x(t) = U(t)x0, where x(t) is the solution of (3.1) (see [36, 49, 51–53]). Let operators A(t) be indefinite (that is, with spectra having non-empty intersections both with the positive and the negative semi-axis). We introduce a family of indefinite metrics [·, ·]t by: [x, y]t = (J(t)x, y), where (·, ·) is the usual inner product on H, J(t) = P1(t)−P2(t), P1(t) = ∫∞ 0+ dEλ(t), and P2(t) = ∫ 0 −∞ dEλ(t).1 Assume that dimP1(t) and 1Eλ(t) is the spectral function of A(t) V. Khatskevich, I. Karelin, L. Zelenko 487 dimP2(t) are stabilizing for large t. Under rather general conditions the operator U(t) : (H, [·, ·]0) → (H, [·, ·]t) is a plus-operator (as an example consider a strong sufficient condition of positivity of the derivative of the solution x(t) along the trajectory, that is, d dt([x(t), x(t)]t) ≥ 0). We can write the condition “U(t) is a plus-operator” as an equivalent condition (2.15). Let us study the dichotomy of the solutions: consider the set of all y ∈ H such that [U(t)y, U(t)y]t ≤ 0. (3.2) Since U(t) is a plus-operator, the vector y is non-positive: ‖P2(0)y‖ ≥ ‖P1(0)y‖. Defining the operator K− by the equality K−P2(0)y = P1(0)y, and extending it to the whole space H2 as a contraction, we getK− ∈ K−. By (3.2) suchK− are in domHT . By Theorem 2.1 domHT = −(imFT ∗)∗. Consider the intersection of all such sets over all t ≥ 0. Let us show that if all of these sets are non-empty, then their intersection is also non-empty. Since U(t) is a plus-operator, the leading coefficient of the corresponding pencil is non-negative, therefore all these sets are compact and convex. It implies the non-emptiness of their intersection. Let us denote the intersection by Z. By (3.2) all trajectories x(t) starting in this set remain non-positive for all t (with respect to [·, ·]t), all x(t), starting at non-negative vectors x0 remain non-negative for all t since U(t) is a plus- operator. Finally, all trajectories x(t) starting at non-positive vectors which are not in Z, at some moment t (see (3.2)) become non-negative (change their sign). This is a generalization of the classical method of Weyl circles. Now let us consider a particular case of dynamical problems. In [36], problem (1) is studied under the following conditions. The operators A(t) are self-adjoint in the Hilbert space H with the inner product (·, ·) and have the same domain of definition D ⊆ H for all t ∈ R + = [0,∞). The Cauchy problem (3.1) is assumed to be uniformly well-posed, i.e., there exists a bounded linear operator (an evolution operator) U(t) such that the relation x(t) = U(t)x0 holds for any solution x(t) of problem (ref(1)) with x(0) = x0 ∈ D. If y0 /∈ D, then y(t) = U(t)y0 is called a generalized solution. Let L2,ω(R+,H) be the set of functions x: R + → H square-integrable in the sense of Bochner with respect to a positive locally integrable weight w = w(t). By N we denote the set of all generalized solutions belonging to L2,ω(R+,H). We set N0 = {h ∈ H : h = y(0), y ∈ N}. Let [x, y]t be an indefinite (depending on t) metric in H determined as follows: [x, y]t = (J(t)x, y), x, y ∈ H, 488 Operator pencils of the second order... where J(t) = P1(t) − P2(t), P1(t) = ∫ +∞ 0 dEλ(t), P2(t) = ∫ 0 −∞ dEλ(t), and Eλ(t) is the spectral function of the operator A(t). For each t ∈ R +, we denote C− t = {y0 ∈ H : [U(t)y0, U(t)y0] ≤ 0}. We shall say that the bicone C− t has rank d, d ≤ ∞, if it contains a subspace L ⊂ H such that dimL = d and does not contain subspaces of larger dimensions. We assume that J(t) is strongly differentiable and consider the deriva- tive of the solution x(t) along the trajectory: [x(t), x(t)]′t = 2Re [A(t)x(t), x(t)]t + (J ′(t)x(t), x(t)). We assume that this derivative is qualified to be positive, i.e., inf ‖z‖=1 { Re [A(t)z, z]t + 1 2 (J ′(t)z, z) } ≥ ω(t) > 0 for t ∈ R +. Under these conditions, the dichotomy of solutions to the Cauchy problem 3.1) was established in [36]. The proof was essentially based on the nonemptiness and compactness in the weak operator topology of the sets domHU(t); but, in the proof of the nonemptiness of these sets, the key point was the condition U11(t)U ∗ 11(t) ≥ U12(t)U ∗ 12(t) (condition (3.3) in Theorem 3.1 [36]) which is equivalent to D = D(t) ≥ 0 in the corresponding pencil (2.12). Recall that D(t) = U11(t)U ∗ 11(t) − U12(t)U ∗ 12(t). Following the argument used in the proof of Theorem 3.1 [36] and replacing its condition (3.3) by a more general inequality (2.23), we obtain the following result. Theorem 3.1. Suppose that the Cauchy problem (3.1) is uniformly well- posed; J(t) is strongly differentiable; the limit lim t→∞ dim (P2(t)H) = d− exists; the derivative of the solution along a trajectory is qualified to be positive; for any t ∈ R + there exists an operator K = K(t) ∈ K1, such that the operator U(t) satisfies inequality (2.23), where S=U22(t)U ∗ 12(t)− U21(t)U ∗ 11(t) and E = U21(t)U ∗ 21(t) − U22(t)U ∗ 22(t). Then the generalized solution y(t) = U(t)y0, y0 ∈ H, has the following properties: (1) N0 ⊃ C− ∞ = ⋂ t∈R+ C− t , where C− ∞ is a bicone of rank d−; V. Khatskevich, I. Karelin, L. Zelenko 489 (2) the inequality ∞ ∫ t ω(s)‖y(s)‖2 ds ≤ I(y) exp ( −2 t ∫ 0 ω(s) ds ) , (3.3) where I(y) = ∞ ∫ 0 ω(s)‖y(s)‖2 ds, holds for any y(t) ∈ N; (3) for any y0 /∈ C− ∞ there exists C(y0) > 0, such that the inequality ‖y(t)‖ ≥ C(y0) exp ( 2 t ∫ 0 ω(s) ds ) , t ∈ R + (3.4) holds. As in [36], we establish the following assertion. Corollary 3.1. If in the conditions of Theorem 3.1 ∞ ∫ 0 ω(t) dt = ∞, (3.5) then N0 = C− ∞ is a closed subspace of H and dimN0 = d−. In [53] we have proved the following theorem on a non-autonomous perturbation of an autonomous dynamical system: Theorem 3.2. Assume that the following conditions hold: in (3.1) the operator A(t) has the form A0 +B(t), where A0 is a self-adjoint operator defined on D0 and bounded from the right; for any t ∈ R+ B(t) ∈ L(H) and the operator functions B(t), B⋆(t) are strongly continuously differ- entiable on D0. Furthermore, assume that the point 0 belongs to a gap of the spectrum σ(A0) and the following estimate holds: ∀ t ∈ R+ : ‖B(t)‖ ≤ d− ω(t), (3.6) where d = dist(0, σ(A0)) and ω(t) is a positive locally integrable function satisfying the condition (3.5). Let P+, P− be the orthogonal projections onto the invariant subspaces of the operator A0, corresponding respec- tively to the positive and to the negative parts of σ(A0). Then the Cauchy problem (3.1), is uniformly well-posed and the following assertions are valid: 490 Operator pencils of the second order... (1) the set C− ∞ coincides with a maximal non-positive subspace L− ∞ of the Krein space (H, V0), where V0 = P+ − P−; (2) for any y0 ∈ L− ∞ the generalized solution y(t) = U(t, 0)y0 to (3.1) satisfies the condition: I(y) = ∞ ∫ 0 ω(s)‖y(s)‖2ds <∞ and estimate (3.3) holds; (3) for any y0 /∈ L− ∞ there exists C(y0) > 0, such that the generalized solution y(t) = U(t, 0)y0 to (3.1) satisfies the inequality (3.4). Let us describe an application of the latter theorem to the following diffusion equation (see [53]) ∂u ∂t = ∂2u ∂x2 −Q(x, t)u (3.7) on the axis −∞ < x < +∞ and t ≥ 0. Denote H = L2(R). We will find a solution to (3.7) satisfying the initial condition: u(x, t)|t=0 = φ(x), (3.8) where φ ∈ H, and the additional condition: u(·, t) ∈ H ∀ t > 0, (3.9) which plays the role of boundary conditions at −∞ and +∞. We can rewrite the boundary value problem (3.7), (3.8), (3.9) in the form of an abstract Cauchy problem: dy dt = A(t)y, (3.10) y(0) = φ, (3.11) where y(t) is the function u(x, t), which is considered as a vector function with the range in the space H : y(t) = u(·, t). For each t ≥ 0 A(t) is an operator defined in H by the differential operation a(t) = d2 dx2 −Q(x, t) · (t ≥ 0). This means that A(t) is the closure of the operator, defined by a(t) on the linear set C∞ 0 (R) of smooth functions with compact supports. Assume that the function Q(x, t) satisfies the following condition: (I) Q(x, t) = p(x) + q(x, t), where p(x) is a real-valued continuous T -periodic function with T > 0, q(x, t) is a real-valued, continuous and V. Khatskevich, I. Karelin, L. Zelenko 491 bounded function on the half-plane Π+ = R × R+, and there exists the partial derivative ∂ ∂tq(x, t) on Π+, continuously depending on t uniformly with respect to x ∈ R. Represent A(t) in the form A(t) = A0+B(t), where A0 is the operator, defined in H by the operation a0 = d2 dx2 − p(x)·, and for each t ≥ 0 B(t) is the operator of multiplication by the function q(x, t) in the space H : B(t)v(x) = q(x, t)v(x). It is easy to check that the operator A0 is self- adjoint and bounded from the right, for any t ∈ R+ B(t) ∈ L(H) and B(t) = B⋆(t), and the operator function B(t) is strongly continuously differentiable on D0. Therefore by Theorem 3.2 the Cauchy problem (3.10), (3.11) is uniformly well-posed on R+. We will call any generalized solution to this problem a generalized solution to the boundary value problem (3.7), (3.8), (3.9). Observe that since p(x) is a periodic function, the spectrum σ(A0) consists of the closures of the zones of stability to the equation d2v dx2 − p(x)v = λv. Assume that the following condition is fulfilled: (J) The point 0 belongs to a gap of the spectrum σ(A0). Denote by P+, P− the orthogonal projections on the invariant sub- spaces of A0, corresponding to the positive and to the negative parts of σ(A0) respectively. Set V0 = P+ − P− and let P− 0 be the non-positive bicone of the Krein space (H, V0). The following theorem is an immediate consequence of Theorem 3.2. Theorem 3.3. Assume that conditions (I), (J) are fulfilled and the func- tion q(x, t) satisfies the following inequality: |q(x, t)| ≤ d− ω(t) ∀ t ≥ 0, where d = dist (0, σ(A0)), ω(t) is a positive locally integrable function satisfying the condition (3.5). Then there exists a maximal non-negative subspace L− ∞ of the Krein space (H, V0) such that the following statements are true: (1) for any initial state φ ∈ L− ∞ the corresponding generalized solu- tion u(x, t) to the problem (3.7), (3.8), (3.9) satisfies the following condition: I(u) = ∞ ∫ 0 ω(τ)dτ ∞ ∫ −∞ |u(x, τ)|2 dx <∞, 492 Operator pencils of the second order... and the following estimate holds: ∞ ∫ t ω(τ)dτ ∞ ∫ −∞ |u(x, τ)|2dx ≤ I(u)exp ( −2 t ∫ 0 ω(s) ds ) ; (2) for any initial state φ /∈ L− ∞ there exists C(φ) > 0, such that the corresponding generalized solution u(x, t) to the problem (3.7), (3.8), (3.9) satisfies the inequality: ∞ ∫ −∞ |u(x, t)|2dx ≥ C(φ)exp ( 2 t ∫ 0 ω(s) ds ) . 3.2. The case of discrete time Now consider the case of discrete time. In the paper [53] we considered a linear fractional transformation of the ball K− of all angular operators corresponding to the set of all maxi- mal non-positive subspaces of a Krein space, i.e, of a Hilbert space H with an indefinite metric, for which both positive and negative components can be infinite-dimensional in general ([7, 31, 34, 59]). This transformation is generated by a continuous linear operator U in H (so called bistrict plus- operator: see for example [7, 31, 34, 35, 59]). We did not suppose U to be continuously invertible. The weak compactness of the image and the domain of K− by the generated linear fractional transformation FU has been established ([53, Theorem 2.1]). We apply the above results to the study of dichotomous behavior of solutions to a non-autonomous linear difference equation in a Hilbert space H yn+1 = Anyn (yn ∈ H, n = 0, 1, 2, . . . ), (3.12) where An are linear bounded operators acting in the space H. This equa- tion describes a non-autonomous dynamical system, in which the inte- ger n plays a role of the discrete time. We do not assume that the operators An are continuously invertible. This means that the corre- sponding dynamical system is irreversible in general. The dichotomous behavior means that solutions with initial values belonging to some sub- space of the phase space H (the stable subspace) stabilize themselves to zero at infinity, but all the other solutions grow infinitely. For differen- tial equations in Hilbert and Banach spaces this property was studied in [35,49,51,53,73,74]. We turn now to the linear difference equation of the form (3.12) in a Hilbert space H. Assume that that all the operators An in this V. Khatskevich, I. Karelin, L. Zelenko 493 equation belong to the class L(H). Consider the evolution operator U(n,m) (n,m ∈ Z+, n ≥ m) of equation (3.12). Recall that this is the operator which associates with each y0 ∈ H the solution yn of equa- tion (3.12) satisfying the initial condition yn|n=m = y0, that is U(n,m)y0 = An−1 ·An−2 · · · · ·Am+1 ·Amy0, (3.13) if n > m and U(n, n) = I, We shall denote briefly U(n) = U(n, 0). Along with equation (3.12) consider the following sequence of difference equations in H: zn+1 = A⋆ N−nzn, (3.14) where N ∈ Z+, n = 0, 1, . . . N . For each fixed N ∈ Z+ denote by ŨN (n,m) the evolution operator of this equation. In view of (3.13), these operators are connected with the evolution operator of equation (3.12) in the following manner: Ũn+m−1(n,m) = U⋆(n,m), hence for m = 0 U⋆(n) = Ũn−1(n). (3.15) In the space H consider a sequence of indefinite metrics of the form [x, y]n = (Vnx, x) (n ∈ Z+), (3.16) where each operator Vn belongs to L(H), it is self-adjoint, it is continu- ously invertible and satisfies the following conditions: (A) The numbers d+ Vn , d−Vn do not depend on n (see subsection 2.10); (B) The sequence of operators {Vn}n∈Z+ is uniformly bounded with re- spect to the operator norm and moreover: sup n∈Z+ ‖Vn‖ = 1. Let us introduce the following Definition 3.1. We call the sequence of quadratic forms [U(n+ 1, n)y0, U(n+ 1, n)y0]n+1 − [y0, y0]n (n ∈ Z+) the increment of the sequence of the quadratic forms [y0, y0]n with respect to the difference equation (3.12) and denote it by ∆(3.12)([y0, y0]n). 494 Operator pencils of the second order... From (3.16), (3.12) we obtain that ∆(3.12)([y0, y0]n) = [Any0, Any0]n+1 − [y0, y0]n = ((A⋆ nVn+1An − Vn)y0, y0). (3.17) In the analogous manner we obtain: ∆(3.14)([y0, y0]n) = [A⋆ N−ny0, A ⋆ N−ny0]n+1 − [y0, y0]n = ((AN−nVn+1A ⋆ N−n − Vn)y0, y0). (3.18) In the sequel we shall assume the following condition to be satisfied for both the metrics (3.16) and equation (3.12): (C) There exists a non-increasing sequence of positive numbers {ǫn}n∈Z+ such that ∆(3.12)([y0, y0]n) ≥ ǫn‖y0‖ 2 ∀n ∈ Z+, y0 ∈ H, that is A⋆ nVn+1An − Vn ≥ ǫnI ∀n ∈ Z+. As a consequence of condition (C) we obtain that [U(n+1)y0, U(n+1)y0]n+1− [U(n0)y0, U(n0)y0]n0 ≥ n ∑ k=n0 ǫk‖U(k)y0‖ 2 ∀ y0 ∈ H, n ≥ n0. (3.19) Consider the following bicones, connected both with the metrics (3.16) and the evolution operator of equation (3.12): C− n = {y0 ∈ H : [U(n)y0, U(n)y0]n ≤ 0} (n ∈ Z+). (3.20) Using the latter definition and property (3.19) it is easy to show that for the family of these bicones the property of monotonicity holds: C− n+1 ⊆ C− n . (3.21) We set C− ∞ = ⋂ n∈Z+ C− n . (3.22) V. Khatskevich, I. Karelin, L. Zelenko 495 Lemma 3.1. Assume that, besides conditions (A), (B) and (C), the condition ∆(3.14)([y0, y0]k) ≥ 0 ∀ k ∈ {0, 1, . . . , N}, y0 ∈ H (3.23) is satisfied, that is AN−kVk+1A ⋆ N−k ≥ Vk ∀ k ∈ {0, 1, . . . , N}, N ∈ N. Then the set C− ∞, defined by (3.22), contains a maximal non-positive subspace L− ∞ of the Krein space (H, V0). Proof. In view of condition (C), condition (3.23) with N = n − 1 and equality (3.15), the inequalities [U(n)y, U(n)y]n ≥ [y, y]0, [U⋆(n)y, U⋆(n)y]n ≥ [y, y]0 ∀ y ∈ H, n ∈ N. are valid. This means that each operator U(n) is a J-biexpansive op- erator. Let M− n be the set of maximal subspaces of the bicone C− n , defined by (3.20), and M− n be the set of angular operators K− in the Krein space (H, V0), corresponding to the subspaces L− ∈M− n . By The- orem 2.1 of [53], each of the sets M− n is compact with respect to the weak operator topology defined in L(H− 0 ,H + 0 ), where H − 0 ,H + 0 are the negative and the positive components of the canonical decomposition of Krein space (H, V0). At the same time, property (3.21) implies that M− n+1 ⊆ M− n . Then the intersection ⋂ n∈N M− n is non-empty. Hence a subspace L− ∞ ∈ ⋂ n∈N M− n is the desired maximal non-positive subspace of the Krein space (H, V0) contained in C− ∞. Let {ǫn}n∈Z+ be the sequence of numbers used in condition (C) and L2,ǫ(Z+,H) be the Hilbert space of sequences ȳ = {yn}n∈Z+ of vectors yn ∈ H satisfying the condition ∞ ∑ n=1 ǫn‖yn‖ 2 <∞ with the inner product (ȳ, z̄)ǫ = ∞ ∑ n=1 ǫn(yn, zn). Denote by N the set of solutions of the difference equation (3.12) which belong to L2,ǫ(Z+,H), i.e., N = {ȳ = {yn}n∈Z+ ∈ L2,ǫ(Z+,H) | yn+1 = Anyn ∀n ∈ Z+}. (3.24) 496 Operator pencils of the second order... Let N0 be the “slice” of the set N at the moment n = 0, i.e. N0 = {y0 | (y0, y1, . . . , yn, . . . ) ∈ N}. (3.25) We turn now to the main result of this subsection. Theorem 3.4. Assume that besides conditions (A), (B) and (C), the condition ∞ ∑ n=0 ǫn = +∞ (3.26) is satisfied, where {ǫn}n∈Z+ is the sequence used in condition (C). Then: (i) The set C− ∞ is a closed subspace of H and, moreover, C− ∞ = N0; (3.27) (ii) For any y0 ∈ C− ∞ for the solution yn = U(n)y0 of equation (3.12) the following estimate holds: ∞ ∑ k=n+1 ǫk‖yk‖ 2 ≤ I(y0) n ∏ k=0 (1 + ǫk) −1, (3.28) where I(y0) = ∞ ∑ k=1 ǫk‖yk‖ 2; (3.29) (iii) For any y0 /∈ C− ∞ there exists C(y0) > 0 such that the solution yn = U(n)y0 of equation (3.12) satisfies the following inequality: ‖yn‖ 2 ≥ C(y0) n ∏ k=0 (1 + ǫk); (3.30) (iv) If, in addition, condition (3.23) is satisfied, the set C− ∞ is a maximal non-positive subspace of the Krein space (H, V0). Proof. First of all, let us prove assertion (iii). Assume that y0 /∈ C− ∞. Then, in view of (3.20), (3.21), (3.22), there exists n0 ∈ Z+ such that [U(n0)y0, U(n0)y0]n0 > 0. (3.31) Using condition (B), we obtain from property (3.19) the following in- equality: ‖U(n+ 1)y0‖ 2 ≥ [U(n0)y0, U(n0)y0]n0 + n ∑ k=n0 ǫk‖U(k)y0‖ 2 ∀n ≥ n0, (3.32) V. Khatskevich, I. Karelin, L. Zelenko 497 which can be rewritten in the form: Yn+1 − Yn ≥ ǫn+1Yn (n ≥ n0), (3.33) where Yn = [U(n0)y0, U(n0)y0]n0 + n ∑ k=n0 ǫk‖U(k)y0‖ 2. (3.34) Inequalities (3.31) and (3.33) imply that Yn ≥ [U(n0)y0, U(n0)y0]n0 n ∏ k=n0 (1 + ǫk+1) (n ≥ n0). Taking into account (3.31), (3.32) and (3.34), we obtain from the latter inequality the desired estimate (3.30) with some positive constant C(y0). So, we have proved assertion (iii). Let us prove assertion (i). Observe that estimate (3.30) and condition (3.26) imply that, if y0 /∈ C− ∞, the solution yn = U(n)y0 (n ∈ Z+) of equation (3.12) does not belong to the space L2,ǫ(Z+). This means that N0 ⊆ C− ∞. (3.35) Let us prove the inverse inclusion. Assume that y0 ∈ C− ∞. Then, by definitions (3.20), (3.21) of the sets C− n and C− ∞, [U(n)y0, U(n)y0]n ≤ 0 ∀n ∈ Z+. This fact and property (3.19) imply that n ∑ k=n0 ǫk‖U(k)y0‖ 2 ≤ [U(n0)y0, U(n0)y0]n0 ∀n ≥ n0, (3.36) hence the solution yn = U(n)y0 (n ∈ Z+) of equation (3.12) belongs to the space L2,ǫ(Z+), i.e., y0 ∈ N0. So, we have proved the inclusion C− ∞ ⊆ N0 which, together with inclusion (3.35), implies equality (3.27). Since the set C− ∞ is closed in H as the intersection of the closed sets C− n , and the set N0 is linear (by (3.24), (3.25)), then C− ∞ is a closed subspace of the space H. We have proved assertion (i). We turn now to the proof of assertion (ii). Assume, as above, that y0 ∈ C− ∞. Let us turn n → ∞ in inequality (3.36) and afterwards sub- stitute there n for n0. Then, in view of condition (B), we obtain the inequality: ∞ ∑ k=n+1 ǫk‖U(k)y0‖ 2 ≤ ‖U(n)y0‖ 2 ∀n ∈ Z+. 498 Operator pencils of the second order... Denote Zn = ∞ ∑ k=n+1 ǫk‖U(k)y0‖ 2. (3.37) Then the latter inequality can be rewritten in the form: Zn+1 ≤ (1 + ǫn+1) −1Zn ∀n ∈ Z+. This estimate and (3.37) imply the desired estimate (3.28), in which I(y0)) is expressed by (3.29). We have proved assertion (ii). Assertion (iv) follows from the obvious inclusion C− ∞ ⊆ C− 0 and Lemma 3.1. References [1] H. Abou-Kandil, V. Ionescu and G. Jank, Matrix Riccati equations in control and system theory, Birkhauser Verlag, Basel, 2003. [2] D. Alpay and V. Khatskevich, Linear fractional transformations: basic properties, applications to spaces of analytic functions and Schroeder’s equation // Int. J. Appl. Math. 2 (2000), N 4, 459–476. [3] B. D. O. Anderson and E. I. Jury, Generalized Bezoutian and Sylvester matrices in multivariable linear control // IEEE Trans. Automat. Control. AC-21 (1976), 551–556. [4] T. Ya. Azizov and M. D. Dragileva, Invariant subspaces of operator-commutative families of operators in a space with indefinite metric // (in Russian), Voronezh. Gos. Univ. Trudy Nauchn.-Issled. Inst. Mat. VGU, N 17, Teor. Operator. Urav- nenii (1975), 3–7. [5] T. Ya. Azizov, E. I. Iokhvidov, and I. S. Iokhvidov, The relation between the Cayley-Neumann and Potapov-Ginzburg transforms // (in Russian), Functional analysis, N 20, Ulyanovsk. Gos. Ped. Inst., Ulyanovsk, 1983, 3–8. [6] T. Ya. Azizov and I. S. Iokhvidov, Linear operators in spaces with an indefinite metric, John Wiley & Sons, 1989. [7] T. Ya. Azizov and I. S. Iohvidov, Foundations of the linear operator theory in spaces with indefinite metric // (in Russian), Nauka, Moscow, 1986. [8] T. Ya. Azizov, V. A. Khatskevich, and D. M. Shŏıkhet, On the number of fixed points of a holomorphism // (Russian) Sibirsk. Mat. Zh. 31 (1990), N 6, 192–195; translation in Siberian Math. J. 31 (1990), N 6, 1040–1042 (1991). [9] T. Andô, Truncated moment problems for operators // Acta Sci. Math. (Szeged), 31 (1970), 319–334. [10] I. N. Baker, Fractional iteration near a fixpoint of multiplier 1 // J. Austral. Math. Soc., 4 (1964), 143–148. [11] C. C. Cowen, Iteration and the solution of functional equations for functions analytic in the unit disk // Trans. Amer. Math. Soc. 265 (1981), N 1, 69–95. [12] C. C. Cowen and B. D. MacCluer, Linear fractional maps of the ball and their composition operators // Acta Sci. Math. (Szeged), 66 (2000), N 1–2, 351–376. V. Khatskevich, I. Karelin, L. Zelenko 499 [13] C. C. Cowen and B. D. MacCluer, Schroeder’s equation in several variables // Taiwanese J. Math.7 (2003), N 1, 129–154. [14] R. Curtain and H. Zwart, An introduction to infinite-dimensional linear systems theory, Texts in Applied Mathematics, 21, Springer-Verlag, New York, 1995. [15] L. E. Faybusovich, Matrix Riccati inequality: existence of solutions // Systems Control Lett. 9 (1987), N 1, 59–64. [16] I. Gohberg, P. Lancaster, and L. Rodman, Matrices and Indefinite Scalar Prod- ucts, Birkhauser Verlag, 1983 [17] G. Gomez and L. Lerer, Generalized Bezoutian for analytic operator function and inversion of strucstructured operators, System and Networks: Mathematical Theory and Applications (U. Helmke, R. Mennicken, J. Saures, eds.), Academie Verlag, 1994, 691–696. [18] J. Hadamard, Two works on iteration and related questions // Bull. Amer. Math. Soc. 50 (1944), 67–75. [19] J. Haimovici and L. Lerer, Bezout operators for analytic operator functions. A general concept of Bezout operators // Integral Equations and Operator Theory, 21 (1995), 33–70. [20] T. E. Harris, Some mathematical models for branching processes in: Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability, 1950, pp. 305–328. University of California Press, Berkeley and Los Angeles, 1951. [21] J. W. Helton, Operators unitary in an indefinite metric and linear fractional transformations // Acta Sci. Math. (Szeged), 32 (1971), 261–266. [22] I. S. Iokhvidov, Unitary and self-adjoint operators in spaces with an indefinite metric (in Russian), Ph. D. Thesis, Kharkov State University, 1950. [23] I. S. Iokhvidov, Banach spaces with a J-metric and certain classes of linear oper- ators in these spaces // (in Russian), Bul. Akad. Stiince RSS Moldoven, (1968), N 1, 60–80. [24] I. S. Iokhvidov, On a class of linear fractional transformations (in Russian), in: Sbornik statej po funkzional’nym prostranstvam i operatornym uravneniyam, (eds S. G. Krein and V. I. Sobolev, Voronezh, Izdatel’stvo Voronezhskogo Go- sudarstvennogo Universiteta, 1970), pp. 18-44. [25] I. S. Iokhvidov, Studies in the theory of linear operators in spaces with a J-metric, and the theory of Toeplitz forms (in Russian), Habilitation Degree Thesis, FTINT AN USSR, Kharkov, 1967. [26] V. Ionescu, C. Oară, and M. Weiss, Generalized Riccati theory and robust control, Chichester, Wiley, 1999. [27] I. Karelin and L. Lerer, Generalized Bezoutian, factorization of rational matrix functions and matrix quadratic equations // Operator Theory: Advances and Applications, 122 (2001), 303–321. [28] I. Karelin and L. Lerer, Matrix quadratic equations, column/row reduced factor- izations and the inertia theorem for matrix polynomials // Int. J. Appl. Math. Comput. Sci. 11 (2001), N 6, 1285–1310. [29] I. Karelin, L. Lerer and A. C. M. Ran, J-symmetric factorizations and algebraic Riccati equation // Operator Theory: Advances and Applications, 124 (2001), 319–360. 500 Operator pencils of the second order... [30] S. Karlin and J. McGregor, Embedding iterates of analytic functions with two fixed points into continuous groups // Trans. Amer. Math. Soc. 132 (1968), 137–145. [31] V. Khatskevich, On the symmetry of properties of plus-operator and its conjugate operator // (in Russian), Functional analysis (Ulyanovsk, Russia), 14 (1980), 177-186. [32] V. A. Khatskevich, On J-semiunitary operators in a Hilbert J-space // (in Rus- sian), Mat. Zametki, 17 (1975), N 4, 639–647. [33] V. A. Khatskevich, Fixed points of generalized linear-fractional transformations // (in Russian), Izv. Akad. Nauk SSSR, Ser. Mat. 39 (1975), N 5, 1130–1141. [34] V. Khatskevich, On characteristic spectral properties of focused operators // (in Russian), Proceedings of Academy of Science of Armenian Republic (Doklady Acad. Nauk Arm. S.S.R.), 79 (1984), 102–105. [35] V. Khatskevich, Some global properties of fractional-linear transformations in: Nonselfadjoint operators and related topics (Beer Sheva, 1992), 355–361, Oper. Theory Adv. Appl., 73, Birkhäuser, Basel, 1994. [36] V. Khatskevich, Generalized fractional linear transformations: convexity and compactness of the image and the pre-image; applications // Studia Math. 137 (1999), 169–175. [37] V. Khatskevich, M.I. Ostrovskii, and V. Shulman, Extremal problems for opera- tors in Banach spaces arising in the study of linear operator pencils // Integral Equations and Operator Theory, 51 (2005), 109–119. [38] V. Khatskevich, M.I. Ostrovskii, and V. Shulman, Linear fractional relations for Hilbert space operators // Math. Nachrichten, 279 (2006), N 8, 875–890. [39] V. Khatskevich, S. Reich, and D. Shoikhet, One-parameter semigroups of fractional-linear transformations in: Operator theory, system theory and related topics, (Beer-Sheva/Rehovot, 1997), 401–411, Oper. Theory Adv. Appl. 123, Birkhäuser, Basel, 2001. [40] V. Khatskevich, S. Reich, and D. Shoikhet, Schröder’s functional equation and the Koenigs embedding property // Nonlinear Analysis: Theory, Methods & Ap- plications, 47 (2001), 3977–3988. [41] V. Khatskevich, S. Reich, and D. Shoikhet, Abel–Schröder equations for lin- ear fractional mappings and the Koenigs embedding problem // Acta Sci. Math. (Szeged), 69 (2003), 67–98. [42] V. Khatskevich and V. Senderov, Basic properties of linear fractional mappings of operator balls: Schröeder’s equation in: Operator theory and its applications (Winnipeg, MB, 1998), 331–344, Fields Inst. Commun. 25, Amer. Math. Soc. Providence, RI, 2000. [43] V. Khatskevich and V. Senderov, The Abel-Schröder equation for linear-fractional mappings of operator balls // (Russian), Dokl. Akad. Nauk, 379 (2001), N 4, 455– 458. [44] V. Khatskevich and V. Senderov, On the convexity, compactness, and nonempti- ness of images and preimages of linear fractional operator relations // Doklady Mathematics, 69 (2004), 409–411. [45] V. Khatskevich and V. Senderov, Abel–Schröder type equations for maps of op- erator balls // Functional Differential Equations, 10 (2003), 239–258. V. Khatskevich, I. Karelin, L. Zelenko 501 [46] V. Khatskevich and V. Senderov, Fractional-Linear Transformation of Operator Balls, Applications to Dynamical Systems // Acta Math. Sinica, English Series, to appear (2006). [47] V. Khatskevich, V. Senderov, and V. Shulman, On operator matrices generating linear fractional maps of operator balls // Israel Conference Proceedings, Con- temporary Math. 64 (2004), 93–102. [48] V. Khatskevich and D. Shoikhet, One version of implicit function theorem for holomorphic mappings // C. R. Acad. Sci. Paris, Sér. I Math., 319 (1994), N 6, 599–604. [49] V. Khatskevich and V. Shulman, Operator fractional-linear transformations: con- vexity and compactness of image; applications // Studia Math. 116 (1995), 189– 195. [50] V. A. Khatskevich and V. S. Shulman, On linear operator pencils and inclusions of images of balls in: Trends in Banach Spaces and Operator Theory (Memphis, TN, 2001), edited by A. Kamińska, Contemporary Math. 321, pp. 115–124, Amer. Math. Soc., Providence, RI, 2003. [51] V. Khatskevich and L. Zelenko, Indefinite metrics and dichotomy of solutions of linear differential equations in Hilbert spaces // Chinese J. Math. 24 (1996), N 2, 99–112. [52] V. Khatskevich and L. Zelenko, The fractional-linear transformations of the op- erator ball and dichotomy of solutions to evolution equations in: Recent develop- ments in optimization theory and nonlinear analysis (Jerusalem, 1995), 149–154, Contemp. Math. 204, Amer. Math. Soc., Providence, RI, 1997. [53] V. Khatskevich and L. Zelenko, Bistrict plus-operators in Krein spaces and di- chotomous behavior of irreversible dynamical systems in: Operator theory and related topics, II (Odessa, 1997), 191–203, Oper. Theory Adv. Appl., 118, Birkhäuser, Basel, 2000. [54] V. Khatskevich and L. Zelenko, Plus-operators in Krein spaces and dichotomous behavior of irreversible dynamical systems with a discrete time, Studia Mathe- matica, to appear. [55] G. Koenigs, Recherches sur les intégrales de certaines équations fonctionnelles // Annales Sci. École Normale Sup. (Ser. 3) 1 (1884), supplement, 3–41. [56] M. G. Krein, The theory of self-adjoint extensions of semi-bounded Hermitian transformations and its applications, I // (Russian) Mat. Sbornik, N.S. 20(62) (1947), 431–495. [57] M. G. Krein, The theory of self-adjoint extensions of semi-bounded Hermitian transformations and its applications, II // (Russian), Mat. Sbornik, N.S. 21(63) (1947), 365–404. [58] M. G. Krein, On an application of the fixed point principle in the theory of lin- ear transformations of spaces with an indefinite metric // (in Russian), Uspehi Matem. Nauk (N.S.), 5 (1950), N 2(36), 180–190. [59] M. G. Krein, A new application of the fixed-point principle in the theory of oper- ators in a space with indefinite metric // (in Russian), Dokl. Akad. Nauk SSSR, 154 (1964), 1023–1026. [60] M. G. Krein and I. E. Ovcharenko, Q-functions and sc-resolvents of nondensely defined Hermitian contractions // (Russian), Sibirsk. Mat. Zh. 18 (1977), N 5, 1032–1056; English transl.: Siberian Math. J. 18 (1977), 728–746. 502 Operator pencils of the second order... [61] M. G. Krein and Yu. L. Shmulyan, Plus-operators in a space with indefinite met- ric // (in Russian), Matem. Issled. (Kishinev), 1 (1966), N 1, 131–161; English transl. in Amer. Math. Soc. Transl. Ser. 2, 85 (1969), 93–113. [62] M. G. Krein and Yu. L. Shmulyan, J -polar representation of plus-operators // (in Russian), Matem. Issled. (Kishinev), 1 (1966), N 1, 172–210; English transl. in Amer. Math. Soc. Transl. Ser. 2, 85 (1969), 115–143. [63] M. G. Krein and Yu. L. Shmulyan, On linear-fractional transformations with operator coefficients // (in Russian), Matem. Issled. (Kishinev), 2 (1967), N 3, 64–96; English transl. in Amer. Math. Soc. Transl. Ser. 2, 103 (1974), 125–152. [64] G. A. Kurina, Control of a descriptor system in an infinite interval // (Russian), Izv. Ross. Akad. Nauk Tekhn. Kibernet. (1993), N 6, 33–38, English transl.: J. Comput. Systems Sci. Internat. 32 (1994), N 6, 30–35. [65] P. Lancaster and L. Rodman, Algebraic Riccati equations, New York, Oxford University Press, 1995. [66] I. Lasiecka and R. Triggiani, Control theory for partial differential equations: continuous and approximation theories. I. Abstract parabolic systems, Encyclope- dia of Mathematics and its Applications, 74, Cambridge University Press, Cam- bridge, 2000. [67] L. Lerer and L. Rodman, Bezoutian of rational matrix functions, J. Funct. Anal. 141 (1996), 1–36. [68] L. Lerer and M. Tismenetsky, The Bezoutian and the eigenvalue separation prob- lem // Integral Equations and Operator Theory, 5 (1982), 386–445. [69] P. Lévy, Fonctions à croissance régulière et itération d’odre fractionnaire // Ann. Mat. Pura Appl. 5 (1928), 269–298. [70] M. M. Malamud, Extensions of Hermitian sectorial operators and dual pairs of contractions // (in Russian), Dokl. Akad. Nauk SSSR, 305 (1989), N 1, 35–41; translation in Soviet Math. Dokl. 39 (1989), N 2, 253–259. [71] M. M. Malamud, On some classes of extensions of sectorial operators and dual pairs of contractions in: Recent advances in operator theory (Groningen, 1998), 401–449, Oper. Theory Adv. Appl. 124, Birkhuser, Basel, 2001. [72] A. S. Markus, Introduction to the spectral theory of polynomial operator pencils. Translated from the Russian by H. H. McFaden, Translation edited by Ben Silver. With an appendix by M. V. Keldysh. Translations of Mathematical Monographs, volume 71. American Mathematical Society, Providence, R.I., 1988. [73] A. D. Maizel, On stability of solutions to systems of differential equations // (in Russian), Proceedings of Ural Politechnic Institute (Ural. Politechn. Instit. Trudy, Russia), 51 (1954), 20–54. [74] J. L. Massera and J. J. Schäffer, Linear differential equations and function spaces, Acad. Press, New York, London, 1966. [75] M. A. Naimark, Unitary permutation operators in the space Πκ // (in Russian), Dokl. Akad. Nauk SSSR, 149 (1963), 1261–1263. [76] M.I. Ostrovskii, Extremal problems for operators in Banach spaces arising in the study of linear operator pencils, II // Integral Equations and Operator Theory, 51 (2005), 553–564. [77] R. Phillips, Dissipative operators and hyperbolic systems of partial differential equations // Trans. Amer. Math. Soc. 90 (1959), 193–254. V. Khatskevich, I. Karelin, L. Zelenko 503 [78] R. Phillips, Dissipative operators and parabolic partial differential equations // Comm. Pure Appl. Math. 12 (1959), 249–276. [79] R. Phillips, The extension of dual subspaces invariant under an algebra in: Proc. Internat. Sympos. Linear Spaces (Jerusalem, 1960), Jerusalem Academic Press, Jerusalem; Pergamon, Oxford, 1961, pp. 366–398. [80] L. S. Pontryagin, Hermitian operators in spaces with indefinite metric // (in Russian), Izvestiya Akad. Nauk SSSR, Ser. Matem. 8, (1944), 243–280. [81] C. Scherer, The solution set of the algebraic Riccati equation and the algebraic Riccati inequality //, Linear Algebra Appl. 153 (1991), 99–122. [82] V. Senderov and V. Khatskevich, Normed J -spases and certain classes of lin- ear operators in these spaces //(in Russian), Matematicheskie Issledovaniya (Kishinev), 8 (1973), N 3, 56–75. [83] Yu. L. Shmulyan, On divisibility in the class of plus-operators // Mat. Zametki, 74 (1967), 516–525. [84] Yu. L. Shmulyan, A Hellinger operator integral // (Russian), Mat. Sb. (N.S.) 49 (91) (1959), 381–430; English transl. in Amer. Math. Soc. Transl. Ser. 2, 22 (1962), 289-337. [85] S. L. Sobolev, Motion of a symmetric top with a cavity filled with a liquid // (in Russian), ZhPMTF, 3 (1960), 20–55. [86] E. Vesentini, Semigroups of linear contractions for an indefinite metric // Mem. Mat. Accad. Lincei, 2 (1994), 53–83. [87] V. A. Yakubovich, Conditions for semiboundedness of a quadratic functional on a subspace of a Hilbert space // Vestn. Leningr. Univ., Math. 14 (1982), 321–325. Contact information Victor Khatskevich Department of Mathematics ORT Braude Academic College College Campus, P.O.Box 78 Karmiel 21982 Israel E-Mail: victor_kh@hotmail.com Irina I. Karelin Lincoln str. 2/3, Haifa, Israel, 34369 E-Mail: Irina@generic-imaging.com Leonid Zelenko Department of Mathematics University of Haifa 31905 Haifa, Israel E-Mail: zelenko@math.haifa.ac.il