In this paper, we show the existence of solutions for the strongly indefinite elliptic system −∆u=λu+f(x, v) in Ω, −∆v=λv+g(x, u) in Ω, u=v= 0, on∂Ω, where Ω is a bounded domain inRN (N ≥3) with smooth boundary,λk0&lt

ISSN: 1072-6691. URL: http://ejde.math.txstate.edu or http://ejde.math.unt.edu ftp ejde.math.txstate.edu (login: ftp)

EXISTENCE RESULTS FOR STRONGLY INDEFINITE ELLIPTIC SYSTEMS

JIANFU YANG, YING YE, XIAOHUI YU

Abstract. In this paper, we show the existence of solutions for the strongly indefinite elliptic system

−∆u=λu+f(x, v) in Ω,

−∆v=λv+g(x, u) in Ω, u=v= 0, on∂Ω,

where Ω is a bounded domain inR^N (N ≥3) with smooth boundary,λk₀<

λ < λk₀+1, whereλk is thekth eigenvalue of−∆ in Ω with zero Dirichlet boundary condition. Both cases whenf, g being superlinear and asymptoti- cally linear at infinity are considered.

1. Introduction

In this paper, we investigate the existence of solutions for the strongly indefinite elliptic system

−∆u=λu+f(x, v) in Ω,

−∆v=λv+g(x, u) in Ω, u=v= 0, on∂Ω,

(1.1) where Ω is a smooth bounded domain inR^N,N ≥3,λk₀< λ < λk₀+1, whereλk is thekth eigenvalue of−∆ in Ω with zero Dirichlet boundary condition.

Problem (1.1) withλ= 0 was considered in [5, 6], where the existence results for superlinear nonlinearities were established by finding critical points of the functional

J(u, v) = Z

Ω

∇u∇v dx− Z

Ω

F(x, v)dx− Z

Ω

G(x, u)dx. (1.2) A typical feature of the functionalJ is that the quadratic part

Q(u, v) = Z

Ω

∇u∇v dx

is positive definite in an infinite dimensional subspaceE⁺ ={(u, u) :u∈H₀¹(Ω)}

ofH₀¹(Ω)×H₀¹(Ω) and negative definite in its infinite dimensional complimentary

2000Mathematics Subject Classification. 35J20,3 5J25.

Key words and phrases. Strongly indefinite elliptic system; existence.

c

2008 Texas State University - San Marcos.

Submitted April l7, 2008. Published May 28, 2008.

Supported by grants 10571175 and 10631030 from the National Natural Sciences Foundation of China.

1

subspace E⁻ ={(u,−u) :u∈H₀¹(Ω)}, that is,J is strongly indefinite. A linking theorem is then used in finding critical points ofJ.

In the case thatλlies in between higher eigenvalues, the parameterλaffects the definiteness of the corresponding quadratic part

Qλ(u, v) = Z

Ω

(∇u∇v−λuv)dx of the associated functional

Jλ(u, v) = Z

Ω

(∇u∇v−λuv)dx− Z

Ω

F(x, v)dx− Z

Ω

G(x, u)dx, (1.3) of (1.1) defined onH₀¹(Ω)×H₀¹(Ω). A key ingredient in use of the linking theorem is to find a proper decomposition ofH₀¹(Ω)×H₀¹(Ω) into a direct sum of two subspaces so that Qλ is definite in each subspace. Obviously, Qλ is neither positive definite inE⁺nor negative definite inE⁻. So we need to find out a suitable decomposition ofH₀¹(Ω)×H₀¹(Ω).

We first consider the asymptotically linear case. Such a problem has been exten- sively studied for one equation, see for instance, [4, 10, 11] and references therein.

For asymptotically linear elliptic system, we refer readers to [8]. Particularly, in this case, the Ambrosetti-Rabinowtz condition is not satisfied, whence it is hard to show a Palais-Smale sequence is bounded. So one turns to using Cerami condi- tion in critical point theory instead of the Palais-Smale condition, various existence results for asymptotically linear problems are then obtained. By a functional I defined onE satisfies Cerami condition we mean that for any sequence{u_n} ⊂E such that|I(u_n)| ≤Cand (1 +ku_nk)I⁰(u_n)→0, there is a convergent subsequence of{u_n}. For the asymptotically linear system (1.1), it is strongly indefinite and the nonlinearities do not fulfill the Ambrosetti-Rabinowitz condition. To handle the problem, we assume:

(A1) f, g∈C(Ω×R,R),f(x, v) =o(|v|), g(x, u) =o(|u|) uniformly forx∈Ω as

|u|,|v| →0 andtf(x, t)≥0, tg(x, t)≥0.

(A2) There exist positive constants l, m, such that lim_t→±∞^f(x,t)_t = l and limt→±∞g(x,t)

t =m.

(A3) λ±√

ml6=λ_k for anyk∈N.

(A4) There existsu₀∈span{ϕ_k0+1, ϕ_k0+2, . . .} with R

Ω|∇u₀|²−λ(u₀)²dx= ¹₂ such that

Z

Ω

(|∇u₀|²−λu²₀)dx−min(l, m) Z

Ω

u²₀dx <0.

Theorem 1.1. Suppose(A1)-(A4), problem(1.1)has at least a nontrivial solution.

Condition (A4) holds, for example, if min(l, m) > λk₀+1−λ, we choose u0 = αϕk+1for someα >0, thenR

Ω|∇u0|²−λu²₀dx−min(l, m)R

Ωu²₀dx= (λk₀+1−λ−

min(l, m))R

Ωu²₀dx <0.

Theorem 1.1 is proved by the following linking theorem with Cerami condition in [3], which is a generalization of usual one in [2], [9].

Lemma 1.2. Let E be a real Hilbert space with E = E1⊕E2. Suppose I ∈ C¹(E,R), satisfies Cerami condition, and

(I1) I(u) = ¹₂(Lu, u) +b(u), where Lu=L₁P₁u+L₂P₂uand L_i :E_i →E_i is bounded and selfadjoint, i=1,2.

(I2) b⁰ is compact.

(I3) There exists a subspaceE˜ ⊂Eand setsS ⊂E, Q⊂E˜ and constantsα > ω such that

(i)S⊂E1 andI|S ≥α, (ii)Qis bounded andI|∂Q ≤ω, (iii)S andQlink.

ThenI possesses a critical valuec≥α.

Next, we consider superlinear case. We assume that

(B1) f, g∈C(Ω×R,R),f(x, v) =o(|v|), g(x, u) =o(|u|) uniformly forx∈Ω as

|u|,|v| →0.

(B2) There exists a constantγ >2 such that

0< γF(x, v)≤vf(x, v), 0< γG(x, u)≤ug(x, u), whereF(x, v) =Rv

0 f(x, s)dsandG(x, u) =Ru

0 g(x, u)ds.

(B3) There exist p, q > 1,_p+1¹ + _q+1¹ > ^N_N⁻², constants a1, a2 > 0, such that

|f(x, v)| ≤a1+a2|v|^q,|g(x, u)| ≤a1+a2|u|^p.

Theorem 1.3. Assume (B1)-(B3), then (1.1)has at least one solution.

We remark that in [6], it also considered the subcritical superlinear problem

−∆u=λv+f(v) in Ω,

−∆v=µu+g(u) in Ω, u=v= 0, on∂Ω.

(1.4)

The functional corresponding to (1.4) is no longer positive definite inE⁺, but it is negative definite inE⁻. It is different from our case.

In section 2, we prove Theorem 1.1. While Theorem 1.3 is showed in section 3.

2. Asymptotically linear case

Let H := H₀¹(Ω), it can be decomposed as H = H¹ ⊕H², where H¹ = span{ϕk₀+1, ϕk₀+2. . .},H²= span{ϕ1, ϕ2. . . ϕk₀}and ϕk is the eigenfunction re- lated to λk. Let Pi be the projection of H on the subspace Hⁱ, i = 1,2, then we define foru∈H a new norm by

kuk²= Z

Ω

|∇(P1u)|²−λ(P1u)²dx− Z

Ω

|∇(P2u)|²−λ(P2u)²dx,

it is equivalent to the usual norm ofH₀¹(Ω). To find out the subspaces ofH×H such that the quadratic part

Qλ(u, v) = Z

Ω

(∇u∇v−λuv)dx of the functional

Jλ(u, v) = Z

Ω

(∇u∇v−λuv)dx− Z

Ω

F(x, v)dx− Z

Ω

G(x, u)dx is positive or negative definite on it, we denote

E₁₁={(u, u) :u∈H¹}, E₁₂={(u,−u) :u∈H¹}, E₂₁={(u, u) :u∈H²}, E₂₂={(u,−u) :u∈H²}.

Therefore,H×H =E11⊕E12⊕E21⊕E22. We may write for any (u, v)∈H×H that

(u, v) = (u11, u11) + (u12,−u12) + (u21, u21) + (u22,−u22), (2.1) where

u11=P1(u+v

2 )∈H¹, u21=P2(u+v

2 )∈H², u12=P1(u−v

2 )∈H¹, u22=P2(u−v

2 )∈H².

It is easy to check thatQ_λis positive definite inE₁₁⊕E₂₂and negative definite in E₁₂⊕E₂₁, so we denoteE₊=E₁₁⊕E₂₂ andE₋=E₁₂⊕E₂₁ for convenience.

Then

Jλ(u, v) =ku11k²+ku22k²− ku12k²− ku21k²− Z

Ω

F(x, v)dx− Z

Ω

G(x, u)dx, (2.2) it isC¹onH×H.

Lemma 2.1. The functional J_λ satisfies the Cerami condition.

Proof. It is sufficient to show that any Cerami sequence is bounded, a standard argument then implies that the sequence has a convergent subsequence. We argue indirectly. Suppose it were not true, there would exist a Cerami sequence zn = {(un, vn)} ⊂H×H ofJλ such thatkznk → ∞. Let

wn= zn

kz_nk = ( un

kz_nk, vn

kz_nk) = (w_n¹, w²_n), we may assume that

(w_n¹, w²_n)*(w¹, w²) inH×H, (w_n¹, w²_n)→(w¹, w²) inL²(Ω)×L²(Ω), w_n¹ →w¹, w²_n→w² a.e. in Ω.

We write as the decomposition (2.1) thatun =P2

i,j=1uⁿ_ij and correspondingly, w¹_n=P2

i,j=1wⁿ_ij. We claim that (w¹, w²)6= (0,0). Otherwise, there would hold

|hJ_λ⁰(un, vn),(uⁿ₁₁, uⁿ₁₁)i| ≤ kJ_λ⁰(un, vn)k·k(uⁿ₁₁, uⁿ₁₁)k ≤ kJ_λ⁰(un, vn)k·k(un, vn)k →0;

(2.3) that is,

kuⁿ₁₁k²− Z

Ω

f(x, vn)uⁿ₁₁dx− Z

Ω

g(x, un)uⁿ₁₁dx→0 (2.4) implying

kwⁿ₁₁k²− Z

Ω

f(x, vn) vn

vn

kznk uⁿ₁₁ kznkdx−

Z

Ω

g(x, un) un

un

kznk uⁿ₁₁

kznkdx→0. (2.5) Therefore,

kwⁿ₁₁k²≤C Z

Ω

[(w¹_n)²+ (w_n²)²]dx+o(1), (2.6) which yields kw₁₁ⁿk → 0. Similarly, kwⁿ₁₂k → 0, kw₂₁ⁿk → 0 and kwⁿ₂₂k → 0 as n→ ∞. Consequently, wn →0. This contradicts to kwnk = 1. Hence, there are three possibilities: (i)w¹6= 0, w²6= 0; (ii)w¹6= 0, w²= 0; (iii)w¹= 0, w²6= 0. We show next that all these cases will lead to a contradiction. Hence,kznkis bounded.

In case (i), we claim that (w¹, w²) satisfies

−∆w¹=λw¹+lw², in Ω,

−∆w²=λw²+mw¹, in Ω, w¹=w²= 0, on∂Ω.

(2.7)

Indeed, let

pn(x) =

(_f(x,vn(x))

v_n(x) ifvn(x)6= 0,

0 ifvn(x) = 0, (2.8)

and

qn(x) =

(_g(x,un(x))

u_n(x) ifun(x)6= 0,

0 ifun(x) = 0. (2.9)

Since 0≤pn, qn ≤M for someM >0, we may suppose thatpn * ϕ, qn * ψ in L²(Ω) andpn →ϕ,qn →ψ a.e in Ω. The factw¹(x)6= 0 implies un(x)→ ∞ and consequently, qn(x)→m. Similarly,w²(x)6= 0 yieldsvn(x)→ ∞and pn(x)→l.

Hence,ϕ(x) =l ifw²(x)6= 0 andψ(x) =mifw¹(x)6= 0.

SinceJ_λ⁰(un, vn)→0, for any (η1, η2)∈H×H, we have Z

Ω

∇vn∇η1−λvnη1dx− Z

Ω

g(x, un)η1dx→0, (2.10) Z

Ω

∇un∇η2−λunη2dx− Z

Ω

f(x, vn)η2dx→0. (2.11) It follows fromkznk → ∞that

Z

Ω

∇w_n¹∇η₂−λw¹_nη₂dx− Z

Ω

p_n(x)w²_nη₂dx→0, (2.12) Z

Ω

∇w²_n∇η1−λw_n²η1dx− Z

Ω

qn(x)w_n¹η1dx→0. (2.13) Noting pnw_n², qnw_n¹ are bounded in L²(Ω), we may assumepnw²_n * ξ(x), qnw¹_n * ζ(x) inL²(Ω) andp_nw_n² →ξ(x),q_nw¹_n→ζ(x) a.e. in Ω. We deduce from the fact w²_n → w², w¹_n → w¹, p_n → ϕ and q_n → ψ a.e. in Ω that ξ = ϕw² = lw² and ζ=ψw¹=mw¹. Letn→ ∞in (2.12) and (2.13) we see that (w¹, w²) solves (2.7).

Let ˜w²=q

l

mw², then (w¹,w˜²) solves

−∆w¹=λw¹+√

mlw² in Ω,

−∆ ˜w²=λw˜²+√

mlw¹ in Ω, w¹= ˜w²= 0, on∂Ω,

(2.14)

which implies

−∆(w¹+ ˜w²) = (λ+√

ml)(w¹+ ˜w²) in Ω,

w¹+ ˜w²= 0 on∂Ω. (2.15)

Ifw¹+ ˜w²6= 0, this contradicts to (A3). Ifw¹+ ˜w²= 0, then

−∆w¹= (λ−√

ml)w¹ in Ω,

w¹= 0 on∂Ω. (2.16)

This again contradicts to (A3).

For case (ii), we derive from (2.12) thatR

Ωpn(x)w_n²η2dx→0 and thenw¹solves

−∆w¹=λw¹ in Ω,

w¹= 0 on∂Ω, (2.17)

which is a contradiction to the assumption that λk₀ < λ < λk₀+1. Similarly, we

may rule out case (iii). The proof is complete.

Next, we show that Jλ has the linking structure. Denotez0 = (u0, u0), where u0is given by assumption (A4), thenkz0k²= 1. Let [0, s1z0] ={sz0: 0≤s≤s1}, MR={z=z⁻+ρz0:kzk ≤R, ρ≥0}, ˜H = span{z0} ⊕E−,S=∂Bρ∩E+. Lemma 2.2. There exist constants α >0 and ρ >0, such that J_λ(u, v)≥αfor (u, v)∈S.

Proof. By (A1) and (A2), for anyε >0 there isCε>0 such that

|F(x, t)| ≤ε|t|²+C_ε|t|^p, |G(x, t)| ≤ε|t|²+C_ε|t|^p for some 2< p < _N^2N₋₂. It implies that for (u, v)∈S,

J_λ(u, v)≥(1

2 −ε)kz⁺k²−C_εkz⁺k^p. (2.18)

The assertion follows.

Lemma 2.3. There existsR > ρsuch that J_λ(u, v)≤0 for(u, v)∈∂M_R.

Proof. Forz∈∂MR, we writez =z⁻+rz0 with kzk=R,r >0 or kzk< Rand r= 0. Ifr= 0, we havez=z⁻ and

Jλ(u, v) =−1

2kz⁻k²− Z

Ω

[F(x, v) +G(x, u)]dx≤0 (2.19) sinceF(x, t), G(x, t)≥0.

Suppose now that r >0. We argue by contradiction. Suppose the assertion is not true, we would have a sequence{zn} ∈∂MR, zn=ρnz0+z⁻_n, ρn>0,kznk=n such thatJλ(zn)>0. We writezn= (un, vn) = (ρnu0+φn, ρnu0+ψn), then

Jλ(zn) =1 2ρ²_n−1

2kz_n⁻k²− Z

Ω

F(x, vn) +G(x, un)dx >0, (2.20) that is

J_λ(z_n) kznk² = 1

2( ρ²_n

kznk² −kz⁻_nk² kznk²)−

Z

Ω

F(x, v_n) +G(x, u_n)

kznk² dx >0. (2.21) Since F, G ≥ 0, then we have ρn ≥ kz⁻_nk. The fact ^ρ2n_kz^+kz−nk2

nk² = 1 implies ¹₂ ≤

ρ²_n

kznk² ≤ 1. Assume _kz^ρ2n

nk² → ρ²₀ > 0, hence ρn → +∞. We may also assume

φ_n

kznk * ξ1,_kz^ψn

nk * ξ2 inH and _kz^φn

nk →ξ1,_kz^ψn

nk →ξ2 a.e. in Ω. Ifx∈Ω such that ρ₀u₀(x) +ξ₁(x)6= 0, thenu_n(x) =ρ_nu₀(x) +φ_n(x)→ ∞. Similarly, ifx∈Ω such that ρ₀u₀(x) +ξ₂(x)6= 0, we havev_n(x) =ρ_nu₀(x) +ψ_n(x)→ ∞. It follows from

(2.21) that 0< 1

2 ρ²_n kznk² −1

2 kz_n⁻k² kznk² −

Z

Ω

[F(x, v_n) v_n² ( v_n

kznk)²+G(x, u_n) u²_n ( u_n

kznk)²]dx

≤ 1 2

ρ²_n kznk² −1

2 kz_n⁻k² kznk² −

Z

{ρ0u0+ξ26=0}

F(x, vn) v²_n ( vn

kznk)²dx +

Z

{ρ₀u₀+ξ₁6=0}

G(x, u_n) u²_n ( u_n

kznk)²dx

(2.22)

Letz=ρ₀z₀+ξ⁻ withξ⁻= (ξ₁, ξ₂) and take limit in (2.22), we get 1

2(ρ²₀kz0k²− kξ⁻k²)− l 2

Z

{ρ0u₀+ξ₂6=0}

(ρ₀u₀+ξ₂)²dx

−m 2

Z

{ρ0u0+ξ16=0}

(ρ0u0+ξ1)²dx≥0.

(2.23)

There are two cases: eitherξ⁻ = (ξ₁, ξ₂)∈E₁₂, that is, ξ₁ =−ξ₂ ∈ H¹ or ξ⁻ = (ξ₁, ξ₂)∈E₂₁, that is,ξ₁=ξ₂∈H². In both cases we haveR

Ω(u₀ξ₁+u₀ξ₂)dx= 0.

By (2.23), we obtain 0≤1

2(ρ²₀kz0k²− kξ⁻k²)−min(l, m) Z

Ω

(ρ²₀u²₀+ξ²₁)dx

≤ρ²₀( Z

Ω

|∇u0|²−λu²₀dx−min(l, m) Z

Ω

u²₀dx)−1

2kξ⁻k²−min(l, m) Z

Ω

ξ₁²dx

<0,

(2.24)

a contradiction.

Proof of Theorem 1.1. Let L(u, v) = (v, u), we may check that L is a bounded selfadjoint operator on H ×H and that E₁₁, E₁₂, E₂₁.E₂₂ are invariant subspace of L, so both E+ and E₋ are invariant subspace of L. (I1) of Lemma 1.2 then holds. (I2) follows from the Sobolev compact imbeddings; (i) and (ii) in (I3) are consequences of Lemma 2.2 and Lemma 2.3. The proof of (iii) in (I3) can be found in [2] and [9]. The proof of Theorem 1.1 is complete.

3. Superlinear case

Let φ1, φ2, φ3, . . . be the eigenfunctions of −∆ in Ω with Dirichlet boundary condition, which consist of the orthogonal basis of L²(Ω). We assume that the eigenfunctions are normalized inL²(Ω); i.e,R

Ωφ_iφ_jdx=δ_ij. Thus, L²(Ω) =

u=

∞

X

k=1

ξ_kφ_k:

∞

X

k=1

ξ_k²<∞ , and

(u, v)_L2=

∞

X

k=1

ξ_kη_k, withu=P∞

k=1ξkφk,v =P∞

k=1ηkφk. For u∈L²(Ω), we define operator (−∆)^r/2 by

(−∆)^r/2u=

∞

X

k=1

λ^r/2_k ξkφk

with domain

D((−∆)^r/2) = Θ^r(Ω) =

∞

X

k=1

ξkφk:

∞

X

k=1

λ^r_kξ²_k<∞

for r ≥ 0. It is proved in [7] that Θ^r(Ω) = H₀^r(Ω) = H^r(Ω) if 0 < r < ¹₂, Θ^1/2(Ω) =H₀₀^1/2(Ω), Θ^r(Ω) =H₀^r(Ω) if ¹₂ < r≤1, and Θ^r(Ω) =H^r(Ω)∩H₀¹(Ω) if 1< r≤2. For r≥0, Θ^r(Ω) is a Hilbert space with inner product

(u, v)_Θr(Ω)= (u, v)_L2+ ((−∆)^r/2u,(−∆)^r/2v)_L2. Let

E^r(Ω) = Θ^r(Ω)×Θ^2−r(Ω), 0< r <2,

we choose r > 0 such that 2 < p+ 1 ≤ _N^2N_−2r and 2 < q+ 1 ≤ _N+2r−4^2N . By the Sobolev embedding, the inclusionE^r(Ω),→L^p+1(Ω)×L^q+1(Ω) is compact.

The quadratic formQλ(u, v) =R

Ω(∇u∇v−λuv)dx can be extended toE^r(Ω) since

Z

Ω

∇u∇v dx=

∞

X

k=1

λkξkηk=

∞

X

k=1

λ

r 2

kξkλ¹⁻

r 2

k ηk, it implies

| Z

Ω

∇u∇v dx| ≤ {

∞

X

k=1

λ^r_kξ_k²}^1/2{

∞

X

k=1

λ^2−r_k η²_k}^1/2=kukΘ^rkvk_Θ^2−r. A direct calculation shows that forz∈E^r(Ω),

Qλ(z) = 1

2(Lz, z)E^r, where

L=

0 (−∆)^1−r−λ(−∆)^−r

(−∆)^r−1−λ(−∆)^r−2 0

, (3.1)

which is a bounded and self-adjoint operator inE^r(Ω). In order to determine the spectrum ofL, we note thatE^r(Ω) is the direct sum of the spacesEk, k= 1,2, . . ., whereEk is the two-dimensional subspace ofE^r(Ω), spanned by (φk,0) and (0, φk).

An orthonormal basis ofEk is given by 1

√2(λ⁻

r 2

k φk,0), 1

√2(0, λ

r 2−1 k φk) .

EveryEk is invariant underL, and the restriction ofLonEkis given by the matrix L^k =

0 λ^1−r_k −λλ^−r_k λ^r−1_k −λλ^r−2_k 0

.

The eigenvalue of L^k is µ^±_k = ±(1−λλ⁻¹_k ). Therefore, µ⁺_k < 0 and µ⁻_k > 0 if k= 1, . . . , k0; while µ⁺_k >0 andµ⁻_k <0 ifk=k0+ 1, . . .. Furthermore,

µ^±_k → ±1 ask→ ∞.

Let H⁺(H⁻) be the subspace spanned by eigenvectors corresponding to positive (negative) eigenvalues ofL_k, then

E^r(Ω) =H⁺⊕H⁻.

BothH⁺ andH⁻ are infinite dimensional. Now we introduce an equivalent norm k · k∗ onE^r(Ω) by

1

2kzk²_∗= (Lz⁺, z⁺)−(Lz⁻, z⁻), wherez^±∈H^±. Then the functional corresponding to (1.1) is

I(z) =1

2(Lz, z)_Er(Ω)−Γ(z) forz= (u, v)∈E^r(Ω), where

Γ(z) = Z

Ω

F(x, v)dx+ Z

Ω

G(x, u)dx.

Lemma 3.1. The functional I satisfies the (PS) condition.

Proof. Let{z_n}be a (PS) sequence ofI inE^r(Ω), we need only to show that{z_n} is bounded. Since

M +εkznk ≥I(zn)−1

2hI⁰(zn), zni

≥(1 2 −1

γ)(

Z

Ω

|un||g(x, un)|dx+ Z

Ω

|vn||f(x, v_n)|dx)−C,

(3.2)

we have Z

Ω

|un||g(x, un)|dx+ Z

Ω

|vn||f(x, v_n)|dx≤C+εkznk. (3.3) We writez_n^±= (u^±_n, v^±_n), then

kz_n^±k²−εkz_n^±k ≤ |hLzn, z_n^±i −I⁰(z_n)z^±|

=|hΓ⁰(z_n), z_n^±i|

=| Z

Ω

g(x, un)u^±_n dx+ Z

Ω

f(x, vn)v_n^±dx|

≤ { Z

Ω

|g(x, un)|^p+1p }^p+1p ku^±_nk_Lp+1+{ Z

Ω

|f(x, vn)|^q+1q }^q+1q kv_n^±k_Lq+1

≤C{1 +{ Z

Ω

|g(x, un)||un|}^p+1p +{ Z

Ω

|f(x, vn)||vn|}^q+1q }kz^±_nkE^r

(3.4) Dividing (3.3) bykz^±_nkE^r, we obtain

kz_n^±kE^r ≤C{1 +{ Z

Ω

|g(x, un)||un|}^p+1p +{ Z

Ω

|f(x, vn)||vn|}^q+1q }. (3.5) It follows from (3.3) and (3.5) that

kz^±_nkE^r ≤C{1 +{C+εkznkE^r}^p+1p +{C+εkz_n^±kE^r}^q+1q }, (3.6) which implies thatkznkE^r is bounded. The proof is complete.

Proof of Theorem 1.3. The proof will be completed by verifying the conditions in Lemma 1.2. We denoteE¹=H⁺ and E²=H⁻, b(z) = Γ(z) and Lis defined by (3.1). Apparently, (I1) and (I2) of Lemma 1.2 hold. Now, we verify (I3).

Forρ >0, lets1> ρands2be positive constants to be specified later. Lete^± be the eigenvectors corresponding to the positive eigenvalue and negative eigenvalue of L¹ respectively and set [0, s₁e⁺] ={se⁺: 0≤s≤s₁},Q= [0, s₁e⁺]⊕( ¯B_s2∩H⁻), H˜ = span{e⁺} ⊕H⁻,S =∂B_ρ∩H⁺.

By assumption (B3), for anyε >0 there existsCε>0 such that G(x, u)≤εu²+C(ε)|u|^p+1, f(x, v)≤εv²+C(ε)|v|^q+1,∀u, v∈R, which implies

I(z⁺)≥(1

2−ε)kz⁺k²−C(ε)kz⁺k^p+1−C(ε)kz⁺k^q+1

for z⁺ ∈E⁺. Thus, we may fix ρ >0 and α >0 such that I(z)≥αonS. This proves (i) of (I3) in Lemma 1.2.

Next we show that for suitable choices ofs1 ands2,I(z)≤0 on∂Q. Note that the boundary ofQin ˜H consists of three parts, i.e,∂Q= {Q∩{s= 0}}∪{Q∩{s= s1}} ∪ {[0, s1e⁺]⊕(∂Bs₂∩H⁻)}. It is obvious that I(z)≤0 onQ∩ {s= 0}since I(z)≤0 for (u, v)≤H⁻ and Γ(z) is nonnegative. For the remaining parts of∂Q, we writez=z⁻+se⁺∈H˜, then

I(z) = 1 2s²−1

2kz⁻k²−Γ(z⁻+se⁺). (3.7) We may show as in [6] that

Γ(z⁻+se⁺)≥Cs^β−C1, (3.8) whereβ = min{p+ 1, q+ 1}. Therefore,

I(z⁻+se⁺)≤1

2s²−Cs^β+C1−1

2kz⁻k². (3.9) Chooses1 sufficient large such that

ψ(s) = 1

2s²−Cs^β+C1≤0 ∀s≥s1,

and then choose s₂ large such that s²₂ >2 max_s≥0ψ(s), then we getI(z) ≤0 on

∂Q. This proves (ii) of (I3) in Lemma 1.2. SinceS and ∂Qare link. The proof is

complete.

References

[1] P. Bartolo, V. Benci and D. Fortunato,Abstract Critical Point Theorems and Applications to Some Nonlinear Problem with ”Strong” resonance at infinity, Nonlin. Anal.7(1983), 981- 1012.

[2] V. Benci and P.H. Rabinowitz, Critical Point Theorem for Indefinite Functionals, Invent.

Math.52(1979), 241-273.

[3] G.B. Li, A. Suzukin,An asymptotically periodic Schro¨ınger equation with indefinite linear part, Comm. Contemp. Math.4(2002),763-776.

[4] G.B. Li, H.S. Zhou,The Existence of a Positive Solution to Asymptotically Linear Scalar Field Equations, Prc. R. Soc. Edinb. A130(2000),81-105.

[5] D G de Figueiredo, P.L.Felmer,On superquadratic elliptic systems, Trans. Amer. Math.

Soc343(1994), 99-116.

[6] J. Hulshof and R. van der Vorst,Differential Systems with Strongly Indefinite Variational Struture, J. Funct. Anal.114(1993),32-58.

[7] J.L. Lions and E. Magenes,Non-homogeneous Boundary Value Problems and Applications I, Springer Verlag. 1972.

[8] G.B. Li and J.F. Yang,Asymptotically Linear Elliptic Systems, Comm. P.D.E.29(2004),925- 954.

[9] P.H. Rabinowitz, Minimax Theorems and Applications to Partial Differential Equations, AMS Memoirs65(1986).

[10] C.A. Stuart, H.S. Zhou,Applying the Mountain Pass Theorem to Asymptotically linear El- liptic Equation OnR^N, Comm. P.D.E.24(1999),1731-1758.

[11] H.S. Zhou,An Application of a Mountain Pass Theorem, Acta. Mathematica Sinica. 18 (2002), 27-36.

Jianfu Yang

Department of Mathematics, Jiangxi Normal University, Nanchang, Jiangxi 330022, China

E-mail address:jfyang [email protected]

Ying Ye

Department of Mathematics, Jiangxi Normal University, Nanchang, Jiangxi 330022, China

E-mail address:[email protected]

Xiaohui Yu

China Institute for Advanced Study, Central University of Finance and Economics, Beijing 100081, China

E-mail address:yuxiao [email protected]