µ= (N−pp )p is the best constant of the Hardy inequality [4

ISSN: 1072-6691. URL: http://ejde.math.txstate.edu or http://ejde.math.unt.edu ftp ejde.math.txstate.edu

EXISTENCE OF SOLUTIONS FOR QUASILINEAR ELLIPTIC SYSTEMS INVOLVING CRITICAL EXPONENTS AND

HARDY TERMS

DENGFENG L ¨U

Abstract. Using variational methods, including the Ljusternik-Schnirelmann theory, we prove the existence of solutions for quasilinear elliptic systems with critical Sobolev exponents and Hardy terms.

1. Introduction and statement of main results We consider the critical quasilinear elliptic system

−∆pu−µ|u|^p−2u

|x|^p = 1

p^∗Fu(u, v) +Gu(u, v), x∈Ω,

−∆pv−µ|v|^p−2v

|x|^p = 1

p^∗F_v(u, v) +G_v(u, v), x∈Ω, u=v= 0, x∈∂Ω,

(1.1)

where Ω ⊂ R^N is a bounded domain with smooth boundary ∂Ω, 0 ∈ Ω, ∆pu= div(|∇u|^p−2∇u) is the p-Laplacian operator, N ≥p²,2 ≤p≤q < p^∗, p^∗ = _N^{N p}_−p denotes the Sobolev critical exponent, F, G ∈ C¹(R⁺×R⁺,R) are homogeneous functions of degrees p^∗ and q, respectively. R⁺ = [0,+∞), (Fu(u, v), Fv(u, v)) =

∇F,(Gu(u, v),Gv(u, v)) =∇G, 0≤µ <µ, ¯¯ µ= (^N−p_p )^p is the best constant of the Hardy inequality [4]:

¯ µ Z

Ω

|u|^p

|x|^pdx≤ Z

Ω

|∇u|^pdx,

for all u∈ W₀^1,p(Ω), where W₀^1,p(Ω) is defined as the completion of C₀^∞(Ω) with respect to the norm kuk = (R

Ω|∇u|^pdx)^1/p. For µ ∈ [0,µ), it follows from the¯ Hardy inequality that

kukµ=Z

Ω

|∇u|^p−µ|u|^p

|x|^pdx^1/p

2000Mathematics Subject Classification. 35J92, 35J50, 35B33.

Key words and phrases. Quasilinear elliptic system; variational method; critical exponent;

Hardy term; multiple solutions.

c

2013 Texas State University - San Marcos.

Submitted February 15, 2012. Published January 30, 2013.

1

defines a norm inW₀^1,p(Ω) equivalent to its usual norm. The best Sobolev constant is defined as

Sµ= inf

u∈D^1,p(R^N)\{0}

R

R^N(|∇u|^p−µ^|u|_|x|^pp)dx (R

R^N|u|^p∗dx)^p/p∗ , µ∈[0,µ).¯ (1.2) In recent years, much attention has been focused on singular problems involving both the Hardy potential and the Sobolev critical term. For example, see [7, 13, 16, 18, 19, 20, 23, 26] and the references therein. In [9], Ding and Xiao consider thep-Laplacian system

−∆pu= 2α

α+β|u|^α−2u|v|^β+λ|u|^q−2u, x∈Ω,

−∆pv= 2β

α+β|u|^α|v|^β−2v+δ|v|^q−2v, x∈Ω, u=v= 0, x∈∂Ω,

(1.3)

where p ≤q < p^∗, α, β > 1, α+β = p^∗. Using standard tools of the variational theory and the Ljusternik-Schnirelmann category theory, in [9] sufficient conditions onλ, δ are given for (1.3) to have at least cat_Ω(Ω) positive solutions. This result extended the result of Alves and Ding in [2] where the single equation case was studied. Hsu [17] obtained the existence of two positive solutions for (1.3) including a sublinear perturbation of 1< q < p < N. Recently, Shen and Zhang extended the results in [25] to a general class of homogeneous functions and obtained similar results. For similar problems, we refer the reader to [3, 6, 8, 10, 11, 12, 14, 15, 21, 22, 24] and the references therein.

In this paper, motivated by [2, 9, 17, 25], we shall extend these results to the case containing a general class of homogeneous nonlinearities and Hardy terms. To the best of our knowledge, problem (1.1) has not been considered before. Thus, it is necessary for us to investigate the related singular critical systems.

The following assumptions are used in this article:

(F0) F ∈C¹(R⁺×R⁺,R) andF(tu, tv) =t^p∗F(u, v)(t >0) holds for all (u, v)∈ R⁺×R⁺,

(F1) F_u(0,1) =F_v(1,0) = 0,

(F2) F_u(u, v)≥0, F_v(u, v)≥0 for allu, v≥0,

(F3) the 1-homogeneous function (u, v)7→F(u^p1∗, v^p1∗) is concave for all (u, v)∈ R⁺×R⁺.

(G0) Gisq-homogeneous for somep≤q < p^∗, (G1) Gu(0,1) =Gv(1,0) = 0.

To present our results, we define

λ= max{G(u, v) :u, v ≥0, u^q+v^q= 1}, (1.4) δ= min{G(u, v) :u, v≥0, u^q+v^q = 1}. (1.5) If Y is a closed subset of a topological space X, we denote, by catX(Y), the Ljusternik-Schnirelmann category of Y in X, namely the least number of closed and contractible sets in X which cover Y. We say that a weak solution (u, v) ∈ W₀^1,p(Ω)×W₀^1,p(Ω) of problem (1.1) is nonnegative ifu, v≥0 in Ω.

The main results of this paper are stated in the following two theorems whose conclusions are new (to the best of our knowledge).

Theorem 1.1. Suppose(F0)–(F3), (G0)–(G1)are satisfied, and one of the follow- ing two conditions holds:

(I) ¯p < q < p^∗ with p¯= maxn

p,_b(µ)^N ,^{p(2N−pb(µ)−p)}_N−p o

,0≤µ <µ¯ andλ, δ >0;

(II) q = p, 0 ≤ µ ≤ ^Np−1(N_pp^−p2) and λ, δ ∈ (0,¹_pΛ₁), where Λ₁ is the first eigenvalue of (−∆p, W₀^1,p(Ω)).

Then problem (1.1)has at least one nonnegative solution.

Theorem 1.2. Suppose(F0)–(F3), (G0)–(G1)are satisfied, and one of the follow- ing two conditions holds:

(I) ¯p < q < p^∗ with p¯= maxn

p,_b(µ)^N ,^{p(2N−pb(µ)−p)}_N−p o

,0≤µ <µ;¯ (II) q=p,0≤µ≤ ^Np−1_p^(Np^−p2).

Then there existsΛ>0 such that problem (1.1)has at least cat_Ω(Ω) distinct non- negative solutions forλ, δ∈(0,Λ).

Remark 1.3. Our Theorem 1.1 is a generalization of [16, Theorem 1.1] from quasi- linear elliptic equations to quasilinear elliptic systems.

Remark 1.4. Theorem 1 in [9] is the special case of our Theorem 1.2 corresponding toµ= 0, F(u, v) = 2|u|^α|v|^β, α+β =p^∗andG(u, v) =λ|u|^q+δ|v|^q. In this paper, different from [25], we can deal with F(u, v) which possesses both coupled and uncoupled terms. For example, let

F(u, v) =au^p∗+

k

X

i=1

b_iu^αiv^βi+cv^p∗,

wherea, bi, c≥0,αi, βi >1,αi+βi=p^∗. F(u, v) obviously satisfies (F0)–(F3).

This article is organized as follows. In Section 2, some notation and the mountain pass levels are established and Theorem 1.1 is proven. We present some technical lemmas which are crucial in the proof of Theorem 1.2 in Section 3. Theorem 1.2 is proven in Section 4.

2. Preliminaries and proof of Theorem 1.1

Throughout this paper,C, Ci will denote various positive constants whose exact values are not important. And → (respectively *) denotes strong (respectively weak) convergence. O(ε^t) denotes |O(ε^t)|/ε^t ≤ C, om(1) denotes om(1) → 0 as m → ∞. L^s(Ω), f or(1 ≤ s < +∞), denotes Lebesgue spaces, the norm L^s is denoted by | · |s for 1 ≤ s < +∞. Let Br(x) denote a ball centered at x with radiusr. The dual space of a Banach spaceE will be denoted by E⁻¹. We define the product space E :=W₀^1,p(Ω)×W₀^1,p(Ω) endowed with the norm k(u, v)kE =

kuk^p_µ+kvk^p_µ^1/p .

In view of (F1), (G1), we can extend the function F(u, v) and G(u, v) to the whole R² by consideringF(u, v) =F(u⁺, v⁺), G(u, v) = G(u⁺, v⁺), where u⁺ = max{u,0} and v⁺ = max{v,0}. It is easy to check that F(u, v) and G(u, v) ∈ C¹(R²). Therefore, we always considerF(u, v) andG(u, v) as these extensions.

A pair of functions (u, v)∈E is said to be a weak solution of problem (1.1) if Z

Ω

(|∇u|^p−2∇u∇ϕ1−µ|u|^p−2uϕ₁

|x|^p +|∇v|^p−2∇v∇ϕ2−µ|v|^p−2vϕ₂

|x|^p )dx

− 1 p^∗

Z

Ω

(Fu(u, v)ϕ1+Fv(u, v)ϕ2)dx− Z

Ω

(Gu(u, v)ϕ1+Gv(u, v)ϕ2)dx= 0, for all (ϕ₁, ϕ₂) ∈ E. Using (F0)-(G1) and well-known arguments, we know that the weak solutions of (1.1) are precisely the critical points of the C¹-functional Iλ,δ:E→Rgiven by

Iλ,δ(u, v)

= 1 p

Z

Ω

(|∇u|^p−µ|u|^p

|x|^p +|∇v|^p−µ|v|^p

|x|^p)dx− 1 p^∗

Z

Ω

F(u, v)dx− Z

Ω

G_λ,δ(u, v)dx.

We notice that, in the definition of Iλ,δ, we are denoting Gλ,δ(u, v) := G(u, v) for (u, v) ∈ R². We shall write Gλ,δ instead of G to emphasize that the main theorems depend on the value of the parametersλandδdefined in (1.4) and (1.5), respectively.

The functionalI∈C¹(E,R) is said to satisfy the (P S)ccondition if any sequence {zm} ⊂E such that as m→ ∞, I(zm)→c,I⁰(zm)→0 strongly inE⁻¹ contains a subsequence converging inE to a critical point ofI. In this paper, we will take I=Iλ,δ andE =W₀^1,p(Ω)×W₀^1,p(Ω).

In this section, we will find the range of c where the (P S)c condition holds for the functionalIλ,δ. First, let us define

SF = inf

(u,v)∈E\{(0,0)}

n R

Ω|∇u|^p−µ^|u|_|x|^p_p +|∇v|^p−µ^|v|_|x|^p_pdx (R

ΩF(u, v)dx)^p/p∗ : Z

Ω

F(u, v)dx >0o . (2.1) Lemma 2.1. Suppose(F0)–(F3), (G0)–(G1) are satisfied, then the functionalI_λ,δ satisfies the(P S)c condition for allc < _N¹S_F^N/p, provided eitherp < q < p^∗orq=p and the parameterλdefined in (1.4)belongs to(0,¹_pΛ1), whereΛ1>0 denotes the first eigenvalue of(−∆p, W₀^1,p(Ω)).

Proof. Let {(um, v_m)} ⊂ E such that I_λ,δ⁰ (u_m, v_m)→ 0 and I_λ,δ(u_m, v_m) → c <

1

NS_F^N/p. Now, we firstly prove that{(um, vm)}is bounded in E. Ifp < q < p^∗, it suffices to use the definition ofIλ,δ to obtainC1>0 such that

c+C1k(um, vm)kE+om(1)≥Iλ,δ(um, vm)−1

qhI_λ,δ⁰ (um, vm),(um, vm)i

=1 p−1

q

k(um, v_m)k^p_E+1 q− 1

p^∗ Z

Ω

F(u_m, v_m)dx

≥q−p

pq k(um, vm)k^p_E,

which implies that{(um, vm)} ⊂Eis bounded. Whenq=p, in this case, it follows that

Z

Ω

Gλ,δ(um, vm)dx≤λ Z

Ω

(|um|^p+|vm|^p)dx≤ λ

Λ₁k(um, vm)k^p_E, and therefore,

c+C₁k(u_m, v_m)k_E+o_m(1)≥I_λ,δ(u_m, v_m)− 1

p^∗hI_λ,δ⁰ (u_m, v_m),(u_m, v_m)i

= 1

Nk(um, vm)k^p_E− p N

Z

Ω

G(um, vm)dx

≥ 1 N

1−pλ

Λ₁

k(u_m, v_m)k^p_E.

Sincepλ <Λ₁, the boundedness of{(u_m, v_m)} follows as in the first case.

So {(u_m, v_m)} is bounded in E. Going if necessary to a subsequence, we can assume that

(um, vm)*(u, v), in E, (u_m, v_m)→(u, v), a.e. in Ω,

(um, vm)→(u, v), inL^s(Ω)×L^s(Ω),1≤s < p^∗, asm→ ∞. Clearly, we have that

Z

Ω

Gλ,δ(um, vm)dx= Z

Ω

Gλ,δ(u, v)dx+om(1). (2.2) Moreover, a standard argument shows thatI_λ,δ⁰ (u, v) = 0. Thus, we obtain

I_λ,δ(u, v) =1

pk(u, v)k^p_E− 1 p^∗

Z

Ω

F(u, v)dx− Z

Ω

G_λ,δ(u, v)dx

=1 p−1

q

k(u, v)k^p_E+1 q − 1

p^∗ Z

Ω

F(u, v)dx≥0.

(2.3)

Let (˜u_m,v˜_m) = (u_m−u, v_m−v). Then by the Brezis-Lieb Lemma [5], we have k(˜u_m,v˜_m)k^p_E=k(u_m, v_m)k^p_E− k(u, v)k^p_E+o_m(1). (2.4) By the same method as in [11, Lemma 8], we obtain

Z

Ω

F(um, vm)dx= Z

Ω

F(u, v)dx+ Z

Ω

F(˜um,˜vm)dx+om(1). (2.5) By (2.2),(2.3),(2.4),(2.5) and the weak convergence of (um, vm), we have

c+o_m(1) =I_λ,δ(u, v) +1

pk(˜u_m,v˜_m)k^p_E− 1 p^∗

Z

Ω

F(˜u_m,v˜_m)dx

≥ 1

pk(˜u_m,˜v_m)k^p_E− 1 p^∗

Z

Ω

F(˜u_m,˜v_m)dx.

(2.6)

Using thatI_λ,δ⁰ (u_m, v_m)→0 and (2.2), (2.4), (2.5), we obtain om(1) =hI_λ,δ⁰ (um, vm),(um, vm)i

=k(um, vm)k^p_E− Z

Ω

F(um, vm)dx−q Z

Ω

Gλ,δ(um, vm)dx

=hI_λ,δ⁰ (u, v),(u, v)i+k(˜u_m,˜v_m)k^p_E− Z

Ω

F(˜u_m,v˜_m)dx.

Recalling thatI_λ,δ⁰ (u, v) = 0, we can use the above equality and (2.6) to obtain

m→∞lim k(˜um,˜vm)k^p_E=k= lim

m→∞

Z

Ω

F(˜um,˜vm)dx, c≥1 p− 1

p∗

k= 1

Nk, wherek≥0.

In view of the definition ofSF, we deduce that k(˜um,v˜m)k^p_E≥SF

Z

Ω

F(˜um,˜vm)dx^p/p∗ .

Taking the limit, we obtaink≥S_Fk^p/p∗. So, ifk >0, we conclude thatk≥S_F^N/p and therefore

1

NS^N/p_F ≤ 1

Nk≤c < 1 NS_F^N/p,

which is a contradiction. Hencek= 0 and therefore (um, vm)→(u, v) inE.

For allµ∈[0,µ), we consider the limiting problem¯

−∆_pU −µU^p−1

|x|^p =U^p∗−1, inR^N \ {0}, U >0, inR^N\ {0},

U →0, as|x| →+∞.

(2.7)

From [1], we know that problem (2.7) has a ground stateUp,µ, which is unique up to scaling. That is, all ground states must be of the form

Vp,µ,ε(x) =ε^−N−pp Up,µ

x ε

=ε^−N−pp Up,µ

|x|

ε

, ε >0, (2.8) that satisfy

Z

R^N

(|∇Vp,µ,ε(x)|^p−µ|Vp,µ,ε(x)|^p

|x|^p )dx= Z

R^N

|Vp,µ,ε(x)|^p∗dx=S_µ^N/p, (2.9) whereSµ is the best Sobolev constant given in (1.2).

Moreover, the ground stateUp,µ is radially symmetric and decreasing, and the following asymptotic properties at the origin and infinity for Up,µ(r) andU_p,µ⁰ (r) hold:

lim

r→0⁺r^a(µ)Up,µ(r) =c1>0, lim

r→0⁺r^a(µ)+1|U_p,µ⁰ (r)|=c1a(µ)≥0,

r→+∞lim r^b(µ)Up,µ(r) =c2>0, lim

r→+∞r^b(µ)+1|U_p,µ⁰ (r)|=c2b(µ)>0,

where c1 and c2 are positive constants depending only on N, p, µ, and a(µ), b(µ), the zeros of the function h(t) = (p−1)t^p−(N−p)t^p−1+µ, t ≥0, which satisfy 0≤a(µ)< b(µ)≤ ^N−p_p−1.

After a direct calculation, we infer thatt_min= ^N−p_p is the unique minimal point of h(t) andh(^N_p^−p) =−¯µ+µ < 0. Moreover, h⁰(t)<0 for all 0< t < tmin and h⁰(t)>0 for all t > tmin. That is, h(t) is decreasing on the interval (0, tmin) and increasing on the interval (tmin,+∞). Thus 0≤a(µ)<^N_p^−p < b(µ).

In addition, using [11, Lemma 3] and the homogeneity ofF, we obtainA, B >0 such that

S_F = k(AVp,µ,ε, BVp,µ,εk^p_E (R

R^NF(AVp,µ,ε, BVp,µ,ε)dx)^p/p∗ = A^p+B^p

(F(A, B))^p/p∗ · Sµ^N/p

|Vp,µ,ε|^p_p∗

, from this and (2.9), we have

S_F = A^p+B^p

(F(A, B))^p/p∗S_µ. (2.10)

We define a cut-off functionφ(x)∈C₀^∞(R^N) such thatφ(x) = 1 if|x| ≤R;φ(x) = 0 if|x| ≥ 2R and 0 ≤φ(x)≤1, whereB2R(0) ⊂Ω and setuε = _|φV^{φ(x)Vp,µ,ε}

p,µ,ε|_p∗, where

Vp,µ,ε was defined in (2.8). So, |uε|p^∗ = 1. Thus, we can get the following results from [26, Lemma 2.2] (or [16]):

kuεk^p_µ=Sµ+O(ε^pb(µ)+p−N), (2.11) Z

Ω

|uε|^ξdx≈











ε^{(b(µ)−N−pp )ξ}, if 1≤ξ < _b(µ)^N , ε^{N−N−pp ξ}|lnε|, ifξ= _b(µ)^N , ε^{N−N−pp ξ}, if _b(µ)^N < ξ < p^∗,

(2.12)

whereA≈B meansC1B≤A≤C2B.

AsIλ,δis not bounded below onE, we need to studyIλ,δon the Nehari manifold:

Nλ,δ=

(u, v)∈E\ {(0,0)}:hI_λ,δ⁰ (u, v),(u, v)i= 0 .

Note thatN_λ,δ contains every nonzero solution of problem (1.1), and we define the minimaxc_λ,δ as

cλ,δ = inf

(u,v)∈Nλ,δ

Iλ,δ(u, v).

Next, we present some properties ofcλ,δandNλ,δ. Their proofs can be done as [27, Theorem 4.2]. First of all, we note that there existsρ >0 such that

k(u, v)kE≥ρ >0, ∀ (u, v)∈ Nλ,δ. (2.13) It is standard to check thatIλ,δ satisfies the mountain pass geometry, so we can use the homogeneity ofF and Gto prove thatc_λ,δ can be alternatively characterized by

c_λ,δ= inf

γ∈Γ max

t∈[0,1]I_λ,δ(γ(t)) = inf

(u,v)∈E\{(0,0)}max

t≥0 I_λ,δ(t(u, v))>0, (2.14) where Γ = {γ ∈ C([0,1], E) : γ(0) = 0, I_λ,δ(γ(1)) < 0}. Moreover, for each (u, v) ∈ E\{(0,0)}, there exists a unique t^∗ > 0 such that t^∗(u, v) ∈ Nλ,δ. The maximum of the functiont7→Iλ,δ(t(u, v)), fort≥0, is achieved att=t^∗.

Lemma 2.2. Suppose that (F₀)−(F₃) and (G₀)−(G₁) hold, p < q < p¯ ^∗ with

¯

p = maxn

p,_b(µ)^N ,p(2N−pb(µ)−p) N−p

o

, 0 ≤ µ < µ¯ and λ, δ defined in (1.4), (1.5) are positive, then cλ,δ < _N¹S_F^N/p. The same result holds if q=p,0≤µ≤ ^Np−1_p^(N−pp ²⁾

andλ, δ∈(0,¹_pΛ1).

Proof. We can use the homogeneity ofF andGto get, for any t≥0, I_λ,δ(tAu_ε, tBu_ε) =t^p

p(A^p+B^p)kuεk^p_µ−t^p∗

p^∗F(A, B)−t^qG_λ,δ(A, B)|uε|^q_q. We shall denote the right-hand side of the above equality byh(t) and consider two distinct cases.

Case 1: p < q < p¯ ^∗ with ¯p = maxn

p,_b(µ)^N ,p(2N−pb(µ)−p) N−p

o

. From the fact that lim_t→+∞h(t) =−∞andh(t)>0 whentis close to 0, there existst_ε>0 such that

h(tε) = max

t≥0 h(t). (2.15)

Let

g(t) = t^p

p(A^p+B^p)ku_εk^p_µ−t^p∗

p^∗F(A, B), t≥0,

and notice that the maximum value ofg(t) occurs at the point

˜t_ε=(A^p+B^p)kuεk^p_µ F(A, B)

_{p∗ −p}¹ . So, for eacht≥0,

g(t)≤g(˜t_ε) = 1 N

(A^p+B^p)kuεk^p_µ (F(A, B))^p/p∗

N/p

, and therefore

h(tε)≤ 1 N

(A^p+B^p)kuεk^p_µ (F(A, B))^p/p∗

^N/p

−t^q_εGλ,δ(A, B)|uε|^q_q. (2.16) We claim that, for someC₂>0, there holds

t^q_εGλ,δ(A, B)≥C2.

Indeed, if this is not the case, we have thatt_εm →0 for some sequenceε_m→0⁺, then

0< cλ,δ ≤sup

t≥0

Iλ,δ(tAuε_m, tBuε_m) =Iλ,δ(tε_mAuε_m, tε_mBuε_m)→0,

which is a contradiction. So, the claim holds, and we infer from (2.16), (2.10), (2.11) and (2.12) that

h(tε)≤ 1 N

A^p+B^p (F(A, B))^p/p∗

Sµ+O(ε^pb(µ)+p−N)N/p

−C2|uε|^q_q

≤ 1

NS_F^N/p+O(ε^pb(µ)+p−N)−C2|uε|^q_q

≤ 1

NS_F^N/p+O(ε^pb(µ)+p−N)−O(ε^{N−N−pp q}).

(2.17)

By ¯p < q < p^∗, we obtain pb(µ) +p−N > N − ^N−p_p q. Thus, from the above inequality we conclude that, for eachε >0 small, there holds

cλ,δ ≤sup

t≥0

Iλ,δ(tAuε, tBuε) =h(tε)< 1 NS^N/p_F .

Case 2: q=pand 0≤µ≤ ^Np−1_p^(N−pp ²⁾. In this case, we have thath⁰(t) = 0 if and only if

(A^p+B^p)kuεk^p_µ−pG_λ,δ(A, B)|uε|^p_p=t^p∗−pF(A, B).

Since we supposeλ < ¹_pΛ1, we can use Poincar´e inequality to obtain pGλ,δ(A, B)|uε|^p_p≤pλ(A^p+B^p)|uε|^p_p

<Λ₁(A^p+B^p)|u_ε|^p_p

≤(A^p+B^p)kuεk^p_µ. Thus, there existstε>0 satisfying (2.15).

Arguing, as in the first case, we conclude that, from (2.17), forε >0 small, there holds

h(tε)≤ 1

NS_F^N/p+O(ε^pb(µ)+p−N)−C2|uε|^p_p

= (₁

NS_F^N/p+O(ε^pb(µ)+p−N)−O(ε^p|lnε|), b(µ) = ^N_p,

1

NS_F^N/p+O(ε^pb(µ)+p−N)−O(ε^p), b(µ)> ^N_p.

Ifb(µ) =N/p, thenpb(µ) +p−N =p, so ε^pb(µ)+p−N =o(ε^p|lnε|). Ifb(µ)> N/p, thenpb(µ) +p−N > p, soε^pb(µ)+p−N =o(ε^p). Choosingε >0 small enough, we have

cλ,δ ≤sup

t≥0

Iλ,δ(tAuε, tBuε) =h(tε)< 1 NS^N/p_F . On the other hand, it is easy to verify that the function

g(t) = (p−1)t^p−(N−p)t^p−1+µ, t≥0

has the only minimal point ¯t = ^N_p^−p and is increasing on the interval (¯t,+∞).

Thus, forN ≥p² we deduce that N

p ≤b(µ)⇔g(N

p)≤g(b(µ)) = 0⇔0≤µ≤ N^p−1(N−p²)

p^p .

This concludes the proof.

Using Lemmas 2.1 and 2.2, we can prove our first result.

Proof of Theorem 1.1. SinceI_λ,δ satisfies the geometric conditions of the mountain pass theorem, there exists {(u_m, v_m)} ⊂ E such that I_λ,δ(u_m, v_m) → c_λ,δ, and I_λ,δ⁰ (um, vm)→0. It follows from Lemmas 2.1 and 2.2 that{(um, vm)} converges, along a subsequence, to a nonzero critical point (u, v) ∈ E of Iλ,δ. If we then denote, byu⁻ = max{−u,0} andv⁻= max{−v,0}, the negative part ofuandv, respectively, we obtain

0 =hI_λ,δ⁰ (u, v),(u⁻, v⁻)i

=−k(u⁻, v⁻)k^p_E− 1 p^∗

Z

Ω

(F_u(u, v)u⁻+F_v(u, v)v⁻)dx

− Z

Ω

(G_u(u, v)u⁻+G_v(u, v)v⁻)dx

≤ −k(u⁻, v⁻)k^p_E.

It thus follows that (u⁻, v⁻) = (0,0). Hence, u, v ≥ 0 in Ω. The theorem 1.1 is

thus proven.

We finalize this section with the study of the asymptotic behavior of the minimax levelcλ,δ as both the parametersλ, δ approach zero.

Lemma 2.3. lim_λ,δ→0+cλ,δ =c0,0= _N¹S_F^N/p.

Proof. We first prove the second equality. It follows fromλ=δ= 0 thatG_0,0≡0.

IfA, B, uε, gεandtεare the same as those in the proof of Lemma 2.2, we have that (tεAuε, tεBuε)∈ N0,0. Thus

c0,0≤I0,0(tεAuε, tεBuε)

= 1 N

A^p+B^p

(F(A, B))^p/p∗kuεk^p_µ^N/p

= 1 N

A^p+B^p (F(A, B))^p/p∗

Sµ+O ε^pb(µ)+p−NN/p

.

Taking the limit asε→0⁺ and using (2.10), we conclude thatc_0,0≤_N¹S_F^N/p.

In order to obtain the reverse inequality, we consider{(um, vm)} ⊂E such that I0,0(um, vm) → c0,0 and I_0,0⁰ (um, vm) → 0. It is easy to show that the sequence {(um, vm)} is bounded inE and therefore

hI_0,0⁰ (um, vm),(um, vm)i=k(um, vm)k^p_E− Z

Ω

F(um, vm)dx=om(1).

It follows that

m→∞lim k(um, vm)k^p_E=l= lim

m→∞

Z

Ω

F(um, vm)dx.

Taking the limit in the inequalitySF(R

ΩF(um, vm)dx)^p/p∗≤ k(um, vm)k^p_E, we con- clude, as in the proof of Lemma 2.1, thatN c0,0=l≥S_F^N/p. Hence,

c0,0= lim

m→∞I0,0(um, vm) = lim

m→∞

1

pk(um, vm)k^p_E− 1 p^∗

Z

Ω

F(um, vm)dx

= 1 Nl≥ 1

NS_F^N/p, and thereforec0,0=_N¹S_F^N/p.

We proceed now to the calculation of lim_λ,δ→0+c_λ,δ. Let {λm},{δm} ⊂ R⁺ such that λm, δm → 0⁺. Since δm, defined in (1.5), is positive, we have that Gλ_m,δ_m(u, v)≥0 whenever (u, v) is nonnegative. Thus, for this kind of function, we have thatIλ_m,δ_m(u, v)≤I0,0(u, v). It follows that

cλ_m,δ_m = inf

(u,v)6=(0,0)max

t≥0 Iλ_m,δ_m(t(u, v))

≤ inf

(u,v)6=(0,0),(u,v)≥0max

t≥0 Iλ_m,δ_m(t(u, v))

≤ inf

(u,v)6=(0,0),(u,v)≥0max

t≥0 I_0,0(t(u, v)) =c_0,0.

In the last equality above, we used the infimum c_0,0, which can be attained at a nonnegative solution. The above inequality implies that

lim sup

m→∞

cλ_m,δ_m ≤c0,0. (2.18)

On the other hand, it follows from Theorem 1.1 that there exists{(u_m, v_m)} ⊂E such that

I_λm,δm(u_m, v_m) =c_λm,δm, I_λ⁰

m,δ_m(u_m, v_m)→0.

Since cλ_m,δ_m is bounded, the same argument performed in the proof of Lemma 2.1 implies that{(um, vm)} is bounded inE. Since (um, vm)≥0, we obtain 0≤ R

ΩG_λm,δm(u_m, v_m)dx≤λ_mR

Ω(|um|^q+|vm|^q)dx, from which it follows that

m→∞lim Z

Ω

G_λm,δm(u_m, v_m)dx= 0. (2.19) Lettm>0 be such thattm(um, vm)∈ N0,0. Since (um, vm)∈ Nλ_m,δ_m, we have that

c_0,0≤I_0,0(t_m(u_m, v_m))

=Iλ_m,δ_m(tm(um, vm)) +t^q_m Z

Ω

Gλ_m,δ_m(um, vm)dx

≤Iλ_m,δ_m(um, vm) +t^q_m Z

Ω

Gλ_m,δ_m(um, vm)dx

=cλ_m,δ_m+t^q_m Z

Ω

Gλ_m,δ_m(um, vm)dx.

If{tm} is bounded, we can use the above estimate and (2.19) to obtain c_0,0≤lim inf

m→∞ c_λm,δm. Using this and (2.18), we obtain

c0,0≤lim inf

m→∞ cλm,δm ≤lim sup

m→∞

cλm,δm ≤c0,0. Thus,c0,0= lim_m→∞cλ_m,δ_m.

It remains to check that{tm} is bounded. A straightforward calculation shows that

t_m= k(um, vm)k^p_E R

ΩF(um, vm)dx _{p∗ −p}¹

. (2.20)

Since (um, vm)∈ Nλm,δm, we obtain k(um, v_m)k^p_E

= Z

Ω

F(um, vm)dx+q Z

Ω

Gλ_m,δ_m(um, vm)dx≤S⁻

p∗ p

F k(um, vm)k^p_E^∗+om(1).

Hence k(um, vm)k^p_E ≥C3>0, and therefore from the above expression, it follows that R

ΩF(um, vm)dx ≥C4 >0. Thus, the boundedness of{(um, vm)} and (2.20) imply that{tm}is bounded. This completes the proof.

3. Some technical lemmas

In this section, we will recall and prove some lemmas which are crucial in the proof of Theorem 1.2. The first lemma is standard, and its proof follows adapting arguments found in [27].

Lemma 3.1. Let {(um, vm)} ⊂E such thatR

ΩF(um, vm)dx= 1 and

m→∞lim k(um, vm)k^p_E=SF. Then there exist{rm} ⊂(0,+∞) and{ym} ⊂R^N such that

ωm(x) = (ω¹_m(x), ω_m²(x)) =r

N−p

mp (um(rmx+ym), vm(rmx+ym)) (3.1) contains a convergent subsequence, denoted again by {ωm}, such that ωm→ω in D^1,p(R^N)× D^1,p(R^N). Moreover, asm→ ∞, we haver_m→0 andy_m→y∈Ω.

Up to translations, we may assume that 0 ∈Ω. Since Ω is a smooth bounded domain of R^N, we can chooser > 0 small enough such that Br =Br(0) = {x∈ R^N :d(x,0)< r} ⊂Ω and the sets

Ω⁺_r ={x∈R^N : dist(x,Ω)< r}, Ω⁻_r ={x∈R^N : dist(x, ∂Ω)> r}

are homotopically equivalent to Ω. Let W_0,rad^1,p (Br) =

u∈W₀^1,p(Br) :uis radial , Erad(Br) =W_0,rad^1,p (Br)×W_0,rad^1,p (Br).

We thus define the functional I_Br(u, v) = 1

p Z

B_r

(|∇u|^p−µ|u|^p

|x|^p +|∇v|^p−µ|v|^p

|x|^p)dx

− 1 p^∗

Z

Br

F(u, v)dx− Z

Br

Gλ,δ(u, v)dx for (u, v)∈Erad(Br), and set

m_λ,δ= inf

(u,v)∈N_λ,δ^Br

I_Br(u, v), where

N_λ,δ^Br :={(u, v)∈E_rad(B_r)\ {(0,0)}:hI_B⁰_r(u, v),(u, v)i= 0}.

Clearly,m_λ,δ is nonincreasing in λ, δ. Note thatm_λ,δ>0 for allλ, δ >0.

Arguing, as in the proof of Lemma 2.3 and Theorem 1.1, we obtain the following result.

Lemma 3.2. Suppose(F0)-(F3), (G0)–(G1)are satisfied. Then the infimum m_λ,δ is attained by a positive radial function (u_λ,δ, v_λ,δ) ∈ E_rad whenever p < q < p¯ ^∗ with p¯= maxn

p,_b(µ)^N ,p(2N−pb(µ)−p) N−p

o

,0≤µ <µ¯ andλ, δ >0, or q=p,0≤µ≤

N^p−1(N−p²)

p^p and λ, δ∈(0,¹_pΛ_1,rad), and where Λ_1,rad >0 is the first eigenvalue of the operator (−∆pu, W_0,rad^1,p (Br)). Moreover,

m_λ,δ < 1

NS_F^N/p, lim

λ,δ→0⁺m_λ,δ= 1 NS_F^N/p. We introduce the barycenter mapβ:Nλ,δ→R^N as

β(u, v) =S_F^−N/p Z

Ω

F(u, v)x dx.

This map has the following property.

Lemma 3.3. If(F0)–(F3), (G0)–(G1), then there existsλ^∗>0such thatβ(u, v)∈ Ω⁺_r whenever(u, v)∈ Nλ,δ, λ, δ∈(0, λ^∗) andIλ,δ(u, v)≤mλ,δ.

Proof. Arguing by contradiction, we suppose that there exist {λm},{δm} ⊂ R⁺ and {(um, vm)} ⊂ Nλ_m,δ_m such that λm, δm → 0⁺ as m → ∞, Iλ_m,δ_m(um, vm)≤ mλ_m,δ_m, butβ(um, vm)6∈Ω⁺_r.

From {(um, vm)} ⊂ Nλm,δm and Iλm,δm(um, vm) ≤ mλm,δm, it follows that {(um, v_m)} is bounded inE. Moreover,

0 =hI_λ⁰_m,δm(um, vm),(um, vm)i

=k(um, vm)k^p_E− Z

Ω

F(um, vm)dx−q Z

Ω

Gλm,δm(um, vm)dx.

Sinceλm→0, we can use the boundedness of {(um, vm)}to get 0≤

Z

Ω

Gλ_m,δ_m(um, vm)dx≤λm

Z

Ω

(|um|^q+|vm|^q)dx→0, from which it follows that

m→∞lim k(um, vm)k^p_E= lim

m→∞

Z

Ω

F(um, vm)dx=k≥0.

Notice that

c_λm,δm ≤I_λm,δm(u_m, v_m)

=1

pk(um, vm)k^p_E− 1 p^∗

Z

Ω

F(um, vm)dx− Z

Ω

Gλ_m,δ_m(um, vm)dx

≤mλ_m,δ_m.

Recalling thatc_λm,δm andm_λm,δm both converge to _N¹S_F^N/p, we can use the above expression and R

ΩGλ_m,δ_m(um, vm)dx→0 again to conclude thatk=S_F^N/p. That is,

m→∞lim k(um, v_m)k^p_E =S_F^N/p= lim

m→∞

Z

Ω

F(u_m, v_m)dx. (3.2) Let t_m = (R

ΩF(u_m, v_m)dx)^−1/p∗ > 0 and notice that t_m(u_m, v_m) satisfies the hypotheses of Lemma 3.1. Using Lemma 3.1, there exist sequences{r_m} ⊂(0,+∞) and {ym} ⊂ R^N satisfyingrm→ 0, ym →y ∈ Ω. We thus have that ωm →ω in D^1,p(R^N)× D^1,p(R^N).

The definition ofβ(u, v), (3.2), the strong convergence of{ωm}, and Lebesgue’s Theorem provide

β(um, vm) =t^−p_m^∗S_F^−N/p Z

Ω

F(tm(um, vm))xdx

= (1 +om(1)) Z

Ω

F(tmum, tmvm)xdx

= (1 +om(1)) Z

Ω

F(ωm)(rmx+ym)dx

= (1 +om(1))Z

Ω

F(ω)¯ydx+om(1) . Since ¯y∈Ω andR

ΩF(ω)dx= 1, the above expression implies that

m→∞lim dist (β(um, vm),Ω) = 0.

Such contradictsβ(um, vm)6∈Ω⁺_r.

According to Lemma 3.2, for each λ, δ >0 small, the infimum mλ,δ is attained by a nonnegative radial functionσ_λ,δ= (u_λ,δ, v_λ,δ)∈ N_λ,δ^Br. We consider

I_λ,δ^mλ,δ={(u, v)∈E:I(u, v)≤m_λ,δ}

and define the functionγ: Ω⁻_r →I_λ,δ^mλ,δ by setting, for each y∈Ω⁻_r, γ(y) =

(σλ,δ(x−y), ifx∈Br(y),

0, otherwise. (3.3)

A change of variables and straightforward calculations show that the mapγis well defined. Sinceσ_λ,δis radial, we have thatR

BrF(u_λ,δ, v_λ,δ)xdx= 0. Hence, for each y∈Ω⁻_r, we obtain

(β◦γ)(y) =S^−N/p_F Z

Ω

F(u_λ,δ(x−y), v_λ,δ(x−y))xdx

=S^−N/p_F Z

Ω

F(u_λ,δ(t), v_λ,δ(t))(t+y)dt

=S^−N/p_F Z

Ω

F(u_λ,δ(t), v_λ,δ(t))ydt=yα_λ,δ, whereα_λ,δ=S_F^−N/pR

ΩF(u_λ,δ(t), v_λ,δ(t))dt.

Along the way of proving Lemma 3.3, we can check easily the following.

Lemma 3.4. If λ, δ→0⁺, thenαλ,δ→1.

Proof. By Lemma 3.2, we have m_λ,δ= 1

p Z

B_r

|∇uλ,δ|^p+|∇vλ,δ|^p−µ|u_λ,δ|^p+|v_λ,δ|^p

|x|^p

dx

− 1 p^∗

Z

Br

F(uλ,δ, vλ,δ)dx− Z

Br

Gλ,δ(uλ,δ, vλ,δ)dx

< 1 NS^N/p_F . As before, R

BrG_λ,δ(u_λ,δ, v_λ,δ)dx → 0. Thus, I_B⁰

r(u_λ,δ, v_λ,δ) = 0, and the above expression and the same arguments used in the proof of Lemma 3.2 imply that

Z

Ω

F(uλ,δ, vλ,δ)dx→S^N/p_F .

The above equality and the definition of α_λ,δ imply thatα_λ,δ →1. The lemma is

thus proven.

Next we defineHλ,δ: [0,1]×(Nλ,δ∩I_λ,δ^mλ,δ)→R^N by H_λ,δ(t,(u, v)) =

t+1−t αλ,δ

β(u, v).

Lemma 3.5. Suppose(F0)–(F3), (G0)–(G1)are satisfied. There then existsλ^∗∗>

0 such that

H_λ,δ [0,1]×(N_λ,δ∩I_λ,δ^mλ,δ)

⊂Ω⁺_r (3.4)

for allλ, δ∈(0, λ^∗∗).

Proof. Arguing by contradiction, we suppose that there exist sequences {λm}, {δ_m} ⊂ R⁺ and t_m ∈ [0,1],(u_m, v_m) ∈ (N_λ,δ∩I_λ,δ^mλ,δ) such that λ_m, δ_m → 0⁺ as m → ∞ and H_λm,δm(t_m,(u_m, v_m)) 6∈ Ω⁺_r for all m, up to a subsequence t_m → t₀ ∈ [0,1]. Moreover, the compactness of Ω and Lemma 3.3 imply that, up to a subsequence,β(um, vm)→y∈Ω. From Lemma 3.4αλ_m,δ_m →1, so we can use the definition ofHλ,δ to conclude that Hλ_m,δ_m(tm,(um, vm))→y ∈Ω, which

is a contradiction. The lemma is proven.

4. Proof of Theorem 1.2 We begin with the following lemma.

Lemma 4.1. If (u, v)is a critical point of Iλ,δ on Nλ,δ, then it is a critical point of Iλ,δ inE.

Proof. The proof is almost the same as [22, Lemma 3.2] and is thus omitted here.

Lemma 4.2. Suppose (F0)–(F3), (G0)–(G1) are satisfied. Then any sequence {(um, vm)} ⊂ Nλ,δ such that Iλ,δ(um, vm) → c < _N¹S_F^N/p and I_λ,δ⁰ (um, vm) → 0 contains a convergent subsequence for λ, δ >0 if q > p andλ, δ ∈(0, λ^∗) if q=p for some small λ^∗>0.