We also discuss the cases where the Nehari set is not a mani- fold

ISSN: 1072-6691. URL: http://ejde.math.txstate.edu or http://ejde.math.unt.edu

MULTIPLICITY OF POSITIVE SOLUTIONS FOR A GRADIENT TYPE COOPERATIVE/COMPETITIVE ELLIPTIC SYSTEM

KAYE SILVA, STEFF ˆANIO MORENO SOUSA

Abstract. We study the existence of positive solutions for gradient type co- operative, competitive elliptic systems, which depends on real parametersλ, µ.

Our analysis is purely variational and depends on finer estimates with respect to the Nehari sets, in fact, we determine the extremal parameterλ^∗(µ) for which the Nehari set is a manifold and hence standard variational techniques can be applied. We also discuss the cases where the Nehari set is not a mani- fold.

1. Introduction

In this work we study the gradient type cooperative or competitive elliptic system

−∆u=µu+λv+f(x)|u|^p−2u in Ω,

−∆v=λu+µv−g(x)|v|^q−2v in Ω, u=v= 0 on∂Ω,

(1.1)

whereλ, µare real parameters, Ω is a bounded domain inR^N with smooth boundary

∂Ω, N ≥ 1, 2 < q < p < 2^∗, 2^∗ = 2N/(N −2). We look for weak solutions in X :=H₀¹(Ω)×H₀¹(Ω).

Gradient type systems and cooperative or competitive type systems have been studied by many authors: see for example the works of deFigueiredo [8], Cl´ement et al. [5], Alves et al. [1], Bozhkov and Mitidieri [3], Wenming [14], Costa and Magalh˜aes [6], da Silva [7] and the references therein. Such systems appear in many phenomena in Physics, Chemistry, Biology, etc. (see for example Brown [4] and the references therein), in particular they are related to reaction-diffusion systems that appear in chemical and biological phenomena.

Our plan is to study system (1.1) with respect to the parametersλ, µonly by a variational method. We will provide a relation between the parameters λ, µ with some topological properties of the Nehari set and the existence of solutions to problem (1.1). From now on, a solution to (1.1) is a critical point to the energy functional Φλ,µ:X →Rwhich is defined by

Φλ,µ(u, v) = 1 2

Z

|∇u|²+ Z

|∇v|²

−µ 2

Z

|u|²+ Z

|v|²

2010Mathematics Subject Classification. 35A02, 35A15, 35B32.

Key words and phrases. Gradient systems; cooperative/competitive systems;

Nehari manifold; variational methods; extremal parameter.

c

2020 Texas State University.

Submitted December 21, 2019. Published January 23, 2020.

1

−λ Z

uv+1 q

Z

g|v|^q−1 p Z

f|u|^p.

Depending on the values of the parameters λ, µ, the energy functional Φλ,µ is unbounded from below and from above. This kind of behavior is similar to that of indefinite problems (see Berestycki et al. [2]) and we should expect multiplicity of solutions (see Ouyang [11]). In fact, by analyzing the Nehari set associated to Φλ,µ:

Nλ,µ={(u, v)∈ X : Φ⁰_λ,µ(u, v)(u, v) = 0},

one is lead to the conclusion that for some parameters, the Nehari setNλ,µis in fact a manifold which split in two disjoint setsN_λ,µ⁺ ,N_λ,µ⁻ satisfyingN⁺λ,µ∩ N_λ,µ⁻ =∅ andN_λ,µ⁺ ∩ N⁻λ,µ=∅, where

N_λ,µ⁺ ={(u, v)∈ X : Φ⁰⁰_λ,µ(u, v)(u, v)²>0}, N_λ,µ⁻ ={(u, v)∈ X : Φ⁰⁰_λ,µ(u, v)(u, v)²<0}.

This suggest multiplicity of solutions, more precisely, the existence of critical points to Φλ,µin each manifoldN_λ,µ⁺ andN_λ,µ⁻ . In fact, let (λ1, φ1) be the first eigenpair of

−∆ in Ω with Dirichlet boundary conditions. We consider the following hypothesis onf andg:

(H1) f, g∈L^∞(Ω) and g(x)>0 a.e. x∈Ω,f(x)≥0 a.e.x∈Ω andf 6= 0.

Our main result reads as follows.

Theorem 1.1. Assume (H1) and that µ < λ1. Then there exists 0 < λ1(µ) <

λ^∗(µ)<∞such that

(1) For each λ∈ (−∞, λ^∗(µ)] problem (1.1)has at least one positive solution (¯u_λ,µ,¯v_λ,µ)∈ N_λ,µ⁻ .

(2) For eachλ∈(λ₁(µ), λ^∗(µ)]problem (1.1)has at least one positive solution (u_λ,µ, v_λ,µ)∈ N_λ,µ⁺ .

By a positive solution we mean that both coordinates are positive functions. The parameters λ₁(µ), λ^∗(µ) which appears in Theorem 1.1 are the so-called extremal parameters (see Il’yasov [9]) and they describe the topological changes on the Nehari set with respect to λ, µ. In fact, if λ < λ₁(µ) we have that N_λ,µ⁺ = ∅, while if λ≥λ^∗(µ), thenNλ,µis no longer aC¹ manifold. They can be found through the study of the so-called nonlinear Rayleigh quotient

R_µ(u, v) :=

R(|∇u|²+|∇v|²)−µR

(|u|²+|v|²)

Ruv +

Rg|v|^q−R f|u|^p R uv . Since Nλ,µ is no longer a manifold when λ≥λ^∗(µ), the technique used to prove Theorem 1.1 can not be used to prove existence of solutions in this case, therefore, we need a finer analysis over the Nehari sets. In this work we deal only with the case whereNλ,µ is a manifold (and its limiting case), although some recent works of Il’yasov and Silva [10], Silva and Macedo [13] suggests multiplicity of solutions forλ > λ^∗(µ).

This article is organized as follows: in Section 2 we study the fiber maps associ- ated to Φλ,µ and the extremal parameters. In Section 3 analyze some topological properties of the energy functional. In Section 4 we show existence of two positive solutions to equation (1.1).

2. Non-linear generalized Rayleigh quotient and extremal parameters

In this Section we establish some notation and technical results which will be used throughout the paper. In particular we study the Nehari set and its decomposition and we analyze the values of the parametersλ, µfor whichNλ,µ is a manifold.

From now on we denotew:= (u, v)∈ X. We equipX with the norm kwk=Z

|∇u|²+ Z

|∇v|^21/2 .

Ifw∈ X, we denote

kwk²₂=kuk²₂+kvk²₂. For (λ, µ)∈R², we recall the definition of the Nehari set

Nλ,µ={w∈ X \ {0}: Φ⁰_λ,µ(w)w= 0}.

Note that the Nehari set can be written as

Nλ,µ=N_λ,µ⁺ ∪ N_λ,µ⁰ ∪ N_λ,µ⁻ , where

N_λ,µ⁺ ={w∈ X \ {0}: Φ⁰_λ,µ(w)w= 0, Φ⁰⁰_λ,µ(w)w²>0}, N_λ,µ⁰ ={w∈ X \ {0}: Φ⁰_λ,µ(w)w= 0, Φ⁰⁰_λ,µ(w)w²= 0}, N_λ,µ⁻ ={w∈ X \ {0}: Φ⁰_λ,µ(w)w= 0, Φ⁰⁰_λ,µ(w)w²<0}.

Lemma 2.1. IfN_λ,µ⁺ ,N_λ,µ⁻ are nonempty sets thenN_λ,µ⁺ ,N_λ,µ⁻ areC¹manifolds of codimension1inX. Moreover,w∈ N_λ,µ⁺ ∪N_λ,µ⁻ is a critical point of(Φλ,µ)_|N+

λ,µ∪N_λ,µ⁻

if and only if wis a critical of Φ_λ,µ.

Since all critical points of Φλ,µbelongs toNλ,µ, in order to find critical points to Φ_λ,µ in X, we restrict our attention to critical points of Φ_λ,µ over Nλ,µ, however, to apply Lemma 2.1 we need to understand the Nehari setsN_λ,µ⁺ ∪ N_λ,µ⁻ andN_λ,µ⁰ . In fact, whenN_λ,µ⁺ ∪ N_λ,µ⁻ 6=∅andN_λ,µ⁰ =∅it is easy to show existence of solutions to problem (1.1), however, whenN_λ,µ⁰ we have to provide a more finer analysis over the Nehari sets.

Forλ, µ∈Randw∈ X we introduce

Hλ,µ(w) =kwk²−µkwk²₂−2λ Z

uv.

First, let us characterize the Nehari set by using the Fibering Method of Po- hozaev (see [12]): for eachw∈ X \ {0}, defineψ_λ,µ,w: [0,∞)→Rbyψ_λ,µ,w(t) = Φ_λ,µ(tw).

Proposition 2.2. For each λ, µ∈ R and w∈ X \ {0}, the function ψλ,µ,w is of class C^∞ on (0,∞). Moreover, the only three cases where ψλ,µ,w has a critical point are:

Case 1: H_λ,µ(w)>0.

(i) There is only one critical point at t⁻_λ(w)∈(0,∞), and this point satisfies ψ⁰⁰_λ,w(t⁻_λ,µ(w))<0 if only if R

f|u|^p>0;

Case 2: Hλ,µ(w) = 0.

(ii) ψ_λ,µ,w is constant equal to zero if and only if R

f|u|^p,R

g|v|^q= 0;

(iii) There is only one critical point at t⁻_λ(w)∈ (0,∞) and this point satisfies ψ⁰⁰_λ,w(t⁻_λ,µ(w))<0 if only if R

f|u|^p,R

g|v|^q>0;

Case 3: H_λ,µ(w)<0.

(I) if R

f|u|^p = 0 and R

g|v|^q > 0 then there is only one critical point at t⁺_λ,µ(w)∈(0,∞)which satisfiesψ⁰⁰_λ,µ,w(t⁻_λ,µ(w))>0;

If R

f|u|^p>0 andR

g|v|^q >0 there are two possibilities:

(II) There are only two critical points for ψλ,w. One critical point at t⁻_λ,µ(w) withψ⁰⁰_λ,µ,w(t⁻_λ,µ(w))<0and the other one att⁺_λ,µ(w)withψ⁰⁰_λ,µ,w(t⁺_λ,µ(w))>

0. Moreover ψλ,µ,w is decreasing over the intervals [0, t⁺_λ,µ(w)],[t⁻_λ(w),∞]

and increasing over the interval[t⁺_λµ(w), t⁻_λ,µ(w)];

(III) The functionψλ,µ,w has only one critical point which is an inflection point att⁰_λ,µ(w). Moreover,ψλ,µ,w is decreasing;

We start with the study ofN_λ,µ⁺ . Observe from Proposition 2.2 that ifN_λ,µ⁺ 6=∅ then there exist (λ, µ)∈R²andw∈ X such thatH_λ(w)<0 or equivalently

R|∇u|²+R

|∇v|²−µ R

|u|²+R

|v|² 2R

uv < λ,

therefore we are led to the study of the function λ_min(µ;w) := kwk²−µkwk²₂

2R

uv , w∈ X, Z

uv >0. (2.1) Now we turn our attention to the Nehari setN_λ,µ⁰ . From Proposition 2.2 we have that ifw∈ N_λ,µ⁰ , then

Z

f|u|^p>0, and Z

uv >0.

Let us introduce the set (the open subset ofX) X+ :=

w∈ X : Z

f|u|^p>0, Z

uv >0 ,

soN_λ,µ⁰ ⊂ X+. For eachw∈ X+, consider the corresponding so-called scalar fibered Rayleigh quotient (see Il’yasov [9])

Rµ(tw) =

R(|∇u|²+|∇v|²)dx−µR

(|u|²+|v|²)dx Ruv dx

+t^q−2R

g|v|^qdx−t^p−2R

f|u|^pdx

Ruv dx .

As was shown in [9] we have that

w∈ N_λ,µ⁰ , if and only if Rµ(w) =λ, d

dtR(tw)_|t=1= 0.

and hence the extremal values of [0,∞)3t7→Rµ(tw) provide regions of parameters where N_λ,µ⁰ =∅. Assume thatµ < λ1 and observe that [0,∞)3t7→Rµ(tw) has two extremal values. The first one is a local minimum attained att= 0, indeed, it corresponds toλmin(µ;w) as defined in (2.1): one can easily see that

λmin(µ;w)≥1− µ

λ₁ >0, ∀µ < λ1, ∀w∈ X+.

The second one corresponds to a local maximum which can be computed by using standard calculus in the following way:

d

dtRµ(tw) = (q−2)t^q−3R

g|v|^qdx−(p−2)t^p−3R

f|u|^pdx

Ruv dx = 0, t >0,

if and only if

(q−2) Z

g|v|^qdx−(p−2)t^p−q Z

f|u|^γdx= 0, and hence

t0(w) =(q−2)R

g|v|^qdx (p−2)R

f|u|^pdx _p−q¹

(2.2) is the critical point of [0,∞)3t7→R_µ(tw) which corresponds to a global maximum.

Therefore we have the nonlinear generalized Rayleigh quotient λmax(µ;w) := max

t≥0 Rµ(tw) = 1 2R

uv

kwk²−µkwk²₂+Cp,q

R g|v|^qdx^p−2p−q

Rf|u|^pdx^q−2_p−q

,

whereCp,q>0 is given by

Cp,q =q−2 p−2

^q−2p−q

−q−2 p−2

^p−2p−q

.

Remark 2.3. We observe here that to study the scalar fibered Rayleigh quotient, there is no need to assume thatw∈ X+; however, [0,∞)3t7→Rµ(tw) has a global maximum if, and only ifw∈ X+. Furthermore, note that if µ < λ1 andR

uv >0, thenλmin(µ;w) is just the local minimum of [0,∞)3t7→Rµ(tw) which is attained att= 0.

The functions λmin(µ;w) and λmax(µ;w) have the following geometrical inter- pretation, with respect to the fiber maps, which can be proved from Proposition 2.2 and their definitions.

Proposition 2.4. The following holds:

(1) For each µ < λ₁ andλ∈R we have thatN_λ,µ⁻ 6=∅. MoreoverN_λ,µ⁺ 6=∅ if, and only ifλ > λ_min(µ, w) andµ < λ₁.

(2) For each µ < λ₁ and w ∈ X+ we have that: λ_max(µ;w) is the unique parameter λ > 0 for which the fiber map ψ_λ,w has a critical point with second derivative zero att(w). Ifλ_min(µ, w)< λ < λ_max(µ;w), thenψ_λ,µ,w satisfies II) of the Proposition 2.2 while if λ > λ_max(µ;w), then ψ_λ,w is decreasing and has no critical points.

Let us consider the following critical values:

λ^∗₁(µ) = inf

λmin(µ;w) :w∈ X : Z

uv dx >0 , (2.3) λ^∗(µ) = inf

λ_max(µ;w) :w∈ X₊ . (2.4) Lemma 2.5. For each µ < λ₁ it holds0< λ^∗₁(µ)< λ^∗(µ)<+∞. Moreover,

(i) λ^∗₁(µ) =λ1−µ.

(ii) There exists a minimizer w^∗ := (u^∗, v^∗) ∈ X+ of (2.4), which means λ_max(µ;w^∗) =λ^∗(µ).

Proof. (i) Indeed, we have

λmin(µ;w)≥ 1− µ

λ₁ kwk²

2R

uv, (2.5)

and

Z

uv≤ kuk₂kvk₂≤ 1

√λ₁kuk 1

√λ₁kvk

= 1 λ1

kukkvk ≤ 1 λ1

R

|∇u|²+|∇v|² 2

= 1 λ1

kwk² 2 . Then we obtain

λ₁≤inf k(u, v)k² 2R

uv . (2.6)

It follows from (2.6) and (2.5) that

λ^∗₁(µ)≥λ₁−µ.

SinceR

φ²₁>0 andλmin(µ, φ1, φ1) =λ1−µit follows thatλ^∗₁(µ) =λ1−µ.

(ii) Now let us prove there exists w^∗ ∈ X+ such that λ^∗(µ) = λmax(µ;w^∗).

Choose a sequencewn:= (un, vn)∈ X+ such thatλmax(µ, wn)→λ^∗(µ) asn→ ∞ and since λmax(µ, tw) = λmax(µ;w) for t > 0, we can assume without loss of generality thatkwnk= 1 and thereforewn* winX andwn→winL^p(Ω)×L^q(Ω).

Note thatR

uv >0 because λmax(µ;wn)≥

1− µ λ₁

kwnk² 2R

u_nv_n = 1− µ

λ₁ 1

2R

u_nv_n, ∀n,

and on the contrary, we would haveλ_max(µ;w_n)→+∞which is clearly a contra- diction. It follows thatu, v6= 0. We claim thatR

f|u|^p>0, indeed suppose on the contrary thatR

f|u|^p = 0. Since λ^∗_max(µ;w_n)≥C_p,q

R g|v_n|^qdx^p−2_p−q Rf|un|^pdx^q−2_p−q

, ∀n, (2.7)

and R

g|v|^q > 0 we conclude that λmax(µ;wm) → +∞ which is a contradiction, therefore R

f|u|^p >0. We denote by ¯u= _kwk^u , ¯v = _kwk^v , then ¯w = (¯u,¯v) satisfies kwk¯ = 1, R

¯

u¯v >0 andR

f|u|¯^p,R

g|¯v|^q >0. We claim thatw_n→win X, indeed if not, by the weak lower semi-continuity of the norm, we have

λmax(µ; ¯w)<lim inf

n→∞ λmax(wn) =λ^∗(µ) (2.8) which is an absurd and henceλ^∗(µ) = λmax(µ; ¯w). By defining w^∗ = ¯w the proof

is complete.

Proposition 2.6. Let µ < λ1, then N_λ⁰∗(µ),µ 6= ∅. Moreover, each w ∈ N_λ⁰∗(µ),µ

satisfies

2 −∆u−µu−λ^∗(µ)v

−pf(x)|u|^p−2u= 0, 2 −∆v−µv−λ^∗(µ)u

+qg(x)|v|^q−2v= 0, (2.9) Proof. From Lemma 2.5 there exists w ∈ X \ {0} such that λmax(µ;w) = λ^∗(µ).

From the definition ofλmax(µ;w) it follows thatN_λ⁰∗(µ),µ6=∅. To prove that each

w∈ N_λ⁰∗(µ),µsatisfies (2.9), we observe thatλ⁰_max(µ;w) ¯w= 0 for all ¯w= (¯u,v)¯ ∈ X, hence we obtain

0 =2Z

∇u∇¯u−µ Z

u¯u−λ^∗(µ) Z

vu¯

−pq−2 p−2

^p−2_p−qR g|v|^q Rf|u|^p

^p−2_q−2Z

f|u|^p−2u¯u,

0 =2Z

∇v∇¯v−µ Z

vv¯−λ^∗(µ) Z

u¯v

−qq−2 p−2

^p−2p−qR g|v|^q R f|u|^p

^p−2q−2Z

g|v|^q−2v¯v.

(2.10)

Forw∈ N_λ⁰∗(µ),µ we have q−2

p−2

^p−2p−qR g|v|^q Rf|u|^p

^p−2q−2

= 1. (2.11)

Then, from (2.10) and (2.11) we conclude the proof.

Corollary 2.7. Let µ < λ1. Then

(i) For eachλ∈R we have thatN_λ,µ⁻ 6=∅.

(ii) N_λ,µ⁺ 6=∅ if, and only ifλ > λ^∗₁(µ).

(iii) N_λ,µ⁰ 6=∅ if, and only ifλ≥λ^∗(µ).

Proof. (i) Givenλ∈Rthere existswn:= (un, vn)∈ X+such thatkunk= 1,vn 6= 0 andvn →0 inH₀¹(Ω), so

n→∞lim H_λ,µ(w_n)≥ lim

n→∞

1− µ λ1

+kvnk²−µkvnk²₂−2λ Z

u_nv_n

= 1− µ

λ1

>0,

then there existsn0∈Nsuch that for alln≥n0we obtainHλ,µ(wn)>0 and from Proposition 2.2 we conclude thatN_λ,µ⁻ 6=∅.

(ii) SupposeN_λ,µ⁺ 6=∅and takew∈ N_λ,µ⁺ . By Proposition 2.2 we conclude that H_λ,µ(w)<0 which implies ^kwk₂^2−µkwkR ²²

uv < λ, thereforeλ^∗₁(µ)< λ.

Now suppose thatλ^∗₁(µ)< λand take w= (φ1, φ1). It follows that λ^∗₁(µ) =kwk²−µkwk²₂

2R

φ²₁ < λ, hence Hλ,µ(w) < 0. Since R

g|φ1|^q > 0, from Proposition 2.2 we conclude that t⁺_λ,µ(w)w∈ N_λ,µ⁺ .

(iii) SupposeN_λ,µ⁰ 6=∅. We know thatw∈ N_λ,µ⁰ if, and only if R_µ(w) =λ, d

dtR(tw)_|t=1= 0,

and therefore by definition ofλ^∗(µ), we conclude thatλ^∗(µ)≤λ.

Now observe from Lemma 2.5 that there existsw^∗∈ X₊such thatw^∗∈ N_λ⁰∗(µ),µ. Moreover there existsw_n := (u_n, v_n)∈ X+ such thatkunk= 1,v_n6= 0 andv_n→0

inH₀¹(Ω), then

n→∞lim λ_max(µ;w_n)≥ lim

n→∞

1 2R

unvn

1− µ λ1

+kvnk²−µkvnk²₂

=∞, therefore, from the continuity of λmax(µ;w) with respect to w, given λ ≥ λ^∗(µ) we there exists w ∈ X+ such that λmax(µ;w) = λ and from Proposition 2.4 we

conclude thatN_λ,µ⁰ 6=∅.

3. Topological properties of the energy functional

In this Section we study the energy functional Φλ,µ, in particular, we show that Φλ,µ has some well know topological properties when restricted to the Nehari set, as for example coerciveness, which allow us to minimize over the Nehari manifolds N_λ,µ⁻ andN_λ,µ⁺ .

Forλ >0 we define

Nˆ_λ,µ⁻ =

w∈ N_λ,µ⁻ :Hλ,µ(w)≤0 .

Proposition 3.1. For each µ < λ₁ andλ∈R, we have the following:

(i) There exists a constantC >0 such thatkwk ≤C for all w∈ N_λ,µ⁺ ∪Nˆ_λ,µ⁻ . (ii) The functionalΦλ,µrestricted toN_λ,µ⁺ ∪N_λ,µ⁻ is coercive that is ifwn∈ N_λ,µ⁻

is such that kw_nk → ∞asn→ ∞, thenΦ_λ,µ(w_n)→ ∞ asn→ ∞.

Proof. Assume thatw_n = (u_n, v_n)∈ N_λ,µ⁺ ∪ N_λ,µ⁻ satisfies kw_nk → ∞. We claim

that Z

un

kw_nk

p→0, as n→ ∞. (3.1)

If not, then there exists ¯C > 0 such that, up to a subsequence, R

u_n kwnk

p

> C.¯ Denote by ¯u_n=_kw^un

nk and ¯v_n= _kw^vn

nk. Sincew_n∈ N_λ,µ⁺ ∪ N_λ,µ⁻ , we have 0 = 1−µZ

|¯un|²+|¯vn|²

−2λ Z

¯

unv¯n+kwnk^q−2 Z

g|¯vn|^q− kwnk^p−2 Z

f|¯un|^p. (3.2) By Sobolev embedding and Poincare’s inequality there exist constantsC1, C2, C3>

0 such thatR

g|¯vn|^q ≤C1,R

|¯un|²+|¯vn|²

≤C2andR

¯

un¯vn≤C3. It follows from (3.2) that

0 = 1−µZ

|¯u_n|²+|¯v_n|²

−2λ Z

¯

u_nv¯_n+kw_nk^q−2 Z

g|¯v_n|^q

− kwnk^p−2 Z

f|¯un|^p

≤1 +|µ|C₂+ 2|λ|C₃+C₁kw_nk^q−2−Ckw¯ _nk^p−2,∀n, which is a contradiction sincep > qand therefore (3.1) is true.

Let us prove (i). Suppose on the contrary that there exists a sequence wn ∈ N_λ,µ⁺ ∪Nˆ_λ,µ⁻ such thatkw_nk → ∞asn→ ∞. From (3.1) we obtain thatR

|¯u_n|²→0 and sinceH_λ,µ(w_n)≤0 andµ < λ₁ we conclude that

0≥1−µZ

|¯u_n|²+|¯v_n|²

−2λ Z

¯ u_nv¯_n

≥ 1− µ

λ1

−2λ Z

¯ un¯vn

→ 1− µ

λ1

>0, n→ ∞,

which is a contradiction and therefore there exists a constant C > 0 such that kwk ≤C for allw∈ N_λ,µ⁺ ∪Nˆ_λ,µ⁻ .

Let us prove (ii): Assume thatkwnk → ∞as n→ ∞. From (3.1) and (3.2) we conclude that

kwnk^q−2 Z

g|¯vn|^q− kwnk^p−2 Z

f|¯un|^p−µ Z

|¯v|²=o(1)−1. (3.3) Now observe that

Φ_λ,µ(w_n) =kw_nk²1 2 −µ1

2 Z

|¯u_n|²+|¯v_n|²−λ Z

¯ u_n¯v_n +kwnk²kw_nk^q−2

q Z

g|¯v_n|^q−kw_nk^p−2 p

Z

f|¯u_n|^p .

(3.4)

Observe that Φ_λ,µ(w) > 0 for all w ∈ N_λ,µ⁻ . If we assume on the contrary that Φ_λ,µ(w_n) does not converge to∞then from (3.4) we are forced to assume that

kwnk^q−2 q

Z

g|¯vn|^q−kwnk^p−2 p

Z

f|¯un|^p−1 2µ

Z

|¯v|²=o(1)−1

2. (3.5) from (3.3) and (3.5) we obtain

kwnk^p−2 Z

f|¯un|^p=o(1) +2 p

q−2 q−p

1−µ

Z

|¯vn|²

, (3.6)

kwnk^q−2 Z

g|¯vn|^q=o(1) + 2 p

q−2 q−p

1−µ

Z

|¯vn|²

. (3.7)

Onceµ < λ1 andq < pit follows from (3.6), (3.7) that kwnk^p−2

Z

f|¯un|^p≤o(1) +2 p

q−2 q−p

1− µ λ1

,

kwnk^q−2 Z

g|¯vn|^q ≤o(1) +2 p

q−2 q−p

1− µ λ₁

,

which is a contradiction and therefore Φ_λ,µ(w_n)→ ∞asn→ ∞.

From Proposition 3.1 we have the following result.

Corollary 3.2. Suppose that µ < λ₁ and λ ∈ R. Then there exists a constant C >0 such thatΦ_λ(w)≥ −C, for allw∈ N_λ,µ⁺ ∪ N_λ,µ⁻ .

Lemma 3.3. For eachµ < λ₁ andλ∈R there exists a constantC >0 such that kwk ≥C, for allw∈ N_λ,µ⁻ . Moreover, ifA⊂ N_λ,µ⁻ is a bounded set, thenkuk_p≥C for each(u, v)∈A.

Proof. Indeed, suppose on the contrary that there existswn= (un, vn)∈ N_λ,µ⁻ such thatkwnk →0. Ifv_n= 0 for allnthe proof is immediate, therefore there is no loss of generality in assuming thatv_n6= 0 for alln. Moreover from Proposition 2.2 we also have that u_n 6= 0 for alln. Define ¯u_n = _kw^un

nk and ¯v_n = _kw^vn

nk and ¯w_n = (¯u_n,¯v_n).

It follows that ¯wn * (u0, v0) in X and ¯wn → (u0, v0) in L^p(Ω)×L^q(Ω). Once wn∈ N_λ,µ⁻ we know that

1−µkw¯nk²₂−2λ Z

¯

unv¯n=kwnk^p−2 Z

f|u¯n|^p− kwnk^q−2 Z

g|¯vn|^q, ∀n, (3.8)

and

1− kw¯nk²₂−2λ Z

¯

unv¯n+ (q−1)kwnk^q−2 Z

g|¯vn|^q−(p−1)kwnk^p−2 Z

f|¯un|^p<0, and hence

(q−2)kwnk^q−2 Z

g|¯vn|^q−(p−2)kwnk^p−2 Z

f|¯un|^p<0, ∀n, which implies

1

kwnk^p−q <p−2 q−2

Rf|¯un|^p R g|¯vn|^q, ∀n, HenceR

g|¯v_n|^q →0 asn→ ∞which combined with (3.8) gives us an absurd since µ < λ₁and thereforeN_λ⁻ is bounded always from the origin.

Now assume thatA⊂ N_λ,µ⁻ is a bounded set. For eachw∈A we have that kwk²−µkwk²₂−2λ

Z uv+

Z

g|v|^q− Z

f|u|^p= 0. (3.9) If on the contrary we can findw_n ∈Asuch that u_n →0 inL^p(Ω), then since Ais bounded, from (3.9) we obtainkw_nk²−µkv_nk²₂+R

g|v_n|^q =o(1) and onceµ < λ₁ we conclude that kwnk =o(1) that is a contradiction and therefore, there exists

C >0 such thatkukp≥Cfor eachw∈A.

4. Existence of solutions in (−∞, λ^∗(µ)]

In this section, by using the properties of the fiber maps, we prove existence of positive solutions to the problem (1.1) forλ∈(−∞, λ^∗(µ)] andµ < λ₁.

Remark 4.1. We claim that there is no non-negative solution of (1.1) forµ > λ₁ and λ > 0. Indeed, take φ₁ ∈ H₀¹(Ω) and let w:= (u, v)∈ X be a non-negative solution for (1.1), then

Z

∇u∇φ1=λ₁ Z

uφ₁=µ Z

uφ₁+λ Z

vφ₁+ Z

f|u|^p−2uφ₁≥µ Z

uφ₁

we obtain

(λ1−µ) Z

uφ1≥0

which implies that u= v = 0, since µ > λ1. Therefore there is no non-negative solution of (1.1) forµ > λ1 and λ >0. Ifw is a positive solution, then the same argument holds for allµ≥λ1 andλ >0.

Forλ∈Rdefine

Mˆλ,µ:={w∈ X :ψλ,µ,w satisfies (I) or (II) of Proposition 2.2}, and

Mˆ⁻_{λ µ} :=

w∈ X \ {0}:H_λ,µ(w)≥0, Z

f|u|^p>0 .

Forλ∈R, letJ_λ,µ⁻ : ˆM_λ,µ∪Mˆ⁻_λ,µ→RandJ_λ,µ⁺ : ˆM_λ,µ→Rbe defined by J_λ,µ⁻ (w) = Φλ,µ(t⁻_λ,µ(w)w), and J_λ,µ⁺ (w) = Φλ,µ(t⁺_λ,µ(w)w).

Remark 4.2. Observe from Proposition 2.2 thatN_λ,µ⁺ ∪ N_λ,µ⁻ ⊂Mˆλ,µ∪Mˆ⁻_λ,µand from Corollary 2.7 we have that N_λ,µ⁺ 6= ∅ if λ > λ1(µ) and N_λ,µ⁻ 6= ∅ if λ ∈ R. MoreoverJ_λ,µ⁻ , J_λ,µ⁺ are the restrictions of Φλ,µ toN_λ,µ⁻ andN_λ,µ⁺ respectively.

We consider the following constrained minimization problems Jˆ_λ,µ⁻ := inf{J_λ,µ⁻ (w) :w∈ N_λ,µ⁻ }, ∀λ∈R, and

Jˆ_λ,µ⁺ := inf{J_λ,µ⁺ (w) :w∈ N_λ,µ⁺ }, ∀λ > λ1(µ).

Proposition 4.3. It holds:

• For each λ∈(λ₁(µ), λ^∗(µ))there there exists w_λ := (u_λ, v_λ)∈ N_λ,µ⁺ such that Jˆ_λ,µ⁺ =J_λ,µ⁺ (wλ).

• For each λ ∈ (−∞, λ^∗(µ)) there there exists w¯λ := (¯uλ,¯vλ) ∈ N_λ,µ⁻ such that Jˆ_λ,µ⁻ =J_λ,µ⁻ ( ¯wλ).

Proof. Firstly, we start with ˆJ_λ,µ⁺ . We may suppose that there exists w_n :=

(un, vn)∈ N_λ,µ⁺ such thatJ_λ,µ⁺ (wn)→Jˆ_λ,µ⁺ . From Proposition 3.1 we havewn * w:= (u, v) inX andwn→win L^p(Ω)×L^q(Ω). Since

Φλ,µ(w)≤lim inf Φλ,µ(wn) = ˆJ_λ,µ⁺ , (4.1) and by Proposition 2.2 we have that ˆJ_λ,µ⁺ <0, we conclude thatw6= 0. We claim that w_n →w in X. Indeed suppose on the contrary that it is false. By one hand note from Proposition 2.2 that H_λ,µ(w) < lim inf_n→∞H_λ,µ(w_n) ≤ 0 and since λ∈(λ₁, λ^∗(µ)) we conclude that w∈Mˆ_λ,µ. On the other hand

0 =ψ⁰_λ,µ,w(t⁺_λ,µ(w))<lim inf

n→∞ ψ⁰_λ,µ,wn(t⁺_λ,µ(w)), and hencet⁺_λ,µ(w)>1 which implies that

J_λ,µ⁺ (w)<lim inf

n→∞ Φλ,µ(w)<lim inf

n→∞ Φλ,µ(wn) = ˆJ_λ,µ⁺ ,

which is a contradiction. Thereforewn →winX,w∈ N_λ,µ⁺ and ˆJ_λ,µ⁺ =J_λ,µ⁺ (w).

Now we consider w_n := (u_n, v_n) ∈ N_λ,µ⁻ such that J_λ,µ⁻ (w_n) → Jˆ_λ,µ⁻ . From Proposition 3.1 we have w_n * w := (u, v) in X and w_n →w in L^p(Ω)×L^q(Ω).

Then from Lemma 3.3 we have thatu6= 0 and hence from Proposition 2.2,t⁻_λ,µ(w) is well defined. We claim thatwn→w inX, so suppose that is not true. Observe that

0 =ψ⁰_λ,µ,w(t⁻_λ,µ(w))<lim inf

n→∞ ψ⁰_λ,µ,wn(t⁻_λ,µ(w)),

and hencet⁺_λ,µ(wn) < t⁻_λ,µ(w) <1 for sufficiently largen in case t⁺_λ,µ(wn) is well defined andtλ,µ(w)<1 in caset⁺_λ,µ(wn) is not defined. In both cases we have

J_λ,µ⁻ (w)<lim inf

n→∞ Φλ,µ(t⁻_λ,µ(w)wn)≤lim inf

n→∞ Φλ,µ(wn) = ˆJ_λ,µ⁻ ,

that is an absurd. Thereforew_n→winX,w∈ N_λ,µ⁻ and ˆJ_λ,µ⁻ =J_λ,µ⁻ (w).

The next Proposition will be useful in order to prove existence of solutions when λ≥λ^∗(µ).

Proposition 4.4. Fix µ < λ1 and takew∈ X \0 such that R

uv >0. LetI ⊂R be an open interval such thatt^∓_λ,µ(w)are well defined for all λ∈I. It holds:

(i) The functions I 3 λ 7→ t^∓_λ,µ(w) are C¹. Moreover, I 3 λ 7→ t⁻_λ,µ(w) is decreasing whileI3λ7→t⁺_λ,µ(w)is increasing.

(ii) The functionsI3λ7→J_λ,µ^∓ (w)are continuous and decreasing.

Proof. (i) For eachw∈ X \0 fixed we define

F(λ, t) =H_λ,µ(tu, tv) +G(v)−F(u).

Sincet^∓_λ,µ(w)w∈ N_λ,µ^∓ , it follows that

F(λ, t^∓_λ,µ(w)w) = 0,

∂

∂tF(λ, t^∓_λ,µ(w))6= 0,

which implies from the implicit function theorem thatt^∓_λ,µ(w) isC¹ and

∂

∂λt^∓_λ,µ(w) = 2R uv ψ⁰⁰_λ,µ,w(t^∓_λ,µ(w)), therefore, _∂λ^∂ t⁺_λ,µ(w)>0 and _∂λ^∂ t⁻_λ,µ(w)<0.

(ii) Indeed,

∂

∂λJ_λ,µ^∓ (w) =− Z

uv;

therefore,J_λ,µ^∓ is decreasing.

Proposition 4.5. For each µ < λ₁, there exists w ∈ N_λ⁺∗(µ),µ and w¯ ∈ N_λ⁻∗(µ),µ

such that Jˆ_λ⁺∗(µ),µ=J_λ⁺∗(µ),µ(w)andJˆ_λ⁻∗(µ),µ=J_λ⁻∗(µ),µ( ¯w).

Proof. Takeλn↑λ^∗(µ) andwn:= (un, vn)∈ N_λ⁻

n,µwith ˆJ_λ⁻

n,µ=Jλ_n,µ(wn). From Lemma 2.1 we have

−∆un−µun−λ^∗(µ)vn−f(x)|un|^p−2un= 0,

−∆vn−µvn−λ^∗(µ)un+g(x)|vn|^q−2vn = 0,

for each n. Using similar arguments to those in Proposition 3.1 and Lemma 3.3, we can show that there exist constants C, c > 0 such that c ≤ kwnk ≤ C. We can suppose without loss generality that wn * w := (u, v) in X and wn → w in L^p(Ω)×L^q(Ω). Hencewn→w6= 0 inX and we conclude that

−∆u−µu−λ^∗(µ)v−f(x)|u|^p−2u= 0,

−∆v−µv−λ^∗(µ)u+g(x)|v|^q−2v= 0, (4.2) We claim thatw∈ N_λ⁻∗(µ),µ. If not, thenw∈ N_λ⁰∗(µ),µ and from Proposition 2.6,

2 (−∆u−µu−λ^∗(µ)v)−pf(x)|u|^p−2u= 0,

2 (−∆v−µv−λ^∗(µ)u) +q|v|^q−2g(x)v= 0. (4.3) From (4.2) and (4.3) we have

(2−p)f(x)|u|^p−2u= 0,

(2 +q)g(x)|v|^q−2v= 0, (4.4)

which impliesw= 0, an absurd. Thereforew∈ N_λ⁻∗(µ),µ and henceJ_λ⁻∗(µ),µ(w)≥ Jˆ_λ⁻∗(µ),µ. To conclude the proof we need to show that J_λ⁻∗(µ),µ(w) = ˆJ_λ⁻∗(µ),µ so

suppose on the contrary that J_λ⁻∗(µ),µ(w) > Jˆ_λ⁻∗(µ),µ. Given ε > 0 there exists z∈ N_λ⁻∗(µ),µ such that

0< J_λ⁻∗(µ),µ(z)−Jˆ_λ⁻∗(µ),µ< ε. (4.5) From Proposition 4.4 we can also findN >0 such that

0< J_λ⁻

n,µ(z)−J_λ⁻∗(µ),µ(z)< ε, ∀n > N. (4.6) From (4.5) and (4.6) we conclude that

Jˆ_λ⁻

n,µ=J_λ⁻∗(µ),µ(w) +o(1)>Jˆ_λ⁻∗(µ),µ+o(1)

> J_λ⁻

n,µ(z)−2ε+o(1)≥Jˆ_λ⁻

n,µ−2ε+o(1),

which is a contradiction and henceJ_λ⁻∗(µ),µ(w) = ˆJ_λ⁻∗(µ),µ. A similar proof can be

carried out for ˆJ_λ⁺∗.

Proof of Theorem 1.1. From Propositions 4.3 and 4.5, it follows that there exist w_λ,µ := (u_λ,µ, v_λ,µ) ∈ N_λ,µ⁺ and ¯w_λ,µ := (¯u_λ,µ,¯v_λ,µ) ∈ N_λ,µ⁻ , such that ˆJ_λ,µ⁺ = J_λ,µ⁺ (wλ,µ) and ˆJ_λ,µ⁻ = J_λ,µ⁻ ( ¯wλ,µ). For simplicity we define w := wλ,µ and ¯w :=

¯

wλ,µ, then from Lemma 2.1 we have thatwand ¯ware solutions of problem (1.1).

Let us prove now thatw and ¯w can be chosen as positive functions. We do it only to ¯wsince forwthe calculations are similar. First, observe that Hλ,µ(|w|)¯ ≤ Hλ,µ( ¯w), where|w|¯ := (|¯u|,|¯v|). We claim thatHλ,µ(|w|) =¯ Hλ,µ( ¯w). Suppose on the contrary thatHλ,µ(|w|)¯ < Hλ,µ( ¯w).

Case 1: λ ∈ (−∞, λ^∗(µ)). From Proposition 2.2 and since |¯u| 6= 0, there exist t⁻ :=t⁻_λ,µ(|w|)¯ >0 such thatt⁻|w| ∈ N¯ _λ,µ⁻ . OnceHλ,µ(|w|)¯ < Hλ,µ( ¯w), we have

0 =ψ⁰_λ,µ,|w|¯(t⁻)< ψ⁰_λ,µ,w¯(t⁻),

which from Proposition 2.2 impliest⁻ <1 and in this caset⁺_λ,µ( ¯w) is defined; we also have thatt⁺_λ,µ( ¯w)< t⁻<1. It follows that

Φλ,µ(t⁻|w|) =¯ (t⁻)²

2 Hλ,µ(|w|) +¯ (t⁻)^q q

Z

g|¯v|^q−(t⁻)^p p

Z f|¯u|^p

< (t⁻)²

2 H_λ,µ( ¯w) +(t⁻)^q q

Z

g|¯v|^q−(t⁻)^p p

Z f|¯u|^p

= Φλ,µ(t⁻w)¯ <Φλ,µ( ¯w) = ˆJ_λ,µ⁻ which is a contradiction and thereforeHλ,µ(|w|) =¯ Hλ,µ( ¯w).

Case 2: λ=λ^∗(µ). Indeed, we claim that ˆJ_λ,µ⁻ = Φλ,µ( ¯w)<0 soHλ,µ( ¯w)<0.

If not, then Hλ,µ( ¯w) ≥0 and by Proposition 2.2 we obtain that ˆJ_λ,µ⁻ ≥ 0 which is an absurd. By the definition of λ^∗(µ) and Propositions 2.2 and 2.4, there exists t:=t_λ,µ(|w¯_λ,µ|)>0 such thatt|w| ∈ N¯ _λ,µ⁻ ∪ N_λ,µ⁰ and hence

0 =ψ_λ,µ,|⁰ _w|¯(t)< ψ⁰_λ,µ,w¯(t).

From Proposition 2.2 it follows thatt <1. Then

Φλ,µ(t|w|)¯ <Φλ,µ(tw)¯ <Φλ,µ( ¯w) = ˆJ_λ,µ⁻ (4.7)

which is a contradiction. ThereforeHλ,µ(|w|) =¯ Hλ,µ( ¯w) which implies that ψ_λ,µ,|⁰ _w|¯(1) =ψ_λ,µ,⁰ _w¯(1) = 0,

ψ_λµ,|⁰⁰ _w|¯(1) =ψ⁰⁰_λ,µ,w¯(1)<0. (4.8) Therefore we can assume thatw,w¯ ≥0. Moreover, one can easily see from (1.1) that the functions u, v,u,¯ ¯v are non-zero. From standard regularity theory we conclude that u, v,u,¯ v¯ ∈ C^1,α(Ω) for some α∈ (0,1) and they are positive everywhere in

Ω.

References

[1] C. O. Alves, D. C. de Morais Filho, M. A. S. Souto;On systems of elliptic equations involving subcritical or critical Sobolev exponents, Nonlinear Anal.42(2000), no. 5, Ser. A: Theory Methods, 771–787. MR 1776922.

[2] Henri Berestycki, Italo Capuzzo-Dolcetta, Louis Nirenberg;Variational methods for indefinite superlinear homogeneous elliptic problems, NoDEA Nonlinear Differential Equations Appl., 2(1995), no. 4, 553–572. MR 1356874.

[3] Yuri Bozhkov, Enzo Mitidieri; Existence of multiple solutions for quasilinear systems via fibering method, J. Differential Equations,190(2003), no. 1, 239–267. MR 1970963.

[4] K. J. Brown;Spatially inhomogeneous steady state solutions for systems of equations describ- ing interacting populations, J. Math. Anal. Appl.,95(1983), no. 1, 251–264. MR 710432.

[5] Ph. Cl´ement, D. G. de Figueiredo, E. Mitidieri; Positive solutions of semilinear elliptic systems, Comm. Partial Differential Equations17(1992), no. 5-6, 923–940. MR 1177298.

[6] D. G. Costa, C. A. Magalh˜aes;A variational approach to subquadratic perturbations of elliptic systems, J. Differential Equations,111(1994), no. 1, 103–122. MR 1280617.

[7] Edcarlos D. da Silva;Multiplicity of solutions for gradient systems using Landesman-Lazer conditions, Abstr. Appl. Anal., (2010), Art. ID 237826, 22. MR 2720025.

[8] Djairo G. deFigueiredo;Nonlinear elliptic systems, An. Acad. Brasil. Ciˆenc.72(2000), no. 4, 453–469. MR 1800060.

[9] Yavdat Ilyasov;On extreme values of Nehari manifold method via nonlinear Rayleigh’s quo- tient, Topol. Methods Nonlinear Anal.,49(2017), no. 2, 683–714. MR 3670482.

[10] Yavdat Ilyasov, Kaye Silva; On branches of positive solutions forp-Laplacian problems at the extreme value of the Nehari manifold method, Proc. Amer. Math. Soc.,146(2018), no. 7, 2925–2935. MR 3787354.

[11] Tiancheng Ouyang;On the positive solutions of semilinear equations∆u+λu+hu^p= 0on compact manifolds. II, Indiana Univ. Math. J.,40(1991), no. 3, 1083–1141. MR 1129343.

[12] S. I. Pokhozhaev;The fibration method for solving nonlinear boundary value problems, Trudy Mat. Inst. Steklov.,192(1990), 146–163, Translated in Proc. Steklov Inst. Math.,1992, no.

3, 157–173, Differential equations and function spaces (Russian). MR 1097896.

[13] Kaye Silva, Abiel Macedo;Local minimizers over the Nehari manifold for a class of concave- convex problems with sign changing nonlinearity, J. Differential Equations,265(2018), no. 5, 1894–1921. MR 3800105.

[14] Wenming Zou;Multiple solutions for asymptotically linear elliptic systems, J. Math. Anal.

Appl.,255(2001), no. 1, 213–229. MR 1813819.

Kaye Silva

Instituto de Matem´atica e Estat´ıstica, Universidade Federal de Goi´as, 74001-970, Goiˆania, GO, Brazil

Email address:[email protected]

Steffˆanio Moreno Sousa

Instituto de Matem´atica e Estat´ıstica Universidade Federal de Goi´as, 74001-970, Goiˆania, GO, Brazil

Email address:[email protected]