Instability Induced by Cross-Diffusion in a Predator-Prey Model with Sex Structure

Volume 2012, Article ID 240432,18pages doi:10.1155/2012/240432

Research Article

Instability Induced by Cross-Diffusion in a Predator-Prey Model with Sex Structure

Shengmao Fu and Lina Zhang

Department of Mathematics, Northwest Normal University, Lanzhou 730070, China

Correspondence should be addressed to Lina Zhang,[email protected] Received 26 July 2011; Accepted 15 January 2012

Academic Editor: Junjie Wei

Copyrightq2012 S. Fu and L. Zhang. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

In this paper, we consider a cross-diffusion predator-prey model with sex structure. We prove that cross-diffusion can destabilize a uniform positive equilibrium which is stable for the ODE system and for the weakly coupled reaction-diffusion system. As a result, we find that stationary patterns arise solely from the effect of cross-diffusion.

1. Introduction

Sex ratio means the comparison between the number of male and female in the species. The sex ratio is generally regarded as 1 : 1. But for wildlife, the sex ratio of species varies with the category, environment condition, community behavior, orientation and heredity, and so forth.

The animal’s sex ratio in the different life history stages may vary with different animals. Take a bird as an example: the number of the older males is larger than that of the older females with the increase in age, which is contrary to the case of the mammal where the number of the older females is larger than that of the older males with the increase in age1,2. Sex ratio is the basis of analyzing the dynamic state of different species, the variation of which has a huge influence on the dynamic state of the species1–7. Abrahams and Dill5have provided evidence that male and female guppies forage differently in the presence of predators and that sexual differences in the energetic equivalence of the risk of predation exist. Eubanks and Miller6have found that female Gladicosa pulchraLycosidaewolf spiders climb trees significantly more often than males in the presence of forest floor predators. It is found that the sex of voles affects the risk of predation by mammals and female voles are more easily predated than male voles in7.

Incorporating the sex of prey in a classical Lotka-Volterra model, Liu et al.1con- sidered the following sex-structure model:

ut b₁v−uD1kukvc₁w, vt vb2−D2−ku−kv−c1w,

wt w−D3c₂uc₂v−c₃w,

1.1

whereu,v, and w are the population densities of the male prey, the female prey, and the predator species respectively. The parametersD₁,D₂, andD₃are their mortality rates,b₁and b₂are the birth rates of the male prey and the female prey,c₁ andc₂ are the predation rate and the conversion rate of the predators, andkandc3are the intraspecific competition rates of the prey and predators. All the parameters in model1.1are positive.

Ifβ b₂−D₂ > 0, then two obvious nonnegative equilibria of model1.1areu0 0,0,0andu1 u1, v1,0, where

u₁ b₁β k

b1D1β, v₁ β D1β k

b1D1β. 1.2

Moreover, model1.1has a positive equilibrium if and only if

R:c2β−kD3>0. H1

In this case the positive equilibrium is uniquely given byu u, v, w, where

u b₁

c₃βc₁D₃ b₁D₁β

c3kc₁c₂, v

D₁β

c₃βc₁D₃ b₁D₁β

c3kc₁c₂,

w c₂β−kD₃ c3kc1c2

.

1.3

It turns out thatRplays an important role in determining the stability ofu1andu1. To be precise,u1 is locally asymptotically stable ifR < 0, whileu is locally asymptotically stable ifR > 0. This shows that a uniform coexistence state exists and is stable when the intrinsic growth rateβof the female prey is larger than the critical valuekD₃/c₂.

Taking account of the inhomogeneous distribution of the prey and the predator in different spatial locations within a fixed bounded domainΩat any given time, and the natural tendency of each species to diffuse to areas of smaller population concentration, Liu and Zhou in8investigated the following weakly coupled reaction-diffusion system:

u_t−d₁Δub₁v−uD1kukvc₁w, x∈Ω, t >0, vt−d2Δvv

β−ku−kv−c1w

, x∈Ω, t >0, w_t−d₃Δww−D3c₂uc₂v−c₃w, x∈Ω, t >0,

∂ηux, t ∂ηvx, t ∂ηwx, t 0, x∈∂Ω, t >0, ux,0 u₀x, vx,0 v₀x, wx,0 w₀x, x∈Ω,

1.4

whereηis the outward unit normal vector of the boundary∂Ωwhich is smooth,∂η ∂/∂η. The homogeneous Neumann boundary condition indicates that the predator-prey system is self-contained with zero population flux across the boundary. The constantsd₁,d₂, andd₃, called diffusion coefficients, are positive, and the initial valuesu0x,v0x, andw0xare nonnegative smooth functions which are not identically zero. Liu and Zhou in8found that the nonnegative constant steady states have the same stability properties as the ODE model 1.1. Therefore, Turing instability cannot occur for this reaction-diffusion system.

However, in model 1.4, only diffusion of each individual species is taken into account. In some cases, the reality is that the female prey is easily predated because of physiological factor, while the male prey can congregate and form a huge group to protect itself from the attack of the predators 7, 9; therefore, the predators tend to keep away from their male prey. Similarly as in 10–12, we model this by the cross-diffusion term Δd3wd4uwfor the predators, whered4 > 0, called the cross-diffusion coefficient. Thus, the cross-diffusion system that we will study is the following:

u_t−d₁Δub₁v−uD1kukvc₁w:G₁u, v, w, x∈Ω, t >0, vt−d2Δvv

β−ku−kv−c1w

:G2u, v, w, x∈Ω, t >0, w_t−Δd₃wd₄uw w−D3c₂uc₂v−c₃w:G₃u, v, w, x∈Ω, t >0,

∂_ηux, t ∂_ηvx, t ∂_ηwx, t 0, x∈∂Ω, t >0, ux,0 u0x, vx,0 v0x, wx,0 w0x, x∈Ω.

1.5

To our knowledge, only a few works investigated the effect of cross-diffusion on population structure and dynamics in the above model. Recently, H. Xu and S. Xu in13investigated the global existence of solutions for the corresponding full SKT model of1.5when the space dimension is less than ten.

An interesting feature of1.5is that the interaction between the predators and the male prey gives rise to a cross-diffusion term. The resulting mathematical model is a strongly coupled system of three equations which is mathematically much more complex than those considered earlier. In this paper, we will show that cross-diffusion can destabilize the uniform equilibriumu which is stable for models 1.1and1.4. Moreover, we will demonstrate that the nonlinear dispersive force can give rise to a spatial segregation of these species.

Our paper is organized as follows. InSection 2, we analyze the local stability of u for1.5and calculate the fixed point index, which is important for our later discussions on the existence of nonconstant positive steady states. InSection 3, we prove global asymptotic stability ofu with d4 0, that is, when no cross-diffusion occurs in the model. This implies that cross-diffusion has a destabilizing effect. InSection 4, we establish a priori upper and lower bounds for all possible positive steady states of1.5. InSection 5, we study the global existence of nonconstant positive steady states of1.5for suitable values of the parameters.

This is done by using the Leray-Schauder degree theory and the results obtained in Sections2,

3, and 4. InSection 6, we discuss the nonexistence of nonconstant positive steady states of 1.5. In the last section, we give a brief discussion about our model.

2. Local Stability Analysis and Fixed Point Index of u

Letu u, v, w^T,Φu d1u, d2v, d3wd4uw^T, andGu G1u, G2u, G3u^T. Then the stationary problem of1.5can be written as

−ΔΦu Gu inΩ; ∂ηu0 on ∂Ω. 2.1

In this section, we study the linearization of2.1atu and calculate the fixed point index.

Similar to14,15, let 0μ₁ < μ₂< μ₃ < μ₄· · · be the eigenvalues of the operator−Δ onΩwith the homogeneous Neumann boundary condition, and letEμibe the eigenspace corresponding toμ_i inH¹Ω. Let{φij :j 1,2, . . . ,dimEμi}be the orthonormal basis of Eμi,X H¹Ω³, andXij{cφij :c∈R³}. Then

X_∞

i1Xi, Xi_dimEμi

j1 Xij. 2.2

LetY C¹Ω³,Y{u∈Y :u, v, w >0 onΩ}, andBC {u∈Y :C⁻¹< u, v, w < Con Ω}forC > 0. Since detΦ_uu d1d2d3d4u >0 for all nonnegativeu,Φ⁻¹_u uexists and det{Φ⁻¹_u u}is positive. Hence,u is a positive solution to2.1if and only if

Fu:u−I−Δ⁻¹

Φ⁻¹_u uGu ∇uΦ_uuu∇u u

0 inY, 2.3

whereI−Δ⁻¹is the inverse ofI−Δunder homogeneous Neumann boundary conditions.

Further, we note thatD_uFu I−I−Δ⁻¹{Φ⁻¹_u uGuu I}andλis an eigenvalue ofD_uFuif and only if, for somei≥1, it is an eigenvalue of the matrix

Bi:I− 1 1μi

Φ⁻¹_u uG_uu I 1 1μi

μ_iI−Φ⁻¹_u uG_uu . 2.4

Writing

H μ

H u;μ

:det

μI−Φ⁻¹_u uGuu

, 2.5

we see that if Hμi/0, then for each integer 1 ≤ j ≤ dimEμi, the number of negative eigenvalues ofD_uFuonXij is odd if and only ifHμi<0. As a consequence, we have the following proposition.

Proposition 2.1see16. Suppose that, for alli≥1,Hμi/0. Then

indexF·,u −1 ^γ, 2.6

where

γ

i≥1,H^μi<0 dimE

μi

. 2.7

To facilitate our computation of indexF·,u, we need to determine the sign of Hμi. In particular, as the aim of this paper is to study the existence of stationary patterns of2.1 with respect to the cross-diffusion coefficientd4, we will concentrate on the dependence of Hμi ond₄. At this point, we note that Hμ det{Φ⁻¹_u u}det{μΦ_uu−G_uu}. Since det{Φ⁻¹_u u}is positive, we will need only to consider det{μΦuu−Guu}. By

G_uu

⎛

⎜⎜

⎝

−ku−D₁−β b₁−ku −c1u

−kv −kv −c1v c₂w c₂w −c3w

⎞

⎟⎟

⎠, Φ_uu

⎛

⎜⎜

⎝

d₁ 0 0

0 d₂ 0

d₄w 0 d₃d₄u

⎞

⎟⎟

⎠, 2.8

we have

det

μΦ_uu−G_uu

C₃d4μ³C₂d4μ²C₁d4μ−detG_uu :C

d₄;μ

, 2.9

where

C3d4 d1d2d3d4u,

C2d4 d1d2c3wd1kd3d4u vd2d3d4u

kuD1β

−d2d4c1uw, C₁d4 d₁kc₃vw d₂c₃

kuD₁β

w d₃d₄u

kuD₁β

kvd₄c₁vwk u−b₁

−c1u−d 2c2wkd4vw c1c2d1vw−d3d4uk vk u−b1, detG_uu vw

−kc3c₁c₂

kuD₁β

c₁c₂kc₃ku−b₁ .

2.10

Notice thatku−b1 < 0; thus detG_uu < 0. We consider the dependence of Con d4. Let

μ₁d4,μ₂d4, andμ₃d4be the three roots ofCd4;μ 0 with Re{μ₁d4} ≤Re{μ₂d4} ≤ Re{μ3d4}. It follows thatμ1d4μ2d4μ3d4 <0 from detGuu< 0. Thus, amongμ1d4,

μ2d4,μ3d4at least one is real and negative, and the product of the other two is positive.

Consider the following limits:

dlim4→ ∞

C₃d4

d₄ d₁d₂u:a₃,

dlim4→ ∞

C2d4

d₄ d₁kuvd₂u

kub1v u

−d₂c₁uw d₁kuvd₂

ku²b₁v−c₁uw :a₂,

dlim4→ ∞

C1d4 d₄ u

kub1v u

kvc₁vwk u−b₁−c₁uk vw−uk vku−b₁ b1v

β−2c1w :a1.

2.11

Therefore,a1<0 if

β <2c₁w. H2

In the following, we restrict our attention toβ <2c1w. In this range, a1<0 andC1d4<0 for all sufficiently larged4. Notice that

dlim4→ ∞

C d4;μ

d4 a₃μ³a₂μ²a₁μμ

a₃μ²a₂μa₁

2.12

and a1 < 0 < a3. A continuity argument shows that, when d4 is large, μ1d4 is real and negative. Furthermore, asμ2d4μ3d4>0,μ2d4andμ3d4are real and positive, and

dlim4→ ∞μ₁d4

−a2−

a²₂−4a1a3

2a3

<0,

dlim4→ ∞μ₂d4 0,

dlim4→ ∞μ3d4

−a2

a²₂−4a₁a₃

2a3 :μ > 0.

2.13

Thus we have the following proposition.

Proposition 2.2. Assume thatH1andH2hold. Then there exists a positive numberd^∗₄such that, whend₄ ≥d^∗₄, the three rootsμ₁d4,μ₂d4,μ₃d4ofCd4;μ 0 are all real and satisfy2.13.

Moreover, for alld₄≥d^∗₄,

−∞<μ1d4<0<μ2d4<μ3d4, C

d₄;μ

<0, μ∈

−∞,μ₁d4

∪

μ₂d4,μ₃d4 , C

d₄;μ

>0, μ∈

μ₁d4,μ₂d4

∪

μ₃d4,∞ .

2.14

Remark 2.3. Proposition 2.2 gives a criterion for the instability of u when μ > μ ₂ and the cross-diffusion coefficientd4is large enough. We further check conditionsH1andH2. Let the parametersd1,d2,d3,b1,D1,k,c1,c2, andc3 be fixed. ConditionH1is equivalent to β > kD₃/c₂, and conditionH2is equivalent toβ >2kc₁D₃/γ₁for someγ₁:c₁c₂−kc₃ >0.

Notice that 2c1/γ1 > 1/c2, so there exists an unbounded region U1 {D3, β ∈ R² : β >

2kc1D3/γ1}, such that for anyD3, β∈U1,u is an unstable equilibrium with respect to 1.5 whenμ > μ ₂and the cross-diffusion coefficientd₄is sufficiently large.

3. Global Asymptotic Stability of u for 1.4

The aim of this section is to prove Theorem 3.2 which shows that model 1.4 has no nonconstant positive steady state no matter what the diffusion coefficientsd₁,d₂, andd₃are;

in other words, diffusion alonewithout cross-diffusioncannot drive instability and cannot generate patterns for this predator-prey model. For this, we will make use of the following result.

Lemma 3.1see17. Letaandbbe positive constants. Assume thatϕ, ψ∈C¹a,∞,ψt≥0, andϕis bounded from below. Ifϕt≤ −bψtandψtis bounded ina,∞, then lim_{t→ ∞}ψt 0.

Theorem 3.2. Let the parametersd1,d2,d3,b1,D1,D3,k,c1,c2,c3, andβbe fixed positive constants that satisfyH1and

b²₁<4kuD 1b₁. H3

Letu, v, wbe a positive solution of 1.4. Then

u·, t−u _L2Ω−→0, v·, t−v _L2Ω−→0,

w·, t−w _L2Ω−→0 as t−→ ∞. 3.1 Proof. Notice from8thatu is uniformly and locally asymptotically stable in the sense of 18. We only need to prove the global stability ofu. Define

V1u, v, w 1 2

Ωu−u ²dxλ

Ω

v−v−vlnv v

dxρ

Ω

w−w−wlnw w

dx,

3.2 whereλu, ρc1u/c 2. Obviously,V1u, v, wis nonnegative andV1u, v, w 0 if and only ifu, v, w u, v, w. The time derivative of V₁u, v, wfor the system1.4satisfies

dV1u, v, w

dt

Ω

u−uu tλv−v

v vtρw−w w wt

dx:−I1t−I2t, 3.3

where I₁t

Ω

d₁|∇u|²λd2v

v² |∇v|²ρd3w w² |∇w|²

dx, I₂t

Ω

D1kukukvc₁wu−u ²λkv−v ²ρc₃w−w ²

2ku−b₁u−uv −v dx,

≥

Ω

D1kuu −u ²λkv−v ²ρc₃w−w ² 2ku−b₁u−uv −v dx.

3.4

If the matrix

⎛

⎜⎜

⎜⎜

⎝

D₁ku 1

22ku−b₁ 0 1

22ku−b1 λk 0

0 0 ρc3

⎞

⎟⎟

⎟⎟

⎠ 3.5

is positive definite, then the quadratic form

D1kuu −u ²λkv−v ²ρc3w−w ² 2ku−b1u−uv −v 3.6 is positive definite. A direct calculation shows that the matrix is positive definite ifH3holds.

Meanwhile, for everyδsuch that 0 < δ < min{c3ρ,4kuD 1b₁−b²₁/4D12ku}, we have

I2t≥δ

Ω

u−u ² v−v ² w−w ² dx. 3.7

Thus,

dV1u, v, w

dt ≤ −δ

Ω

u−u ² v−v ² w−w ² dx. 3.8

Similarly to19, Theorem 2.1, we can prove that the solutionu, v, wis bounded, and so are the derivatives of

Ωu−u ² v−v ² w−w ²dxby the equations in1.4. Using Lemma 3.1, we have

u·, t−u _L²_Ω−→0, v·, t−v _L²_Ω−→0,

w·, t−w _L2Ω−→0 ast−→ ∞. 3.9 By the fact that V₁u, v, w is decreasing fort ≥ 0, it is obvious that u, v, w is globally asymptotically stable, and the proof ofTheorem 3.2is completed.

Remark 3.3. Notice that conditionH3is equivalent to γ₂β > b₁kc3c₁c₂

4k −c₁D₃, γ₂ 3kb₁c₃4kc₃D₁−b₁c₁c₂

4kb1D1 . 3.10

Ifγ2<0, it is easy to verify that−c1/γ2> k/c2. Hence, there exists an unbounded region U₂

D₃, β

∈R²:β > kD₃

c₂ , γ₂β > b₁kc3c₁c₂ 4k −c₁D₃

, 3.11

such that for anyD3, β∈U₂,u is the unique positive steady state with respect to 1.4.

Remark 3.4. From Remarks2.3and3.3, there exists an unbounded region

U₃U₁∩U₂ D₃, β

∈R² :γ₁>0, β >2kc₁ γ1

D₃, γ₂β > b₁kc3c₁c₂ 4k −c₁D₃

, 3.12

such that for anyD3, β ∈U3, cross-diffusion can destabilize the uniform equilibriumu of 1.5whenμ > μ ₂andd₄is sufficiently large.

4. A Priori Estimates

In the following, the generic constantsC, C_∗, and so forth, will depend on the domain Ω and the dimensionN. However, asΩand the dimensionNare fixed, we will not mention the dependence explicitly. Also, for convenience, we will writeΛ instead of the collective constants b1, D1, D3, c1, c2, c3, k, β. The main purpose of this section is to give a priori positive upper and lower bounds for the positive solutions to2.1whenR > 0. For this, we will cite the following two results.

Lemma 4.1Harnack’s inequality20. Letw∈C²Ω∩C¹Ωbe a positive solution toΔwx cxwx 0, wherec ∈ CΩ, satisfying the homogeneous Neumann boundary condition. Then there exists a positive constantC_∗which depends only onc∞such that max_Ωw≤C_∗min_Ωw.

Lemma 4.2maximum principle21. Letg∈CΩ×R¹andbj∈CΩ,j1,2, . . . , N.

iIfw∈C²Ω∩C¹Ωsatisfies

Δwx ^N

j1

bjxwxjgx, wx≥0 in Ω, ∂ηw≤0 on ∂Ω, 4.1

andwx0 max_Ωwx, thengx0, wx0≥0.

iiIfw∈C²Ω∩C¹Ωsatisfies

Δwx ^N

j1

b_jxwxjgx, wx≤0 inΩ, ∂_ηw≥0 on ∂Ω, 4.2

andwx0 min_Ωwx, thengx0, wx0≤0.

Theorem 4.3upper bound. Letdandd^∗be two fixed positive constants. Assume thatd_i≥d, i 1,2,3, and 0≤d4≤d^∗. Then every possible positive solutionu, v, wof 2.1satisfies

maxΩ u≤ b1

k, max

Ω v≤ β

k, max

Ω w≤

1 d^∗b1

dk c2

b1β

c₃k . 4.3

Proof. A direct application of the maximum principle to2.1givesv≤β/konΩ. Letux0

max_Ωu. Using the maximum principle again, we haveb1vx0≥ux0D1kux0 kvx0

c₁wx0. Thus,kux0vx0≤b₁vx0andux0≤b₁/k.

Defineϕd3wd4uw; then,ϕsatisfies

−Δϕw−D3c₂uc₂v−c₃w inΩ, ∂_ηϕ0 on ∂Ω. 4.4

Letϕx1 max_Ωϕ. ByLemma 4.2, we have

−D3c₂ux1 c₂vx1−c₃wx1≥0. 4.5

It follows that

wx1≤ c₂

c₃ux1 vx1≤ c₂ c₃

b₁ k β

k

c2

b1β

c₃k . 4.6

Hence,

ϕx1 d3d4ux1wx1≤

d3d4b1

k c₂

b₁β c3k , maxΩ wmax

Ω

ϕ d3d₄u ≤

1d₄b₁ d₃k

c2

b1β c₃k ≤

1d^∗b₁ dk

c2

b1β c₃k

4.7

for anyd₃≥dand 0≤d₄≤d^∗.

Turning now to the lower bound, we first need some preliminary results.

Lemma 4.4. Let dij be positive constants, i 1,2,3,4, j 1,2, . . ., and let uj, vj, wj be the corresponding positive solution of 2.1 with d_i d_ij. If uj, v_j, w_j → u^∗, v^∗, w^∗ uniformly onΩasj → ∞andu^∗, v^∗, w^∗is a constant vector, thenu^∗, v^∗, w^∗must satisfy

b₁v^∗−u^∗D1ku^∗kv^∗c₁w^∗ 0, β−ku^∗−kv^∗−c₁w^∗0,

−D3c₂u^∗c₂v^∗−c₃w^∗0. 4.8

Moreover, ifu^∗,v^∗, andw^∗are positive constants, thenu^∗, v^∗, w^∗ u, v, w.

Proof. It is easy to see that for allj,

ΩG₁

u_j, v_j, w_j

dx0. 4.9

IfG₁u^∗, v^∗, w^∗> 0, thenG₁uj, v_j, w_j >0 whenjis large sinceuj, v_j, w_j → u^∗, v^∗, w^∗. This is impossible. Similarly,G₁u^∗, v^∗, w^∗ < 0 is impossible. Therefore,G₁u^∗, v^∗, w^∗ 0.

The same argument shows thatβ−ku^∗−kv^∗−c1w^∗ 0 and−D3c2u^∗c2v^∗−c3w^∗ 0.

Consequently,u^∗, v^∗, w^∗ u,v, w.

Lemma 4.5. The system

ut−d1Δub1v−uD1kukv, x∈Ω, t >0, v_t−d₂Δvv

β−ku−kv

, x∈Ω, t >0,

∂ηu∂ηv0, x∈∂Ω, t >0, ux,0≥/≡0, vx,0≥/≡0, x∈Ω,

4.10

has a unique positive constant steady stateu1, v₁which is globally asymptotically stable, whereu₁ andv1are given by1.2.

Proof. Let

V2u, v 1 2

Ωu−u1²dxρ

Ω

v−v1−v1ln v v₁

dx, 4.11

whereρ b1−ku1/kb1b1D1/kb1D1β>0, and letu, vbe a positive solution of4.10. Then a direct computation gives

dV2

dt −

Ω

d₁|∇u|²ρd₂v₁

v² |∇v|² D₁kuku₁kvu−u₁²ρkv−v₁²

dx≤0, 4.12

anddV₂/dt 0 holds if and only ifu, v u1, v₁. ByLemma 3.1, we can conclude that u1, v1is globally asymptotically stable.

Theorem 4.6lower bound. Letdandd^∗ be two fixed positive constants. Assume thatd_i ≥ d, i 1,2,3, and 0 ≤d₄ ≤d^∗. Then there exists a positive constantC CΛ, d, d^∗, such that every possible positive solutionu, v, wof 2.1satisfies

minΩ u, min

Ω v, min

Ω w≥C. 4.13

Proof. If the conclusion does not hold, then there exists a sequence{d1j, d2j, d3j, d4j}^∞_j1with d_1j, d_2j, d_3j ≥dand 0≤d_4j ≤d^∗such that the corresponding positive solutionuj, v_j, w_jof 2.1satisfies

min

minΩ uj, min

Ω vj,min

Ω wj

−→0, asj−→ ∞. 4.14

Moreover, we assume that dij → di ∈ d,∞ for i 1,2,3, and d4j → d4 ∈ 0, d^∗. By Theorem 4.3and the standard regularity theory for the elliptic equations, we may also assume thatuj, v_j, w_j → u, v, winC²Ω³for some nonnegative functionsu, v, w. It is easy to see thatu, v, walso satisfies estimate4.3, and min{min_Ωu, min_Ωv, min_Ωw}0.

Moreover, we observe that, ifd₁, d₂, d₃<∞, thenu, v, wsatisfies2.1.

Next we derive a contradiction for all possible cases.

Firstly, we consider the cased₁, d₂, d₃<∞.

1In view of2.1, min_Ωv0 impliesv0 onΩfrom the Harnack inequality. In this case, by the strong maximum principle and the Hopf boundary lemma, it follows thatuw0 onΩ. This is a contradiction toLemma 4.4. Thus, min_Ωv >0.

2If min_Ω u0, we denoteux0 min_Ωu0. By the maximum principle we have b₁vx0≤ux0D1kux0 kvx0 c₁wx0 0, and so min_Ω v0. This is a contradiction to min_Ωv >0. Thus min_Ωu >0.

3If min_Ω w0, letϕd₃wd₄uw. Then min_Ω ϕ0 andϕsatisfies

−Δϕϕ−D3c2uc2v−c3wd3d4u⁻¹ inΩ, ∂ηϕ0 on∂Ω. 4.15

The Harnack inequality shows that min_Ωϕ0 impliesϕ0 onΩ. Hence,w0 onΩ. From Lemma 4.5, we haveu, v u1, v1. Definewjwj/wj∞; then,uj, vj,wj, wjsatisfies

−d1jΔujb₁v_j−u_j

D₁ku_jkv_jc₁w_j inΩ,

−d2jΔvj vj

β−kuj−kvj−c1wj

inΩ,

−Δ

d3jwjd4jujwj

wj

−D3c2ujc2vj−c3wj

inΩ,

∂ηuj ∂ηvj∂ηwj0 on ∂Ω.

4.16

Similarly to the above, we can prove that there exists a subsequence of{wj}, denoted by itself, and a nonnegative function w, such thatw_j → w inC²Ωand w∞ 1. Moreover, w satisfies

−d3d₄u₁Δww−D3c₂u₁c₂v₁ inΩ, ∂_ηw0 on∂Ω. 4.17 Sincew_∞ 1, by the strong maximum principle and the Hopf boundary lemma, we find thatw > 0 onΩ. Applying the maximum principle again, we have−D3 c₂u₁c₂v₁ 0.

Thus,u1v1D3/c2. Noting thatu1v1β/kin1.2, it follows thatc2β−kD30, which is a contradiction to the conditionRc₂β−kD₃>0.

Next, we consider the remaining cases.

Integrating by parts, we obtain that

b₁

Ωv_jdx

Ωu_j

D₁ku_jkv_jc₁w_j dx,

Ωv_j

β−ku_j−kv_j−c₁w_j dx0,

Ωwj

−D3c2ujc2vj−c3wj

dx0,

4.18

forj1,2, . . .. Moreover,u, v, wsatisfies b₁

Ωvdx

ΩuD1kukvc₁wdx,

Ωv

β−ku−kv−c1w dx0,

Ωw−D3c2uc2v−c3wdx0.

4.19

Ifd1∞, thenusatisfies

−Δu0 in Ω, ∂ηu0 on∂Ω. 4.20 Hence,uis a constant. Ifu0, from4.19, we have in turn thatvw0. This contradicts Lemma 4.4. So,uis a positive constant and either min_Ωv0 or min_Ωw0.

Ifd₂ < ∞. In the case of min_Ωv0, similarly to the arguments of1, we havev0 onΩ. This contradicts the first equation of4.19. Thus min_Ωv >0 and min_Ωw0. Note that wsatisfies

−d3d4uΔww−D3c2u−c3w inΩ, ∂ηw0 on∂Ω. 4.21

Ifd₃ < ∞, the Harnack inequality implies thatw 0 onΩ. Ifd₃ ∞, thenw ia a constant. Since min_Ωw 0, sow 0 onΩ. Therefore, byLemma 4.5,u, v, w u1, v1,0.

Similarly to the arguments of3, we arrive atc₂β−kD₃ 0, which is a contradiction.

Similarly, we can derive contradictions for all the other cases.

5. Existence of Stationary Patterns for the Model 1.5

In this section we discuss the existence of nonconstant positive solutions to 2.1. These solutions are obtained for large cross-diffusion coefficientd₄, with the other parametersd₁, d₂,d₃,b₁,D₁,D₃,k,c₁,c₂,c₃, andβsuitably fixed. Our main result is as follows.

Theorem 5.1. Let the parametersd1,d2,d3,b1,D1,D3,k,c1,c2,c3, andβbe fixed such thatH1, H2, andH3hold. Let μbe given by the limit2.13. Ifμ ∈ μn, μ_n1for some n ≥ 2 and the sum!_n

i2dimEμiis odd, then there exists a positive constantd^∗₄such that2.1has at least one nonconstant positive solution ford4> d^∗₄.

Proof. ByProposition 2.2and our assumption on μ, there exists a positive constant d^∗₄ such that2.14holds ifd₄> d^∗₄, and

μ1d4<0μ1<μ2d4< μ2, μ3d4∈

μn, μ_n1

. 5.1

We will prove that for anyd₄ > d^∗₄,2.1has at least one nonconstant positive solution. The proof, which is by contradiction, is based on the homotopy invariance of the topological degree.

Suppose on the contrary that the assertion is not true for somed4 d4 > d^∗₄. In the following we fixd₄d₄.

Forθ∈0,1, defineΦθ;u d1u, d₂v, d₃wθd₄uw^Tand consider the problem

−ΔΦθ;u Gu inΩ, ∂ηu0, on ∂Ω. 5.2 Thenu is a positive nonconstant solution of2.1if and only if it is such a solution of5.2for θ 1. It is obvious thatu is the unique constant positive solution of 5.2for any 0≤ θ≤1.

As we observed inSection 2, for any 0≤θ≤1,u is a positive solution of5.2if and only if Fθ;u:u−I−Δ⁻¹

Φu−1θ;uGu ∇uΦuuθ;u∇u u

0 in Y. 5.3

It is obvious thatF1;u Fu.Theorem 3.2shows that F0;u 0 has only the positive solutionu in Y . By a direct computation,

D_uFθ;u I−I−Δ⁻¹

Φu−1θ;uG uu I

. 5.4

In particular,

D_uF0;u I−I−Δ⁻¹

D⁻¹Guu I

, 5.5

whereDdiagd1, d₂, d₃and

D_uF1;u I−I−Δ⁻¹

Φ_u⁻¹G_uu I

D_uFu. 5.6

From2.5and2.9we see that H

μ det

Φu−1u C

d₄;μ

. 5.7

In view of2.14and5.1, it follows that H

μ1

H0>0, H

μi

<0, 2≤i≤n, H

μ_i

>0, i≥n1.

5.8

Therefore, zero is not an eigenvalue of the matrixμ_iI−Φ_u⁻¹uG_uufor alli≥1, and

i≥1,H^μi<0 dimE

μ_i ⁿ

i2

dimE μ_i

σ_n, which is odd. 5.9

Thanks toProposition 2.1, we have

indexF1;·,u −1 ^γ −1^σn −1. 5.10

Similarly, we can easily show that

indexF0;·,u −1 ⁰ 1. 5.11

Now, by Theorems4.3and4.6, there exists a positive constantCsuch that, for all 0≤θ≤1, the positive solutions of2.1satisfy 1/C < u, v, w < C. Therefore,Fθ;u/0 on∂BCfor all 0≤θ≤1. By the homotopy invariance of the topological degree,

degF1;·,0, BC degF0;·,0, BC. 5.12

On the other hand, by our supposition, both equationsF1;u 0 andF0;u 0 have only the positive solutionu in BC. Hence, by5.10and5.11, we have

degF1;·,0, BC indexF1;·,u −1, degF0;·,0, BC indexF0;·,u 1.

5.13

This contradicts5.12, and thus we complete the proof ofTheorem 5.1.

Remark 5.2. Assume that all the conditions hold inTheorem 5.1.Theorem 3.2shows thatu is a globally asymptotically stable equilibrium for the system1.4. However,Theorem 5.1 implies that the cross-diffusion system1.5has at least one nonconstant positive steady state.

Our results demonstrate that stationary patterns can be found due to the emergence of cross- diffusion.

6. Nonexistence of Nonconstant Positive Solution of 2.1

In this section, we discuss the nonexistence of nonconstant positive solution of2.1when the cross-diffusion coefficientd₄>0 is small.

Theorem 6.1. If the parametersd₁,d₃,d₄,b₁,D₁,D₃,k,c₁,c₂,c₃, andβsatisfyH1,H3, and

c1d²₄uw

c2 <4d1d3, H4

then the problem2.1has no nonconstant positive solution.

Proof. Assume thatu, v, wis a positive solution of2.1. Letλu, ρc1u/c 2. Multiplying the equations of2.1byu−u, λv−v/v, and ρw−w/w, respectively, and integrating by parts, as in the proof ofTheorem 3.2, we obtain 0−I3−I₄, where

I₃

Ω

d₁|∇u|²λd2v

v² |∇v|²ρwd 3d4u

w² |∇w|²ρd4w

w ∇u· ∇w

dx, I4

Ω

D1kukukvc1wu−u ²λkv−v ²

ρc3w−w ² 2ku−b₁u−uv −v dx.

6.1

ApplyingH3andH4, it is easy to prove thatI₃≥0 andI₄≥0. This implies thatu, v, w u,v, w onΩand the proof is complete.

Remark 6.2. Theorem 6.1shows that the problem2.1has no nonconstant positive solutions if one ofd1andd3is sufficiently large; that is, unlimitedly increasing one of the diffusion rates d₁andd₃will eventually wipe out all nonconstant solutions of2.1. However,Theorem 6.1 does not tell us the effect of the diffusion rated₂on the stationary problem2.1. Using the similar arguments inSection 2, we can find thatd2does not cause instability ofu. Therefore, we conjecture that the problem2.1has no nonconstant positive solutions ifd₂is sufficiently large.

Remark 6.3. Theorems5.1and6.1seem to indicate that diffusion tends to suppress pattern formation, while cross-diffusion seems to help create patterns.

7. Discussion

In this paper, we have introduced a more realistic mathematical model for a diffusive predator-prey system where the prey has a sex structure comprising male and female mem- bers. In this model, we model the tendency of the predators to keep away from the male prey by a cross-diffusion. As a result, our model is a strongly coupled cross-diffusion system, which is mathematically more complex than systems used to model sex-structured predator- prey behavior hitherto 1, 8. What is noteworthy about this model is that, as the cross- diffusion term arises, it is precisely this cross-diffusion that destabilizes the uniform positive equilibrium and gives rise to stationary patterns for the model. Indeed, stationary patterns do not arise for the ODEspatially independentmodel, nor the PDE model without cross- diffusion.

In fact, one can see that this particular cross-diffusion term is also significant from the mathematical point of view. The following system represents the general form of SKT-type cross-diffusion22in the predator-prey model

u_t−Δd₁ud₁₂uvd₁₃uw G₁u, v, w, x∈Ω, t >0, v_t−Δd₂vd₂₁uvd₂₃vw G₂u, v, w, x∈Ω, t >0, wt−Δd3wd31uwd32vw G3u, v, w, x∈Ω, t >0,

∂ηux, t ∂ηvx, t ∂ηwx, t 0, x∈∂Ω, t >0, ux,0 u0x, vx,0 v0x, wx,0 w0x, x∈Ω,

7.1