** Contribution estimation for human interaction recognition**

**5.2 Contribution estimation**

The main contribution of this work is that a contribution estimation method is proposed
to extract major participant for interaction recognition instead of using all participants’ action
information. Since contribution estimation shields the interference from passive actions,
inter-action recognition accuracy is improved. Comparing with previous algorithms using inter-actions
of both two participants, we provide a new way to solve such complex problems. Our method
is tested on UT-interaction dataset and the LIMU dataset^{1} captured by ourselves. Experiment
results show its efficiency and out-stand performance.

(a) Single-contribution

(b) Co-contribution

Figure 5.2: Contribution interaction model construction

interaction model construction for the two cases of “co-contribution” are almost the same, which will be described later. In this chapter, we use histograms of local feature occurrence calculated by BoW to represent human actions. The frame sequences of interactions are represented as histograms, which are used to constuct interaction models.

To explain the procedure of model construction clearer, the interaction samples in UT
dataset are used as examples in the following. In UT dataset, the two participants in these
samples stand at the left and the right sides, respectively. The two participants are defined as
the left participant and the right participant according to their positions in videos. Suppose
that there are total *K* interaction categories, and the *kth category contains* *N** _{k}* training
sam-ples, The action histograms of the left and the right participant in the

*kth category are recorded*

**as h*** _{kL}*(i),

**h**

*(i), k = 1, . . . , K, i = 1, . . . , N*

_{kR}*, total 2N*

_{k}*histograms. We average the his-tograms in training samples to obtain the contribution interaction models, which are recorded*

_{k}**as H**

_{kL}*,*

**H**

_{kR}*, k*= 1, . . . , K.

One contribution interaction model of “single-contribution” interactions is shown at the
bottom of Figure5.2(a). When we create the contribution models, the major actions done by
either the left or the right participants in training samples are adopted. As shown in the figures
on top of Figure5.2(a), the actions marked by yellow rectangles are major actions. Firstly,
we separate the *N**k* training samples into “the left group” and “the right group:” in the left
group the left participants do major actions, while in the right group the right ones do the major
actions. We suppose that the left group contains *N**kL* training samples, and the right group
contains *N** _{kR}* training samples, here

*N*

*=*

_{k}*N*

*+*

_{kL}*N*

*. Then we average the histograms of the left participants in the left group by formula (5.1) to obtain the action histogram of the*

_{kR}**left participant in contribution interaction model, H**

*, in the contribution interaction model of interaction category*

_{kL}*k. Similarly, the right histogram, H*

*kR*, is calculated by averaging the histograms of the right participants’ actions in the right group.

**H***kL* =

*N**kL*

∑

*i=1*

**h***kL*(i). (5.1)

For “co-contribution” interactions, the actions of both two participants in training samples are used to construct contribution interaction models. It should be noted that in some “co-contribution” interaction categories, the actions of two participants are similar, while they are dissimilar in some other “co-contribution” interaction categories.

If the interaction category*k*belongs to a “co-contribution” interaction category, and the two
participants do similar actions, one contribution interaction model is calculated for the category
by averaging the histograms of the left and the right participants in training samples. The
construction procedure is shown in Figure5.2(b). Therefore, two histograms in the contribution
**interaction model are obtained and recorded as H***kL**,***H***kR***. We calculate H***kL*by formula (5.2),

**and H*** _{kR}* is calculated similarly using action histograms of the right participants.

**H***kL* =

*N**k*

∑

*i=1*

**h***kL*(i). (5.2)

For “co-contribution” interaction categories having dissimilar actions, we creat two
con-tribution interaction models for each category to handle its reversibility. Figure5.3 shows two
examples that the major action may be led by the left participant (left figure in Figure5.3) or by
the right participant (right figure in Figure5.3), thus contribution interaction models should be
created for the two situations. If category*k*_{0} belongs to “co-contribution” interaction with
dis-similar actions, the training interaction samples are separated to “left group” and “right group”

according to which participant, the left or the right, leads the progress of interaction. The
inter-action models of the two groups are calculated using the same procedure with Figure5.2(b). It is
supposed that the left group contains*N*_{k}_{0}* _{L}*training samples, and the right group contains

*N*

_{k}_{0}

*training samples, where*

_{R}*N*

_{k}_{0}=

*N*

_{k}_{0}

*+N*

_{L}

_{k}_{0}

*. The histograms of participants in two contribution*

_{R}**interaction models are recorded as H**

^{L}

_{k}_{0}

_{L}*,*

**H**

^{L}

_{k}_{0}

_{R}**and H**

^{R}

_{k}_{0}

_{L}*,*

**H**

^{R}

_{k}_{0}

_{R}**. The H**

^{L}

_{k}_{0}

*in the model of the*

_{L}**left group is calculated by formula (5.3), and H**

^{L}

_{k}_{0}

*is calculated in the same way. Similarly, The*

_{R}**H**

^{R}

_{k}_{0}

_{L}**in the model of the right group is calculated by formula (5.4), and H**

^{R}

_{k}_{0}

*is also calculated similarly using training samples in the right group.*

_{R}**H**^{L}_{k}_{0}* _{L}*=

*N*∑_{k}_{0}_{L}

*i=1*

**h**_{k}_{0}* _{L}*(i). (5.3)

**H**^{R}_{k}_{0}* _{L}*=

*N*∑*k*0*R*

*i=1*

**h***k*0*L*(i). (5.4)

Finally, we obtain the histograms in contribution interaction models of all interaction
cate-gories, recorded as*{***H**_{0L}*,***H**_{0R}*,***H**_{1L}*,***H**_{1R}*,· · ·* *,***H**^{L}_{k}_{0}_{L}*,***H**^{L}_{k}_{0}_{R}*,***H**^{R}_{k}_{0}_{L}*,***H**^{R}_{k}_{0}_{R}*,· · ·* *,***H**_{KL}*,***H**_{KR}*}*.
Sup-pose that total*M*(M > K)**pairs of histograms, we record the histogram set as H**=*{***H***mL**,***H***mR**,*
*m* = 1,*· · ·* *, M}*. We use the contribution interaction models to test one interaction video
be-longing to “co-contribution” or “single-contribution” interaction, and to determine the major
action of “single-contribution” interaction.

Figure 5.3: Co-contribution interaction with dissimilar actions

**5.2.2** **Contribution determination**

Both of two participants do major actions in “co-contribution” interactions, while only one participant does major action in one “single-contribution” interaction. As introduced above, the contribution interaction models are composed of two major actions. Therefore, we determine a testing interaction to be “co-contribution” or “single-contribution” by comparing the actions of participants in the testing sample with contribution interaction models of all categories. If both of two participants’ actions in the sample match with the actions in one model, the interaction sample is co-contribution interaction. Otherwise, it is single-contribution, and the matched action is the major action. The estimation procedure is explained in the following.

The histograms of local feature occurrence are calculated for two participants in the testing
**interaction, recorded as h***tL**,***h***tR*. Then we compare the two histograms with histograms in
contribution interaction models of all categories. The comparison is realized by calculating the
distance between histograms using formula 5.5.

*D** _{mj}* =

*h*

_{tj}*·*

**H**

_{mj}*∥h*_{tj}*∥∥***H**_{mj}*∥.* (5.5)

So we obtain the distance set*{D*_{mj}*}*, here*j* *∈ {L, R}, m*= 1, . . . , M.
Then, we perform estimation by the following 3 steps.

Step 1:

Look for the maximum value of distance *max(D** _{mj}*), and obtain the category

*A*

_{m}_{0}corre-sponding to

*max(D*

*). Then the distance pair*

_{mj}*{D*

_{m}_{0}

_{L}*, D*

_{m}_{0}

_{R}*}*corresponding to category

*A*

_{m}_{0}

is found in distance set*{D*_{mj}*}*.
Step 2:

If category*A*_{m}_{0} belongs to “co-contribution,” and ^{max(D}_{min(D}^{m}^{0}^{L}^{,D}^{m}^{0}^{R}^{)}

*m*0*L**,D*_{m}_{0}* _{R}*)

*< T*, the testing interac-tion is determined to be “co-contribuinterac-tion” interacinterac-tion.

If category *A*_{m}_{0} belongs to “single-contribution,” and ^{max(D}_{min(D}^{m}^{0}^{L}^{,D}^{m}^{0}^{R}^{)}

*m*0*L**,D*_{m}_{0}* _{R}*)

*> T*, the testing interaction is determined to be “single-contribution” interaction.

*max(D*

*m*0

*L*

*, D*

*m*0

*R*)indicates the major participant in the testing interaction.

Step 3:

If*{D*_{m}_{0}_{L}*, D*_{m}_{0}_{R}*}*fits neither of the two situations in step 2, the distance pair*{D*_{m}_{0}_{L}*, D*_{m}_{0}_{R}*}*
are set to 0 in*{D**mj**}*, return to step 1 to select the second largest distance.

The processing continues until the contribution estimation result is given in step 2.

The classification result clearly indicates the contribution class of the testing interaction sample, and determines the major actions in single-contribution interactions.