4.3 Summary
5.1.3 Object-layer update
In order to accurately detect both moving foreground objects and stationary foreground objects in complex situations, we need a novel background model whose structure corresponds to a
5.1 Object-level multi-layered background model
end
Is matches ?
Does the pixel (x,y) belong to ?
Yes Yes
Yes
No
No
Yes No
start
Input image
Select a first pixel
Detect as ithlayer (stationary object) Detect as moving foreground
Select the next pixel
Are all pixels processed?
No
Figure 5.3: The detailed process of layered object detection using the object-level multi-layered background model: For each pixel, all the object-layers are examined to see which layer is matched to the pixel in the order ofLmaxtoL1.
structure of a surveillance scene. In case of the proposed object-level multi-layered background model, by hierarchically conserving the object-layers for stationary objects, the structure of the multi-layered background model can correspond to the structure of the surveillance scene.
Then, to maintain the correspondence relationship of the structures between the multi-layered background model and the surveillance scene, the appearance and disappearance of stationary object need to be accurately detected in order to add/delete an object-layer to/from the multi-layered background model. Therefore, in this subsection, the author explains how to detect the appearance and disappearance of stationary objects and how the corresponding object-layers
Chapter 5. Object-level multi-layered background modeling in complex situations
are added to/deleted from the multi-layered background model.
Object-layer update based on foreground object state analysis
To conserve background models for each stationary object, we need to detect the appearance of stationary objects and create new background models corresponding to newly-observed sta-tionary objects. Here, the author discusses how to detect the appearance of a stasta-tionary object Onew, and to create a new object-layerLnewforOnew. The details of these processes are shown in Figure 5.4.
A stationary object can be defined as an object which stops moving (does not move) for long time. Then, looking at a stationary object frame-by-frame, we can consider that the ob-ject is continuously “staying” at the same position. According to this interpretation, in the proposed multi-layered background modeling, we can translate “stationary” into “being contin-uouslystayingat the same position.” Therefore, by determining whether each object is “mov-ing” or “stay“mov-ing” in each frame of a video sequence, the stationary objects can be detected.
To determine whether an object is “moving” or “staying,” in the proposed multi-layered back-ground modeling, an overlap ratio of blobs between two consecutive images is used as shown in Figure 5.5. By analyzing the topological structures of the pixels detected as “foreground”
according to Eq.(5.3), we can extract blobs of foreground objects {bt1, . . . ,bti, . . . ,btηt}, where ηt is the number of blobs detected at time t. Note that here all the blobs consist of only the foreground pixels. Then, the state of a foreground object corresponding to a blobbti at framet is defined using an overlap ratio as follows.
ω(bti) =
⎧⎨
⎩
Staying if |bti∩bt−1i |
|bti∪bt−1i | ≥,
Moving otherwise,
(5.5)
where,is a threshold for determining whether the blobbti is “staying” or “moving,” andbt−1i
is a blob which corresponds tobti at framet−1. If there are no such blobs which overlap with the blobbti at framet−1, an empty blob whose size is0is employed asbt−1i .
As described above, a stationary object is defined as an object which is continuously staying
5.1 Object-level multi-layered background model
No
No
Yes
No (moving object)
(stationary object candidate)
Yes
Yes start
Multi-layered object detection result
i+1 → i
Extract blobs of foreground objects
0 → i
end
Are all blobs processed?
Select a blob
Select the corresponding blob from the previous result
Is the overlap rate between and larger than ?
Is the blob staying for more than consecutive frames?
Create a new object-layer for , and add it to the multi-layered background model
Figure 5.4: The details of the object-layer update according to the foreground object state anal-ysis: This process creates and adds new object-layers for each stationary object, which is an object continuously “staying” at one position, to the multi-layered background model.
Chapter 5. Object-level multi-layered background modeling in complex situations
: moving : moving
time: t-1 time: t
Overlap Blobs corresponding to each foreground object
(a) Example of a “moving” bus
: staying : moving
time: t-1 time: t
Overlap Blobs corresponding to each foreground object
(b) Example of a “staying” bus
Figure 5.5: Foreground object state analysis: (a) and (b) correspond to the states where a bus is “moving” and “staying,” respectively. The overlap images visualize the similarities of blobs between two consecutive images, and the similarity increases in proportion to the overlap ratio.
In a case where a ratio of deeply-colored area exceeds a threshold, a corresponding blob is determined as “staying.”
at the same position. Therefore, when a blobbticontinuously-determined as “staying” more than λframes based on Eq.(5.3), it is detected as a newly-observed stationary objectOnew=bti. Note that here stationary objects are detected not at pixel-level but at object-level, i.e., blob-level.
This allows the proposed multi-layered modeling to keep the integrity of stationary objects, which is useful in correctly identifying the disappearance of the stationary objects (will be discussed in the later part of this subsection).
When a newly-observed stationary objectOnewis detected, we need to create a new object-layerLnewby constructing background models βnew(x,y) for each pixel(x, y)belonging toOnew. To create and initialize a background modelβ(x,y)new of a pixel(x, y)∈Onew, the author proposes to use a feature set{X(x,y)t−λ+1, . . . ,X(x,y)t }observed in the most recentλframes. The reason why this feature set is used for initialization of the background model is that all the stationary objects
5.1 Object-level multi-layered background model
are detected after continuously “staying” at the same position for more thanλframes. There-fore, when a stationary object is newly-observed, it can considered that at least the most recentλ features are attributed to the stationary object. This is why, in the proposed multi-layered back-ground modeling, each layer can be effectively initialized using the past information. Then, the new object-layer Lnew can be defined as Lnew = {β(x,y)new|(x, y) ∈ Onew}, and the object-level multi-layered background modelB,≺is updated asB+{Lnew},≺.
Object-layer update based on stationary object state analysis
To keep the proposed multi-layered background model reflecting a structure of a surveillance scene, when a stationary object moves away, we need to delete an abandoned object-layer cor-responding to the disappeared object from the multi-layered background model. The details of the detection of disappeared objects and deletion of the corresponding object-layers are illus-trated in Figure 5.6. When an stationary object appears, a corresponding background region is occluded by the stationary object. In contrast, when an existing stationary object moves away, a corresponding background regions, which has been occluded by the stationary object, is uncovered. By using the characteristics of the occlusion described above, the disappearance of an existing stationary object can be detected. In practice, the correspondence relationship between the multi-layered object detection result and the structure of the multi-layered back-ground model is analyzed as shown in Figure 5.7. As shown in Figure 5.7(c), if an existing stationary object Oi has already moved away, then most of the pixels(x, y) ∈ Oi should be judged as background or lower object-layers Lu (Lu ≺ Li) according to Eq.(5.3). Under this assumption, the states of each existing stationary object can be estimated as follows.
Υ(Oi) =
⎧⎨
⎩
Disappeared if ς(Oit)≥ξ, Still existing otherwise,
(5.6) where, ξ is a threshold for determining whether a stationary object Oi is “still existing” or
“disappeared,” andς(Oit)is a function detected as ς(Oi) = 1
|Oi|
(x,y)∈Oi
δ(i,Θ(X(x,y)t )), (5.7)
Chapter 5. Object-level multi-layered background modeling in complex situations
No
Yes
No start
Multi-layered detection result
end
For the pixels , count the pixels matching to a lower layer
Yes
Recognize the disappearance of the stationary object , and delete the corresponding layer
from the multi-layered background model Is the uncovered rate
larger than ?
Figure 5.6: The details of the object-layer update according to the stationary object state analy-sis: This process detects the disappearance of a stationary objects and delete their corresponding object-layers from the object-level multi-layered background model.
5.1 Object-level multi-layered background model
object-layer
L
1L
2L
3Stationary object Initial background
Stationary object
input image at time t
detection result
in the region of L3 (b) an example where the state of a stationary object (= the yellow bus) is “still existing”
(c) an example where the state of a stationary object (= the yellow bus) is “disappeared”
(a) object-level multi-layered background model at time t
input image at time t
detection result
in the region of L3
( is the yellow bus)L3
( is the yellow bus)L3
Figure 5.7: Examples of stationary stationary object state analysis: (a) visualizes the object-level multi-layered background model at timet. (b) and (c) show the examples where the states of a stationary object, i.e., the yellow bus, are “still existing” and “disappeared,” respectively.
where,Θ(X(x,y)t )is a multi-layered detection result according to Eq.(5.3), andδ(i, j)is a func-tion detected as
δ(i, j) =
⎧⎨
⎩
1 if Lj ≺Li, 0 otherwise.
(5.8) According to Eq.(5.6), we can detect the disappearance of existing stationary objects not at pixel-level but at object-level. Then, we can delete all parts of an object-layer corresponding to a disappeared stationary object, if its corresponding region is partly occluded by foreground objects and stationary objects of higher object-layers. This is owing to the integrity of stationary objects, which is maintained by object-level stationary object detection as explained in the first part of this subsection.
Finally, when the disappearance of an existing stationary layer is detected according to Eq.(5.6), its corresponding object-layerLiis deleted from the multi-layered background model.
Chapter 5. Object-level multi-layered background modeling in complex situations
pixel clusters L1
object-layer
pixel clusters object-layer
Stationary object L1
L2
pixel clusters object-layer
L1 L2
L3 object-layer addition
object-layer addition
object-layer deletion object-layer deletion
a stationary object newly-observed
a stationary object disappeared a stationary object
newly-observed
a stationary object disappeared
Figure 5.8: The transition of pixel clusters of each object-layer: Red arrows show how the pixel clusters of lower object-layers are divided for a new object-layer. Blue arrows show how the pixel clusters of an abandoned object-layers are merged into the lower object-layers.
Then, the object-level multi-layered background modelB,≺is updated asB− {Li},≺.