平成２８年度 修士論文
Non‐contact measurement of eyeblink by using Doppler
sensor
学籍番号 １５３２０７２ 氏名 ＡＭＩＲ ＭＡＬＥＫＩ 知能機械工学専攻 内田雅文研究室 指導教員 内田雅文 准教授 副指導教員 桐本 哲郎 教授 提出日 平成２９年０２月２８日
Non‐contact measurement of eyeblink by using Doppler
sensor
Amir Maleki
Graduate School of Informatics and Engineering
The University Of Electro‐Communications
Master Thesis
March 2017
修士論文２８年度
ドップラーセンサーを用いたアイブリンクの非接側定
論文要旨 自然界にはさまざまなゆらぎ現象が確認されるが、それらが統計的に完全に独立で不 規則な過程を示すことは稀である。瞬目はヒトにとって極めて自然な運動の一つであり、 認知科学,ヒューマンコンピュータインタフェース等に応用が期待されている。例えば 認知科学の分野では，ヒトの数々の精神的かつ身体的な運動が 1/f ゆらぎの特性を示し 得ることが確認され、多くの研究者がゆらぎ特性の意味することを見つけようと試みて いる。瞬目は車両および航空機運転者の眠気の探知に有効である為，瞬目の計測の重要 性は高い。従来，瞬目の計測には EOG、赤外線センサ、カメラおよびドップラーセンサ などが 用いられてきた。近年では、グーグルグラスのよう、赤外線センサなどの搭載 されたセンサにより瞬目を計測可能な眼鏡型デバイスがいくつか提案されている。ドッ プラーセンサは、他の手法では問題となる長距離での検出などいくつかの長所がある。 故に、本研究ではドップラーセンサを搭載することによりヒトの瞬目を計測可能な眼鏡 型デバイスの開発を目的とする。 眼鏡型デバイスは運転中に装着可能である事から、運転者の眠気検出が可能であり、 ヒューマンエラーの抑制に寄与できるものと考えられる。実験はドップラーセンサーに より瞬目を計測すると共に、身体の動きを計測するための加速度センサ、および瞬目信 号が正当であることを確認するための EOG を用いて行う。実験結果より、瞬目に関する ドップラー周波数は 2Hz 近辺に高い周波数を含むことが確認された。さらに、観測され るデータに頭や体の動きが含まれてしまう場合、PCA を用いてそれらから瞬目信号を分 離可能であることを確認できた。Non‐contact measurement of eyeblink by
using Doppler sensor
Amir Maleki
Abstract
Many phenomena in the nature exhibit anomalously large temporal fluctuations exceeding what cannot be explained as a consequence of statistically independent random events. Eyeblink is one of the natural parts of human activities which can be used in diverse applications such as Cognitive fields, Human Computer Interfaces etc. For instance, there is a large and growing body of evidence that sequences of psychophysical data fluctuate as 1/ƒ noises and many researchers are trying to find what these fluctuations are suggesting us. Considering the fact that eyeblink has been proposed as a marvelous way for detecting driver’s and pilot’s drowsiness, the importance for eyeblink detection is increasing. There are several ways for eyeblink detection such as the EOG method, Infrared sensors, Cameras, and Doppler sensors. Recently, several types of glasses have been produced, for instance ‘Google glass’, which is able to detect eyeblink using embedded sensors such as infrared sensors. The objective of this thesis is to develop a glass which can be used to measure human eyeblink through embedding a Doppler sensor inside it, considering to the fact that Doppler sensors have a number of advantages, for instance, long distance detection that is a problem in other methods. This glass can be used during driving and flying in order to detect drowsiness and therefore prevent accidents caused by human errors. Furthermore, it can be deployed for measuring several psychological factors during performing tasks in clinical setups. We designed an
experimental setup in which measurement of the eyeblink was conducted using a Doppler sensor accompanied by an accelerometer for body movement measurements and also an EOG (Electrooculography) recording for verifying detected eyeblink signal by Doppler sensor. From our experimental results, we found that the Doppler frequency related to eyeblink contains a dominant frequency near 2 Hz. Finally, we found that when blinking is accompanied with head and body movements, eyeblink signal is separable by deploying the Principal Component Analysis.
Contents
Figure index ... 5 Chapter 1 Introduction ... 8 1.1 Background ... 8 1.2 Motivation ... 9 1.3 Related works on the eyeblink detection ... 10 1.3.1 EOG (Electrooculography) method ... 10 1.3.2 Optical sensors ... 12 1.3.3 IR (Infrared) sensors ... 14 1.3.4 Doppler sensors ... 16 1.4 Aplication ... 22 1.5 Thesis organization ... 23 Chapter 2 Concept ... 24 2.1 Doppler Sensor ... 25 2.2 Accelerometer ... 26 Chapter 3 Measurement system ... 27 3.1 EOG recording ... 28 Chapter 4 Analysis method and results ... 32 4.1 results when the subject’s head and body was steady ... 33 4.2 results when the subject’s head and body was not steady ... 40 Chapter 5 Conclusion and future work ... 54 Acknowledgment ... 56 References... 57 Publications and conferences ... 59
Figure index
Figure 1.1 ORBICULARIS OCULI muscle……….11 Figure 1.2 Infrared proximity sensor built into Google Glass………15 Figure 1.3 Blink detection by calculating the peak value……….…16 Figure 1.4 source and motive receiver……….…17 Figure 1.5 motive source and receiver………..19 Figure 2.1 the concept of the designed glass………..….24 Figure 2.2 Doppler sensor………..25 Figure 2.3 Accelerometer……….….26 Figure 3.1 Experimental system……….…27 Figure 3.2 EOG recording………28 Figure 3.3 MULTI AMPIFIER MA1000‐DIGITECS……….…29 Figure 3.4 Experimental setup when the Doppler sensor was fixed to a stand………29 Figure 3.5 Experimental setup when the Doppler sensor was attached to a pair of glasses……..30 Figure 3.6 three different angles for eyeblink detection when the Doppler sensor was mounted on a fixture and head and body were fixed………...30 Figure 3.7eyeblink detection when when the Doppler sensor was attached to a pair of glasses……….…31 Figure 4.1 Eyeblink detection flow……….32Figure 4.2 Experimental Protocol……….……33 Figure 4.3 a sample of recorded EOG………34 Figure 4.4 Spectrum of the recorded EOG……….…34 Figure 4.5 A sample conscious eyeblink detected by Deppler sensor……….…..35 Figure 4.6 Spectrum of the Sample conscious eyeblink detected by Deppler sensor……….…35 Figure 4.7 A sample unconscious eyeblink detected by Deppler sensor………..36 Figure 4.8 Spectrum of the Sample unconscious eyeblink detected by Deppler sensor………36 Figure 4.9 Mean of 100 samples of conscious eyeblink………37 Figure 4.10 Mean of 100 samples of unconscious eyeblink………..…37 Figure 4.11 Spectrum for mean of 100 samples of conscious eyeblink……….……38 Figure 4.12 Spectrum for mean of 100 samples of unconscious eyeblink………38 Figure 4.13 Experimental results when eyeblink was conscious: (a) ‐45 degree below the eye center, (b) 0 degree and (c) +45 degree above the eye center………39 Figure 4.14 Experimental results when eyeblink was unconscious: (a) measured data from ‐45 degree, below the eye center, (b) 0 degree and (c) +45 degree above the eye center………….….40 Figure 4.15 The arithmetic mean of 100 samples of Doppler signals: (a) when the subject sat in front of the personal computer and performed a routine task and (b) when the subject performed the walking task……….…41 Figure 4.16 The power spectrum of the signals represented in figure 4.14………..…42 Figure 4.17 The accelerometer signals: (a) when the subject sat in front of the personal computer and performed a routine task and (b) when the subject performed the walking task……….…..43
Figure 4.18 The Power Spectrum of the signals represented in figure 4.16 (Accelerometer signals)………43 Figure 4.19 First principal components of the signals when the angle between the sensor and the center of eye is 0°: (a) conscious blinking and (b) unconscious blinking……….……45 Figure 4.20 The power spectrum of the first to sixth principal components of the signal when the subject sat in front of a personal computer and performed a routine task……….….46 Figure 4.21 The power spectrum of the first to sixth principal components of the signal when the subject performed a walking task………47 Figure 4.22 Reconstructed blinking signal when the subject sat in front of a personal computer and performing a routine task………48 Figure 4.23 Reconstructed blinking signal when the subject performed a walking task………49 Figure 4.24 Eyeblink detection procedure when the subject sat in front of a personal computer and performing a routine task………50 Figure 4.25 Eyeblink detection procedure when the subject performed a walking task………..…50 Figure 4.26 The power spectrum of the first to sixth principal components of the signal when the second subject sat in front of a personal computer and performed a routine task………..….51 Figure 4.27 The power spectrum of the first to sixth principal components of the signal when the second subject performed a walking task………....52 Figure 4.28 Eyeblink detection procedure when the second subject sat in front of a personal computer and performing a routine task………..…….53 Figure 4.29 Eyeblink detection procedure when the second subject performed a walking task……….53
Chapter 1 Introduction
1.1 Background
The importance of eyeblink detection has been raised since blinking has been introduced as a complex phenomenon which reflects the influence of higher nervous processes so that different experimental conditions consequences changes in blink patterns. The blink rate decreases significantly while performing visual tasks and this inverse relationship between blink rate and task demands may address mental demands for information processing [1][2]. Recently, development of human‐computer interfaces is attracting worldwide attention of researches [3]. In order to measure the blinking activity there are several techniques such as EOG (Electrooculography), image processing, Infrared proximity sensor, Doppler effect sensor, etc. EOG technique is the most precise method but it needs several electrodes to be attached to the skin and it seems somewhat impractical during implementing every day’s activities. Image processing methods are noninvasive and detect the blinking patterns by focusing on face recognition. However, in condition such as driving in which the light intensity changes time by time the eyeblink detection may be unsuccessful [3]. One common technique of eyelid movement detection utilizes measurements of infrared (IR) light reflected from the surface of the eye. The performance of current IR sensors, however, is limited by their sensitivity toambient infrared noise, by their small field‐of‐view and by short working distances [4]. Doppler sensors are a non‐contact way for measuring eyeblink and it is insensitive to light intensity and it can detect blinking from a considerable long distance. Thus using Doppler sensors has a high potential of becoming a practical way for eyeblink detection. Unfortunately, there are a few researches about the eyeblink detection using Doppler sensor and there is not any concrete method for this type of eyeblink measurement.
1.2 Motivation
Eyeblink may find a verity of applications. As an important example, by detecting the eyeblink rate, the measurement of level of drowsiness in drivers becomes possible and consequently it can drastically reduce the vehicular accidents caused by human failures. Furthermore, eyeblink detection may find applications for measuring several psychological factors, e.g. mental workload, cognitive load, etc. Thus it can be employed for pathological settings and also for improvement of workers efficiency while implementing tasks. The aim of this study is to verify eyeblink detection in several real conditions such as working with computer at office and walking and also develop a method in which the separation of the eyeblink signal from the original signal that is contaminated with head and body movement becomes possible. The speed of a voluntary eyeblink for the closing and opening phases are [52~138 mm/s] and [24~47 mm/s], respectively [5]. Therefore we can estimate that the Doppler frequency of blinking has an approximate range of [1~5.4 Hz]. However, this range can be extended to [1~18 Hz] due to the higher speed of blinking in some human subjects. Our empirical results showed that the head and body movements has the same frequency range as
such as performing tasks, walking, etc. Thus conventional low pass filters are almost inefficient. We deployed Principal Component Analysis in order to extract the blinking signals from the mixed raw signals and we observed that there is a high possibility of separating the eyeblink signals.
1.3 Related works on the eyeblink detection
1.3.1 EOG (Electrooculography) method
One of the most precise methods of eyblink detection is EOG (electrooculography). Eyeblinks are periodic closings and re‐openings of the eyelid. The duration of the eyelid closure is used as the criterion to discriminate an eyeblink from an eye closure. An eyeblink closure duration is usually 300 msec and is more typically 200 msec or less. The EOG records changes in the electrical potentials between the cornea and retina as the eyelid movement occurs. The lid closing over the eye causes a difference in the corneal/retinal potential that is evident in the EOG. Therefore, the EOG can be used to detect eyeblinks. However, the measurement of eyeblink can be achieved by placing electrodes on the lower and upper (or vertical) or the left and right (or horizontal) sites of the orbicularis oculi muscle (above and under the right eye) shown in figure 1.1[6].Figure 1 U EMG, and involunta EMG ma be achie the signa method attached during pe 1.1 ORBICUL Usually EOG s d saccade ey ary eyeblink y be conside ved by mea al exceed t is reliable b d to the face
erforming ev ARIS OCULI signals are c ye movemen k and the ve ered as eyeb asuring the a his average but the prob
e skin of th veryday’s ta muscle, (An ontaminate nt. It is prop rtical electro blink by erro average valu e value (or
blem with th he subjects a sks [7]. atomy of th 2013). d with noise posed that vo ode yields b or. One of th ue of eyebli any thresho his method and it is alm
e Face and N e including in oluntary eye better detect he methods nk signal fo old value) t
is that it ne most imprac Neck, Peter nvoluntary e eblink is dist tion of eyeb for eyeblink or each indi the signal is eeds some ctical for ey M. Prenderg eyeblink artif tinguishable blink, but str k recognition vidual and w s detected. electrodes t yeblink dete gast, facts, from rayed n can when EOG to be ection
1.3.2 Optical sensors
This method is based on optical tracking of eyelid movements to detect eye blinks. Detection is based on matching SIFT (scale‐invariant feature transform) descriptors computed on GPU. First, threshold frame difference inside the eye region locates motion regions. Consequently, these regions are being used to calculate the optical flow. While user blinks, eyelids move up and down and the dominant motion is in vertical direction. This method detects 97% of blinks on their dataset. Most of the false positive detections are the result of gaze lowering and vertical head movements. There are several methods for eyeblink detection, one of them is optical flow estimation. It locates eyes and face position by 3 different classifiers. The algorithm is successful mostly when the head is directly facing the camera. The KLT (Kanade–Lucas–Tomasi) tracker is used to track the detected feature points. This blink detector uses GPU‐based optical flow in the face region. The flow within eyes is compensated for the global face movement, normalized and corrected in rotation when eyes are in non‐horizontal position. Afterwards dominant orientation of the flow is estimated. The flow data are processed by adaptive threshold to detect eye blinks. Authors report good blink detection rate (more than 90%). However this approach has problems with detecting blinks when eyes are quickly moving up and down. The eyelid movements are estimated by normal flow instead of optical flow. It is the component of optical flow that is orthogonal to the image gradient. Authors claim that the computation of normal flow is more effective than the previous methods. Arai et al. present Gabor filter‐based method for blink detection. Gabor filter is a linear filter for extracting contours within the eye. After applying the filter, the distance between detected top and bottom arc in eye region is measured. Different distance indicates closed or opened eye. The problem of arc extraction arises while the person is looking down. Variance map specifies distribution of intensities from the mean value in an image sequence. The intensity of pixels located in eye region changes during the blink, which can be used in detection process. Correlation measures the similarity between actual eye and open eye image. As someone closes eyes during the blink, correlation coefficient decreases. Another blink detection algorithm is based on the fact that the upper and lower part of eye has different distribution of
mean intensities during open eyes and blinks. These intensities cross during the eyelid closing and opening. Liting et al. [8] use a deformable model ‐ Active Shape Model represented by several landmarks as the eye contour shape. Model learns the appearance around each landmark and fits it in the actual frame to obtain new eye shape. Blinks are detected by the distance measurement between upper and lower eyelid. Ayudhaya et al. [9] detect blinks by the eyelid’s state detecting (ESD) value calculation. It increases the threshold until the resulting image has at least one black pixel after applying median blur filtering. This threshold value (ESD) differs while user blinks.
Here we will introduce briefly two proposed methods based on histogram backprojection and the Inner movement detection based on KLT feature tracker.
1.3.2.1 Histogram Backprojection
Histogram backprojection creates a probability map over the image. In other words backprojection determines how well the pixels from the image fit the distribution of a given histogram. Higher value in a backprojected image denotes more likely location of the given object. High percentage of skin color pixels within the eye region represents closed eyes and otherwise eyes are considered opened.
1.3.2.2 Inner Movement Detection
Optical flow is usually used in this method. Optical flow is the relation of the motion
field:
The 2D projection of the physical movement of points relative to the observer to 2D displacement of pixel patches on the image plane. One of the most common methods called KLT tracker selects features suitable for tracking with high intensity changes in both directions.
If a user blinks, the mean displacement of feature points within the eye region should be greater than the displacement of the rest of the points within the face area.
The first step consists of localizing a user’s face and eyes using Haar Cascade Classifier on grayscale image. Then random KLT features within the eye and nose regions are initializes and classified as left ocular, right ocular or non‐ocular. These features are being tracked by KLT tracker. Tracking is reinitialized in regular intervals or in case of loss of many feature points. After that the average displacement is computed separately for three groups of points. Finally, the difference between the left or right ocular and non‐ocular movement displacements is computed. If this difference exceeds a threshold value, a movement within the eye region is anticipated [10].
1.3.3 IR (Infrared) sensors
Another common method of eyeblink recording utilizes infrared (IR) photoelectric sensors. This approach measures IR light reflected from the surface of the eye. A typical IR eyeblink measurement device consists of an IR light emitting diode (LED), which illuminates the eye surface, paired with an IR photodiode that detects IR light reflected back from the eye [11].An ideal IR eyeblink detector should have several important properties. To detect the full range of eyelid movement, the IR LED should completely illuminate the surface of the fully opened eye, and in addition, the field‐of‐view of the IR photodiode should encompass the whole eye area. Some currently used detectors rely on commercial proximity sensors that not only must be positioned close to the eye but also emit a narrow IR beam resulting in incomplete coverage of the full range of eyelid motion. Google glass implements an infrared proximity sensor facing the users’ eye that can be used for blink detection shown in Figure 1.2.
Fig Blinks moved o peak or center po p7, p8 an excluded center po defined a the peak working gure 1.2 Infra s are detecte n the senso not accordin oint in the w nd p9) is calc d from the a oint. If the d as a blink. B k detection v distance are ared proxim 1.3.3.1 Bl ed based on r data stream ng to the fo window (p5 culated. The average calc distance is la Because the varies for ea e small and s ity sensor b link detect n the raw in m and moni ollowing def in Figure 1. e preceding a culation bec arger than a shape of the ach user. IR short respec uilt into Goo ion nfrared prox tored wheth finition. The 3) to the av and subsequ ause their s threshold ra e face and e sensors are tively. ogle Glass (Is ximity senso her the cent distance fr verage value uent points o sensor value anging from eye location e noninvasiv shimaru et
or signal. A s ter point of e rom one sen e of other po of the cente es are often 3.0 ‐ 7.0, th vary, the b e but their f al., 2014). sliding windo each window nsor value o oints (p1, p2 er (p4 and p6 n affected b he center po est threshol field‐of‐view ow is w is a of the 2, p3, 6) are y the oint is ld for w and
F E moving frequenc T relative observed receiver. Figure 1.3 Bl
1.3.4
lectromagne object is ill cy of the retu he Doppler motion betw d frequency ink detectio4 Doppler
etic waves h luminated b urn signal. 1.3.4.1 Do Effect is th ween the s is only the on by calcular sensors
have been u by a continoppler Effe he change in ource and t motion in t ting the pea utilized for nuous wave, ect n the obser the receiver he Line‐Of‐S ak value (Ish the detectio , it generat rved freque r. The relat Sight (LOS) b himaru et al
on of eye b tes a Dopp
ncy of a so tive motion between the ., 2014). blinking. Wh pler shift on ource due to that affect e source and hen a n the o the s the d the
If propagat will dete earlier th the LOS. changed S where f a source is te out from ct the sound han it would Thus the f by this relat ince the orig , th s stationary, the source d coming fro d have if the frequency th tive motion w is t ∆ is th is t Figu ginal freque he observed
as the one as shown b om the sourc e receiver we hat each su where: ∆ the original he change i he velocity ure 1.4 sourc ncy of the s d frequency shown in f elow. As the ce but each ere stationa uccessive wa wavelength n the observ of the recie ce and motiv ource can b becomes: igure 1.4, it e receiver m successive s ary, due to t ave front w 1.1 h of the sour ved frequen ever in the L ve receiver. be expressed t will emit so moves towar sound wave he motion o ould be det
rce ncy LOS d in terms of ound waves rds the sour e will be dete of the receiv tected wou f the wavele s that rce, it ected ver in ld be ength
∆ 1.2 1.3 1.4 Note that this equation only works if the relative velocity of the receiver, is towards the source. If the motion is away from the source, the relative velocity would be in the opposite direction and the equation would become: 1.5 The two equations are usually combined and expressed as: 1.6 If the source is moving towards the receiver, the effect is slightly different. The spacing between the successive wave fronts would be less as seen in the figure 1.5. This would be expressed as:
T N moving a o calculate t Note that this away, the eq Figu is the observed s is only whe quation wou ure 1.5 moti ∆ s the relative d frequency ∆ en the sourc ld be change ve source an e velocity o : ce is moving ed to: nd receiver. 1.7 f the source 1.8 1.9 towards the e e receiver. Iff the source is
1.10 When combined with the previous result, the equation would be expressed as: ∓ 1.11 Notice that this time, the plus/minus symbol is inverted because the sign on top is to be used for relative motion of the source towards the receiver. By combining the previous results, we can derive one equation to use as the Doppler Equation. This is usually expressed as: ∓ 1.12 It must be noticed that the quantities for the velocity of the receiver, , and the velocity of the source, , are only the magnitudes of the relative velocities in (or along) the LOS. In other words, the component of the velocity of the source and the receiver, that are perpendicular to the LOS do not change the received frequency. Secondly, the top sign in the numerator and the denominator are the sign convention to be used when the relative velocities are towards the other. If the source were moving towards the receiver, the sign to use in the denominator would be the minus sign. If the source were moving away from the receiver, the sign to use would be the plus sign. 1.3.4.2 Eyeblink measurement using Doppler Effect By the time of writing this thesis there are only two related works that explain eyblink detection using Doppler effect: One work was conducted by Kamil et al. in 2012. The research was carried out in order to measure the level of drowsiness in drivers. They deployed a multiple receiver microwave Doppler radar. The sensor was consisted of a synthesized signal source with the output frequency of 9 GHz. The system implemented a single side band SSB modulation and a quadrature modulator structure. The modulation frequency was 1 KHz and the antenna was a
In order to find the signal’s characteristics they implemented several methods and they reported that: Through frequency analysis, FFT, of the recorded eyeblink signal, the frequency related to blinking is completely covered by the spectrum of general head movement signal. Implementing Continuous Wavelet Transform (CWT) they reached the same results as FFT. They could not find any useful information in signal phase extraction.
Finally they selected time domain analysis in order to reach the eyeblink signal and introduced a separated wave form for eyeblink detected by Doppler sensor.
As the signal detection procedure after passing the measured signal and signal de‐ noising procedure using multilevel discrete wavelet decomposition, they achieved a signal containing several sinusoidal waveforms. In the next step, they calculated the envelope of the represented signal by using a simple rectangular filter. Finally, they determined a threshold and when the amplitude of the envelope exceeded the threshold they considered the signal as a blink. Unfortunately, they didn’t provide detailed information for their experimental method and also the frequency characteristics of the eyeblink signal remains undetermined.
Another study aimed to detect eyeblink using Doppler sensor was conducted by Youngwook Kim in 2015. He performed several measurements with/without the noise caused by human movement when the sensor was placed near the eyes. Then he analyzed the Doppler signal in the joint time–frequency domain and suggested that the Doppler frequency produced by eye blinking was approximately 115 Hz. Furthermore, he proposed that unconscious and conscious eye blinking exhibited different Doppler characteristics and deployed Principal Component Analysis In order to classify these characteristics. Then he introduced a method in which conscious and unconscious eyeblink can be distinguished by multiplying truncated
Kim implemented PCA method as an image processing method of target recognition for distinguishing the conscious from unconscious blinking and also noise. He represented the sorted eigenvalues, in which two eigenvalues contain more than 97% of the total energy and the reconstructed two‐dimensional images from the one dimensional eigenvectors that corresponded to the highest eigenvalues. He claimed that the first eigenvector was similar to the image of conscious eye blinking, and the second eigenvector was analogous to that of the unconscious eye blinking.
1.4 Aplication
Eyeblink detection may find many applications in a wide variety of the fields. For instance investigation of eye‐blinking activity can have applications in the detection of the unconsciousness of pilots and the drowsiness of drivers. One study was conducted in 2013 by Genis et al. in order to investigate connection between changes in blinking and changes in a driving‐ task difficulty. They implemented a camera located inside a car for blink detection. They reported that changes in task difficulty follow fluctuation in eyeblink frequency [14].
Another study was performed in 1998 in order to measure the Physiological workload reactions to increasing levels of task difficulty. They measured blink intervals when changing a pilot‐task difficulty and they observed that eyeblink interval fluctuates with task difficulty level and this feature can be used for measuring the mental workload level allocated to each task. Furthermore, eyeblink can be used as a simple communication (yes/no) for soldiers in the battlefield and patients with spiral‐cord injuries, where normal verbal communication is not possible. In addition, eye blinking can serve as a communication methodology such as human input into a computer. For example, eye blinking has also been studied for the development of a head set type computer mouse designed for the disabled [15].
1.5 Thesis organization
Chapter 1: Introduction We discuss about the background, application, motivation and related works on eyeblink detection in this chapter. Chapter 2: Concept In this chapter we introduce our concept about making a glass for eyeblink detection. Chapter 3: Measurement system We introduce the measurement method and system for this study and also we give detail information about experimental items. Chapter 4: Analysis method and results In this chapter we represent our analysis method that is based on PCA method. We also show our experimental results in this chapter. Chapter 5: Conclusion The conclusion of this study has been discussed in this chapter. We also have an overlook on the future part of this study.D objective to measu by huma detection This Acce The follo protectio Developing a e of this thes ure the drow an errors. W n and also w elerometer p owing we wi on glass w
a glass whic sis. This glas wsiness stat We deployed we used an plays a part i Figure ill introduce without any
Chapte
ch is able to s can measu e in drivers d a Doppler n accelerom in the analys 2.1 the cone each part o metal par
er 2 Con
o detect ey ure the eyeb and pilots a r sensor, atteter for hea sis and meas
cept of the d
of the glass rt which m
ncept
yeblink in hu blink in diverand therefo tached to a ad and bod surements. designed gla . Where, th may cause uman subje rse condition re prevent a pair of glas y movemen Figure 2.1 d
ass.
e glass itsel some dist
ects is the m ns. It can be
accidents ca sses for eye nt measurem
epicts the gl
lf is a plastic turbance in major used aused eblink ment. lass. c eye n the
electrom the expe
2
F power Pu is 5.8 GH antenna output (I by Samra output, i magnetic wav rimental me2.1 Dopp
igure 2.2 sh ulsed Dopple Hz which co with a 60 & Q) and it akhsh Co. In .e. ves produce easurementspler Sens
ows the Do er Radar (PD orresponds degree con responds to this study w across al ed by the Do s.sor
ppler senso DR) and has to a wavele nical coverag o radial velo we calculate ll the study. Figure 2.2 oppler senso r employed a detection ength of ab ge pattern. ocities betwe d the amplit 2 Doppler se or and there in our expe range up to bout 5.2 cm The outpu een 2.6 cm/s tude of the ensor. efore conseq eriment. The o 10 m. the c. it has an t is a Cohe s and 2.6 m/ Doppler sen quences fau e sensor is a center frequ on‐board d erent quadr /s, manufact nsor’s quadr ults in a low‐ uency dipole ature tured ature
T W and λ is t
2
T power, c accelerat gravity in or vibrat ( he relation b Where is t the wavelen2.2 Accel
he Accelero complete 3‐ tion with a m n tilt‐sensing ion. The sen ) fo between rad the Doppler gth at the celeromete
ometer is AD axis acceler minimum fu g application nsitivity of s or this study dial velocity 2 r frequency, enter frequeer
DXL335 sho rometer wit ull‐scale rang ns, as well as sensor is 300 . Figure 2.3 and Doppler is the rad ency.own in figur th signal con ge of ±3 g. It s dynamic ac 0 mV/g. we 3 Accelerom r frequency 2.1 dial compon re 2.3. The nditioned vo t can measu cceleration r calculated meter. is: nent of the
sensor is a oltage outp ure the stati resulting from the amplitu target’s velo small, thin puts. It mea ic accelerati m motion, s de of these ocity, , low sures on of hock, axes
Figure 3 eyeblink conducte A signals to .1 depicts t detection, a ed by placing All recording o digital by t
Chapte
the experim and accelero g three elect F gs took plac he A/D boarer 3 Mea
mental syste ometer for h trodes place Figure 3.1 Exce using a p rd.
asurem
em. This sys head and bo ed on the sub xperimental personal coment sys
stem consis ody measure bjects face a l system. omputer aftstem
sts of a Do ement and a and using a m er convertin ppler senso an EOG reco multi amplifi ng the anal or for ording ier. logueT and upp electrode to verify MULTI A paced on electrode resistanc 0.3 Sec, H
3.1 E
he EOG reco per sites of
e placed ove the blinking AMPIFIER MA n the upper
es the elect ce and bette HFF = 100 Hz
EOG reco
ording for t the orbicul er the forehe g signals co A1000‐DIGIT and lower s rodes the s er measurem z, Sensitivityording
he eyeblink laris oculi m ead as the g llected by t TECS. As we sites of the urface of th ment of the y = 500 μV, t Figure 3. took place muscle (abo ground elect he Doppler e can see th orbicularis o he skin wasEOG. The s he reference
2 EOG recor
by placing t ove and und trode, repres sensor the he positive a oculi, respe cleaned ca etting for th e setting wa rding. two electro der the righ
sented in fig EOG were r and negative ctively. Befo refully for r he Multi am as standard.
des to the l ht eye) and gure 3.2. In o recorded us e electrodes ore attachin reducing the mplifier was: ower d one order sing a s was g the e skin TC =
Figure 3. D frequenc O 1‐ W 2‐ W 3 shows the Doppler and cy of 1000 H Our experime When the Do When the Do Figure 3.4 E e MULTI AMP Figure 3 Accelerome z by using an ent was cond oppler senso oppler senso Experimenta PIFIER MA10 3.3 MULTI AM eter sensors n AD board ( ducted in tw r was fixed t r was attach al setup whe 000‐DIGITEC MPIFIER MA s recordings (TRM‐7100) wo parts: to a stand, s hed to a pair en the Dopp CS. A1000‐DIGIT s were carr manufactur hown in figu r of glasses, d ler sensor w ECS.
ried out tho red by Interf ure 3.4. depicted in f was fixed to a orough a sa face. figure3.5. a stand. ample
Figure In angles: ‐4 eye and + Doppler Figure 3 e 3.5 Experim n part 1, the 45 degree (u +45 degree sensor was 5 .6 three diff mental setu Doppler sig under the ey (above the e 5 cm. ferent angles on a f p when the nal produce ye), 0 degree eye), shown s for eyeblin ixture and h Doppler sen ed by eyblink e (horizontal in figure 3.6 nk detection head and bod nsor was atta k was measu l to the eye o 6. The distan when the D dy were fixe ached to a p ured from th or at the sam nce between Doppler sens ed. pair of glasse hree differen me level wit n the eye and or was mou es. nt h the d nted
In eye and was 0° ac Figure 3 T 1‐ W 2‐ W n part 2, the Doppler sen cross all mea 3.7eyeblink d hen the mea When subject When subject e Doppler Se nsor was 5 c asurements, detection wh asurement w ts sat in fron ts performe ensor was a cm. In this p , depicted in hen when th was conduct nt of a perso d a walking ttached to t part the ang n figure 3.7. he Doppler s ed in two co onal compute task. the glass an gle between sensor was a onditions: er and perfo nd the distan the eye and attached to a ormed a rou nce betwee d Doppler se a pair of glas tine task. n the ensor sses.
A Doppler s order to d signal rec steady an and body collected eyeblink s frequency was perfo analysis w
Cha
s the proced sensor and E define the do corded by Do d the subject y movement u Doppler sign signal from th y and calculat ormed by de was conductedapter 4
dure of the OG when th ominant frequ oppler signal ts could able using the acc nals, contami hem. Finally, ted the mean efining a thr d using MATL Fi
Analys
eyeblink det e head and uency associa during perfo to move thei celerometer a nated with t we selected t n of them as t eshold value LAB software igure 4.1 Eyeis meth
tection, first, body were s ated with eac rming two ta r heads and b at this stage.he head and the principal the reconstru e. Figure 4.1 . eblink detec
hod and
we recorde steady. Then ch subject. N asks in which bodies arbitra After that, w body movem components ucted eyeblink depicts the ction flow.d result
ed and verifi we analyzed ext, we meas h the head a
ary. We also we applied P ment, in orde
which had th k signal. The eyeblink de
s
ed eyeblink d obtained da sured the eye nd body wer recorded the CA method t er to separat he same dom eyeblink dete etection flow using ata in eblink re not e head to the te the minant ection w. The
4
In head on the EOG procedur each me after the represen recorded Fo points fo consciou figure 4.44.1 resul
n this part w a fixture. Th signals in re for signalasurement a e global max nts the EOG d by the Dop
ollowing the or each eye
s and unco 4 shows the
ts when
we measured he EOG signa order to co l detection and then we ximum of th G recording ppler sensor. F e fact that th blink sample nscious blin correspondthe sub
d both EOG al was passe onfirm each first we calc e cropped h e EOG, this and the r Figure 4.2 Ex he sampling e. We colle nking. Figur ing frequencject’s he
and Dopple d through a h eyeblink' d culated the alf a second is shown in red signal r xperimental g frequency cted 100 sa re 4.3 repre cy spectrumead and b
er signal wh low pass filt detection u global max d of recorde figure 4.2. represents t Protocol. was 1 KHz t amples for eesents a sam m.
body wa
hen the subj ter. In fact w sing Dopple ximum of th ed Doppler s In this figur the original therefore we each measumple of rec
as steady
ects placed we have reco er signal. As e EOG signa signal before re the blue s l eyeblink s e had 1000 rement for cording EOG
y
their orded s the al for e and signal signal data‐ both G andF signal is the origi between are unde rom the Pow located in t nal recorded the eye an er 10 Hz. Fig Figur wer spectru he frequenc d conscious d Doppler s ure 4.3 a sa e 4.4 Spectr m of the EO cies up to 10 eyeblink an ensor was 0 mple of reco rum of the re OG signal we 0 Hz. Figure nd its power 0°. As we ca orded EOG. ecorded EOG e can see th e 4.5 and fig r spectrum n see most G. at most of t gure 4.6 show respectively of the powe the power o w one samp y when the a erful freque of the ple of angle encies
Fig F and its p 0°. As we Figure gure 4.6 Spe igure 4.7 an ower spectr e can see mo 4.5 A samp ectrum of th nd figure 4.8 rum respect ost of the po le conscious e Sample co 8 depicts the ively when t owerful frequ s eyeblink de onscious eye
e case for t the angle be uencies are etected by D eblink detect he original r etween the under 10 Hz Deppler sens ted by Depp recorded co eye and Do z. sor. ler sensor. onscious eye ppler senso eblink r was
Figu A waveform unconsci form and each mea Figure 4 ure 4.8 Spec
As a result ms and it ous eyeblin d their frequ asurement: 4.7 A sample ctrum of the we can se can be gre k. To better uency charac
e unconsciou
Sample unc
ee that con eatly used r understand cteristics we us eyeblink d conscious ey nscious and in order t ding the con e calculated detected by yeblink detec d unconscio o distinguis nscious and the arithme Deppler sen cted by Dep ous eyeblink sh between d unconsciou etic mean of nsor. pler sensor. k have diffe n conscious us eyeblink f 100 sample erent and wave es for
F blinking w igure 4.9 to when the an Fi o figure 4.1 ngle between Figure 4.9 M gure 4.10 M 1 100 10 represent n the eye an Mean of 100 Mean of 100 t the result nd Doppler s 0 samples of samples of u 4.1 ts for both sensor is 0°. conscious e unconscious conscious a eyeblink. s eyeblink. and uncons scious
F in figure selected A unconsci of subjec average opening igures 4.11 a 4.8 and figu those frequ Figure 4 Figure 4.
As a result ous blinking cts and may speed of ey phase, res and Figure 4 re 4.9. For b encies less t 4.11 Spectru .12 Spectrum we can se g are near 2 vary for eac elid has bee pectively. T 4.12 represen better analyz than 20 Hz. um for mean m for mean o
ee that the Hz. Howeve ch individua en suggested Thus as we
nts the pow zing the freq n of 100 sam of 100 samp e dominant er this freque l. As we intr d [52~138 m expected er spectrum quency featu mples of cons ples of uncon t frequencie ency depend roduced in t mm/s] and [2 the obtaine m of the repr ures of the e scious eyebl nscious eyeb
es for both ds only to th he previous 24~47 mm/s ed frequenc esented sign eyeblink we h ink. blink. h conscious he blinking s chapter tha s] for closing cy is within
nals have and speed at the g and n the
F from thre respectiv Figure A detection condition best dete this posit igure 4.13 a ee different vely. 4.13 Experim cen
As we can s n angle bet n that the se ectable angl tion. nd Figure 4. angles; ‐45° mental resul nter, (b) 0 de
ee in this f ween ‐45° ensor is fixed le is 0° beca 14 show the °, 0°, 45°; for lts when eye egree and (c figure, obvio to +45°. Th d and the he use the rad e measured D r both consc eblink was co ) +45 degree ously, the e is means th ead is movin ial velocity o Doppler sign cious and un onscious: (a e above the eyblink is w hat eyeblink ng upward a of the eyelid nal produced conscious ey ) ‐45 degree eye center. well detectab k detection nd downwa d has the m d by eyeblin yeblink, e below the e
ble from a is practical rd. Howeve aximum val k eye wide in a r, the ue at
Figure 4 deg In between and unco blinking blinking t
s
In 1. W 4.14 Experim gree, below n this figure ‐45° to +45 onscious blin is stronger the eyelid m4.2 r
steady
n this part w When the su mental result the eye cen we also se 5° and the b nking have d than that o movement isresults w
we measured bject sat in ts when eye ter, (b) 0 dee that the e best detecta different wa of unconscio accompanie
when the
the eyeblin front of a p blink was un egree and (c) eyblink is w ble angle is aveforms an ous blinking ed with the me subjec
k signal for t ersonal com nconscious: ) +45 degree well detectab 0°. But we nd also the d g due to the muscle movct’s head
two conditio mputer and p (a) measure e above theble from a w observed th detected sig e fact that
ements arou
and bod
ons: performed a ed data from eye center. wide angle r hat the cons gnal of cons in the cons und the eye.dy was n
an everyday m ‐45 range scious scious scious .not
task,2. W In consciou samples Figure 4 fron A the head distorted the signa body mo previous method separatio of ICA on method a When the sub n this part d s eyeblink. for both con 4.15 The arit nt of the pers As we can se d and body m d signal. The al thorough ovements h works by s for signal se on purposes n the signal and finally w bject perform
due to the m Figure 4.15 nditions. hmetic mea sonal compu ee the measu movement a ere are sever a low pass o ave similar some autho eparation, IC s. We exami separation we selected P med a walki more powe 5 and 4.16 n of 100 sam uter and per performed ured signals and we need ral proposed or band pas frequency ors, using co CA (Indepen ned ICA met procedure a PCA method ng task. rful signal in show the mples of Dop rformed a ro d the walkin by the Dop d to find a w d ways for s ss filter, but component ommon filte dent Compo thod in our and therefor d. n conscious arithmetic ppler signals outine task a g task. ppler sensor way to separ ignal separa due to fact ts and as it ers seems a onent Analy study, but d re poor resu s blinking w mean of th s: (a) when t and (b) when are comple rate eyeblin ation. One o that eyebli t has been almost hope ysis) is widel due to the i ults, we tried
e only study he 100 colle he subject s n the subject etely distorte k signal from of them is to nk and head reported in eless. As ano
ly used for s nefficient im d to find ano y the ected at in t ed by m the o pass d and n the other signal mpact other
B the powe W except fr In order accelerom Accelero efore apply er spectrum Figure 4.1
We can see requencies n to better un meter. Figu meter and f ing PCA met of the recor 16 The powe
that in figu near 2 Hz the nderstanding ure 4.17 sh igure 4.18 re thod, in ord rded signals er spectrum re 4.16 (a), e signal has g this featur hows the r epresents its der to study . Figure 4.16 of the signa frequencies most of its p re we also m recorded he s power spe frequency c 6 depict the als represent s from 2 to power in fre measured th ead and bo ctrum. characteristi results. ted in figure o 4 Hz and i equencies be he body mov ody movem ics we calcu e 4.14. in figure 4.1 etween 5 to vement usin ments using ulated 16(b), 7 Hz. g the g the
Figu comput Figure F those fre Now that section w PCA met re 4.17 The er and perfo e 4.18 The P rom the fre equencies be t we know w we have trie hod. accelerome ormed a rou Power Spect quency cha etween 10 t which freque ed to separa eter signals: ( tine task an rum of the s racteristics o to 13 Hz are encies conta ate the eyeb (a) when the d (b) when t signals repre signals). of the head the domina ain the most blink signal f e subject sat the subject p esented in fig and body m ant frequenc t powerful p from the or t in front of t performed t gure 4.16 (A movements cy for this p part of the s iginal measu the persona he walking t Acceleromete we can see articular sub ignal, in the ured signal l task. er e that bject. e next using
4.3 Applying PCA for signal separation
In this section in order to extract the principal components of the original signals we implemented the Principal Component Analysis (PCA) [13]. In the previous parts each original signal was divided into 100 datasets of 1000 point length (or one seconds), thus we obtained 100 samples of eyeblink signals for each measurement. Then we arranged these samples into matrices so that the size of each matrix was 100×1000. Then we calculated the PCA of each matrix. X , ⋯ , ⋮ ⋱ ⋮ , ⋯ , 4.1 So that: . In which: U is the leftsingular vectors Then we calculated the principal components of the matrix, X: E S 4.3 In which E represents the principal components. Figure 4.19 represents the first principal components for the measured signal when the head and body were steady when the angle between the eye and Doppler sensor was 0°.
Figure 4 A same as because the inver A first we s that the compone the recon 4.19 First pr the cente As we can se obtained d of axis rotat rse of repres As the recons selected the dominant ent that hav nstructed ey incipal comp r of eye is 0° ee the first p ata using th tions during sented signa struction pro e first to sixt frequency o e dominant yeblink signa ponents of t °: (a) conscio principal com he arithmeti PCA proced l in figure 4. ocedure for th principal c of eyeblink frequency a al. Figure 4.2 he signals w ous blinking mponents of ic mean of t dure, the obt 10. the case wh components is near 2 at 2 Hz and t 20 and figur when the ang and (b) unc f each meas the signals tained signa hen the head s of each sig Hz, we cho then we calc re 4.21 show gle between onscious bli sured signal in the previ al for uncons d and body gnal and con ose those se culated the w the power the sensor nking. s are exactl ious section scious eyebl were not ste nsidering the elected prin mean of the r spectrum o and y the n. But ink is eady, e fact ncipal em as of the
first to Figure 4 In represen feature o and yell frequenc principal eyeblink. to other separatio mean of sixth princi 4.20 The pow the subject n figure 4.2 nts the Powe of the signal ow lines as cy at 2 Hz an component . The other p movements on and we s them as the ipal compo wer spectru t sat in front 20, the hor er of the sig we selected ssociated w nd from the ts which ha principal com s such as he elected the e reconstruct nents for b m of the firs of a person rizontal axis gnal at each d those freq with the firs experimenta ave the mos
mponents ha ead and bod first to third ted signal as both workin st to sixth pr al computer s represents frequency. uencies up t st to third al results in st of the po ave differen dy movemen d principal c s eyeblink. ng with co rincipal com r and perfor s the frequ To better u to 6 Hz. As w principal c the previou ower at 2 H t dominant nts. Thus we components mputer and ponents of t med a routin uency and t understandin we can see, omponents us section, w z as the rep
frequencies e used this f s and then w d walking t the signal w ne task. the vertical ng the frequ the blue, or have dom we consider t presentative s and they be feature for s we calculate tasks. hen l axis uency range inant those es for elong signal d the
Figure 4 F the walk represen understa 6 Hz. As compone other pri before, w for eyebl 4.21 The pow igure 4.21 r ing task. In t nts the Pow anding the fr
we can see ents have do incipal comp we selected link. Figure 4 wer spectru the represents t this figure, th wer of the
requency ch e the blue a ominant freq ponents are these signa 4.22 and figu m of the firs e subject per he power sp he horizonta signal at haracteristics and purple quency at 2 e the represe ls and calcu ure 4.23 rep st to sixth pr rformed a w pectrum of al axis repre each frequ s of the sign lines associa Hz and they entatives of lated the m resent the re rincipal com walking task. the signal w sents the fre uency and nal we select ated with th y are explai f other mov ean of them esults: ponents of t when the su equency and like figure ted those fr he first and ning the eye ements. The m as the reco the signal w ubject perfo d the vertica 4.20 to b requencies b d fourth prin eblink signal erefore as it onstructed s hen rmed al axis better below ncipal l. The t said signal
Figure 4 In the norm measure the wave blinking task, we due to th toward le and if th value for 4.22 Reconst n figures 4.2 malized ampl d by Dopple eform and a signal when selected th he phase sh eft or right. e selected s r this subject tructed blink 22, the horiz litude of the er sensor, oc also the tim n the subject e largest wa ift induced As the proc signal exceed t was 0.4 V. king signal w and perform zontal axis r e signal. Refe ccurs at the me when it o t sat in fron ave form be by the signa cedure of ey ds this thres Figure 4.24 d when the sub ming a routin epresents th erring back t time near 0 occurs, in fig nt of a perso etween 0.4 t al separation yeblink dete shold value, depicts the e bject sat in fr ne task. he time and to figure 4.2 0.5 s. There gure 4.22, w onal comput to 0.9 s as t n procedure ection first w the eyeblin eyeblink det ront of a per d the vertica 2, the peak o efore, consid which is ass
ter and perf the eyeblink e the signal we defined a nk is detecte tection flow rsonal comp al axis repre of eyeblink s dering the si ociated with formed a ro k signal. How may shift sli a threshold v ed. The thres . puter sents ignal, ize of h the utine wever ightly value shold
Figure 4. In the norm and also when the 1 s as th value wa 23 Reconstr n figure 4.23 malized amp the time w e subject pe e eyeblink s s 0.4 V. Figu ructed blinki 3, the horizo plitude of th when it occu
rformed a w signal. The p ure 4.25 show ng signal wh ontal axis re e signal. Lik urs, in figure walking task, procedure o ws the eybli hen the subj epresents th ke figure 4.2 e 4.23, whic we selected of eyeblink d nk detection ject perform
he time and 22 considerin ch is associa d the largest detection is n flow for th med a walkin the vertica ng the size o ated with th t wave form the same a he walking ta g task.
al axis repre of the wave he blinking s between 0. nd the thres ask: sents eform signal 52 to shold
Figure 4 Figure 4. A waveform 4.24 Eyeblink 25 Eyeblink
As we can s ms exceeded
k detection p
detection p
see in thes d the thresh procedure w and perform rocedure wh e figures th old value. when the sub ming a routin hen the subj he eyeblink bject sat in fr ne task. ject perform k occur bec ront of a per med a walkin cause the p rsonal comp ng task.
peak of sele
puter
F dominan 0.4 V. Figure 4 the In represen lines asso Thus we selected subject s igure 4.26 t nt frequency 4.26 The pow e second sub n figure 4.2 nts the Powe ociated with selected th signals as t at in front o to 4.29 dep for the seco wer spectru bject sat in fr 26, the hor er of the sig h the first to e first to th he reconstr of a personal
icts the res ond subject m of the firs ront of a per rizontal axis gnal at each o third princ hird principa ucted signal computer a
ult for the was also 2 H st to sixth pr rsonal comp s represents frequency. ipal compon l componen l for eyeblin and perform second sub Hz. The thre rincipal com uter and pe s the frequ As we can nents have d nts and then nk during pe med a routine bject. We ob shold value ponents of t rformed a ro uency and t the blue, or dominant fr n we calcula erforming th e task. bserved tha was also de the signal w outine task. the vertical range and y equency at ated the me he task whe t the efined hen l axis ellow 2 Hz. an of n the
Figure 4 In the norm the first the mean 4.27 The pow n figures 4.2 malized amp and second n of them as wer spectru the sec 27, the horiz litude of the principal co s the reconst m of the firs cond subject zontal axis r e signal. In t omponents tructed signa st to sixth pr t performed epresents th his figure th have domin al of eyeblin rincipal com a walking ta he time and he blue and nant frequen nk for the wa ponents of t ask. d the vertica orange lines ncy at 2Hz a alking task. the signal w al axis repre s associated and thus we hen sents d with took
Figure Figure 4.
