March 2020 Wei-Fen Hsieh Human-Robot Interaction Considering User Types by Sensing Human Behavior

(1)

Human-Robot Interaction Considering User Types by Sensing Human Behavior

March 2020

Wei-Fen Hsieh

(2)

I

Chapter 1. Introduction

1.1. Motivation

With technology and artificial intelligence advancements, robotic technologies have been developing for decades, and the relative applications have been adapted for numerous purposes. Robots are introduced in human society and expected to be as a partner in our daily lives. As such, the field of Human-Robot Interaction (HRI) has emerged, which aims to facilitate a robot’s interaction with humans. The main goal of HRI research is to enable a robot to understand people deeper and building up an intimate relationship. Thus, understanding human behavior plays an important role in devising a more natural HRI. In Human-Human communication, we convey and exchange information through both verbal and nonverbal approaches in interactive contexts.

Likewise, it is necessary for a robot to not only understand human behavior but also show social signals according to the context for smooth communication.

Several social robots, such as Pepper from Softbank [1] and Buddy from Blue Frog Robotics [2], have been released and gradually brought into our daily life. Since most of the social robots are designed for general purposes, it is still difficult for a robot to perform sophisticated actions and be exactly human-like, not to mention eliciting continual human responses and build a deeper relationship with a human partner. Therefore, it is essential for a robot to equipped with the ability to interpret human expressions in order to give proper actions to a human partner. In most of the current utilization, social robots engage only in short-term interaction and can merely afford to provide services under common conditions, which is insufficient to deal with their surrounding environments and events.

The field of HRI is now looking to a human-centered perspective approaching a more flexible interaction process. To build up a deeper relationship with humans, robots’

capability of recognizing the human internal state is essential. During the interaction, verbal and nonverbal behavior can reveal a person’s internal state such as emotion, confidence, and attitude. Therefore, if robots are enriched with the ability to understand verbal and nonverbal signals, they can provide more adequate assistant and be less machine-like. Furthermore, socially interactive robots are expected to support a wide range of users who may show obviously different characteristics. As soon as the robot has become a part of our daily lives, it is necessary to endow robots with the ability to treat distinct individuals in various ways even under the same condition. Meanwhile, human behavior and perception are related to lots of individual characteristics such as attitude,

(5)

2

personality traits, and inherited characteristics. These personal traits determine how people perform both externally and internally while facing a robot. Considering influential factors overall, how to promote the natural HRI process by satisfying interindividual differences is one of the emerging issues for current robot design.

On the other hand, from the viewpoint of robot-to-human in HRI, robot expressions are designed to attract human attention and transmit robot intention clearly. Generic robot expression patterns are usually adopted at public places such as a museum, restaurant, and market. However, social robots for domestic usage are still under development to manage individual differences. The differences in human perception show various degrees of sensitivities for perceiving social signals and understand counterparts’ emotion.

Likewise, differences in perception influence sensitivity that humans perceive and how to interpret robot expressions. Human preference and attitude toward robots which are formed by the aforementioned factors, influence the smoothness and persistence of interaction. However, it is impractical to design distinct robot expressions for all users. In addition, robot expression is restricted by the structure of the robot body, which is still hard to perform facial expression and body movement vividly according to all possible situations. Through the variances and complexities in human beings, it is hard to provide assistance toward all possible conditions. Furthermore, the existing HRI dataset for computational analysis does not meet the practical requirement and lack of variability.

The above-described concerns motivated this doctoral study. Therefore, this dissertation presents the idea to construct user types as the basis for a robot to interact with people by analyzing latent associations among people. User types is a term which describes people potentially have similar perception while receiving messages from robots and reacting similarly under certain condition. The similar human experiences, personality, and attitude can yield greater behavioral consistency, making it possible to categorize individuals who perform similar behavior into the same user type. Concluding the aforementioned ideas, user types are proposed to be built from two perspectives.

Firstly, from the viewpoint of robot-to-human, it is necessary for a robot to recognize a human partner’s behaviors and understand a human partner’s intention by detecting internal state and behavior patterns in different conditions. Secondly, from the viewpoint of human-to-robot, the design of robot expressions during the interaction should take user preference into consideration in order to deal with diverse human perception.

Nevertheless, individual differences affect human behavior from numerous levels, which causes difficulties to make the robot satisfy all requirements that each person might have. Studies have shown that people tend to behave in a manner that matches their past decision and experience. It implies that people have a greater tendency to behave similarly

(6)

3

while experiencing a similar situation next time. Furthermore, it is possible to recognize a person’s internal state by comparing it with behavior patterns from the same user type.

Take advantage of the behavioral consistency, this thesis proposes the concept of user types that categorizes the users who behave similarly under particular circumstance into the same type [5].

Accordingly, in order to build proper robot expressions efficiently, this thesis proposes robot expression designs based on user types [3][4][6]. From the aspect of users’

preferences toward robot expressions, it is able to establish criteria for the robot to determine performing expression within the robot’s limited capability. Through exploring from both the psychological and engineering side, this thesis attempts to propose a solution for conventional HRI that lacks in taking individual differences into account from a human-centered perspective. The user types can be adopted as the benchmark for robot emotional expression which should be understandable by human beings. be adopted as the benchmark for robot emotional expression which should be understandable by human beings.

This thesis aims to establish user types as criteria to determine the human internal state and modify robot expression. To achieve this goal, this thesis is constructed by two main parts. 1) the first part of this thesis presents the criteria to categorize people with similar behavior patterns into the same user type. To begin with, the recognition of human behavior in the selected condition: confidence and unconfidence, is performed. 2) In order to build varied robot expressions corresponding to user types, the second part presents robot expression design utilizing user impression preference, which attempts to increase acceptability. Prototypes of robot expression patterns are built based on human behavior understanding while prototypes of robot expression patterns present the capability of conveying internal states such as cute and cool. Subsequently, the discussion of interindividual differences in the aspect of the human internal state is presented. The impression preference toward prototypes of robot expression patterns reveals the necessity of utilizing user types for corresponding robot expression. Afterward, the basic parameters of robot expressions are selected from prototypes of robot expression patterns using user impression preference. Finally, the mutual influences between robot behavior and human internal state are demonstrated to stress the importance of applying user types in HRI.

(7)

4

1.2. Organization of Chapters

This thesis focuses on developing a comprehensive robot model that provides services according to user types. Robot expression patterns considering human internal state and user types are proposed. The robot expression patterns are constructed from two point- of-view, robot-to-human (human perception differences) and human-to-robot (human behavior sensing). Firstly, it is significant for the robot to recognize a human internal state for understanding human intention and giving the proper reactions. Otherwise, the information and condition might be misunderstood and result in unacceptability. Secondly, the investigation of human perception on robot physical attributes has been conducted to provide criteria for robot expression. This thesis comprises five chapters. The chapters' position is depicted in Fig. 1-1. The specific description of each chapter is described as follows.

Chapter 1: This chapter introduces the background, motivation and overall structure of this thesis.

Chapter 2: This chapter describes the concept to establish user types in the aspect of bidirectional interaction between humans and robots. In this thesis, the establishment of the basis of user types is both from the view of robot recognizing human behavior and from the view of humans performing respective perceptions toward robot expression.

Firstly, Section 2.1 explains the main target of this thesis, user types. The concept of user types is purposed to manage individual differences based on the motivations described in Section 1.1, which is anticipated to steady the interaction and pursuit for long term usage.

Section 2.2 explains the necessary competence of a robot from the viewpoint of robot-to- human. Human behavior recognition and human interstate understanding are necessary for a robot to understand human intention considering both verbal and nonverbal expressions.

In addition to bodily expressions, human perception is another important factor affecting the interaction process. The importance of human behavior recognition and human perception are explained in Section 2.1.1 and 2.2.2 respectively. Section 2.2.3 describes the method, Heart Rate Variability, used to estimate a user’s internal level reaction in the interaction. Section 2.3 explains the robot expression from the viewpoint of human-to-robot. To begin with, we introduce the influence of robot expression in HRI.

Subsequently, we explain the importance of human perception in the process of HRI.

Section 2.4 describes the

Chapter 3: presents the human behavior analysis for understanding the human internal

(8)

5

state. Section 3.1 describes the procedure to obtain human behavior (including voice and nonverbal expressions) under selected internal state, confidence and unconfidence.

Section 3.2 describes the analysis of obtained human behavior. Comparison analysis between users has been conducted to observe users with similar behavior patterns, which is helpful to establish the evidence for a robot to detect user types. Subsequently, human behavior similarities are analyzed to provide evidence for establishing user types.

Chapter 4: presents the robot behavior design based on human behavior understanding.

Subsequently, robot expression impressions are analyzed. Section 4.1 describes the procedure of obtaining human behavior and implement selected motion to robot simulator as prototypes of robot expressions. The investigation of impressions toward robot expression patterns are shown in two aspects. Following the introduction of robot expression, the impression analyses including preference toward expression patterns and impression investigation are presented in Section 4.1. In Section 4.2, two experiments are presented to estimate the human perception sensitivity toward robot expression and robot emotion. The construction of user types according to human perception is explained, which shows the necessity of adopting user types to satisfy individual differences.

Chapter 5: The effectiveness of experiment results is concluded, followed by the potential application and prospects of further development.

(9)

6

Fig. 1-1 The illustration of chapters’ position

(10)

7

Chapter 2. Construction of User Types in HRI

This chapter describes the related knowledge and methods used in this thesis in order to compose the research background and motivation of the presented work. To begin with, the concept of proposed user types and the construction of user types are introduced.

Subsequently, the two main research topics of this thesis are described. Firstly, human internal state understanding, this topic focuses on the process for a robot to understand human intention by referring the user types based on human behavior similarity. Secondly, robot expression design, human perception, and impressions are adopted to determine the influential factor in designing the robot’s motion and voice expressions. The related literatures are then discussed from these two aspects.

2.1. User Types in HRI

Construction of User Types

In interpersonal communication both verbal and nonverbal information is transmitted to exchange information; likewise, in the process of HRI (Fig. 2-1). People tend to be more prosocial after synchronizing behaviors with others which have been proven by lots of recent research in neuroscience and cognitive science area [7]. However, when the interlocutor is a robot, the way people perceive verbal or nonverbal information might be different from the human-human interaction. Similarly, how robot’s behaviors are perceived can have relatively different effects on humans comparing with human interlocuter.

Up to now, it is still difficult for most of the current robots to act perfectly lifelike, not to mention some of the robots even do not equip with changeable facial expression.

Fig. 2-1 Illustration of the process of HRI concluding verbal and nonverbal factors

(11)

8

Obviously, there is a huge gap between humans and intelligent robots on the ability to adapt to the interactive partner or circumstance. It is essential to focus on the social signals and the influence of individual factors such as different personality traits and attitudes that could affect how human beings behave when facing a robot. Social robots are expected to assist a wide variety of users. Despite the hardware limitation, how to satisfy individual differences either from the aspect of comprehending human intention or from the aspect of performing preferred expression is still a challenge. Since the conventional researches usually design a standard robot model for several limited situations, it is not enough to meet the feelings of sensitive users who may misunderstand robot intention (Fig. 2-2).

However, the field of HRI is now considering managing the influence of individual differences for satisfying the diverse requirement. In the conventional method, collecting big data is necessary to analyze user characteristics to define the essential element while designing robot expressions. It is not only time consuming, but also impractical to collect all kinds of user characteristics. Due to the difficulties of establishing the whole response according to a predictable situation, the feasible solution is to build robot response patters for a similar condition. Human behavior is deeply associated with the experience, personality, and attitude. Meanwhile, people who have experienced similar conditions tend to behave similarly while experiencing a similar condition again [8][9].

Taking advantage of behavior consistency, this thesis proposes to establish the user types by grouping people with similar behavior into the same user type. The process of building user types is depicted in Fig. 2-3. During the interaction, a robot is able to obtain a new user’s characteristics from the user’s response. By comparing obtained user characteristics with the existing user types in the database, it is possible to determine whether the new user behaves similarly with one of the existing user types or not. A new user is assigned to the matched user type under the condition that the new user behave

Fig. 2-2 Image of generic robot expression toward users

(12)

9

similarly with one of the existing user types. In contrast, under the condition that the new user type does not match with any existing user types, the new user is inserted into the database as a new user type. Continuing the process of establishing the user types database, it is possible for a robot to interact with users with diverse requirements. Fig. 2-4 shows the concept of building user types according to the human similarity in two aspects, human behavior and human perception. Both human behavior analysis and human perception are important factors which influence how robot understand human and how human perceive a robot. As a result, the robot is able to speculate the possible intention of new user by the similar human behaviors which have been recognized in existing user types. user types are practical to compose human behavior and human perception to satisfy the individual requirements. While utilizing user types, users are able to receive their preferred robot expression ( Fig. 2-5).

Fig. 2-3 The process of determining user types of a new user.

Fig. 2-4 The concept of categorizing users into user types by individual characteristics

(13)

10

On the other hand, humans intrinsically have the social capability to perceive the counterpart’s emotion and give the proper response in interpersonal communication.

Moreover, humans synchronize their actions naturally when they are interacting with others verbally and non-verbally. Similarly, people can observe a robot’s behavior and intention while facing a robot. Meanwhile, people have learned to adapt their behavior and to react to bodily expressions transmitted by the counterpart [10]. That is to say, instead of merely considering how to adapt robot behaviors to fit human beings’ reactions, it is far more advisable to utilize human self-adaption to encourage long-term interaction.

If a robot performs the expressions according to the user’s condition, it will be possible to elicit the user’s fondness.

Fig. 2-5 Example of robot expression according to user types.

Adaptive Expression

Type A User type A

Type A

Type X Type B

(14)

11

Importance of Individual Differences in HRI

Human-robot-interaction, an interdisciplinary field, concerns the mutual reactions from both sides. With the noteworthy development of artificial intelligence, computational human-robot interaction has emerged as an interesting candidate. Most existing studies, aiming to construct standardized models for human-robot interaction, discuss from the robot’s aspect which means that the individual is regarded as the particular representative and expects everyone to express the same social signals at the same rhythm, ignoring individuals’ diversification [11]. Personal factors indeed bring an extinct effect during the period of either human-human or human-robot interaction. In other words, even in the same situation, different personality traits and attitudes could influence how human beings behave when facing a robot. Personality, as an important internal attribution, affects how people interact with technology. Individual behavior to a social robot is shaped by their personality and impressions [12]. For example, research has shown that introverted individuals recognized emotions from less visually complex characters better than extroverted individuals [13]. Besides, it has been shown that there is an association between people’s preferences toward robots and their personalities [14][15].

Several studies discussed the connection between personality traits and characteristics in HRI which have indicated the importance to build a customized robot system to satisfy actual needs [16][17]. The individual characteristics are crucial factors to determine how humans perceive robots, which results in stable human-robot interaction or refusal [18].

Thus, the personalized robot is necessary for different individuals to encourage and sustain long-term human-robot interaction[19].

The complexity of human thoughts and expressions makes it a great challenge to would categorize each one in a particular type. Moreover, different personalities performed distinct abilities to recognize emotion expressed by the robot. The existing personality models always simply group people according to their self-evaluation score which obviously lost the distinguish individual features. In order to obtain a satisfactory solution to meet necessities individually, many pieces of researches strive to find the association between individuals, which indicates that one’s preference for the expression styles seems to be tied up with the participant’s personality [20]. The emission of various social signals including verbal and non-verbal features does link to the personality profile [21].

Furthermore, the evidence has shown that the link between one’s personality and the behaviors during social interaction can be classified into particular patterns [22]. To smooth the interaction and attain effective expression, social robots need to be capable of expressing its emotional status appropriately which means both the design of robot

(15)

12

behaviors and the way of being perceived by different individuals are taken into consideration at the same time in this thesis. The understanding of the relation between personal features and performances can demonstrate different individuals’ characteristics and provide prescriptive guidelines for persuading humans to better undertake the appropriate actions in real social scenarios.

2.2. Human Internal State Understanding in HRI

n this section, user types based on human behavior similarity are introduced, which is used to understand users’ internal state. Human internal state understanding is an import issue in HRI which is the basis for a robot to comprehend a human partner’s intention.

People express their internal state through verbal and nonverbal messages. This thesis focuses on the understanding of the human internal state by analyzing nonverbal factors.

Taking advantage of people who have similar experiences tend to behave similarly, the user types are built by human behavior similarity. A robot is able to speculate a new user’s internal state by comparing the existing user types in the database. The necessary methods and relevant literatures are introduced.

Human Behavior Recognition

In human-human communication, we convey information through various way including verbal traits and nonverbal behavior. From most direct spoken utterance to a subtle body movement, these all reflect our internal state. Therefore, understanding human behavior plays an important role in devising advanced human-robot interaction model. Focusing on the social signals and the various individual factors is essential for the present HRI. However, up to now, it is highly challenging to create a socially flexible robot able to interact with humans in natural appropriate behaviors. The robot requires understanding human internal states and personality and subsequently generate adaptive verbal and nonverbal combined behavior.

In this study, human behavior under confident and unconfident condition is selected as the analysis target. Confidence is demonstrated as one of the important internal states to determine an individual’s attitude and outlook toward the environment, which leads to significant influences on human decision-making [23]. Moreover, confidence is also a prime key to the steady relation between individuals and likewise between people and technologies [24]. Expressions of confidence in human communication are expressed as

(16)

13

a statement with conveyed nonverbal factors such as gestures, tones, facial expressions, being a complement to the verbal communication [25]. Among the nonverbal signals, the head pose is the fundamental factor that enables the robot to become more social by showing engagement. The head orientation is often used to approximately track the gaze direction which is another important social signal in the face-to-face interaction [26][27].

Especially, humans use specific head motion patterns to express intention and emphasize utterance content, which demonstrates the consistency of human utterance [28][29]. Head movements have been proven to carry numerous information and be as a social signal to imply internal states [30]. Consequently, the head movement serves as the indicator to present a specific pattern based on the affective state [31]. In short, the head motion can express personal emotion and attitude for a matter. Therefore, the head movement is selected to be the non-verbal analysis data for human behavior recognition.

Meanwhile, verbal signals carry significant information in social interaction such as emotions. Verbal communication is a key aspect of human social behavior and therefore is critical for human-robot interaction [32]. While communicating with others, besides the utterance, vocal cues such as voice pitch and intonation show the attitudes and emotions of the speaker [33]. Voice characteristics have an impact on how listeners perceive a speaker’s attitude and impression establishment [34]. In this research, 8 potent verbal factors that show better results in the former experiment are taken as the analysis data this time. We removed the verbal factors, like the volume which is related to the individual discrepancy and showed less persuasive results. The voice data used for human behavior analysis are analyzed by Praat which is a software package for speech analysis in phonetics [35]. We extracted the voice factors including the effective value of sound pressure, highest sound pressure, lowest sound pressure, sound pressure range, average voice pitch, highest voice pitch, lowest voice pitch, and pitch range.

Considering both voice and nonverbal factors have proven an important role in social interaction and personal traits, rather than using either, it is essential to combine them to effectively investigate the current challenges. Thanks to the rapid development of robot’s using scenarios, advanced social needs have been drawing considerable attention. If human behavior under certain and uncertain condition can be detected, the situation, for example, the voice recognition error, the unfunctional system can be recovered by conducting the unconfident expression style, helping sustain long-term human-robot interaction In contrast, while the robot receiving a confident response from a human, the robot can infer that the conversation can continue. The consequence of human confidence analysis is not only crucial to understand the individual’s personality and behavior patterns systematically but also makes it possible to optimize the HRI model. It’s our goal

(17)

14

to provide insights into the construction of a comprehensive and customized robot to advance HRI.

Human Perception

Human perception is a process people make sense of the stimuli from the environment through the recognizing, organizing, and interpreting sensory information arousing the following reaction for the stimuli [36]. Especially, emotion perception is an important parameter in human-human interaction [37]. During communication, people perceive and express emotional information through various channels, including nonverbal factors (facial expressions, body movement) and voice factors (prosody, utterance) [38]. It affects a wide range from how we interpret social signals to how we build the impression of others and the ability to recognize emotion. Emotions can be perceived through physiological sensory processes. How people interpret interlocutors’ physical changes through sensory systems is responsible for understanding others’ emotional states and providing evidence for the subsequent reactions [39]. Some people are comparatively more alert to nonverbal cues and better in identifying what those cues mean. [40].

In human-robot communication, perception is not only a key point to if a human can properly recognize the intention robot expressed and continue the process but also an important factor that influences the way people perceive body movement and emotion transmitted from robot. All these evidence have shown that it is fundamental for robot developer to design robot expressions. However, researches on HRI always pays attention to the study on the evaluation of robot appearance and behavior designs, having ignored the primary problem, people’s perception toward these factors. At present, various expression styles have been designed, intending to make the robot seem more natural and friendly. It has been proved that there are both similarities and differences between interpersonal interaction and human-robot interaction [41]. The perceptive sensitivity of social nonverbal correlates to personal traits and circumstance factors as well. It is not uncommon for a robot which intends to behave friendly; on the contrary, the robot is evaluated as strange by users in some situations [42]. It has been demonstrated that even a minor inconsistency of the robot’s behavior can greatly affect the way humans will perceive it and react to it [43]. The evidence shows the understanding of human perception is thus considered as a significant factor for the robot.

In addition, understanding the human perception dimension of robot expression is necessary to design the robot’s social characteristics on social robots for avoiding potential misunderstanding. User perception or acceptance requires an in-depth

(18)

15

understanding of the future human-robot relationship. For the robot, not only the appearance but also the way of emotional expression can influence users’ evaluations of it [44][45][46]. During the human-robot interaction, it is inevitable for the robot to show different emotional expressions to accomplish specific tasks or motivate long-term interaction. Many pieces of research have revealed that verbal and nonverbal factors show a prominent influence on human’s perception of the robot’s internal state [47]. Therefore, it is unreasonable to completely imitate the pattern in human-human social interaction for the robot. Several studies have demonstrated that human perception results can directly influence the impression and trust level for the robot [48][49]. Simply using the same response style for all could lead to a misinterpretation of the robot’s behaviors due to personal distinction. Conclude the abovementioned evidence, human perception is a prominent factor that affects if people can perceive emotion, verbal and nonverbal behavior properly interpreting the message from robot well. Nevertheless, considering the diversity of perception, it is our goal to utilize the characterizes of human perception and individual differences in social human-robot interaction.

Heart Rate Variability

During communication, the social signals that the conversational partner expressed can stimulate one’s nonverbal reaction. In order to understand the human internal state toward robot expression during the interaction, heart rate variability (HRV) is utilized to further explore the internal level reaction toward robot expression. HRV is a measure of the interval between two consecutive heartbeats, which is used as a marker of Autonomic Nervous System (ANS) to indicates the process of emotion regulation and stimuli. In an electrocardiogram, the sharpest peak part is denoted as R-wave while the interval between R-waves is denoted as the R-R interval (Fig. 2-6). The power spectral density is calculated from RR interval data by 100-sec window size and shift size is 1 sec. To analyze HRV frequency spectrum, the low-frequency (LF; 0.05Hz - 0.14Hz) and high-frequency (HF;

0.05Hz - 0.14Hz) components of HRV are computed by Fast Fourier transform. LF is associated with both the sympathetic nervous system and parasympathetic nervous system, and HF is associated with parasympathetic nervous [50][51].

To adopt the HRV, most researches focus on the LF/HF ratio which is regarded as one of the most useful HRV markers[52]. It was suggested that the LF/HF ratio represents the sympathovagal balance which reflecting ANS activity [53]. For a noise elimination, a linear interpolation is performed in error values in the heartbeat data. Generally, the R-R interval is decided by the normal heart rate interval. In rare cases, outliers are outputted

(19)

16

because of facial movement. Thus, the heart rate under 40 and over 120 are defined as sickly bradycardia and tachycardia [54]. Therefore, the normal interval of heart rate is the range between 500ms to 1,333ms.HRV has been extensively used in investigating human emotional reactions during human-robot interaction [55][56]. Besides, HRV has also been used as an index to measure human impressions to the robot’s behavior style [57]. In this thesis, LF/HF ratio is used as a stress indicator to observe what kind of expression discomfort/comfort the interaction process, which is expected to reveal the relation between robot’s nonverbal features and the human’s internal preference for future construction of robot expressions [58][59].

Fig. 2-6 Illustration of heartbeat interval (R-R interval).

R-R Interval

(20)

17

2.3. Robot Expression Design Facilitating Communication

Nowadays, the social robot is expected to be a conversational partner and widely used in different aspects to improve the quality of our life, such as integrating the robot into smart home [60][61]; providing the elderly and children for whether medical assistance or education assistance [62][63]. Besides, for providing services, robots are placed at museum or mall as guides [64][65], During the process of HRI, it is essential for the robot to perform proper responses in order to transmit the correct information. Through bodily expression, voice, and facial expression, the robot is able to express social signals conveying emotional information like a human. An interesting interaction process can arouse a human partner’s interest and continue to play with a robot; on the other hand, an inappropriate response might dampen a human partner’s interest. Social robots are required to have human-like expression and interactivity. Some humanoid robots are designed with human-like facial features such as Actroid-SIT [66] and RoboThespian [67]. These robots are capable of conveying a robot’s emotion by facial expression, human-like face, or different pattern emotional signs such as heart and wink.

In this thesis, I utilized “Pepper” robot and its simulator in the experiment. Pepper, a humanoid robot developed by Aldebaran as an emotional robot, is able to communicate with the human through voice, touch, and emotions by its designed function to understand people’s feeling and react accordingly. Different from the robot with facial expressions mentioned above, Pepper is difficult to show its emotional state (internal world) because of the inability to display facial expression. For these robots like Pepper, while interacting with the human, bodily expression is the best way to display understandable emotional information. How to make a robot without facial expression can also convey emotion while interacting with a human is the problem which this thesis aims to solve. It is acceptable that the robot has a constant posture or motion if the robot is placed in public places as a service robot. However, if Pepper plays an everyday role at home, people might feel revulsion to the unchanged facial expression and the empty look. Furthermore, if a robot only has a fixed behavior pattern with the same greeting every day, it might disappoint the human.

Except for the appearance, robots should also be endowed with social skills which can ensure the interaction goes smoothly by verbal and nonverbal messages. Since the robot is expected to live among human, how the robot behaves when responding to human distinctively affect how people feel about the robot. Referred to social learning theory,

(21)

18

imitation is considered as a skill for people to learn new posture, motion, and response while interacting with people [68]. Designing the robot to imitate human behavior and way of speaking will help it adapt to society because it’s an instinctive strategy in human social interaction. Robot imitation thus has been widely used for different purposes.

Mimicking could allow better emotional contagion and make robots more lively [69].

Many pieces of researches used sensors to capture human motion to implement robot imitation for teleoperating [70][71][72][73]. Researches not only provide the method but also let the robot learn and realize more postures through imitation [74]. The assistant robot has been used by companies to assist autistic children to learn imitation [75].

Human-like not just means that the robot is physically similar to humans but also the behaviors displayed by the robot are similar to that performed by humans. Comparing to the designed motion and gesture which are not changed corresponding to different conditions, the same motion with only small changes such as the different speeds and angles might change people’s impression of robot expressions [76]. In this thesis, the movement exaggeration, movement speed, and voice pitch are modified to obtained people’s impressions toward the robot’s expression patterns.

Displaying Robot’s Internal State

Through imagination and empathy, people are able to anthropomorphize animals, planets, materials, and objects around us, attributing internal states to non-human others [77]. In HRI, anthropomorphism is an important factor to increase the acceptability of robots, which attributes thought and internal states to robots [78]. This theory makes it plausible for people to build a relationship with a robot even although its cognitive, behavioral and interactive capabilities are simpler than other living creatures. In the study of [79], an elderly man saw a robotic doll as if his ex-wife, and he reported that the robotic doll gave him comfort and a space for conversation. The perception of the robot possessing human-like traits is a decisive factor that affects people’s attitudes and responses [80]. Also, it has been shown that the robot’s human-likeness could positively affect people’s engagement with those robots [81].

Different interpretations of anthropomorphism have led to different research topics of human-like robots. Lots of researches pursue the android robot with a human-like appearance and strive to build human-like properties like skin and facial expressions.

However, the uncanny valley hypothesis suggests when a robot's appearance becomes almost human-like, but is still distinguishable from it, people's emotional response instantly become strong revulsion [82]. As for the humanoid robot like Pepper, instead of

(22)

19

being close to a real human, the robots’ appearances are usually designed to be attractive and non-threating.

During the interaction, people interpret the robot’s expressions as its internal state unconsciously [83]. In order to avoid confusion, it is important for the robot to perform bodily expressions properly for transmitting its internal state and intention. However, the limitation of robot hardware might cause ambiguity of expressions [84]. As described in Section 2.2.1., confidence expression is important to show a trusting attitude in the conversation. In the procedure of human-robot interaction, it is both substantial for a robot to recognize human conditions and inform its internal state such as the confidence level of notifying information as well [85]. Taking advantage of the confident expression, robots could be considered to be able to take a compensation strategy in the erroneous situation caused by humans or robots. In contrast, hesitating behavior conveys the robot’s uncertainty which is supposed to reduce ambiguity toward human interpretation caused by the limitation of robot ability. However, it has been shown that appearances do not affect a human’s ability to recognize a robot/agent’s expression [86].

Meanwhile, as described in Section 2.2.2, the human perception of a robot’s expression is of equal importance. Referring to [87], empathy can be broadly defined as “an observer's reacting emotionally because he perceives that another is experiencing or is about to experience an emotion”. People are able to perceive other’s emotions and understand its meaning, leading to emotion contagion [88]. If people can’t perceive the robot’s expression properly, it will be merely increasing the people’s distrustfulness of the robot’s intention. Once people perceived the robot’s internal state, their attitudes are inevitably affected by the internal state they perceived, which is made use of eliciting people’s emotions and conveying a message for a specific purpose [89][90][91]. Not just conveying emotions, the robot’s internal state can be used to express its personality and playing a specific role, such as parenting style at home [92], teaching style in primary school and camp [93]. By adopting the variations in bodily expression and voice features, robots are able to manifest different personalities for the purpose of maintaining the interaction process. For example, empathic expression makes people feel the interaction more engaging and helps maintain the process of interaction longer than non-empathic expression [94].

Most of the researches have built postures that convey basic emotions such as happiness and sadness on robots, adopting adopted positive and negative as the standard for designing robot behavior. It is reported that the postures designed for some expressions such as anger/fear, happiness/surprise, and sadness/fear, were difficult for people to distinguish them clearly [95]. Meanwhile, positive and negative states include

(23)

20

diverse emotions and images, which are hard to be evaluated. To increase the variability, cute and cool styles are selected as the robot expression in this thesis. The impression evaluation revealed that cute and cool styles are considered to be more distinctly performed than other styles, such as “smart” or “competitive” styles. The cute expression denotes positive connotations as cheerful and approachable atmosphere, meanwhile, it is able to give a good first impression while greeting someone approaching. On the other hand, cool expression establishes a stable and confident manner. The two expression styles we selected to establish the robot’s behavior traits have formed different impressions than just a dull robot response.

2.4. Summary

The goal of this thesis is to construct user types that are expected to be used as a benchmark for a robot while interacting with a user. To achieve this goal, theories in HRI are combined with the researches including human behavior detecting and human internal state estimating. This chapter introduced the construction of user types and reviewed the relevant literatures. During the interaction, it is not only important to understand the response from the users. But it is also important to understand how a user receives the robot expressions.

Human behavior detecting has been researched thoroughly for the purpose of understanding human motion. However, the human internal state behind human behavior has not been examined comprehensively. On the other hand, many pieces of research have studied impressions of robot appearance and motions. However, it has not been investigated widely how people perceive voice and motion factors in robot expressions, which is the basis of impressions toward robot expressions. In addition, researches have also investigated generally shared impressions of the robot while the individual difference has not been considered well.

In the next two chapters, user types will be introduced from two aspects respectively.

In Chapter 3, from the viewpoint of human to robot, user types are for a robot to determine a person’s internal state. We will present the procedure of obtaining human behavior under selected internal state, confidence and unconfidence. The user types built on human behavior similarities will be explained. In Chapter 4, we will introduce the user types from the viewpoint of the robot-to-human and present the analysis of different perception levels towards robot expressions. Finally, the mutual influences between robot behavior and human internal state are demonstrated to stress the importance of applying user types in HRI.

(24)

21

Chapter 3. User Types Based on Analysis of Human Behavior

This chapter presents the procedure of building user types based on analyzing human behavior. Human behavior is directly linked to intention, making it possible for a robot to determine human intention by recognizing human behavior. A person’s confident expressions are essential factors for the robot to trigger the subsequent reaction after posing questions or facing erroneous situations, which is described in Section 2-1. Human behavior under the selected human internal states, confidence and unconfidence, are obtained by designed questions trying to elicit participant’s certain and uncertain attitudes which are performed in both voice and nonverbal approaches.

3.1. Obtaining Human Behavior under Confident and Unconfident Condition

A conversation-like setting was used to explore participants’ confidence behavior. Both verbal and nonverbal behaviors are obtained for confidence estimation. To prevent the influence of human factors, MMDAgent, which is a toolkit for dialogue interaction systems [96], was set to be the agent to ask questions. Two experiments with the same experiment setup are conducted to collect the data while the number of participants and the questionnaire used are different. To avoid ambiguity, there are no duplicate questions used at these two experiments.

The experiments are conducted in the laboratory room at Tokyo Metropolitan University. Fig. 3-1 shows the experiment scene. The participants are assigned to answer the questions asked by MMDAgent. Participants are instructed to give a response whether they knew the correct answer or not. The response timing was guided at the moment MMDAgent completed asking the question. Participants’ behavior and voice information during the response time are recorded and used to the analysis of the extraction of properties to distinguish the confident and unconfident conditions. Afterwards, a 5-point Likert-type scale questionnaire from 1 (most unconfident) to 5 (most confident) was answered by participants to grade the confidence level about the answer they had given for each question.

(25)

22

Fig. 3-1 Illustration of experiment scene and procedure.

The motion capturing sensor Kinect V2 was utilized to track participants’ skeleton data during their response time. As shown in Fig. 3-1, the Kinect was placed behind the computer to record the participants frontally. During the trial, upper body joint positions (Head, Neck, Spine Shoulder, Shoulder) are recorded for further analysis. Moreover, a microphone (SONY, ICD-SX1000) was set to record the voice information. The interval of response time was annotated by the experimenter. Fig. 3-2 shows the sensors used in this experiment.

Although all joints of the upper body have been recorded, arm and shoulder movements are confined to the participant’s sitting posture. To be specific, the action might be restricted since the participant was sitting in front of the desk, and the movement below shoulder would be different when the participant was at standing posture. In addition, the hand gesture consists of intricate categories and a wide variety of meaning not to mention considering individual differences, which is hard to be distinguished in the condition of subtle expression [97]. Taking all these into consideration, in this study, head movement, which is an important social signal to present the human internal state, was especially focused on confidence identification. Joints positions (Head, Neck, Spine Shoulder) captured from Kinect are used for further calculation to analyze head movement.

Fig. 3-2 Sensors used in this experiment (Kinect V2 and SONY, ICD-SX1000 )

(26)

23 - First trial

Ten participants (8 males, 2 females) are recruited at the first trial. All of them are college students from the engineering department. Totally acquired valid data acquired from 10 participants including two females and eight males, aged from 22 to 23. The participants are all native Japanese speakers. All participants have no prior knowledge of the experiment questions and did not attend the experiment before.

There are 8 questions in the first experiment, which is listed at Table 3-1. To increase the number of uncertain answers, the questions are designed as the basic knowledge in history and science categories that have appeared in the high school courses.

Table 3-1 Example of questions used in the experiment. Difficulty level was separated to elicit behaviors in condition of confidence or unconfidence.

Difficulty

level Question Content

Easy

あなたの名前は何ですか？

(What's your name?) あなたは今何歳ですか？

(How old are you? )

あなたの出身地はどこですか？

(Where do you live now?)

あなたの血液型はなんですか？

(What's your blood type?)

Difficult

平等院鳳凰堂はどこにある？

(Where is the location of Byodo-in Temple?) 福沢諭吉が設立した大学は？

(What's the university that Fukuzawa Yukichi established?)

現在の千円札に描かれている人物は？

(Who is printed on thousand-yen bills now? ) 東京スカイツリーの高さは？

(What is the height of Tokyo Skytree)

(27)

24 - Second trial

Nine participants (5 males, 4 females) were recruited at the second trial. Among them, 2 participants had attended the first trial. However, participants did not have prior knowledge of the new questions which were distinct from the questions at the first trial.

Thirty questions were designed for the second trial (see Appendix A for the detail of questions used in the second trial). There was no duplicate question included. The difference from the question at first trial is the difficulty level which has been separated obviously to attempt to elicit an equal quantity of certain and uncertain expression. Two hundred and forty pieces of data have been obtained. For those participants whose data were more than 20 pieces were used for behavior similarity evaluation. From two trials, regardless of the discarded data which were graded as 3 (neutral) and the missing data, 285 pieces of data have been obtained totally. Among all participants, there were eight participants whose data were more than 20 pieces.

(28)

25

3.2. Analysis of Obtained Data

For evaluation of collected data, WEKA data mining software (Waikato Environment for Knowledge Analysis) was utilized to conduct data processing [98]. The goal of the evaluation was to seek a satisfactory classification algorithm to automatically recognize the confidence level based on human behaviors. The algorithm including Multinomial Logistic Regression (Logistic), Multi-layer perceptron (MLP), support vector classifier (SVM), and Locally weighted learning (LWL) were used for the analysis. The data corpus was separated as different data set to validate two different validation procedures.

- Condition 1: Classification of participants with less than 20 pieces of data.

- Condition 2: The comparatively certain and uncertain behavior which can be classified by the most confident and unconfident expression.

- Condition 3: Analysis of human behavior similarity using data from those participants who possessed more than 20 pieces.

The skeleton data were influenced by the distance between a participant and Kinect, and the individual’s height. To minimize the influence of variance, preliminary normalization is necessary. In this study, since the analysis was centered on head movement, Head, Neck, and Spine_Shoulder joints (Kinect’s Spine_Shoulder joint is located on the top of sternum) have been selected for calculation. Nonverbal features including the velocity of the Head joint and the angle of the head movement were used.

For the purpose of eliminating the movement range affected by the distance between the participant and Kinect sensor, distance between the Head and Spine joints was set as unit length to conduct coordinates transformation. Unit length (U) which is the Euclidean distance between Head joint and Spine_Shoulder joint was used for normalization. The equation is defined in equation (1), where H is the Head joint, and SS is the Spine_Shoulder joint.

𝑈 = √(𝐻_𝑥− 𝑆𝑆_𝑥)²+ (𝐻_𝑦− 𝑆𝑆_𝑦)²+ (𝐻_𝑧− 𝑆𝑆_𝑧)² (1)

The coordinates of the 𝑖 -th skeleton joint at 𝑡 -th frame are denoted as 𝐽_𝑖 =

[𝐽_𝑖^𝑥 , 𝐽_𝑖^𝑦, 𝐽_𝑖^𝑍](t). The coordination transformation of necessary joints is defined in equation (2), where J denotes the joint coordinate obtained from Kinect at a specific frame, U denotes the unit length (distance between head and spine shoulder), and 𝑇𝑟𝑎𝑛𝑠₋𝐽_𝑖^{(𝑥,𝑦,𝑧)}

represents the coordinate after transformation.

(29)

26 𝑇𝑟𝑎𝑛𝑠₋𝐽_𝑖^{(𝑥,𝑦,𝑧)}(𝑡) =^(𝐽^𝑖

(𝑥,𝑦,𝑧)

(𝑡)−𝑆𝑆_𝑖^{(𝑥,𝑦,𝑧)}(𝑡))

𝑈(𝑡) (2) Afterward, the velocity of head movement was calculated by the coordinate after transformation in equation (3), where V denotes the velocity, t denotes the 𝑡-th frame, and the frame_interval denotes the period between 𝑡-th and (𝑡 − 𝑖)-th frame.

𝑉_{ℎ𝑒𝑎𝑑}^{(𝑥,𝑦,𝑧)}(𝑡) = ^{(𝑇𝑟𝑎𝑛𝑠}⁻^𝐽^{ℎ𝑒𝑎𝑑}

(𝑥,𝑦,𝑧)

(𝑡)− 𝑇𝑟𝑎𝑛𝑠−𝐽_{ℎ𝑒𝑎𝑑}^{(𝑥,𝑦,𝑧)}(𝑡−1))

𝑓𝑟𝑎𝑚𝑒_𝑖𝑛𝑡𝑒𝑟𝑣𝑎𝑙 (3) As for the angle of head movement, the angle was calculated between the origin position where at the moment participant started answering and the head movement during the answer period. The calculation is denoted in the equation (4) where 𝑉𝑠 denotes the head-neck vector at the start of response, and 𝑉𝑒 denotes the head-neck vector during the response time at 𝑡 − 𝑡ℎ frame.

cos 𝜃(𝑡) = 𝑉𝑠(𝑡)∙𝑉𝑒(𝑡)

|𝑉𝑠(𝑡)||𝑉𝑒(𝑡)| (4) The maximum of head movement’s angle and velocity in x, y, and z-axis were used as the nonverbal features for the evaluation. To evaluate the verbal features, the voice data obtained from the recorder was trimmed from the point participant started answering to the finished point. Eight verbal features were then extracted from the clip. According to the evaluation questionnaire, verbal and non-verbal data scaled with 1-point were labeled as the most unconfident, and data scaled with 5-point were labeled as the most confident.

The data scaled 2-point were labeled as relative uncertainty and data scaled 4-point were labeled as comparative certainty.

Classification of Confidence Level

- First condition

We conducted the classification of participants with less than 20 pieces of data. Nine participants’ data were less than 20 pieces. This evaluation is for the purpose of validating when encountered with a new user. The model probably has little knowledge of the new user’s behavior pattern. Under this condition, if it is possible to classify the data correctly with high accuracy the use of the selected features is proven to be essential.

Table 3-2 shows the accuracy for classifying participant data with fewer data to

(30)

27

simulate the situation if encountered new users. The percentage of classification accuracy showed in the table was round off to the 2nd decimal place. The best classification accuracy is 72.88 % performed by MLP. The second-ranked accuracy is 69.49 % performed by SVM. Both accuracies are higher than 66%. From the confusion matrix, it is shown that most of the confidence data were classified correctly. Though almost 60%

of unconfidence data were correctly classified, the incorrectly classified data was probably because of unbalanced data. Table 3-3 shows confusion matrix of the classification result for condition 1 using MLP. It describes that most of the data denoted as confidence were classified correctly. However, half of the data denoted as unconfidence were classified incorrectly. It implied that there might be other latent factors influencing the classification of unconfidence data.

- Second condition

In order to identify subtle behavior differences, it is important to recognize human behavior as comparatively confident and unconfident. Since behaviors of comparatively confident and unconfident are weaker human internal state than most confident and unconfident, there are possibly similar behavior patterns with the extreme condition.

Therefore, we assumed that if the model can classify features in an extreme level of confidence and unconfidence, comparatively confident and unconfident probably can be classified by the same model.

The selected classifiers were trained by the most confident and unconfident features.

Subsequently the relative confident and comparatively unconfident features were used as the test set for each classifier. Table 3-4 shows the accuracy percentage for correctly

Table 3-2 The accuracy for classifying the participant data with less than 20 pieces.

Method Logistic MLP SVM LWL

Accuracy

(%) 66.10 % 72.88 % 69.49 % 62.71 %

Table 3-3 Confusion matrix of the classification result for condition 1 evaluation using MLP.

Method: MLP Data classified as Certain

Data classified as Uncertain

Certain 34 3

Uncertain 13 9

(31)

28

classifying relative uncertainty and comparative certainty features. The better classification accuracy is 66.67 % performed by Logistic and SVM. However, the overall classification accuracy was comparatively low. The result indicates that further investigation is necessary to observe additional features for better classification of subtle human behavior to build behavior recognition at any confidence level.

User Types Construction Based on Behavior Similarity

From the evaluation result in the first condition, the results indicate that it is possible to classify the confidence level of participants with fewer data in the model. From the process shown in Fig. 3-3, the obtained data are used to compare the similarity among the participants in order to estimate if the participants can be attributed to the same user type or not. Consequently, if the new user can be analyzed by the model which has already been trained by the data from another with similar behavior patterns, the classification accuracy might be increased. Therefore, in the third condition, the behavior similarities between participants were analyzed. Eight participants with more than 20 pieces of data were used for evaluation. MLP and SVM, the model performed relatively steady and had better accuracy in the previous condition, were used to present the classification accuracy.

The model was trained by each participant to classify others’ data.

Fig. 3-3 The process of analyzing human behavior similarity between participants is used to examine whether the participants belong to the same user type or not.

Table 3-4 The accuracy for correctly classifying relative uncertainty and comparative certainty features.

Method Logistic MLP SVM LWL

Accuracy

(%) 66.67 % 65.00 % 66.67 % 58.33 %

(32)

29

Table 3-5 and Table 3-6 show the results depicted in the heat map using MLP and SVM respectively. From the two tables, the different accuracies have demonstrated that higher similarity exists among a few participants. In Table 3-5, the results show that participants 5 and 8 with high accuracies of similarities. When the data of participant 5 were used as training data, the accuracy of classifying participant 8 was 82.14%. On the other hand, when the data of participant 8 were used as training data, the accuracy of classifying participant 5 was 89.66%. Likewise, similarities between participants 5 and 8 performed by the SVM classifier are shown with great accuracy (Table 3-6). When the data of participant 5 were used as training data, the accuracy of classifying participant 8 was 82.14%. On the other hand, when the data of participant 8 were used as training data, the accuracy of classifying participant 5 was 93.10%. The results indicate that participants 5 and 8 behave similarly at a certain level while expressing confidence. Based on the behavior similarity, participants 5 and 8 are attributed to the same user type.

Meanwhile, high similarity shown in Table 3-6 suggests that participants 1, 2 and 4 had an equal number of confident and unconfident data while participants 5 and 8 had relatively more unconfident data. Moreover, participant 7 was the only participant who had more unconfident data than the confident, accounting for up to 65 percent of participant 1’s own data. This may cause a low accuracy on participant 7’s result compared with other participants. The results imply that the SVM classifier is probably unsuitable to analyze the data with high bias.

Table 3-5 The heat map of accuracy percentage classified by MLP while the vertical index denotes the participant data used as training set and horizontal index

denotes the participant data used as test.

1 2 3 4 5 6 7 8

1 51.61% 51.72% 35.71% 51.72% 71.43% 52.17% 53.57%

2 63.33% 72.41% 57.14% 62.07% 35.71% 60.87% 60.71%

3 50% 48.39% 53.57% 34.48% 46.43% 52.17% 32.14%

4 40% 83.87% 55.17% 20.69% 42.86% 52.17% 25%

5 60% 70.97% 55.17% 64.29% 60.71% 39.13% 82.14%

6 60% 38.71% 58.62% 53.57% 93.10% 39.13% 82.14%

7 46.67% 58.06% 34.48% 67.86% 13.79% 39.29% 21.43%

8 56.67% 77.42% 65.52% 57.14% 89.66% 60.71% 34.78%

Test data Training No.

data No.

Classification accuracy 0% 100%

(33)

30

3.3. Summary

The procedure of obtaining human behaviors under confident and unconfident condition has been presented. The analysis of human behavior is shown in order to build criteria for a robot to understand human confidence. Several analyses have been conducted, MLP classification performed better in classifying the data. From the result of analyzing the human behavior similarity, people with similar behavior have been presented in which users with similar behavior are attributed to the same user types (Fig.

3-4). The user types can be used for a robot to detect user internal state by analyzing whether a new user matches one of the existing user types or not. Despite its limitations, this chapter has paved the way to design an adaptive interactive robotic technology according to each person’s traits. The model of certain and uncertain according to the integration of non-verbal and verbal features will allow for a wider range of movement as well as making the robots’ behaviors more natural and thus perfect the social HRI.

Fig. 3-4 The image of the condition when the features of a new user are matched with one of the existing user types. The new user is then set as the matched user type.

Table 3-6 The heat map of accuracy percentage classified by SVM while the vertical index denotes the participant data used as training set and horizontal index denotes

the participant data used as test set.

1 2 3 4 5 6 7 8

1 80.65% 79.31% 57.14% 93.10% 64.29% 34.78% 82.14%

2 76.67% 75.86% 57.14% 93.10% 60.71% 34.78% 75%

3 60% 70.97% 57.14% 93.10% 64.29% 34.78% 75%

4 36.67% 74.19% 44.83% 34.48% 46.43% 34.78% 21.43%

5 53.33% 77.42% 51.72% 60.71% 60.71% 39.13% 82.14%

6 70% 61.29% 55.17% 57.14% 93.10% 47.83% 78.57%

7 40% 32.26% 24.14% 42.86% 3.45% 39.29% 21.43%

8 53.33% 74.19% 58.62% 57.14% 93.10% 50% 43.48%

Test data Training No.

data No.