Identity Verification for Attendees of Large-scale Events Using Face Recognition of Selfies Taken with Smartphone Cameras

全文

(1)Electronic Preprint for Journal of Information Processing Vol.26. Paper on Consumer Devices. Identity Verification for Attendees of Large-scale Events Using Face Recognition of Selfies Taken with Smartphone Cameras Akitoshi Okumura1,a) Takamichi Hoshino1 Susumu Handa1 Eiko Yamada1 Masahiro Tabuchi1 Received: February 28, 2018, Accepted: August 3, 2018. Abstract: This paper proposes an identity-verification system for attendees of large-scale events using face recognition from selfies taken with smartphone cameras. Such a system has been required to prevent illegal resale such as ticket scalping. The problem in verifying ticket holders is how to simultaneously verify identities efficiently and prevent individuals from impersonating others at a large-scale event at which tens of thousands of people participate. We developed two ticket ID systems for identifying the purchaser and holder of a ticket that uses two face-recognition systems, i.e., tablet-based and non-stop face-recognition systems that require ID equipment such as a tablet terminal with a camera, card reader, and ticket-issuing printer. Since these systems were proven effective for preventing illegal resale by verifying attendees at large concerts of popular music groups, they have been used at more than 100 concerts. However, simplifying the ID equipment is necessary from an operational view-point. It is also necessary to secure clear facial photos from a technical view-point of face-recognition accuracy because face recognition fails when unclear face photos are obtained, i.e., when individuals have their eyes closed, not looking directly forward, or have hair covering their faces. We proposed a system that uses attendees’ selfies as input photos for face recognition, which simplifies ID equipment by only requiring smartphone cameras, enabling clear face photos to be obtained the allowing attendees to take their own photos. The system achieved 97.5% face-recognition accuracy in preliminary tests of 240 photos taken under various brightness and background conditions. Keywords: face recognition, biometrics, identity verification, illegal ticket resale prevention, selfie, smartphone app. 1. Introduction Identity verification is required in more situations than before. Let us take an example of a case in which many people are admitted to an event. It used to be that in such cases, having a document, such as a ticket or an attendance certificate, checked was sufficient to gain entry; the need for personal authentication was not seriously considered due to the limited amount of time for admitting all participants. Many events with high ticket prices had designated seating, so it was not necessary to assume that some tickets may have been counterfeit. However, the advent of Internet auctions in recent years has made it easier to buy and sell tickets at the individual level. This has resulted in an increase in illegal ticket scalping, i.e., tickets being purchased for resale purposes [1]. Equity in ticket purchasing is required not only by ticket purchasers but also by event organizers and performers [2]. Consequently, event organizers have had to deal with complaints about malicious acts by undesignated individuals who take advantage of fans by buying and selling tickets on the Internet. In many cases, therefore, any ticket buying and selling outside the normal sales channels is prohibited. Ticket sales terms now often stipulate that tickets are invalid when people apply for them 1 a). NEC Solution Innovators, Ltd., Kawasaki, Kanagawa 213–8511, Japan [email protected]. c 2018 Information Processing Society of Japan . using a pseudonym or false name and/or false address, or when they have been resold on an Internet auction or through a scalper. Illegally resold tickets have in fact been invalidated at amusement parks [3] and concert halls [4]. Verification has therefore become a more important social issue than ever before. Legally verifying a person’s identity requires verifying two points: that “the person actually exists (reality)” and “the person is who he/she claims to be (identity)” [5]. The foundation of reality is the family registry. There are limited situations in which “reality” needs to be strictly confirmed, but there are many situations in which “identity” needs to be verified. This personal-authentication confirmation is called “identity” and is often used in the same sense as verification. Current personal authentication can be done with three methods: (1) knowledge certification using information only the person in question knows, such as a password or personal identification number, (2) possession of certification, i.e., such as an ID card or driver’s license, and (3) biometric authentication by confirming a person’s fingerprints face, etc. . . Although knowledge and certification are widely used, such as at bank terminals and in e-commerce, identity can be transferrable with both methods if the individuals purchasing tickets and those entering the event match. Therefore, neither can be an effective means to prevent individuals from impersonating others. Many anticipate that biometric authentication can be a means of solving these prob-.

(2) Electronic Preprint for Journal of Information Processing Vol.26. lems [6], [7]. One advantage of biometric authentication is that there is no risk of biological information being lost or forgotten. Also, biometric authentication can be considered a means to prevent individuals from impersonating others because it uses person-specific biological information. Biometric authentication verifies identity by matching pre-registered biometric information and collation information obtained through a sensor. For example, both vein authentication used in financial institutions [8] and fingerprint authentication used in national and local governments [9] require dedicated biometric-information sensors. Biometric authentication is thus not acceptable for verifying the identity of the purchaser and holder of a ticket because it is not practical for ordinary people to have their biological information registered in advance at home and checked at event sites on the day of the event with such sensors. On the other hand, a sensor for face recognition is a normal camera that ordinary people can easy handle. Face recognition has been put into limited practical use for verifying identity such as entrance and exit control for rooms, immigration control, and reception control [10], [11], [12]. The problem in verifying ticket holders is how to simultaneously verify identities efficiently and prevent individuals from impersonating others at a large-scale event at which tens of thousands of people participate. To solve this problem, we developed two ticket ID systems that identify the purchaser and holder of a ticket by using face-recognition software, which require ID equipment such as a tablet terminal with a camera, card reader, and ticket-issue printer [13], [14]. Since the two systems were proven effective for preventing illegal resale by verifying attendees at large concerts of popular music groups, they have been used at more than 100 concerts [15]. However, simplifying the ID equipment is necessary from an operational view-point. It is also necessary to secure the cooperation of individuals in obtaining facial photos from a technical view-point of face-recognition accuracy because face recognition fails when the unclear facial photos of individuals are obtained, i.e., when the individuals have their eyes closed, are not looking directly forward, or have hair covering their faces. This paper proposes an identity-verification system that uses attendees’ selfies as input photos for face recognition, which simplifies ID equipment by using the attendees’ smartphone cameras and secure clear facial photos from attendees by allowing them to take their own photos. The remainder of the paper is as follows. In Section 2, we survey related work on electronic ticketing systems, walk-through systems, and face-recognition software. In Section 3, we describe problems with the two conventional systems of verifying the identity of ticket holders at largescale events using face recognition. In Section 4, we present our identity-verification system for attendees of large-scale events using face recognition from selfies taken with the attendees’ smartphone cameras. In Section 5, we report on the methods of preliminary tests, and the results of face-recognition accuracy for 240 photos taken under various brightness and background conditions. In Section 6, we discuss our proposed system regarding its feasibility of solving the problems with the conventional systems and conditions of brightness and background, and consider the outlook for future issues. We conclude the paper in Section 7.. c 2018 Information Processing Society of Japan . 2. Related Work 2.1 Electronic Ticketing Systems Electronic ticketing systems have made it unnecessary to issue physical admission tickets and the admission procedure more efficient at large-scale events. These systems provide an electronic ticket, which is a barcode or QR code displayed on a smartphone or tablet terminal instead of a paper ticket. One such system offers an electronic tear-able ticket that is invalidated when it is used for admission in the same way as a normal paper ticket is invalidated when it is physically torn [16]. However, an ordinary electronic ticket is not effective in preventing impersonation because it is transferable. Therefore, an electronic system has been investigated that electrically verifies identities of attendees to control admission. A transfer-prohibited electronic ticket system with anonymity makes it possible to prohibit ticket-transfer with an interactive signature and undeniable signature [17]. Although this system presented promising experimental results, it has not been put into practical use because it uses an IC card securing a secret key for the ticket purchaser on the presupposition that the IC card is never transferred to people other than the purchaser. The system cannot practically prevent impersonation because the presupposition cannot be always arranged. 2.2 Walk-through System Walk-through systems have been widespread for efficient admission at station and airport gates. A large number of passengers can quickly go through the automatic gates by placing their IC cards or QR codes on reading sensors. Since the systems are based on IC cards and QR codes which are transferable to people other than the purchaser, they cannot practically prevent impersonation. To prevent illegal rides, regulations on passenger operations of railway companies prohibit passengers from boarding with illegal ticket such as borrowing commuter passes of other passengers [18]. Several automatic gates at railway station are designed to emit a sound and/or a light when passengers go through the gate with children-fare tickets. However, they do not completely work for preventing illegal rides. As a walk-through system using biometrics authentication, finger-vein authentication resulted in high throughput equal to that of automatic ticket gates [19]. The system makes it possible to quickly recognize finger-vein patterns when passengers pass their hands over the dedicated biometric sensor. However, the sensor is a bottleneck for putting the systems into practical use. At the identity verification of large-scale events, it is necessary for ordinary people to have their biological information registered in advance at home and checked at event sites on the day of the event with the sensor. The cost and portability are bottlenecks for using walk-through systems. 2.3 Face-recognition Software Many face-recognition software programs have been developed [20], [21]. The face-recognition software that the Ticket ID System uses is the high-speed and high-precision commercial product NeoFace [22]. NeoFace exhibited the highest performance evaluation in the Face Recognition Vendor Test 2014.

(3) Electronic Preprint for Journal of Information Processing Vol.26. Fig. 1. Outline of face-recognition process.. conducted by the U.S. National Institute of Standards and Technology (NIST) [21], [22]. The face-recognition process is outlined in Fig. 1. In the process, registration photos are compared with collation photos to determine whether they show the same person [23]. The Ticket ID System compares registered photos of applicants with collation photos of individuals entering the event venue. First, face detection is executed by detecting and processing the facial areas for each photo. Next the facial-feature points of the detected areas — the eyes, nose, mouth edges, and so forth — are processed to carry out facial-point detection. Finally, the obtained facial-point positions are used to normalize the size and positions of the facial areas and measure their similarity between a registered and collation photo during the collation process. When the similarity measure exceeds a certain threshold, the face recognition is regarded as successful. The threshold was set in accordance with that of NIST personal identity searches [13]. When NeoFace is implemented in a commercially available tablet terminal, the recognition result is displayed with regard to the facial-photo information of 100,000 people within about 0.5 seconds [13].. 3. Ticket ID System Using Face Recognition 3.1. Problems in Verifying Identity of Ticket Holders at Large-scale Events Thorough verification for preventing individuals from impersonating others is in a trade-off relationship with efficient verification. The problem in verifying ticket holders is how to simultaneously verify identities efficiently and prevent individuals from impersonating others at a large-scale event in which tens of thousands of people participate. The solution should be suitable within practical operation costs for various sized events held in various environments including open air. As a practical solution combining efficiency, scalability, and portability for a large-scale event, our Ticket ID System involves a tablet-based face-recognition system [13] and non-stop face-recognition system [14] using NeoFace. Since both systems were proven effective for preventing illegal resale by verifying attendees at large concerts of popular music groups, they have been used at more than 100 concerts [15]. However, simplifying the ID equipment for decreasing operational costs and securing clear facial photos for improving face-recognition accuracy are necessary [14]. 3.2 Tablet-based Face-recognition System The tablet-based face-recognition system supports identity verification of attendees. A venue attendant checks-in by placing. c 2018 Information Processing Society of Japan . Fig. 2 Ticket-verification procedure with Ticket ID System.. Fig. 3. Tablet-based face-recognition system.. his/her membership card on the card reader and executes facerecognition by taking his/her photos. Figure 2 shows the ticketverification procedure from the first step of ticket application to the last step of admission supported with the Ticket ID System. Step 1: Tickets to popular events are often sold on a lottery basis at fan clubs or other organizations where membership is registered. Individuals applying for tickets register their membership information as well as their facial photos. At that time, they are advised of the privacy policy in effect regarding the handling of the photo and other personal information and the verification of their identity on the day of the event. In the same way as for an ordinary ID photo, the registered facial photo is a clearly visible frontal photo taken against a plain background. The face must not be obstructed by a hat, sunglasses, mask, muffler, etc. . . , or by excessively long hair or a flashed peace sign [24]. Step 2: Event organizers notify ticket winners, i.e., successful applicants that have been selected. Since resale risk is high, applicants may only be notified of their selection and not sent the tickets in advance. Step 3: On the day of the event, venue attendants use a membership card reader or other means to verify that individuals entering the venue are successful applicants, as shown in Fig. 3. Step 4: At the event, the attendant uses face-recognition software to confirm that the photo taken at the time of appli-.

(4) Electronic Preprint for Journal of Information Processing Vol.26. Table 1. Comparison of tablet-based and non-stop face-recognition systems.. Fig. 4 Non-stop face-recognition system.. cation and the registered photo show the same person. The attendant explains the verification through face recognition to the attendees and instructs them where to stand in front of the terminal. Then, they execute the face-recognition process using the terminal to confirm the attendees are those who applied for the tickets. Step 5: The admission procedure is carried out in accordance with the face-authentication results. 3.3 Non-stop Face-recognition System The non-stop face recognition system takes attendees’ photos to recognize their faces while they are walking. The photos of walking attendees could not be used for face recognition because the attendees are not always directly facing cameras nor opening their eyes when the system takes their photos. To solve this problem, this system was developed, as shown in Fig. 4, based on the tablet-based face-recognition system with the following improvements: 1) Spacing between a card reader and verification place A card reader is set up about 1.5 m ahead of a venue attendant. Attendees walk toward the attendant after the attendants check-in by placing their membership cards on the card reader. The attendees do not have to stop for their identity verification. 2) Face recognition by using two different photos The system takes two photos of a walking attendee after an interval of about 0.5 seconds with two external IP cameras set up in noticeable places for attendees. The system collates a registered facial photo of the attendee with the two photos taken from different angles at different times. The system determines that identity verification is successful when either photo is identical to the registered photo. 3.4 Problems with Conventional Systems Table 1 compares the tablet-based and non-stop facerecognition systems as the conventional systems [14]. While the systems were used at more than 100 concerts, it was found that it was necessary to simplify the ID equipment from an operational view-point and ensure clear facial photos from a technical viewpoint of face-recognition accuracy. 1) Simplifying ID equipment These conventional systems require a large amount of equipment such as card readers, cameras, and tablet terminals at event. c 2018 Information Processing Society of Japan . Fig. 5 Admission by using identity-verification app.. sites. Several concerts require printers for ticket issuing on the day of the event. For example, one hundred twenty sets of the tablet-based face-recognition systems, each including a check-in system were used for a pop concert [13]. The equipment took much time and were expensive to carry in, set up, and carry out. Therefore, it is important to simplify the equipment for identity verification. 2) Securing clear facial photos Though the conventional systems achieved 90% recognition accuracy, they do not always work when attendees have their eyes closed, are not directly facing a camera, or have their faces covered by their hair. Therefore, it is important to secure clear facial photos from attendees.. 4. Identity Verification System Using Face Recognition from Selfies 4.1 Using Selfies with Attendees’ Smartphone Cameras We propose an identity-verification system for attendees using face recognition from selfies taken with their smartphone cameras The proposed system solves the following problems with the conventional systems: 1) Simplifying ID equipment Attendees open an identity-verification app with their smartphone cameras to show a venue attendant the verification result, as shown in Fig. 5. The app recognizes attendees’ selfies and makes it possible for them to check-in. It is not necessary for an event organizer to prepare card readers for check-in, cameras for taking facial photos of attendees, tablet-terminals for face recognition, and ticket-issuing printers..

(5) Electronic Preprint for Journal of Information Processing Vol.26. tography is useful for face recognition under dark conditions, the front-facing built-in cameras, which are used for selfies are not currently equipped with a flash. Therefore, the app provides a soft-flash screen for selfies under dark conditions, which has a wide white area except for a small viewfinder with the highest brightness, as shown in Fig. 6 (right). The soft-flash screen makes it possible for attendees to execute face recognition by illuminating their faces.. Fig. 6 Self-photographing screen (left) and soft-flash screen (right).. 2) Securing clear facial photos Selfies are helpful for securing clear facial photos because it is possible for attendees to take their own photos with the identityverification app. The app helps attendees take acceptable photos by showing their registered facial photos as good examples together with an instruction message, as shown in Fig. 6 (left). The message suggests that they should directly face the camera without closing their eyes and having their faces covered by their hair. They can re-try if they were not successful. 4.2 Requirements for Using Selfies 1) Pre-screening of Attendee’s Operational Skills Not all attendees possess smartphones or always have sufficient skills to operate a smartphone camera for selfies. Therefore, attendees who would like to enter an event venue with the app have to be checked as to whether they succeeded in identity verification with the app in advance of the event. Successful attendees are allowed to verify themselves by using their selfies with the app on the day of the event. Unsuccessful attendees as well as those who do not want to use the app at an event site can arrange to be verified with conventional systems. The pre-screening makes it possible for an event organizer to estimate the necessary equipment and number of venue attendants by determining the number of attendees verified with the proposed and conventional systems. 2) Preventing impersonation Selfies should be checked when and where they were taken because it is possible for attendees to use ticket winners’ smartphones with which identity verification succeeded by using the winners’ selfies in advance. Therefore, the app is designed to ensure the time and location of selfies, i.e., whether the selfies were appropriately taken at the right place and time of the event, by using the built-in clock and GPS of smartphones. Selfies should be also checked whether they were photos of real ‘live’ face or non-real ‘fake’ face because it is possible for attendees to spoof face recognition systems by presenting facial photos of the ticket winners in front of the camera. To guard against such spoofing, a venue attendant visually ensures that the attendees’ faces match those of selfies at the final step of the identity verification procedure. 3) Self-photographing under dark conditions Face recognition from selfies at an event site is impossible when it is too dark to detect the facial areas. Though flash pho-. c 2018 Information Processing Society of Japan . 4.3 Identity-verification Procedure The proposed system meets the above-mentioned requirements according to the following operational procedure from the first step of ticket application to the last step of admission supported by the app: Step 1: Individuals applying for tickets register their membership information as well as their facial photos in the same manner as the conventional systems [13], [14]. The photos are stored in the membership database. Step 2: After an event organizer notifies ticket winners, i.e., successful applicants that have been selected, the winners can download the identity-verification app. The organizer gives permission to attendees who succeeded in the identity verification by using their selfies with the app in advance of the event. The permitted attendees can enter the event venue with the app. Step 3: The permitted attendees open the app to execute face recognition by taking their selfies at a specified date and place. They show a venue attendant a message displayed on their smartphone when the verification was successful, as shown in Fig. 5. Checking the execution date and place prevents attendees from borrowing or obtaining ticket winners’ smartphones with which identity verification succeeded in advance. Step 4: A venue attendant carries out the admission procedure in accordance with the message on the smartphone display after visually ensuring that the attendees’ faces match those of the selfies. When the attendant invalidates a ticket on the attendees’ smartphone, the same as with an electronic tearing ticket [16], the information is transmitted to an attendeemanagement server of the event organizer. 4.4 Configuration of Identity-verification App The identity-verification app consists of four modules, i.e., face recognition, time-location verification, identity-verification control, and check-in, as shown in Fig. 7. In the face-recognition module, the face-photographing function makes it possible for attendees to take selfies even under dark conditions by means of a soft-flash screen. This module stores the encoded facial image of the attendee that is registered at the time of ticket application. The module collates the selfie with the registered facial photo then transmits the recognition result to the identity-verification control module together with the selfie and registered photo. When the module transmits a signal of face-photographing to the timelocation verification module, that module extracts the time and location data. This module checks whether the extracted data are consistent with the pre-stored time and location data of the event.

(6) Electronic Preprint for Journal of Information Processing Vol.26. data, the transmitted selfie and registered photo, and a sentence prompting the attendee to re-try identity verification, as shown in Fig. 9 (center). When a venue attendant carries out the admission procedure on the attendee’s smartphones, the app transmits the attendee’s check-in information to the event organizer as well as displaying an appreciation message on the display, as shown in the Fig. 9 (right).. 5. Preliminary Tests. Fig. 7 Configuration of identity-verification app.. Fig. 8 Flowchart of identity-verification app.. Fig. 9 Messages of success (left), failure (center), and invalidation (right).. at which an attendee can participate as a ticket winner. After the check, the module transmits the verification result to the identityverification control module. Figure 8 show a flowchart of the identity-verification app. The identity-verification control module determines that the verification result is successful if both results of face-recognition and time-location verification were successful. Otherwise, the identity-verification control module determines that the verification result is a failure. This module transmits the result to the check-in module together with the selfie and registered facial photo. The check-in module stores the attendee’s data, such as name and seat, in advance. This module generates a success message from the attendee’s data, the transmitted selfie and registered photo, and a sentence telling the attendee to enter the venue if the verification was successful, as shown in Fig. 9 (left). Otherwise, it generates an unsuccessful message from the attendee’s. c 2018 Information Processing Society of Japan . 5.1 Face-recognition Parameters The proposed system should be evaluated before the identityverification app is used for actual events from the view-point of face-recognition accuracy. Though time-location verification is reliable because commercial smartphones have practically proven results, face-recognition accuracy has to be scrutinized regarding the feasibility under actual event conditions because attendees take photos of themselves using their smartphones in various environments. Face recognition is controlled using intrinsic, extrinsic, and operational parameters [13]. The intrinsic parameters are of the physical nature of the face and independent of the observer. They include age, expression, and facial paraphernalia such as facial hair, glasses, and makeup. Extrinsic parameters are related to the appearance of the face. They include lighting, pose, background, and imaging such as resolution and focus. Operational parameters are related to the interaction between attendants and attendees. They include how many times the face-recognition process should be repeated per attendee until his/her identity is verified, whether an attendee should stop for the face-recognition process, and whether an attendee should face the camera. The proposed system makes it possible for attendees to control the intrinsic and operational parameters using selfies they have taken of themselves. However, attendees are not able to control several extrinsic parameters such as resolution, lighting, and background. Though resolution is not a problem for a commercial smartphone from the view-point of face recognition, it is necessary to evaluate face-recognition accuracy under actual venue conditions with regard to brightness and background conditions. 5.2 Methods The identity-verification app was developed based on the tablet-based face-recognition system [13]. The face recognition software is NeoFace, and the similarity threshold was set in accordance with that of NIST personal identity searches which achieved the lowest false reject rate (FRR) of 0.3% in processing the visa photo image database at a false accept rate (FAR) of 0.1% in the same manner as conventional systems [13]. The app can be installed on smartphones commercially available in Japan with Android OS and iOS. As preliminary tests, 30 examinees executed face recognition with the app under different brightness and background conditions. The examinees participated in the tests according to the following steps: Step 1: Examinees registered their membership information as well as their facial photos. The photos were stored in the membership database. Step 2: In the same manner as ticket winners, they could download the identity-verification app on their smartphones. They.

(7) Electronic Preprint for Journal of Information Processing Vol.26. Table 3 ID equipment necessary for event organizers.. Fig. 10 Backgrounds of indoor (left-up), outdoor (right-up), crowds (left-down), and under-umbrella (right-down). Table 2 Face-recognition results under various brightness and background conditions.. were permitted to operate the app at any time and place for the tests. They had their operational skills pre-screened after downloading the app. This means that they acquired enough skill to operate the app. Step 3: They started the app to execute face recognition under the following two conditions: bright enough to detect their faces and too dark to detect them. Under the dark conditions, they used the soft-flash screen for face recognition. Both conditions contained four backgrounds, i.e., indoor, outdoor, crowds, and under-umbrella, as shown in Fig. 10. This means that eight selfie patterns were tested for each examinee, i.e., taken under the two conditions multiplied by four backgrounds. The total number of selfies were 240, i.e., 8 patterns multiplied by 30 examinees. 5.3 Results The identity-verification app was downloaded and operated normally without any problems by all examinees. The facerecognition accuracy was 97.5% (the FRR was 2.5%) since 6 photos failed in face recognition among the 240 photos. There was no false acceptance within the 240 photos (the FAR was 0.0%). Table 2 lists the results in the form of the number of failure photos/total number of photos with regard to the two brightness and four background conditions. One failure under the bright outdoor condition was due to the examinees closing his/her eyes during photographing. There were no failure photos due to the fact that examinees were not looking directly forward or that they had hair covering their faces. Five failures in crowds were due to facedetection errors. Since the five images contained several people behind the examinees, the faces of the different people were detected for face recognition. There was no failure photo among those taken with soft-flash or under-umbrella.. 6. Discussion 6.1 Problems with Conventional Systems 1) Simplifying equipment Table 3 shows ID equipment that event organizers have to pre-. c 2018 Information Processing Society of Japan . pare for using conventional systems (the table-based face recognition system and non-stop face recognition system) and the proposed system. Before event days, all systems require ticket-application servers for individuals to apply for tickets by registering their membership information as well as their facial photos. On the event days, the conventional systems require card readers, personal computers (PCs) as well as displays for attendees’ checkin. After the check-in, the table-based face recognition system requires tablet terminals with built-in cameras, and the non-stop face recognition system requires PCs and external cameras. Both systems require printers for issuing ticket during the admission procedure. All three systems require an attendee-management server to record attendees who checked-in on the event days. The proposed system does not require event organizers to prepare any equipment for check-in, face recognition, or ticket-issuing because they are supplanted by attendees’ smartphones. The proposed system can be helpful for simplifying ID equipment compared with the conventional systems. 2) Securing clear facial photos The identity-verification app was downloaded and operated normally for the preliminary tests with the commercial smartphones of all examinees. It was not difficult for the examinees to operate the app. The face-recognition accuracy was 97.5%, which was higher than 90% of the tablet-based face-recognition system and 91% of the no-stop face recognition system, as respectively mentioned in Sections 3.2 and 3.3. The higher accuracy was achieved by self-photographing and pre-screening of attendee’s operational skills. Self-photographing is regarded as helpful to secure clear facial photos because examinees could control the intrinsic parameters such as their expressions, facial hair, and facial directions. There was only one failure due to the examinees closing his/her eyes during photographing, and no failure photos of examinees not looking directly forward or having hair covering their faces. The pre-screening of attendee’s operational skills understandably resulted in higher accuracy because the attendees can learn how to take photos acceptable for face recognition, i.e., how their faces could be successfully recognized at the step of gaining permission of the app, as mentioned in Section 4.3..

(8) Electronic Preprint for Journal of Information Processing Vol.26. required to perform certain tasks such as smiling and moving their sight line upward at the time of face recognition. It is difficult for attendees to prepare fake facial photos or videos in advance because the tasks are randomly chosen among various detectable facial movements. When it is necessary to detect face liveness, NeoFace Monitor can be used as the face recognition software of the app. Fig. 11. Photographing abreast.. 6.2 Self-photographing under Various Conditions 1) Background conditions Face detection exhibited a problem in that faces of incorrect people were detected when selfies contained other people behind the examinees. This can be practically solved by choosing the face with the largest face area among all the detected faces. Detected face areas may be equal when two people are photographed abreast intentionally, as shown in Fig. 11. The app will be improved with a re-try function with a message telling attendees to take a photo of one person again when it detects same-sized faces. 2) Brightness conditions The soft-flash screen made it possible for attendees to execute face recognition even under conditions in which it was too dark to detect their faces. In general, brightness is more than 1,000 lux in bright offices such as department stores, 750 lux under shopping arcades at night or just after sunset, 200 lux under street lights or in bedrooms, and 30 lux in moonlight or candlelight [27]. The soft-flash screen provided a brightness of more than 80 lux for face recognition. Since the soft-flash illuminated the face of only the person close to the smartphone screen, it prevented the detection of those of other people in the crowd. 6.3 Preventing Impersonation Face recognition systems are vulnerable to spoof attacks involving fake faces. A secure system requires face-liveness detection to guard against such spoofing [28]. The proposed system can be more secure by taking following approach: 1) Visual detection of designated backdrop patterns On the event day, attendees are required to take selfies in front of designated backdrop patterns such as black banners in front of the main gate and polka-dot screens at the check-in gates. The designated backdrop patterns are helpful scenic cues for a venue attendant because he/she can easily distinguish fake facial photos from those of attendees by glancing at the backdrop pattern of the textures and colors. It is difficult for attendees to prepare fake facial photos including the backdrop patterns in advance, which change depending on the event. 2) Detection of designated backdrop patterns The designated backdrop patterns can be detected with analysis techniques of their textures and colors [28]. When it is necessary to for a venue attendant to assist in visual detection, the techniques can be applied to the face recognition software of the app. 3) Detection of facial expression movements NeoFace Monitor can detect certain movements of facial expressions [29]. When it is implemented to the app, attendees are. c 2018 Information Processing Society of Japan . 7. Future Issues The preliminary tests ensured the feasibility for simplifying ID equipment by using selfies with attendees’ smartphones. They also clarified that selfies are helpful for securing clearer images than with conventional systems. However, the proposed system should be more practically evaluated. After improving face detection, we will conduct the following tests to implement the proposed system for actual large-scale events. 7.1 Pseudo Attack Tests The FAR should be more carefully examined from the viewpoint of preventing impersonation. It is necessary to evaluate the robustness with pseudo attack tests. The tests should include a disguise test and lookalike test. A disguise test makes people’s facial appearances as similar as possible to those of different people by using facial paraphernalia such as facial hair, glasses and makeup. A lookalike test is conducted for those, such as twins or similar looking siblings, with a similar physical nature regarding facial appearances. The disguise test will reveal tricks and help in creating operational manuals for venue attendants. A lookalike test will disclose the technical limitation of current face recognition techniques and help in establishing next-generation technology. 7.2 Evaluation of Identity Verification Procedure Each step of the identity-verification procedure mentioned in Section 4.3 should be evaluated from the view-points of attendees and attendants except for the first step, which is the same as that of the conventional systems. At Steps 2 and 3, it is necessary to evaluate the number of identity-verification trials as well as the accuracy and time of each trial, i.e., how often attendees fail in identity verification by using the app and how many attendees succeed in the identity verification at the first trail, second trial, and so on, together with the time spent operating the app. It is important to clarify the tolerable number of trials, which is how many trials it will take for attendees to give up using the app before they succeed in their identity verification at Steps 2 and 3. We are investigating attendees’ opinions through questionnaire surveys on the identity verification procedure and user interface of the app to improve the proposed system from the view-point of attendees. At Step 4, we measure the operation time per person, i.e., how long attendants should spend in the admission procedure per attendee to estimate the number and allocation of necessary venue attendants according to the event scale. Larger-scale tests are planned at actual event sites where the proposed system is expected to be used. All the modules of the identity-verification app will be checked as a rehearsal for actual events. At the event sites, we will install conventional systems.

(9) Electronic Preprint for Journal of Information Processing Vol.26. for attendees who did not download the identity verification app in advance and those who gave up using the app because of smartphone problems on the event day. We are also developing operational guidelines for dealing with disruptive individuals.. [12]. [13]. 8. Conclusion We developed an identity-verification system for attendees of large-scale events using face recognition from selfies. The proposed system simplifies ID equipment by using attendees’ smartphone cameras and ensures clear facial images from attendees by allowing them to take their own photos. The system achieved 97.5% face-recognition accuracy in preliminary tests involving 240 photos taken by 30 examinees under various brightness and background conditions. The soft-flash screen made it possible for attendees to execute face recognition under conditions in which it would be too dark to detect faces. Though the preliminary tests ensured the feasibility of our system for practical use, the proposed system should be evaluated on a larger scale from the view-points of attendees and venue attendants. After improving face detection, we will conduct attack tests and evaluate the identity verification procedure for actual large-scale events. We are developing a clear guideline for the system introduction by estimating the number and allocation of necessary venue attendants through larger-scale tests at actual sites. Acknowledgments Thanks are expressed to all the personnel related to our systems, especially to Mr. Kiyoshi Sugiyama, President of NEC Solution Innovators, Ltd. for his encouragement and support. References [1] [2] [3]. [4]. [5]. [6] [7] [8] [9]. [10] [11]. The National Consumer Affairs Center of Japan: Internet auction, available from http://www.kokusen.go.jp/soudan topics/data/ internet3.html. (in Japanese) The Asahi Shimbun, The Yomiuri Shimbun: No illegal ticket resale, full-page advertisements (2016), available from http://www.tenbaino.jp/. (in Japanese) Nihon Keizai Shimbun Electronic Version: NEC’s face recognition prevents illegal reselling of Momoiro Clover-Z tickets (2014), available from http://www.nikkei.com/article/ DGXMZO80664930Y4A201C1H56A00/. (in Japanese) JE fandom: Johnny’s Tracks Illegally Sold Tickets for Arashi’s Japonism Tour (2015), available from https://jnewseng.wordpress.com/ 2015/10/25/johnnys-tracks-illegally-sold-tickets-for-arashisjaponism-tour/. Japan Information Economic and Social Promotion Association: 2012 Information Security Promotion Business survey report, Survey Research on Social Infrastructure Construction Using Attribute Information for Identification, p.16 (Mar. 2013). (in Japanese) Imaoka, H., Mizoguchi, M. and Hara, M.: Biometrics technology to preserve safety and security, Information Processing, Vol.51, No.12, pp.1547–1554 (2010). (in Japanese) Seto, Y.: Trends and prospects in biometric security authentication technology, Information Processing, Vol.47, No.6, pp.571–576 (2006). (in Japanese) Soto, M.: Using biometric authentication technology in Japanese financial institutions, Information Processing, Vol.47, No.6, pp.577– 582 (2006). (in Japanese) Sakamoto, S.: Present Status and Prospects of Biometric Products and Solutions, NEC Technical Report, Vol.5, No.3 (2010), available from http://www.nec.com/en/global/techrep/journal/g10/n03/pdf/ 100303.pdf. Zhao, W., Chellappa, R., Rosenfeld, A. and Phillips, P.J.: Face Recognition: A Literature Survey, ACM Computing Surveys, pp.399–458 (2003). Jafri, R. and Arabnia, H.R.: A Survey of Face Recognition Techniques, Journal of Information Processing Systems, Vol.5, No.2, pp.41–67 (2009).. c 2018 Information Processing Society of Japan . [14]. [15] [16]. [17] [18] [19]. [20] [21] [22] [23] [24] [25] [26] [27]. [28] [29]. Face Recognition Technology Evaluation Committee for Immigration: Demonstration experiment results on face recognition technology for Japanese going and returning from abroad (2014), available from http://www.moj.go.jp/content/001128805.pdf. (in Japanese) Okumura, A., Hoshino, T., Handa, S., Nishiyama, Y. and Tabuchi, M.: Identity Verification of Ticket Holders at Large-scale Events Using Face Recognition, Journal of Information Processing, Vol.25, pp.448– 458 (June 2017). Okumura, A., Hoshino, T., Handa, S., Nishiyama, Y. and Tabuchi, M.: Improving Identity Verification for Ticket Holders of Large-scale Events using Non-stop Face Recognition System, Journal of Information Processing, Vol.8, No.1, pp.1–8 (2018). (in Japanese) JSAI: The 2015 Gold Award for Field Innovation from the Japanese Society for Artificial Intelligence (2016), available from https://www. ai-gakkai.or.jp/about/award/#INNOVATION. (in Japanese) fringe watch: “tixee” ticketless management system realized “Denshi Mogiri” service for smart phones without barcode or QR code, available from http://watch.fringe.jp/2013/0613200357.html. (in Japanese) Kaine, I., Sasaki, R. and Saito, T.: The Proposal and Evaluation of a Transfer Prohibited Electronic Ticket System with Anonymity, Information Processing, Vol.47, No.7, pp.2267–2278 (2006). (in Japanese) JR East: Boarding with invalid or no ticket, Part II – Chapter 7 – Section 3 – Subsection 2 of the Regulations on Passenger Operations. Nagasaka, A.: New Development of Biometrics Authentication for Application as Social Infrastructure–Walk-through Type of Finger Vein Authentication, The Sixth Symposium on Biometrics, Recognition and Authentication (2016). FACE RECOGNITION HOMEPAGE VENDORS, available from http://www.face-rec.org/vendors/. NIST: Face Projects; available from https://www.nist.gov/programsprojects/face-projects. NEC: NEC Face Recognition, available from http://jpn.nec.com/ face/. (in Japanese) Imaoka, H.: NEC’s Face Recognition Technology and Applications, IPSG SIG Technical Report, Vol.2013-CVIM-187, No.38, pp.1–4 (2013). (in Japanese) Every Entertainment Inc.: Momoclo Tickets: The Face Image Format Standards, available from https://momoclo-ticket.jp/mp/ps. (in Japanese) Tapirs Inc.: Face recognition, available from https://www.tapirs.co. jp/face-authentication.html. (in Japanese) Tapirs Inc.: Ticket issuing on the day, available from https://www. tapirs.co.jp/unveil-ticket-detail-on-the-exact-show-day.html. (in Japanese) National Institute of Advanced Industrial Science and Technology: Let’s make the brightness standards of Japan, available from http://www.aist.go.jp/science town/standard/standard 03/standard 03 04.html. (in Japanese) Chakraborty, S. and Dhrubajyoti, D.: An Overview of Face Liveness Detection, International Journal on Information Theory (IJIT), Vol.3, No.2, pp.11–25 (Apr. 2014). NEC: NeoFace Monitor, available from https://jpn.nec.com/ biometrics/face/product/neoface monitor.html. (in Japanese).

(10) Electronic Preprint for Journal of Information Processing Vol.26. Akitoshi Okumura received his B.E. and M.E. degrees in precision engineering from Kyoto University, Kyoto, Japan in 1984 and 1986. He joined NEC Corp. in 1986 for researching natural language processing, speech translation, and AI robots at Central Research Laboratories. He joined DARPA machine translation project as a visiting scientist at the University of Southern California, Los Angeles, USA in 1993. He received his Ph.D. in computer science from Tokyo Institute of Technology, Tokyo, Japan in 1999. He is a senior vice president of NEC Solution Innovators, Ltd. He is the recipient of the 2005 Nagao Award from AAMT, 2007 METI Minister Award from Fuji Sankei Business i, 2010, 2015, and 2016 Field Innovation Awards from JSA, 2008 Kiyasu Special Industrial Achievement Award, 2017 Industrial Achievement Award and 2017 IPSJ Yamashita SIG Research Award from IPSJ.. Takamichi Hoshino received his B.E. degree from Tokai University, Kanagawa, Japan in 1983. In 1983, he joined NEC Informatec Systems, Ltd. His current interests include the business promotion and development of physical security systems related to advanced technologies. He is the recipient of the 2015 Gold Award for Field Innovation from JSAI.. Susumu Handa received his B.S. degree in physics from Chuo University, Tokyo, Japan in 1984 and his Ph.D. in computer science from Kyushu Institute of Technology, Fukuoka, Japan in 2001. He is currently a manager in the Advanced Technology Solutions Division at NEC Solution Innovators, Ltd. His current interests include the business promotion and development of computer systems related to advanced technologies, especially face recognition and visualization in a high-performance computing area. He is the recipient of the 2015 Gold Award for Field Innovation from JSAI.. Eiko Yamada received her B.E. and M.E. degrees from Ochanomizu University, Tokyo, Japan in 1990 and 1992. She has been engaged in developing software systems at NEC Solution Innovators, Ltd. Her current interests include developing software for face recognition. She is the recipient of the 2016 Award for Field Innovation from JSAI.. c 2018 Information Processing Society of Japan . Masahiro Tabuchi received his B.E., M.E., and Ph.D. degrees in computer science from Waseda University, Japan in 1987, 1989, and 1993. He joined NEC Corporation in 1993 as a Researcher at NEC C&C Research Labs. and is now a senior expert of business development in the Advanced Technology Solutions Division at NEC Solution Innovators, Ltd. He is interested in human augmentation by using cognitive science and artificial intelligence. He is the recipient of the Best Paper Award for Young Researchers of IPSJ National Convention in 1987 and IPSJ Yamashita SIG Research Award in 1994. He is a member of IPSJ and the Institute of Electronics, Information and Communication Engineers..

(11)