A Rhythmical Tap Approach for Sending Data across Devices
Abstract
This paper proposes a new user interface technique to specify sending data across digital devices. In this approach, users specify what to send from what device to what device by tapping them rhythmically. This technique is easy to operate, low implementation cost, applicable to a wide range of devices, and scalable by adding numerous rhythmical tap sequences. We confirmed the feasibility of this approach through preliminary experiments.
Author Keywords
Rhythmical taps; Ad-hoc network connections; Cross- device interaction.
ACM Classification Keywords
H.5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous;
Introduction
With the advent of various mobile devices such as smartphones, tablet PCs, and watches, people are now surrounded by many digital appliances. If we can easily establish ad-hoc network connections among these devices, this would be effective in situations where switching projected documents or sending data to other people in meetings, sending documents to printers, or searching Web pages using text sent by other devices.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored.
Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from
MobileHCI '16 Adjunct, September 06-09, 2016, Florence, Italy
© 2016 ACM. ISBN 978-1-4503-4413-5/16/09…$15.00 DOI: http://dx.doi.org/10.1145/2957265.2961851 Hirohito Shibata
Research & Technology Group, Fuji Xerox Co. Ltd.
430 Sakai, Nakai, Kanagawa, 259-0157, Japan
Junko Ichino
Graduate School of Engineering, Kagawa University
Hayashicho 2217-20, Takamatsu, Kagawa, 761-0396, Japan [email protected]
Tomonori Hashiyama Shun’ichi Tano
Graduate School of Informatics and Engineering,
The University of Electro- Communications
Chofugaoka 1-5-1, Chofu, Tokyo, 182-8585, Japan [email protected] [email protected]
However, to send data from one device to another, users need to setup network connections and specify a source device and a target device for sending data.
Moreover, it is desirable to allow specifying what kind of data is sent from the source device to the target device. Users might want to send a file, a page, a mail address, a URL, or just text to achieve more elaborate cooperation among devices depending on a situation.
We need an easy user interface technique to send data among various devices with different sizes and different OSs in a uniform manner.
Related Work
To easily send data among devices, various user interface techniques have been proposed. They are roughly divided into four types.
The first ones are techniques to connect displays visually [2,5,15]. After integrating display spaces, users can send data across devices using a drag-and-drop user interface.
It is intuitive, but the procedure to send data is not easy.
Users have to physically connect devices and perform a drag-and-drop operation. Additionally, because users must have to devices to connect them, we cannot adopt this technique for heavy devices or devices fixed to environments. Moreover, it is difficult to implement a drag-and-drop interface among different OSs.
The second techniques specify connecting devices by using special gestures such as moving devices closely each other [12], tossing devices [16], and drag using hands [8]. They all provide an easy intuitive user interface. However, they need special sensors to detect the proximity or position of devices. Additionally, we cannot adopt these techniques to unmovable devices which cannot be held by hands.
The third techniques point to multiple devices by using pens with IDs [8], fingers [14], or special buttons [4].
They can be applied to unmovable devices. However, they need additional devices such as pens, buttons, or sensors to detect a finger print. Because their implementation cost is not low, we cannot assume that a wide range of devices implement these techniques in the future.
The forth techniques use synchronous gestures to specify the connection of devices [9]. In these techniques, synchronous events across devices such as shaking devices [3], bumping devices [1], and pressing and releasing buttons [11], are used as key gestures to connect the devices. They are easy to implement.
However, synchronous shaking and bumping gesture cannot be applied to heavy devices or fixed devices.
Moreover, as a more important thing, these techniques do not have scalability that can be extended to specify various types of coordination of what kind of data are sent.
We summarize requirements of our interaction technique for sending data among devices.
The procedure needs to be easy (Easiness).
It should be implemented at low cost so that various devices can adopt our framework (Low-cost).
It should be applied to a wide range of devices in a uniform manner (Applicability).
It should provide scalability in specifying what to send to meet future user needs (Scalability) Previous approaches do not satisfy the above all requirements. We aim to provide a new user interface
technique for sending data across devices that satisfies the above all four requirements at the same time.
Rhythmical Tap Approach
In our approach, we develop a solution based on the framework of the synchronous gesture because it is easy to operate (Easiness) and its implementation cost is low (Low-cost).
In addition, we develop our technique based on touch gesture because touch sensors on panels are now widely used and we can assume that many devices will provide touch sensors more and more. Moreover, touch gesture can be adopted by various devices from small ones to large ones and from mobile ones to fixed ones.
Therefore, our approach will be applicable to various types of devices (Applicability).
However, as we described in the previous section, current approaches of synchronous gestures have the problem that they do not have the scalability to specify what to send. They connect devices based on a fact that same events simultaneously occurs at different devices. To increase coordination types, we need to give a variety in event types.
To give the variety of events to synchronous tapping, we expand a single tap to a sequence of taps on a temporal axis. In our approach, we connect multiple devices if rhythmical taps occur across multiple devices and send data from the one tapped at the first tap to the other, where we call this rhythmical tap approach.
Rhythmical patterns are diversified and we can create various rhythms. If we specify what to send by rhythmical taps, we can meet a new user need by adding a new rhythm (Scalability).
Figure 1 shows a simple example of this approach. Let’s consider the situation that tap events T1, T2, and T3
occur in devices A, B, and A in this order. Let L1=T2-T1
and L2=T3-T2. If |L1-L2| is small enough, then we consider this tap sequence is rhythmical and data corresponding to the rhythm are sent from A to B.
Although cross-device interaction using digitized visible light and touch is proposed [7], our technique is unique in using users' rhythmical taps to judge the type of communication among devices.
We consider why we adopted rhythmical taps to specify what to send. First, almost all people can tap
rhythmically for multiple devices, if the rhythm is not too complicated. This means they can easily specify sending data among devices.
Second, it is difficult to perform rhythmical taps with other people. It is difficult to play a session with those who do not have an intention to cooperate with other people. This means it is difficult to steal data by breaking into other people’s session intentionally.
Finally, if there is a conductor and people have an intention to cooperate with others under the conductor, they can perform rhythmical taps together easily. This means users can send data to multiple users at the same time. For example, in a meeting, a presenter might send a file to all audiences. In this case, the presenter becomes a conductor, decides a rhythmical tap sequence (e.g., T1T2T3 of Figure 1), assigns conductor's taps (e.g., T1 and T3 of Figure 1) and audiences' taps (e.g., T2 of Figure 1), and performs rhythmical taps with audiences by keeping rhythm as a conductor. Then the presenter’s file will be sent to Figure 1: An example of
rhythmical taps
audiences' devices. This means our approach can be extended to one-to-many data sending.
As a network architecture, in this paper we assume a server-client architecture, that is, all digital devices are connected and registered to a server. However, we can easily extend this framework as a peer-to-peer service without any server or a cloud service that all digital devices are connected to a cloud server.
In our current study, we focus on providing theoretical feasibility of our approach rather than implementation.
We also focus on a user interface technique to specify what to send from what device to what device.
Additionally, we have not decided how to assign rhythms to ways of coordination. We are currently focusing on how we can precisely detect intended rhythmical taps without detecting unintended spontaneous tap sequences.
Feasibility
The following are steps that we conducted to confirm the feasibility of the rhythmical tap approach.
1. We must understand what kind of rhythmical taps are preferred or easy to operate. We conducted subjective evaluation of rhythmical taps and selected some of them that seemed to be reasonable as instruction methods when sending data.
2. Users cannot perform rhythmical taps precisely. To understand the distribution of the accidental errors, we collected users' actual data of rhythmical taps.
3. We considered how to detect rhythmical taps based on the analysis of the users’ actual rhythmical taps.
4. If the detecting method detects many false detections, that is, if it detects users’ spontaneous tap sequences by chance frequently, this method cannot be used in real world. Therefore, we conducted a simulation to check how many false detections occurred in our daily situation.
Selecting rhythmical tap sequences
To understand preference or easiness to operate, we conducted a subjective evaluation of rhythmical taps.
The participants were 15 people (13 men, 2 women).
Their ages were from 22 to 26 (avg. 23.8).
We selected 99 rhythmical taps, where the tap count of each rhythm was less than six. They are systematically selected considering some features of rhythms such as tap counts of dominant hand, the count of pause, and successive tap counts. The participants evaluated each rhythmical taps in five-point scaling for four criteria:
ease of tapping in specified speed (the interval of taps was 150ms), ease of tapping in high speed, ease of memorizing, and friendliness. Figure 2 is a sample of the evaluation sheets. As the notation of rhythms, "A"
stands for a tap by a dominant hand, "B" stands for a tap by a non-dominant hand, "C" stands for a tap by both hands, and "-" stands for a pause.
We evaluated rhythmical taps by total scores of the four criteria and selected eight rhythms (ABAB, ABA, ABABA, AB-AB, AB-C, AB-A, ABA-A, and AAB). We also added three rhythms (AB-B, ABA-C, and ABB) by taking into account the symmetry of both hands.
Collecting rhythmical taps
To collect users' actual rhythmical taps, we asked participants to tap rhythms in various conditions.
Figure 2: A sample of an evaluation sheet
The experimental design was a four-way factorial design. The first factor was the directions of display surfaces with two levels: Horizontal and Vertical. The second factor was rhythms with eleven levels: ABA, AAB, ABB, ABAB, AB-A, AB-B, AB-C, ABABA, ABA-A, ABA-C, and AB-AB. The third factor was hands of the first taps with two levels: Regular (starting with dominant hand) and Reverse (starting with non-
dominant hand). The fourth factor was speeds with four levels: Preferred speed, Slow, Fast, and Specified speed.
The participants were 15 people (13 men, 2 women).
Their ages were from 22 to 26 (avg. 23.6). They were all right-handed. All participants tapped rhythmical taps in all conditions. The order of conditions for each participant was randomized.
The participants tapped on the panel of Microsoft Surface Pro 3 (Windows 10). All tap events were saved in logs with timestamps. We collected 8678 valid participants' rhythmical taps (5.16 per condition and per participant). We analyzed them to determine the algorithm to detect rhythmical taps.
The tapping speed is fast in Fast, Specified, Preferred, and Slow in this order. The interval of each tap was 115.4, 139.2, 217.1, and 391.6ms in each tapping speed respectively.
Detecting method
Before presenting a method to detect rhythmical taps, we explain notations. A tap is expressed as Ti as shown in Figure 3. A tap Ti is also used as the time when Ti
occurred. To express the device of Ti, we use a notation of device(Ti).
The third tap and the fourth tap of AB-C should be tapped simultaneously, and the fourth tap and the fifth tap of ABA-C should be also tapped simultaneously.
Analyzing the difference of these participants' simultaneous taps, the upper 95% limit was 46.3ms (Dmax). Therefore, we consider that two taps are tapped simultaneously if the time difference of these taps is less than this value.
Regarding the time difference of the first two taps of all rhythms, the lower 95% limit was 47.0ms (Smin) and the upper 95% limit was 451.3ms (Smax). If two taps occur within this time interval, new rhythmical taps might start from the two taps.
To detect rhythmical taps, we assume that all tap events include a tapped device and time. To consider a tap sequence as a rhythmical one, we use two criteria:
the validity of devices and the validity of intervals.
Regarding the validity of devices, we check the order of tapped devices. For example, in rhythm ABA, the device of the first tap is different from the device of second tap and same as the device of the third tap.
Regarding the validity of intervals, we check the length of intervals between taps. We estimate L2, L3, and L4
based on L1. Table 1 shows estimation formulas for each rhythm based on a single regression analysis. If actual L2, L3, and L4 are contained in the 95%
prediction interval calculated by the estimation formulas and standard errors, we considered the tap sequence as a rhythmical one.
The criteria for judgment are different for every rhythm.
We present just one example. Regarding ABA-C, a tap Figure 3: Notation of taps and
intervals.
sequence T1T2T3T4T5 is considered as a rhythmical one, if these taps satisfy the following requirements.
(The validity of devices) device(T1)≠device(T2), device(T1)=device(T3), device(T4)≠device(T5) and (device(T1)=device(T4) or device(T1)=device(T5)).
(The validity of the first interval) Smin≦ L1≦ Smax.
(The validity of the second interval) Letting [min2, max2] be 95% prediction interval of L2 based on L1
in ABA-C, T3 is contained in [T2+min2, T2+max2].
(The validity of the third interval) Letting [min3, max3] be 95% prediction interval of L3 based on L1
in ABA-C, T4 is contained in [T3+min3, T3+max3].
(The validity of the simultaneous taps) 0 ≦ L4≦ Dmax.
We implemented this detection method and tried to detect rhythmical taps for the actual rhythmical taps collected in the previous experiment. As a result, this method covered 84.15% of the participants’ actual rhythmical taps.
Possibility of false detection
Next, we present a simulation to measure how many false detections occurred in a "natural" work situation.
We made a model of this "natural" work situation based on event logs of PC operations in our previous study [13]. The logs were collected from eight office workers in 2007. We hypothesize that the use of mobile devices would be less than the use of PCs and tapping in mobile devices would be less than the clicking in PCs.
In a model of our simulation, all persons own five mobile devices. The period of the study is five week days. Basic data as for how users use their digital devices are determined based on the data obtained from the analysis of the above PC operation logs. For example, the users use digital devices 3 hours 27 minutes a day. They may use multiple devices at the same time. Average duration time of the devices was 17.68 minutes and the taps on the devices occur 3.86 times per minute.
We varied the number of persons and measured how many false detections occurred in each situation. Table 2 shows the count of false detections in each condition for each rhythm. For ABAB, AB-C, ABABA, ABA-A, ABA- C, and AB-AB, which include at least four taps and are colored in Table 2, no false detection count was found for less than 10 persons. In the case of 10 persons (i.e., 50 devices), only one false detection was found in ABAB and AB-AB.
Table 1: Estimation of L2, L3, and L4 based on L1 for each rhythm.
Rhythm Estimation of L2 Estimation of L3 Estimation of L4
ABA 0.91 L1 + 37.2 AAB 1.03 L1 - 7.1 ABB 0.85 L1 + 43.7
ABAB 0.96 L1 + 14.1 0.95 L1 + 10.6 AB-A 1.54 L1 + 183.0
AB-B 1.48 L1 + 195.9 AB-C 1.45 L1 + 207.9
ABABA 0.93 L1 + 22.5 0.97 L1 - 0.5 0.92 L1 + 36.4 ABA-A 0.97 L1 + 28.0 1.54 L1 + 166.5
ABA-C 0.95 L1 + 34.1 1.44 L1 + 170.0 AB-AB 1.59 L1 + 146.5 0.95 L1 + 0.3
We can say that our rhythmical tap approach is a promising framework to send data among devices in a small group with 10 persons using 50 devices, if rhythmical taps include at least four taps.
Discussion
To check the validity, we predicted second and further tap intervals based on the first interval by using a single regression analysis. It is the most simple prediction method. We can adopt more accurate prediction method. Or we can adopt a discrimination analysis to detect rhythms by looking at whole taps simultaneously, not looking at each tap one-by-one like the method of this paper. Trying to adopt these
methods, we can expect false detections would become fewer and our approach would become safer.
In this paper, we focused on confirming the theoretical feasibility of rhythmical tap approach. To implement this approach, we need to resolve some challenges. The first one is a problem of network delays. In this paper, we considered that there were no network delays to collect tap events from all devices. However, bluetooth or wireless LAN causes 1-20ms delays to send data among devices. Taking into account such network delays, the cover rate and the false detection rate of the detection method will be changed.
As a next challenge, we need to consider a peer-to- peer network architecture as an easy ad-hoc network.
Our approach of this paper can be extended to a peer- to-peer network protocol. We need to verify it does not cause the increase of network traffic.
The method presented in this paper can detect collaborative rhythmical taps of one-to-many data
sending as it is. However, it seems difficult to tap rhythmically with people together. Therefore, we might need to lower the threshold to detect rhythmical taps if the system anticipates collaborative tapping.
Additionally, it might be desirable that the system should give feedback to users whether their taps were too fast or too slow when they failed in collaborative
rhythmical taps.
Finally, all participants in our experiments were in their 20s and most of them were right-handed. We need to gather more diversified people such as younger or elder people or left-handed people.
Conclusion
This paper proposed a new user interface technique to specify sending data across digital devices. In this approach, users specify what to send from what device to what device by tapping multiple devices rhythmically.
In the first experiment, we conducted a subjective evaluation of rhythmical taps and selected 11 rhythms.
In the second experiment, we collected participants' actual rhythmical taps for the 11 rhythms. By analyzing the logs of rhythmical taps, we set up a reasonable detection method that covered a wide range (84.15%) of the participants' actual rhythmical taps and false detection did not occur frequently (at most once in a week) for rhythms including four or more taps.
Although it remains many challenges to make this approach practical, we could confirm the feasibility of this approach in preliminary experiments.
As the future work, we need to improve our detection algorithm by using more sophisticated prediction methods or discrimination analysis. Additionally, we need to consider implementing this approach in a peer- Table 2: Results of simulation
to measure how many false detections occur in a work situation
Rhythm Person 1 5 10 20
ABA 3 11 57 233
AAB 2 15 44 193
ABB 1 44 205 1653
ABAB 0 0 1 7
AB-A 4 39 112 446 AB-B 6 107 497 3390
AB-C 0 0 0 0
ABABA 0 0 0 0
ABA-A 0 0 0 2
ABA-C 0 0 0 0
AB-AB 0 0 1 29
to-peer network as more realistic network architecture.
Finally, we need to implement this approach in real devices and compare the effectiveness with other methods to send data between devices.
Trademarks
Microsoft, Windows, and Surface are trademarks or registered trademarks of Microsoft Corporation.
All brand names and product names are trademarks or registered trademarks of their respective
companies.
References
1. Hinckley, K. Synchronous gestures for multiple persons and computers. Proc. UIST '03, ACM (2003), 149-158.
2. Hinckley, K., Ramon, G., Guimbretiere, F.,
Baudisch, P., and Smith, J. Stitching: Pen gestures that span multiple displays. Proc. AVI '04, ACM (2004) 23-31.
3. Homquist, L.E., Schiele, M.B., Alahuhta, S.P., Beigl, M., and Gellersen, H. Smart-Its friends: A
technique for users to easily establish connections between smart artefacts. Proc. UbiComp '01, Springer (2001), 116-122.
4. Iwasaki, Y., Kawaguchi, N., and Inagaki, Y. Touch- and-Connect: A connection request framework for ad-hoc networks and the pervasive computing environment. Proc. PerCom ‘03, IEEE (2003).
5. Marquardt, N., Hinckley, K., and Greenberg, S.
Cross-device interaction via micro-mobility and f- formations. Proc. UIST '12, ACM (2012), 13-22.
6. Morris, M.R., Huang, a., Paepcke, A., and Winograd, T. Cooperative gestures: Multi-user gestural interactions for co-located groupware.
Proc. CHI '06, ACM (2006), 1201-1210.
7. Ogata, M., Sugiura, Y., and Imai, M.: FlashTouch:
Touchscreen communication combining light and touch. Proc. SIGGRAPH 2015, ACM Press (2015).
8. Radle, R., Jetter, H.C., Marquardt, N., Reiterer, H., and Rogers, Y.: HuddleLamp: Spatially-aware mobile displays for ad-hoc around-the-table collaboration. Proc. ITS '14, ACM Press (2014).
9. Ramos, G., Hinckley, K., Wilson, A., and Sarin, R.
Synchronous gestures in multi-display
environments. Human-Computer Interaction, 24 (1-2), (2009), 117-169.
10. Rekimoto, J. Pick-and-drop: A direct manipulation technique for multiple computer environments.
Proc. UIST'97, ACM (1997), 31-39.
11. Rekimoto, J., Ayatsuka, Y., and Kohno, M.
SyncTap: An interaction technique for mobile networking. Proc. MobileHCI '03, Springer (2003).
12. Rekimoto, J., Ayatsuka, Y., Kohno, H., and Oba, H.
Proximal Interactions: A direct manipulation technique for wireless networking. Proc. INTERACT '03, Springer (2003), 511-518.
13. Shibata, H: Measuring the efficiency of the
introduction of large displays and multiple displays.
IPSJ Journal, 50 (3), (2009), 1204-1213. [in Japanese]
14. Sugiura, A., and Koseki, Y. A user interface using fingerprint recognition: Holding commands and data objects on fingers. Proc. USIT '98, ACM (1998), 71-79.
15. Tandler, P., Prante, P., Muller-Tomfelde, C., Streitz, N., and Steinmetz, R. ConnecTables: Dynamic coupling of displays for the flexible creation of shared workspaces. Proc. UIST '01, ACM (2001).
16. Yatani, K., Tamura, K. Hiroki, K., Sugimoto, M., and Hashizume, H. Toss-It: Intuitive information transfer techniques for mobile devices using toss and swing actions. IEICE Transactions on Information & Systems, E89-D (1), (2006).
