Tatsuo Nak ajimaand Atsushi Hasegaw a
Departmentof Information and Computer Science
Waseda University
3-4-1 Okub o Shinjuku Tokyo 169-8555 JAP AN
Abstract
Inthispap er,weprop oseuniversalinteractionfornet-
worked home appliances, which is a simple mechanism
to llthegap b etween traditionaluser interface systems
and advanced user interaction devices. Our system en-
ables us to control appliances in a uniformway at any
places, andthe systemallowsusto select suitable input
andoutput devicesaccording toourpreferences andsit-
uations. Also, the devices can b e changed dynamically
according toauser's preferences.
1 Introduction
In ubiquitous computing environments[6], one of the
mostimp ortantproblemsishowto interact withavari-
etyofobjectsemb eddingcomputers. Theinteractionb e-
tweenusandcomputersemb eddedinvariousobjectshas
b een develop ed by several research groups[1, 4]. These
devices enable us to interact with emb edded computer
more naturally. However, current standard middleware
comp onentsfornetworkedhomeapplianceshaveadopted
traditionalstandardgraphicaluserinterfacesystemssuch
asJavaAWTorGTK+. Therefore,itisnoteasytocon-
trol home appliances fromadvanced interaction devices
such as PDAs, cellularphones, or a variety of research
prototyp esdescrib ed ab ove. Also, naturalinteractionis
changed according toauser's current situation. For ex-
ample, ifauser is co oking adish. S/he likes to control
appliances via voices, but if s/he is watching TV on a
sofa,aremotecontrollermayb eb etter. Thismeansthat
themostappropriateinteractiondeviceshouldb edynam-
icallychosen accordingto a user's current situationand
preference, andtheselectionofinteractiondevicesshould
b econsistentwhethers/heislivinginanyspaces suchas
at home,inoÆces,orinpublicspaces.
Inthispap er,weprop oseuniversalinteractionfornet-
workedhomeappliances,whichisauserinterfacesystem
to ll the gap describ ed ab ove. Our system allows us
to control various appliances in a uniform way at any
places,andthesystemenables anapplicationtousetra-
ditional standard graphical user interface systems such
as Java AWT or GTK+, but a user can navigate the
interface through avarietyofdevicessuch asPDAs, cel-
lular phones, or advanced technologies. We show that
it isp ossible torealize the goalvery easilybased onthe
statelessthin-clientsystemsuchasCitrixMetaframe,Mi-
crosoftTerminalServer,SunMicrosystemsSunRay,and
AT&T VNC(Virtual Network Computing)system. W e
havebuiltaprototyp esystem,andshownthatausercan
use avarietyof interactiondevices carriedby him. The
prototyp e systemis currently integrated with ourhome
computingsystem[5]thathaveimplementedHAVi(Home
Audio/VideoInterop erability)[3],whichisastandarddis-
tributed middleware sp ecication for home appliances,
and showsthatoursystemisusefulto controlhomeap-
pliances.
2 Design and Implementation
2.1 Universal Interaction
In our approach, we call the proto col that can b e
universally used for the communication b etween in-
put/output interaction devices and appliances universal
interaction. Universalinteractionenablesustocontrola
varietyofhomeappliancesinauniformway. Thismeans
thatourb ehaviorisnotrestrictedaccordingtowherewe
areor which appliancewe liketocontrol. Therefore,our
approachprovidesverynaturalinteractionwithhomeap-
pliances.
The output events pro duced by appliances are con-
verted to universal output interaction events, and the
events are translated for resp ective output interaction
devices. Also, input events generated in input interac-
tiondevices are converted to universalinput interaction
events,andtheeventsarepro cessed byapplicationsexe-
cuted inappliances.
A universal interaction proxy that is called the Uni-
Int proxy describ ed in the next section plays a role to
convert b etween the universal interaction proto col and
input/outputeventsofresp ective interactiondevicesina
genericway. Theproxyallowsustouseanyinput/output
interaction devices to control appliances if theeventsof
thedevicesareconvertedtotheuniversalinteractionpro-
to col. This approach oers thefollowing three very at-
tractive characteristics.
Therstcharacteristicisthatinputinteractiondevices
and outputinteractiondevicesare chosenindep endently
accordingtoauser'ssituationandpreference. Forexam-
ple, auser canselect theirPDAs fortheir input/output
interaction. Also, the user maycho ose his/her cellular
phones as their inputinteraction devices, and television
displaysas his/her output interaction devices. The user
maycontrol appliancesbyhis/her gesture bynavigating
augmentedrealworldgeneratedbywearabledevices.
Thesecondcharacteristicisthatourapproachenables
ustocho osesuitableinput/outputinteractiondevicesac-
cording to a user's preference. Also, these interaction
devices are dynamicallychanged accordingto the user's
current situation. For example, auser who controls an
applianceby his/her cellularphone as aninput interac-
tion device willchange theinteractiondevice to avoice
Proceedings of the 22 nd International Conference on Distributed Computing Systems (ICDCS’02)
1063-6927/02 $17.00 © 2002 IEEE
workcurrently.
The third characteristic is that any applications exe-
cutedinappliancescanusetheanyuserinterfacesystems
iftheuser interface systems sp eak theuniversalinterac-
tion proto col. In ourapproach,we currently adopt key-
b oard/mouseeventsasuniversalinputeventsandbitmap
imagesasuniversaloutputevents. Theapproachenables
ustousetraditionalgraphicaluserinterfaceto olkitssuch
as JavaAWT,GTK+, and Qt for interfacing with any
interaction devices. Infact, most of standard sp ecica-
tionsforconsumerelectronicsliketorecentlyadoptJava
AWT fortheirGUI standards. Thus, our approach will
allow us to control various future consumer electronics
fromv ariousinteractiondevices withoutmo difyingtheir
application programs. The characteristic is very desir-
able b ecause it is very diÆcult to c hange existing GUI
standards.
2.2 System Arc hitecture
Our system uses the thin-client system to transfer
bitmap imagesto drawgraphical user interface, and to
pro cess mouse/keyb oard events for inputs. The usual
thin-clientsystemconsistsofaviewerand aserver. The
server is executed ona machine where anapplicationis
running. The applicationimplementsgraphical user in-
terface by usingatraditional user interfacesystem such
as the Xwindowsystem. Thebitmapimagesgenerated
bytheuser interfacesystemare transmittedto aviewer
that are usually executed on another machine. On the
otherhand,mouseandkeyb oard eventscaptured bythe
viewerareforwardedtotheserver. Theproto colb etween
theviewerandtheserveraresp eciedasastandardpro-
to col. In the pap er, we callthe proto col the universal
interactionprotocol. Thesystemisusuallyusedtomove
auser's desktopaccordingtothelo cationofauser[2],or
showsmultipledesktopsonthesamedisplay,forinstance,
b oth MS-WindowsandtheXWindowsystem.
In our system, we replace the viewer of a thin-client
system to the UniInt(Universal Interaction) proxy that
forwardsbitmapimagesreceivedfromaUniIntserver to
anoutputdevice. Inourapproach,aserver ofanythin-
client systems can b e used as the UniInt server. Also,
UniIntproxyforwards inputeventsreceived froman in-
put interactiondevicetotheUniIntserver.
Oursystemconsistsofthefollowingfourcomp onents.
Inthefollowingparagraphs,weexplainthesecomp onents
indetails.
HomeApplianceApplication
UniIntServer
UniIntProxy
Input/Output Devices
Home appliance applications generate acontrol panel
for currently available appliances to control them. F or
example,ifTViscurrentlyavailable,theapplicationgen-
erates user interface fortheTV.Ontheother hand,the
applicationgeneratesthecomp osedGUIforTVandVCR
ifb othTVandVCRarecurrently available.
byawindowsystemusingthe universalinteraction pro-
to col to a UniInt proxy. Also, it forwards mouse and
keyb oardeventsreceivedfromaUniIntproxytothewin-
dow system. In our current implementation, we need
nottomo difyexisting servers ofthin-clientsystems,and
anyapplicationsrunningon windowsystems supp orting
aUniIntserver canb e controlled inoursystemwithout
mo difyingthem.
TheUniIntproxyisthemostimp ortantcomp onentin
our system. The UniInt proxy converts bitmap images
receivedfromaUniIntserver accordingtothecharacter-
istics ofoutputdevices. Also,itconvertseventsreceived
frominputdevicesto mouseorkeyb oardeventsthatare
complianttotheuniversalinteractionproto col. TheUni-
Intproxycho osesacurrentlyappropriateinputandout-
put interactiondevices forcontrollingappliances. Then,
theselectedinputdevicetransmitsaninputplug-inmo d-
ule, and theselected output devicetransmits anoutput
plug-in mo dule to the UniIntproxy. The input plug-in
mo dulecontainsaco de totranslateeventsreceivedfrom
theinputdevice to mouseorkeyb oard events. Theout-
put plug-in mo dule contains a co de to convert bitmap
imagesreceived fromaUniIntserver to imagesthat can
b e displayedonthescreen ofthetargetoutputdevice.
The last comp onent is input and output interaction
devices. An input device supp orts the interaction with
a user. The role of an input device is to deliver com-
mandsissued by auser to control home appliances. An
outputdevicehasadisplaydevicetoshowgraphicaluser
interface tocontrolappliances.
Inourapproach,theUniIntproxyplaysaroletodeal
withtheheterogeneityofinteractiondevices. Also,itcan
switch interactiondevicesaccordingto auser's situation
or preference. This makes it p ossible to p ersonalize the
interactionb etween auserandappliances.
3 Conclusion
This pap erhasdescrib ed anew user interface system
tollthegapb etweentraditionalgraphicaluserinterface
systems and adv anced input/output interaction devices
for networked homecomputing. We havealso describ ed
theeectivenessofoursystembydemonstratingoursys-
temtocontrolourhomecomputingsystem.
References
[1] W.Ark,D.ChristopherDryer,D.J.Lu,\TheEmotionMouse",In
Pro ceedingoftheHCIInternationalconference,1999.
[2] AndyHarter,AndyHopp er,PeteSteggles,AndyWard,PaulWeb-
ster,\TheAnatomyofaContext-AwareApplication",InPro ceed-
ingsofthe5th Annual ACM/IEEEInternational Conferenceon
MobileComputingandNetworking,1999.
[3] HAVi Consortium,
"HAViSp ecication: Sp ecicationoftheHomeAudio/VideoIn-
terop erability(HAVi)Architecture,http://www.havi.org/
[4] H. Ishii, B.Ullmer, \Tangible Bits: TowardsSeamless Interfaces
b etweenPeople,BitsandAtoms",InPro ceedingsofConferenceon
HumanFactorsinComputingSystems,1997.
[5] T.Nakajima,\SystemSoftwarefor AudioandVisualNetworked
HomeAppliancesonCommo dityOp eratingSystems",InPro ceed-
ingsoftheIFIP/ACMInternationalConferenceonDistributedSys-
temPlatforms-Middleware2001,2001.
[6] MarkWeiser, \The Computer for the 21st Century", Scientic
American,Vol.265,No.3,1991.
