research-article
Authors: Hiroyuki Ishihara and Shiro Kumano
ICMI '20 Companion: Companion Publication of the 2020 International Conference on Multimodal Interaction
October 2020
Pages 433 - 441
Published: 27 December 2020 Publication History
- 0citation
- 56
- Downloads
Metrics
Total Citations0Total Downloads56Last 12 Months12
Last 6 weeks2
New Citation Alert added!
This alert has been successfully added and will be sent to:
You will be notified whenever a record that you have chosen has been cited.
To manage your alert preferences, click on the button below.
Manage my Alerts
New Citation Alert!
Please log in to your account
Get Access
- Get Access
- References
- Media
- Tables
- Share
Abstract
To analyze human interaction behavior in a group or crowd, identification and device time synchronization are essential but time demanding to be performed manually. To automate the two processes jointly without any calibration steps nor auxiliary sensor, this paper presents an acceleration-correlation-based method for multi-person interaction scenarios where each target person wears an accelerometer and a camera is stationed in the scene. A critical issue is how to remove the time-varying gravity direction component from wearable device acceleration, which degrades the correlation of body acceleration between the device and video, yet is hard to estimate accurately. Our basic idea is to estimate the gravity direction component in the camera coordinate system, which can be obtained analytically, and to add it to the vision-based data to compensate the degraded correlation. We got high accuracy results for 4 person-device matching with only 40 to 60 frames (4 to 6 seconds). The average timing offset estimation is about 5 frames (0.5 seconds). Experimental results suggest it is useful for analyzing individual trajectories and group dynamics at low frequencies.
References
[1]
Shervin Ardeshir and Ali Borji. 2018. Integrating egocentric videos in top-view surveillance videos: Joint identification and temporal alignment. In Proceedings of the European Conference on Computer Vision (ECCV). 285--300.
Digital Library
[2]
Chiraz BenAbdelkader, Ross Cutler, Harsh Nanda, and Larry Davis. 2001. Eigengait: Motion-based recognition of people using image self-similarity. In International conference on audio-and video-based biometric person authentication. Springer, 284--294.
Digital Library
[3]
Frederik Brudy, Christian Holz, Roman R"adle, Chi-Jui Wu, Steven Houben, Clemens Nylandsted Klokmose, and Nicolai Marquardt. 2019. Cross-Device Taxonomy: Survey, Opportunities and Challenges of Interactions Spanning Across Multiple Devices. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (CHI). ACM.
Digital Library
[4]
Rainer E Burkard, Mauro Dell'Amico, and Silvano Martello. 2009. Assignment problems. Springer.
Digital Library
[5]
Laura Cabrera-Quiros, Andrew Demetriou, Ekin Gedik, Leander van der Meij, and Hayley Hung. 2018. The MatchNMingle dataset: a novel multi-sensor resource for the analysis of social interactions and group dynamics in-the-wild during free-standing conversations and speed dates. IEEE Transactions on Affective Computing (2018).
[6]
Laura Cabrera-Quiros and Hayley Hung. 2016. Who is where?: Matching people in video to wearable acceleration during crowded mingling events. In Proceedings of the 24th ACM international conference on Multimedia (ACMMM). ACM, 267--271.
Digital Library
[7]
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 7291--7299.
[8]
Sarah D'Angelo and Andrew Begel. 2017. Improving communication between pair programmers using shared gaze awareness. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (CHI). ACM, 6245--6290.
Digital Library
[9]
Chenyou Fan, Jangwon Lee, Mingze Xu, Krishna Kumar Singh, Yong Jae Lee, David J Crandall, and Michael S Ryoo. 2017. Identifying first-person camera wearers in third-person videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5125--5133.
[10]
Andreas Fender, Philipp Herholz, Marc Alexa, and Jörg Müller. 2018. OptiSpace: Automated Placement of Interactive 3D Projection Mapping Content. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (CHI). ACM.
Digital Library
[11]
Martin A Fischler and Robert C Bolles. 1981. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, Vol. 24, 6 (1981), 381--395.
Digital Library
[12]
Shaogang Gong, Marco Cristani, Chen Change Loy, and Timothy M Hospedales. 2014. The re-identification challenge. In Person re-identification. Springer, 1--20.
[13]
Kensei Jo, Mohit Gupta, and Shree K Nayar. 2016. DisCo: Display-camera communication using rolling shutter sensors. ACM Transactions on Graphics (TOG), Vol. 35, 5 (2016), 150.
Digital Library
[14]
Brett R Jones, Hrvoje Benko, Eyal Ofek, and Andrew D Wilson. 2013. IllumiRoom: peripheral projected illusions for interactive experiences. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (CHI). ACM, 869--878.
Digital Library
[15]
Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. 2015. Panoptic studio: A massively multiview system for social motion capture. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 3334--3342.
Digital Library
[16]
Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. 2016. Deeper depth prediction with fully convolutional residual networks. In Proceedings of the Fourth international conference on 3D vision (3DV). IEEE, 239--248.
[17]
Ville M"akel"a, Mohamed Khamis, Lukas Mecke, Jobin James, Markku Turunen, and Florian Alt. 2018. Pocket transfers: Interaction techniques for transferring content from situated displays to mobile devices. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (CHI). ACM.
Digital Library
[18]
Alessandro Masullo, Tilo Burghardt, Dima Damen, Toby Perrett, and Majid Mirmehdi. 2019. Who Goes There? Exploiting Silhouettes and Wearable Signals for Subject Identification in Multi-Person Environments. In Proceedings of the IEEE International Conference on Computer Vision Workshops.
[19]
Alaeddine Mihoub and Grégoire Lefebvre. 2017. Social Intelligence Modeling Using Wearable Devices. In Proceedings of the ACM International Conference on Intelligent User Interfaces (IUI). ACM.
Digital Library
[20]
Anton Milan, Stefan Roth, and Konrad Schindler. 2013. Continuous energy minimization for multitarget tracking. IEEE transactions on pattern analysis and machine intelligence, Vol. 36, 1 (2013), 58--72.
Digital Library
[21]
Daniel Olguin Olguin, Peter A Gloor, and Alex Pentland. 2009. Wearable sensors for pervasive healthcare management. In 2009 3rd International Conference on Pervasive Computing Technologies for Healthcare. IEEE, 1--4.
[22]
Thomas Plotz, Chen Chen, Nils Y Hammerla, and Gregory D Abowd. 2012. Automatic Synchronization of Wearable Sensors and Video-Cameras for Ground Truth Annotation--A Practical Approach. In Proceedings of the 16th International Symposium on Wearable Computers (ISWC). IEEE, 100--103.
Digital Library
[23]
Mahsan Rofouei, Andrew Wilson, AJ Brush, and Stewart Tansley. 2012. Your phone or mine?: fusing body, touch and device sensing for multi-user device-display interaction. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (CHI). ACM, 1915--1918.
Digital Library
[24]
Osamu Shigeta, Shingo Kagami, and Koichi Hashimoto. 2008. Identifying a moving object with an accelerometer in a camera view. (2008), 3872--3877.
[25]
Frank Spitzer. 2013. Principles of random walk. Vol. 34. Springer Science & Business Media.
[26]
Thiago Teixeira, Deokwoo Jung, and Andreas Savvides. 2010. Tasking networked cctv cameras and mobile phones to identify and localize multiple people. In Proceedings of the 12th ACM international conference on Ubiquitous computing. ACM, 213--222.
Digital Library
[27]
Alessandro Vinciarelli, Maja Pantic, and Hervé Bourlard. 2009. Social signal processing: Survey of an emerging domain. Image and vision computing, Vol. 27, 12 (2009), 1743--1759.
Digital Library
[28]
Andrew D Wilson and Hrvoje Benko. 2014. Crossmotion: fusing device and image motion for user identification, tracking and device association. In Proceedings of the ACM International Conference on Multimodal Interaction (ICMI). ACM, 216--223.
Digital Library
[29]
Ryo Yonetani, Kris M Kitani, and Yoichi Sato. 2015. Ego-surfing first-person videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5445--5454.
[30]
Liang Zheng, Yi Yang, and Alexander G Hauptmann. 2016. Person re-identification: Past, present and future. arXiv preprint arXiv:1610.02984 (2016).
Index Terms
Gravity-Direction-Aware Joint Inter-Device Matching and Temporal Alignment between Camera and Wearable Sensors
Computing methodologies
Artificial intelligence
Computer vision
Computer vision problems
Matching
Object identification
Tracking
Computer vision tasks
General and reference
Cross-computing tools and techniques
Estimation
Measurement
Document types
General conference proceedings
Hardware
Communication hardware, interfaces and storage
Sensors and actuators
Human-centered computing
Ubiquitous and mobile computing
Ubiquitous and mobile computing theory, concepts and paradigms
Ubiquitous and mobile devices
Index terms have been assigned to the content through auto-classification.
Recommendations
- Camera–projector matching using unstructured video
This paper presents a novel approach for matching 2-D points between a video projector and a digital camera. Our method is motivated by camera–projector applications for which the projected image needs to be warped to prevent geometric distortion. Since ...
Read More
- Joint Depth and Color Camera Calibration with Distortion Correction
We present an algorithm that simultaneously calibrates two color cameras, a depth camera, and the relative pose between them. The method is designed to have three key features: accurate, practical, and applicable to a wide range of sensors. The method ...
Read More
- Camera-to-camera mapping for hybrid pan-tilt-zoom sensors calibration
SCIA'07: Proceedings of the 15th Scandinavian conference on Image analysis
Video surveillance becomes more and more extended in industry and often involves automatic calibration system to remain efficient. In this paper, a video-surveillance system that uses stationary-dynamic cameras devices is presented. The static camera is ...
Read More
Comments
Information & Contributors
Information
Published In
ICMI '20 Companion: Companion Publication of the 2020 International Conference on Multimodal Interaction
October 2020
548 pages
ISBN:9781450380027
DOI:10.1145/3395035
- General Chairs:
- Khiet Truong
University of Twente, the Netherlands
, - Dirk Heylen
University of Twente, the Netherlands
, - Mary Czerwinski
Microsoft Research, USA
, - Program Chairs:
- Nadia Berthouze
University College London, United Kingdom
, - Mohamed Chetouani
Sorbonne University, France
, - Mikio Nakano
C4A Research Institute, Japan
Copyright © 2020 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [emailprotected]
Sponsors
- SIGCHI: ACM Special Interest Group on Computer-Human Interaction
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 27 December 2020
Permissions
Request permissions for this article.
Check for updates
Author Tags
- accelerator
- gravity direction
- person-device matching
- time synchronization
- video
Qualifiers
- Research-article
Conference
ICMI '20
Sponsor:
- SIGCHI
ICMI '20: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION
October 25 - 29, 2020
Virtual Event, Netherlands
Acceptance Rates
Overall Acceptance Rate 453 of 1,080 submissions, 42%
Contributors
Other Metrics
View Article Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
Total Citations
56
Total Downloads
- Downloads (Last 12 months)12
- Downloads (Last 6 weeks)2
Other Metrics
View Author Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in
Full Access
Get this Publication
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderMedia
Figures
Other
Tables
