The 24th International Conference on Pervasive Computing and Communications (PerCom 2026), one of the top conferences in ubiquitous computing, will be held in Pisa, Italy, from March 16 to 20, 2026.

We have presented the following work in the demo session:

Shape-N-Motion: Fine-Grained Hand Object Manipulation Recognition with Ultrasonic and IMU

Abstract:

Finger-based object manipulations, such as smartphone operations, are important indicators reflecting cognitive and motor functions, and recognition using wearable sensors has been increasingly demanded, particularly in the healthcare field.
However, conventional sensors widely used for this purpose, such as IMU sensors and passive microphones, have difficulty directly capturing local cues near the fingers.

Ultrasonic sensing is a promising approach because, through active sensing, it can reflect geometric changes and contact states near the fingertips and thereby directly capture such local information [1]. On the other hand, during object manipulation, the reflected signals become more complex, making it difficult to distinguish cues derived from subtle differences in finger movements.

In this study, we propose a recognition method that does not rely on explicit hand-shape reconstruction by using the reflection patterns generated by contact with objects themselves as discriminative features. Furthermore, by combining this method with an IMU sensor, we also incorporate arm- and wrist-trajectory information, thereby enabling recognition of diverse finger-based object manipulations.

On a 45-class benchmark focused on fine-grained finger-based object manipulations, we compared the proposed method with existing methods, including IMU + passive microphone fusion methods and ultrasonic-only methods. The results showed that the proposed method achieved better performance than the existing methods. These findings demonstrate an effective framework for recognizing fine-grained finger-based object manipulations.