Seeing with the Hands: Sensory Substitution for Manual Interactions

Our team at the University of Chicago engineered a wearable that lets users see with their hands: a miniature camera on the palmar side captures what the hand faces, and an electrotactile display renders it as tactile images felt on the skin. This sensory substitution is developed for Blind and Low Vision users. Hands move fast, reach from any angle, and preshape to grasp, so this new perspective opens a way to support manual tasks in everyday life without vision.

The perspective. A fresh perspective for sensory substitution: putting the camera on the palmar side and the tactile image on the back of the hand.
The ergonomics. In our study the hands’ perspective suited ergonomic reaching, and relocating the array to the hand prioritises dexterity.
Not a replacement. It combines with the traditional eyes’ perspective; given the choice, all participants used both.

A new tactile perspective designed for grasping objects

Instead of substituting for the eyes, we put cameras on the palmar side of the hand, aimed at the objects you reach for.

A person using their eyes to perceive their surroundings, with a blue light indicating the direction of their gaze. The same person also uses their hands to see. They are reaching into a cabinet with the device that has a tactile feedback system. An inset shows a tactile image on the back of the hand, represented by a grid of red and green dots. — Typical sensory substitution interfaces are thought to substitute the eyes for assisting with perceiving one’s surroundings (e.g., navigation). We explore a new tactile perspective with **cameras on the palmar side**: the side of the hand facing toward objects to grasp for **hand manipulation**.

A blind person is walking downstairs outdoors, using a white cane. They use the hand device to look for handrails to hold on. — Our study participants envision using our device for various manual tasks, such as **finding the handrail while walking down the stairs**, identifying and grabbing ingredients inside a shelf while cooking, retrieving an object that fell on the floor, etc.

Implementation

A palm-facing camera and a thin electrotactile grid on the back of the hand turn what the hand sees into patterns you can feel.

A photo of a hand wearing the device, with a black strap on the wrist and golden strips on the back of the hand. A callout pointing to the palmar side of the wrist strap is labeled camera, with a zoom-in picture of the camera. A callout pointing to the strips is labeled electrotactile display. — The image under the hand captured by the camera is segmented and translated into tactile patterns, using computer vision techniques, in real-time. The tactile patterns are felt on the back of the hand through a custom electrotactile display (5×6) that is flexible and thin (0.1 mm).

A diagram of a cross section of the skin, titled a tactile pixel on the electrotactile display. Two electrodes are shown labeled with plus and minus correspondingly. An arrow pointing from plus electrode goes to the minus electrode. The arrow is labeled with a lightning icon and the word pulse current. The arrow intersects with a drawing of a tactile receptor, with more receptors drawn nearby. — Each “tactile pixel” on the electrotactile display consists of a pair of electrodes. Whenever the pixel is supposed to be felt, our system passes tiny current pulses (3-5 mA) between the electrodes, which stimulate the tactile receptors under the skin, causing a sense of slight touch.

User study

Blind and low-vision participants used the device for grasping and everyday manual tasks, often after only about 15 minutes of practice.

A collage of study photos in which each shows a different person while blindfolded, reaching their hand for different tasks in front of a table with different objects. — Our user studies including Blind & Low Vision participants focus on daily tasks (e.g., grasping objects). Without extensive training (~15 min), most participants completed not having to touch unintended objects. Compared to using the eyes' perspective, participants found the hands' provides a more ergonomic way of seeing.

A photo showing two people are shaking their hands. Blue rays extend from the head and from the hand. Call-outs from both consist of close up photos overlaid with a grid of red and white dots. — We further let participants **choose** which tactile perspective to use for a handshaking task. All chose to use both devices, and most stated that the **eyes' provides an overview**, and the **hands' provides details**. We were really impressed by how they were able to fluently utilize these new tactile senses!

Team

This is research from the Human-Computer Integration Lab at the University of Chicago by:

Publication

This work is published at ACM CHI 2025 (see paper on ACM DL). Authors’ print is available here (PDF).

Shan-Yuan Teng*, Gene S-H Kim*, Xuanyou Liu*, and Pedro Lopes (*equal contribution). 2025. Seeing with the Hands: A Sensory Substitution That Supports Manual Interactions. In CHI Conference on Human Factors in Computing Systems (CHI ’25), April 26–May 01, 2025, Yokohama, Japan. ACM, New York, NY, USA, 14 pages. https://doi.org/10.1145/3706598.3713419

@inproceedings{teng2025seeing,
  title={Seeing with the Hands: A Sensory Substitution That Supports Manual Interactions},
  author={Teng, Shan-Yuan and Kim, Gene S-H and Liu, Xuanyou and Lopes, Pedro},
  booktitle={Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems},
  year={2025},
  doi={10.1145/3706598.3713419}
}

Resources

Press kit: High resolution photos and unannotated video are available here.

Source code: Available on GitHub.

Funding: This research is supported by Google Award for Inclusion Research.

Overview