It is now possible to use computer software to recreate pictures that the eye is seeing using only signals from the visual cortex. Researchers from ATR Computational Neuroscience Laboratories in Japan have described using functional magnetic resonance imaging (fMRI) signals from the visual cortex to reconstruct letters and symbols seen by a subject. They asked subjects to look at 400 random black and white images on a 10 x 10 grid and processed the fMRI signals to map blood flow changes in the brain associated with individual pixels. They were then able to use these maps to reconstruct new images that the subjects looked at. In one test, the researchers were able to reconstruct the letters N-E-U-R-O-N presented to the subjects. This is the first example where such images have been created without using a previously defined set of ‘training’ images.
Although the present study focussed only on the reconstruction of relatively simple black and white patterns, the researchers believe the approach could be extended to reconstruct visual images based on colour, motion or texture. The technique could eventually be used to improve neural prosthetics or to enhance brain-machine interfaces. Other applications could include the reconstruction of mental images that a subject is thinking about, but not actually looking at, such as illusions and dreams which are believed to take place in the early visual cortex.
The study is published in the journal Neuron.