#planet3d

This post looks at research in both invasive and non invasive methods for detecting what is going on inside our brains. It examines developments in speech reconstruction, health monitoring , silk brain implants, and capturing the visions of your imagination.

Speech Reconstruction

Electrodes were implanted surgically into a subject’s brain to simulate and then  understand computer generated speech usually by counting from one to ten. With the help of computational neural networks, simple words were reconstructed that were 75 percent accurate  just through the pattern of neurons firing. This is a very complex procedure, as information had to pass between layers of chips and human brain tissue to infer and reconstruct the sounds.  The authors stated “ Reconstructing speech from the human auditory cortex creates the possibility of a speech neuroprosthetic to establish a direct communication with the brain.”

Various layering between implant, brain tissue and computation to infer speech

What was so cool about this experiment is it is accompanied by a neuroacoustic library of code and recordings  using ielectroencephalography (EEG), electrocorticography (ECoG) and magnetoecnephalography (MEG).

If you click this link, or this one, you can actually hear squeaky weird reconstructed counting of one to nine, or synthesized human speech created by neurons firing in the brain through specific patterns.

Touch

In 2016 in an experiment with direct brain implantation at the University of Pittsburg, scientists put sensors in a mind-controlled robotic arm producing the sensation of touch in a  paralyzed man. This let him experience subtle pressure in his own fingertips when the artificial ones were touched.

University of Pittsburgh researcher Robert Gaunt prepares Nathan Copeland for research testing.TIMOTHY BETLER/UPMC/PITT HEALTH SCIENCES VIA AP

Nathan Copeland had a broken spinal chord, so small chips were implanted in his brain to relay electrical signals that produce movement. Sensors were packed inside the prosthetics, then electrodes were implanted in Copeland’s brain at the site that controls haptic sensation. Electrically stimulating those cells worked even a decade after his accident, meaning the cells were still partially viable. 

New Implant Technology

Instead of clunky electrodes and wires, thin silk, meaning  all the way down to  2.5 micrometers (0.00025cm) has been developed. This means, as people in the fashion world say,  that thinner is indeed better.

 How silk electrodes are implanted onto the brain

Since the brain is basically wet anyway, if saline water is put on top of the implant the silk dissolves, leaving the electrodes exposed and stable, even on top of the curvy part of the brain. The silk breaks down easily, and is reabsorbed back into the body because in essence silk is a fibrous protein. 

Silk dissolving on the brain like a pair of faded, bleached jeans.

But it can take a while to dissolve, so graphene, which dissoles more quickly, is also being tried out.

Graphene with electrodes sandwiched in between computer circuits

OpenWater

The most astonishing breakthroughs are continuing to occur in Mary Lou Jepsen’s OpenWater company, a company I have been tracking for a about a year and wrote about here.

Jepsen’s background is in electrical engineering and optics, and includes stints at Apple, Google, Oculus Rift and Facebook. She surmised that the medical diagnostic imaging world, dominated by fMrIs, CT Scans and PET technologies had not really been updated for decades. Using consumer electronics imagining systems and electronics, she reduce the size of an apparatus to scan the brain to about that of a thick headband. 

With Jepsen’s background in imaging and optics, she understood that the body is translucent to red light, x-rays, and  gamma rays. Out of all of these light sources, red light is the cheapest. but it scatters or diffracts quite a bit, and loses strength and focus. Her solution to this problem is to use holography to recapture the scattering of lightwaves, which can then be mathematically inverted to strengthen the signal. The wavelength of the light that is refocused is 1 micron, which means its possible to record it directly into a hologram. Using technology first developed by Texas Instruments in the 1990s that uses an ultrasonic ping, the light is subtlety changed, and two beams of the same color light make a holographic interference pattern. The hologram is then decoded with blasts of light at one million times a second with double stacked camera chips.

Japanese scientists also trying to figure out what you might be seeing in your brain when you are dreaming.

The brain and body can be scanned out voxel by voxel in different resolutions . Jepsen’s  tests surpassed fMRIs, MRIs, all the way down to a few microns, which means they can focus through skull and brain to a single neuron. This means neuron states can be read and written using light alone, without using an invasive probe.

OpenWaters new camera, ultrasonic and laser chips

Currently the testing is going on with rats, but here is where it gets totally interesting. Jepsen brings the research back to Jack Gallants UC Berkeley Lab that I have written about in great depth over the years. She believes if the size can be brought down and the image and resolution brought up, we can look at images of thought. She also referenced the work done by Japanese dream researchers (see image above) to bring up those images that usually remain locked inside one’s head. She states a five percent false negative reading, so its pretty accurate. She also mentioned focused ultrasound surgery without incision, very Star Trekky.  

Jepsen states telepathy is within reach now, and the technologies to assist that are big data and machine learning. She actually asked “what if we could dump the raw images in our brains out to each other? How do we handle being able to see each other’s thoughts and dreams?” She sees it as the “democratization of healthcare and telepathy”, a very optimistic view.

As new devices are developed to ‘read’ thoughts without invasive means, AI is speeding along developing imitation newscasters. As tech monitoring moves inward towards cognition, simulation and simulacra moves outward towards artifice and puppetry.

Arnav Kapur, wearing AlterEgo selects which channel to watch on TV just by thinking about it. Looks like the History Channel won the challenge.

AlterEgo Reads Your Thoughts

A new device named “AlterEgo” developed by MIT student Arnav Kapur uses subtle neuromuscular movements on the face, known as inner articulatory muscles to interpret thought. AlterEgo  describes itself as a “non-invasive, wearable, peripheral neural interface that allows humans to converse in natural language with machines, artificial intelligence assistants, services, and other people without any voice—without opening their mouth, and without externally observable movements—simply by articulating words internally”

The articulatory muscles are able to be interpreted into thought for words from the speech system. This occurs when one subconsciously forms words one sees when reading text. The device picks up very subtle myoelectrical pulses and it was painstakingly trained in the lab to interpret about 100 words, such as the numbers 1 to 9, and the words add, subtract, reply and call. 

Once the device was able to pick up myoelectrical pulses, Kapur focused on developing a data set to train AlterEgo to recognize signal signatures for various words. It was a laborious process — someone had to sit in a lab wearing the device and silently speak specific words until the computer mastered them.

Going into a store Kapur is able to keep a running tab on his bill with thre prices showing up as neon green numbers on screen- photo courtesy MIT Media Lab

The device cannot read one’s mind, and can only work if the wearer chooses to have it work. The point is to give  feedback to the user “through audio, via bone conduction,  without disrupting the user's usual auditory perception, and making the interface closed-loop. This enables an human-computer interaction that is subjectively experienced as completely internal to the human user—like speaking to one's self.” Its a great idea to use for those who are speech impaired or paralyzed for sure.

AIs Deliver the News In China

This AI Zhang Zhao displays typical traits of humility. even though he is an English language AI, where traits of humility are rare

Using machine learning, NLG (natural language generation a sub-branch of natural language processing — NLP) and GANs (Generative Adversarial Networks), researchers in China have created digital composites created from  footage from  human hosts that read the day’s news using synthesized voices.

Mr. Zhang’s doppleganger does not look too happy about being created just to lower production costs

Then there is Zhang’s competitor, Qui Hao also modeled thorough AI but only  for Chinese language speakers

Qui Hao gets the Star Wars treatment

Hao is slightly more chunky, has no glasses and has more wing tipped ears than Zhao

Qui Hao Alpha (the real one) in front  pleads with his boss (off camera)  to be able to keep his job against his AI competitor (behind) because he (Alpha) has ‘richer emotions”

According to Forbes Magazine, DARPA just laid 18.3 million on a brain start up. The lucky bride is Paradromics who are working on “Massively Parallel Neural Interfaces & Real–Time Decoding” or as they put it, “Broadband For the Brain”.  Their goal is to treat the connectivity disorders of “blindness, deafness, paralysis, ALS, and amputation” and that is really great. Very laudable. Then they drop the big “if” - “So what if patients with connectivity disorders could connect their brains directly to a computer?” 

Well, as the developers say themselves, its a bit of a pickle because computers work in serial processing, and brains in parallel processing.

Their idea is to take CMOS technology with microtechnology to deal with the issue of single channel vs. massively parallel channels.

Paralysis, they believe, is only a communications error., a very novel and systems analysis perspective. They will be offering “65,000, 32 kHz microwire electrodes for massively–parallel neural recordings.”

Yikes! Sounds painful. They immediately backpedal by posting “Notice: This device is not intended nor approved for human use. It is a scientific tool intended for research purposes only.” Well then, you can’t have it both ways. 

A custom CMOS array of low–noise, high–speed amplifiers to which bundles of microwire electrodes are connected. Each microwire consists of a metal–in–glass fiber with an outer diameter of less than 20 µm.

Maybe its because that blue thing wit the Paradromics logo  is what they want to put in your head? If you open it up it looks like this:

So from what I can figure out at this point they made a big bulky processor with some itsy thin wires and they want to stick this on people’s brains who are injured, and then connect them to external device. Basically I get it, its like version 1.0 and won’t really work but if they can miniaturize it enough, then it might work. They seemed to have sourced their main material from “an obscure Moldavan company that had developed a way to stretch hot metal and mass-produce coils of extremely thin insulated wires, just 20 microns thick.” Moldavia - wow, now that’s unique. 

The core of the idea - super thin connecting wire from Moldavia

They say that their “Neural Input–Output Bus, or NIOB, will be capable of reading and stimulating brain activity from 1 million neurons with an effective data rate of > 1 Gbps.”

So the company started with a 30 something Stanford guy who is trained in nano-tech and neuro-science, they are brining in some big gun Stanford doctor surgeon, another big gun sensor guy and they are giving it a whirl. 

Actually, I can’t wait to see where there goes. According to the MIT Tech Review, the DARPA money is throwing a wide net,  “looking at “flexible circuits that can be layered onto the brain, sand-size wireless “neurograins,” and holographic microscopes able to observe thousands of neurons at once. “ They have also awarded money to support what they call the Neural Engineering System Design (NESD) program: Brown University; Columbia University; Fondation Voir et Entendre (The Seeing and Hearing Foundation); John B. Pierce Laboratory; and the University of California, Berkeley. 

Now this is where it gets dicey. Up front it sounds all very normal, but with the new appointees of our government agencies, what, exactly does this mean. According to DARPA’s own website, “The program’s first year will focus on making fundamental breakthroughs in hardware, software, and neuroscience, and testing those advances in animals and cultured cells. Phase II of the program calls for ongoing basic studies, along with progress in miniaturizations and integration, with attention to possible pathways to regulatory approval for human safety testing of newly developed devices. As part of that effort, researchers will cooperate with the U.S. Food and Drug Administration (FDA) to begin exploration of issues such as long-term safety, privacy, information security, compatibility with other devices, and the numerous other aspects regulators consider as they evaluate potential applications of new technologies.

The FDA is in charge of privacy and information security for these areas and devices? Really? Wow, that is going to be dodging all sorts of regulatory bullets in terms of other areas of society. Smart move.

Artist Tiffany Trenda’s idea of the full body suit is way different than Teslasuit’s - This is a photo from her performance “Body Code”, 2012 taken in Times Square, NY - which shows once again, the artworld gets it first.

The Kickstarter photos of Tesla Studios commercial body suit, 2015

The Teslasuit is a 52 point climate controlled haptic feedback suit that is fully upgradable and can be used with VR systems like the Oculus Rift, Playstation and Xbox. Their logo says “feel what you play”. It comes equipped with electrodes that “move your muscles around” according to one user. That is because it uses EMS or electro-muscular stimulation.

User Anthony Brown gets a virtual  but real massage through touch the dots

The technology already exists in physical therapy to assist in pain relief and train muscle groups.

The touchsuit design

Its a wireless suit that incorporates tiny electric mesh sensors

Heat mapping and motion sensors

The suit can capture body movements and send them directly to whatever VR application is being used, or at least in theory. I would like to see a number of demonstrations to make sure there are no glitches. It also has a climate control system that responds to what you are doing inside the VR system, ie., the environment becomes hot if you are in the tropics or cold if you are in the virtual Arctic

The Bluetooth enabled control belt

It has its own operating system and apps, like the virtual meeting app where users can transmit virtual hugs. The developers describe it as “durable, washable and comfortable to wear”.  They describe its uses for “ gaming, virtual dating, health, wellbeing, education and virtual technologies, sport and fitness, science and engineering, psychology and real life training simulations, animation and many more.” You know what is coming next, -the sex industry will be all over this in a nanosecond.

The all-male team of the UK based company has a pretty cheeky sense of their ability to be god-like, but all in good humour. 

The only time a girl shows up in the promo videos is an avatar babe who is on the receiving end of a “virtual hug” from robotic hands. The only game that currently works with the suit is a virtual paintball game, but more should be in the pipeline.

Standard military uses - again. Calling DARPA, calling DARPA, MI6, do you read me?

Then there is artist Tiffany Trenda’s 3D printed costume for her piece “Ubiquitous States” that “synchronises heartbeats through touch and sound to probe deeper complexities of human emotions”. What a totally different approach! Her suit has ECG sensors in the fingertips. When Trenda touches someone’s wrist of neck that persons’s vital signals are picked up. Trenda’s are monitored all the time and both are displayed on a front panel in the costume. Both also wear headphones to hear one another’s heartbeat. When they synchronize the panel display turns from black to white.

The piece is a collaboration between 3D Systems, the largest specialised 3D printing company in the world, award winning designer Janne Kyttenan, and  Trenda.

Trenda at the Context Art Fair in Miami, December 2015

What I really like is the artist’s statement - 

In Veléry's The Conquest of Ubiquity, he describes the current state of our techno-obsessed culture where our tools of precision have changed our behaviors. Today, these tools are primarily our screens (smart phones, tablets, computers, etc.). They give us the ability to download, text, talk, and interact anywhere with the surface of the skin and surface of the computer. Thus, we are living in both the physical and the simulated world simultaneously and we are unable to fully engage in the present. Instead, we are hiding behind our devices and masking our primitive desires of having real time physical interactions. This questions the authenticity of our behaviors using these vernacular technologies. In Ubiquitous States, the performance will unite the heartbeats of both artist and the participant and will measure our emotional connections through the virtual.”