ALL RIGHT.
WELCOME, EVERYONE.
WE'LL GO AHEAD AND GET STARTED.
SO, WELCOME TO THE NIH
DIRECTOR'S LECTURE.
THIS IS A VERY SPECIAL LECTURE
AND IT FEATURES LEADING
RESEARCHERS FROM AROUND THE
GLOBE.
NOMINATIONS ARE TAKEN FROM
ACROSS NIH AND TYPICALLY ONLY
ABOUT THREE DIRECTOR'S LECTURERS
ARE CHOSEN EACH YEAR AND IT'S
OUR HONOR TO WELCOME EDDIE CHANG
TODAY TO SPEND THIS TIME WITH
US.
I'M PERSONALLY VERY GRATEFUL.
EDDIE IS SO WELL KNOWN TO US I
WAS THINKING IN NEUROSCIENCE
CIRCLES AT NIH YOU JUST SAY
EDDIE AND EVERYONE KNOWS WHO YOU
ARE SO IT'S LIKE MADONNA TO US.
BEFORE I INTRODUCE, DR. CHANG, A
FEW HOUSEKEEPING ITEMS.
IF YOU'RE ATTENDING THIS LECTURE
FOR CME CREDIT THE CODE IS
57680.
IF YOU'RE WATCHING ONLINE AND
WOULD LIKE TO SUBMIT QUESTIONS,
PLEASE CLICK THE LINK FOR SEND
LIVE FEEDBACK AND YOUR QUESTION
WILL BE RELAYED TO US AND WE'LL
PRESENT THAT TO THE SPEAKER AT
THE END OF THE LECTURE.
AND YOU CAN SUBMIT QUESTIONS AT
ANY TIME DURING THE LECTURE.
SO, DR. CHANG IS THE CHAIR AND
JEANIE ROBERTSON DISTINGUISHED
PROFESSOR OF NEUROLOGICAL
SURGERY AT THE UNIVERSITY OF
CALIFORNIA IN SAN FRANCISCO.
HIS CLINICAL EXPERTISE IS IN
SURGICAL THERAPIES FOR EPILEPSY,
PAIN AND BRAIN TUMORS.
HE SPECIALIZES IN ADVANCED
NEUROPHYSIOLOGIC BRAIN MAPPING
METHODS AND AWAKE SPEECH AND
MOTOR MAPPING TO SAFELY PERFORM
PROCEDURES IN THE ELOQUENT OF
THE BRAINS.
HE'S THE CONSUMMATE SURGEON
SCIENTIST.
HE'S A PRACTICING SURGEON.
HE CHAIRS THE DEPARTMENT AND HE
DOES ALL OF THIS INCREDIBLE
RESEARCH THAT YOU'LL HEAR ABOUT
SHORTLY.
HIS LAB HAS DEMONSTRATED THE
DETAILED FUNCTIONAL ORGANIZATION
OF THE HUMAN SPEECH CORTEX AND
HAS TRANSLATED THESE DISCOVERIES
TOWARDS THE DEVELOPMENT MUCH A
SPEECH PROSTHETIC DEVICE TO
RESTORE COMMUNICATION FOR PEOPLE
LIVING WITH PARALYSIS AND SPEECH
DISORDERS.
HE DIRECTS OR CO-DIRECTS THE
CENTER FOR NEURAL ENGINEERING
AND PROSTHESIS AND THAT'S A
COLLABORATIVE ENTERPRISE OF UCSF
AND BERKELEY AND BRINGS TOGETHER
EXPERTS TO DEVELOP STATE OF THE
ART BIO MEDICAL TECHNOLOGIES FOR
SPEECH DISORDERS.
HE'LL DESCRIBE CRITICAL ADVANCES
IN THE PAST DECADE MADE BY HIS
LAB IN DECIPHERING THE CORTICAL
SIGNALS THAT UNDER LIE OUR
ABILITY TO SPEAK AND UNDERSTAND
WORDS.
HE'S THE RECIPIENT OF MANY
AWARDS AND WAS ELECTED TO THE
NATIONAL ACADEMY OF MEDICINE IN
2020.
I AM SO EXCITED TO WELCOME
DR. CHANG TODAY.
WE LOOK FORWARD TO YOUR TALK.
>> THANK YOU FOR THE
INTRODUCTION AND TO THANK
DR. SHOER FOR THE HONOR TO BE
HERE.
IT'S A PRIVILEGE TO BE AT THE
NIH.
I MEAN THAT FROM THE DEEPEST
PART OF MY HEART AND WANT TO
THANK YOU FOR THE WORK YOU DO IN
PARTICULAR FOR THE COLLABORATION
WE'VE HAD OVER THE LAST TWO
DECADES.
WE'RE JUST GOING TO SWITCH THE
SLIDES HERE.
THE FOCUS OF THE LECTURE IS PART
OF WHAT MY LAB STUDIES.
WE STUDY HOW THE BRAIN PROCESSES
WHAT WE HEAR AND HOW WE CREATE
SPEECH.
I'M FOCUS ON THE LATTER.
THE PROCESS OF THE CORTICAL
CONTROL OF SPEAKING.
HOW WE USE THAT KNOWLEDGE TO
BUILD A TRANSLATIONAL DEVICE FOR
PEOPLE WHO HAVE LOST THE ABILITY
TO COMMUNICATE.
ONE PERSPECTIVE THAT IS REALLY
IMPORTANT TO UNDERSTANDING HOW
WE SPEAK IS THE FUNDAMENTAL
PROCESS OF SPEAKING IS ABOUT HOW
WE SHAPE THE BREATH.
SPEECH STARTS WITH THE AIR FROM
THE LUNGS.
WHEN IT GETS TO THE LEVEL OF
YOUR VOCAL FOLDS IT STARTS
VIBRATIONS THERE AND THAT IS THE
VOICE ENERGY OF OUR WORDS.
THAT VOICE ENERGY THEN GOES UP
THROUGH THE UPPER PART OF THE
VOCAL TRAP, THE LIPS, THE JAW,
THE TONGUE WHERE THE MOVEMENTS
CREATE A FILTER ON THAT SOUND
AND THAT WHAT GIVES RISE TO ALL
THE CONSONANTS AND VOWELS THAT
WE SPEAK.
SO, A LOT OF WHAT I'M GOING TO
BE FOCUSSING ON IS THE CONTROL
OF THE VOCAL TRACK.
HOW DOES THE BRAIN ALLOW US TO
CONTROL THAT ABILITY TO SHAPE
THE BREATH AND IT'S REALLY A
REMARKABLE PROCESS ONE UNIQUE TO
OUR SPECIES TO CREATE WORDS AND
SOMETHING QUITE SPECIALIZED.
WHEN I STARTED THIS WORK, IT WAS
ABOUT 15 YEARS AGO STARTED IN MY
LAB.
IT BECAME IMMEDIATELY OBVIOUS TO
ME WE DID NOT UNDERSTAND VERY
MUCH ABOUT THE ORGANIZATION OR
THE BASIC PRINCIPLES BY WHICH
THE HUMAN BRAIN PROCESSES WORDS.
SO IN ADDITION TO STUDYING HOW
WE HEAR, WE DECIDED TO DO
STUDIES THAT WERE GOING TO BE
DIRECTLY RELATED TO HOW WE
PRODUCE AND ARTICULATE.
AND 99% OF MOTOR NEUROSCIENCE IS
REALLY FOCUSSED ON THE ARMS AND
LIMBS, THE ARM AND HAND.
THE VAST MAJORITY.
BUT HOW MUCH WE ACTUALLY STUDY
THE CENTRAL CONTROL OF SPEAKING
IS VERY UNDER STUDIED.
SO THIS IMAGE IS ACTUALLY AN
ILLUSTRATED FROM HARVEY CUSHING
CONSIDERED THE FATHER OF
NEUROSURGERY.
A STUDENT GAVE ME THIS
ILLUSTRATION WHICH HANGS IN THE
WALL IN MY LAB.
HE WAS DESCRIBING A PATIENT THAT
WAS SHOT IN THE HEAD WITH A
BULLET ENTRY IN THE FOREHEAD AND
DEVELOPED SEIZURES FROM PART OF
THE TEMPORAL LOBE.
THE REASON I'M SHOWING THIS IS
NOT RELATED TO HIS INJURY BUT
IT'S ONE OF THE FIRST
DESCRIPTIONS OF THE BODY
ORGANIZATION OR SOMATIC TOPIC OF
THE HUE PART OF THE BRAIN THAT
CORRESPONDS TO MOVEMENT OF THE
FACE, CORNER OF THE LIPS, THE
JAW, TONGUE, PALLET AND BACK OF
OUR THROAT AND VOCAL FOLDS.
SO BASICALLY THERE'S A MAP IN
ALL OF OUR BRAINS THAT
RECAPITULATES THE SPATIAL
ORGANIZATION FROM FRONT TO BACK
OF THE VOCAL TRACK.
THAT WAS THE STARTING POINT.
THIS WAS IN THE CONTEXT OF
MOVING THE AREAS AND I WAS
INTERESTED IN HOW TO RELATE THE
PHENOMENON IN UNDERSTANDING
SPEECH.
SO THE MAIN TECHNIQUE THAT WE
USE WAS A TECHNIQUE AND WE STILL
USE THIS CALLED ELECTROCO
ELECTROCORTIOGRAPHY AND WE
INCREASED THE DENSITY OF THE
RAYS ON THE SURFACE OF THE
BRAINS ALLOWING US TO ASK DEEPER
QUESTIONS ABOUT THE NEURAL
CODING WITHOUT INTRODUCING
ADDITIONAL RISK.
MORE ELECTRODES BUT ON THE
SURFACE MEANT MORE DATA AND THIS
IS A 16X16, ARRAY.
STILL IS THE MOST HIGHLY
RESOLVED METHOD IN NEUROSCIENCE
IN SPATIAL AND SIMULTANEOUSLY
TEMPORAL RESOLUTION TO
UNDERSTANDING WHAT IS HAPPENING
IN THE CORTEX AND YOU'LL SEE THE
PRE AND POST CENTRAL GYRUS WHEN
SOMEONE IS SPEAKING.
>> SHIP BUILDING IS A
FASCINATING PROCESS.
>> SO THAT PATTERN OF ACTIVITY
IS WHAT WE REFER TO IS THE
NEURAL CODE.
AND WHAT WE HAVE DEEPLY, DEEPLY
FOCUSSED ON FOR THE LAST 15
YEARS IS TRYING TO UNDERSTAND
HOW THAT NEURAL CODE WORKS.
HOW DOES ELECTRICAL ACTIVITY
REFLECTED IN WHAT YOU JUST SAW
IN THAT PARTICULAR PART OF THE
BRAIN.
HOW DOES THAT CORRESPOND TO
DIFFERENT ASPECTS OF SPEECH
CONTROL.
I'M GOING TO TALK A LITTLE BIT
ABOUT HOW WE DECIPHERED THAT
CODE, WHAT ARE THE THINGS WE
LEARNED FROM IT AND THEN HOW WE
LEVERAGE THAT CODE FOR SOMETHING
TRANSLATIONAL.
WE HAD PARTICIPANTS READING A
LIST OF SYLLABLES, KIND OF
BORING, RIGHT?
VERY SIMPLE.
IF YOU LOOK AT THE SYLLABLES
WHEN YOU MAKE THE BA SOUND YOUR
LIPS COME TO CLOSURE THAT CREATE
THE SOUND THE TOP TIP FROM THE
UPPER TOOTH THAT CREATES THE BA
SOUND AND THE GA SOUND IS THE
BACK OF THE TONGUE GOING UP.
THAT'S QUITE INCREDIBLE.
WHEN I LEARNED ALL THIS I WAS
ASTOUNDED.
I HAD NO IDEA HOW WE WERE SPEAK
AND THIS IS SOMETHING WE ALL
LEARN WITHOUT EXPLICIT KNOWLEDGE
WHICH IS QUITE INCREDIBLE.
NONETHELESS, THIS IS HOW WE
PRODUCE THE SOUNDS AND IF YOU
LOOK AT THE ELECTRODES SORTED
FROM TOP TO BOTTOM HERE, THESE
ARE THE SAME SET OF ELECTRODES
ACROSS THE THREE UTTERANCES.
YOU CAN SEE THEY ARE HAVE
OVERLAPPING AND DISTINCT
PATTERNS OF ACTIVATION.
FOR EXAMPLE, IN THE BA SOUND,
THE B SOUND YOU SEE THE
ELECTRODES ACTIVATED HERE BUT
NOT SO MUCH IN THE "D" AND GA
SOUND WHERE IN THIS SOUND THE
SOUND IS ACTIVATED BUT NOT
OTHERS AND THIS GA SOUND HAS AN
ACTIVATION DOWN HERE.
AND THEN THERE'S SOME THAT ARE
OVERLAPPING ACROSS ALL OF THESE.
SO IT'S NOT AS SIMPLE AS THIS
ONE AREA IS ONE SYLLABLE,
ETCETERA.
THERE'S A COMPLEX REPRESENTATION
THAT HAS DISTRIBUTED
INFORMATION.
SO ONE WAY THAT WE TOOK TO TRY
TO UNDERSTAND THIS WAS TO TAKE
ALL THE DIFFERENT SOUNDS, FOR
EXAMPLE, ALL THE LIP SOUNDS LIKE
THE BA, FA, MA, PA, THEY'RE WHAT
WE CALL LABIALS.
WE'RE ABLE TO CORRELATE THOSE
WITH THE ELECTRODES ACROSS
PATIENTS THAT ARE CORRELATED
WITH THAT MOVEMENT.
SO, WHAT THIS IS IS BASICALLY A
SPATIAL PLOT FROM TOP TO BOTTOM
AND THESE ARE ASSOCIATED WITH
THE LABIAL MOVEMENTS WHILE
PEOPLE ARE SPEAKING.
THEN IF YOU LOOK AT A DIFFERENT
SET OF ARTICULATORS, ARTICULATED
SOUND FOR THE LARYNX WE FOUND
ONE AT THE BOTTOM WHICH I SHOWED
YOU EARLIER IN THE CUSHING
ILLUSTRATION.
THERE'S THIS VOCAL FOLD AT THE
HOMUNCULUS AND THEN FOUND AN
ADDITIONAL LARYNGEAL AREA THAT
WAS HARD TO UNDERSTAND.
WE DID MULTIPLE STUDIES TO TRY
TO MAKE SURE THIS IS REAL.
AND I'LL DESCRIBE ONE OF THEM.
BUT WHEN YOU LOOK ACROSS
MULTIPLE PATIENTS WITH HE SEE
THE SEE THE LIP AREA, JAW,
TONGUE, LARYNX AND IT'S
ACTIVATED WHEN WE SPEAK.
THIS IS THE FIRST FUNCTIONAL
ORGANIZATION OF THE SPEECH
MORTAR CORTEX AND RECAPITULATED
THINGS WE THROUGH BEFORE AND
THERE WAS A NEW THING, AN
ADDITIONAL AREA OF THE LARYNX.
ONE OF MY STUDENTS WAS VERY
INTERESTED IN THE FINDING AND
DID HIS WHOLE Ph.D. IN TRYING TO
UNDERSTAND HOW THE VOCAL FOLD
CONTROLLED BY THE BRAINS.
THE VOCAL FOLDS IN ADDITION TO
LIKE PRODUCING THE ENERGY OF THE
VOICE, IT'S WHAT MODULATES THE
PITCH OF MY VOICE.
SO WHEN I STRESS A PARTICULAR
WORD, THAT'S PRIMARILY BECAUSE
I'M CHANGING THE PITCH OF THAT
AND THAT'S BY TENSING THE VOCAL
FOLDS.
SO WHAT BEN DISCOVERED IN A
PAPER IN CELL IN 2018 IS THIS
AREA, IN PARTICULAR THIS
PREVIOUSLY UNKNOWN LARYNGEAL
MOTOR CORTEX IS DIRECTLY
RESPONSIBLE FOR THE VOCAL FOLDS.
HE DID AN EXPERIMENT WHERE
PEOPLE BASICALLY SAID THE SAME
SENTENCE OVER MULTIPLE TRIALS
BUT ON A GIVEN TRIAL THEY STRESS
ONE WORD AND WE DID THESE KIND
OF RANDOMLY.
IN ONE INSTANCE I NEVER SAID SHE
STOLE MY MONEY, I NEVER SAID SHE
STOLE MY MONEY AND CHANGE THIS
AND WE USE THIS IN ENGLISH AND
YOU CAN CHANGE DIRECTLY THE WORD
MEANING.
WHAT BEN FOUND WAS FINDING THIS
CLUSTER OF ELECTRODES ON THE
RIGHT AND LEFT HEMISPHERE
CORRELATED TO THE PITCH OF THE
VOICE.
STRONGLY CORRELATED AND TIGHTLY
CLUSTERED AND CONFIRMED WHAT WE
FOUND FROM OUR EARLIER STUDIES
USING SYLLABLES.
IF YOU LOOK AT ALL THE TRIALS
AND LOOKED AND THE PITCH TRA TRG
OF THE UTTERANCES AND LOOKING AT
THE HIGH FREQUENCY COMPONENT, WE
CALL HIGH GAMMA, THERE'S SINGLE
TRIAL TIME LINE RESPONSE TO WHAT
WORD WHAT -- WAS STRESSED.
THE VOCAL FOLDS ARE TENSING AND
WE VERIFIED.
BEN ALSO SHOWED WHEN PEOPLE ARE
SENDING MALADIES THE DOE RAY ME
KIND OF THING IT'S NOT SPECIFIC
TO SPEECH IT'S SOMETHING MORE
RUDIMENTARY LIKE THE VOCAL FOLD
TENSION.
SOMETHING UNEXPECTED BUT QUITE
INTERESTING WAS THAT WHEN WE
PLAYED BACK PEOPLE'S SPEECH
THROUGH A SPEAKER, WE FOUND
THESE AREAS WERE ALSO ACTIVATED
AND CODING WHAT THEY HEARD.
SO THIS MEANS THERE'S AUDITORY
INFORMATION IN THIS PART OF THE
BRAIN AS WELL.
IT'S ENCODING THE PITCH OF WHAT
PEOPLE HEARD.
IN A SEPARATE SET OF EXPERIMENTS
AND PEOPLE UNDERGOING BRAIN
MAPPING AS PART OF THEIR SLEEP
SURGERIES, WHAT WE DID WAS WE
USED A BREATHING TUBE MY
COLLEAGUES IN E.N.T. USE FOR
MAPPING THE NERVES IN THE NECK
WHEN THEY DISSECT TUMORS HERE.
BUT HERE WE WERE USING THE
ENDOTRACHEAL TUBE THAT HAS WIRES
THAT NON-INVASIVELY TRACK THE
FOLDS AND SEE WHAT PART OF WERE
EVOKING MOVEMENT AND THERE WERE
PARTS THAT EVOKED MOVEMENT.
WE CAN PROVE THIS WAS A MOTOR
AREA NOT JUST A SENSORY AUDITORY
ONE BUT IN FACT A MOTOR AND
SENSORY.
IN FACT, IF YOU INCREASE THE
MAGNITUDE OF THE STIMULATION WE
SEE AN INCREASE IN MAGNITUDE OF
THE ENG.
SO THERE'S A PART OF OUR BRAIN
ON BOTH SIDES THAT CONTROLS THE
VOCAL FOLDS IN A VERY DIRECT WAY
AND WE THINK IT'S THE
CONNECTION.
WHAT ABOUT WHAT'S HAPPENING
ABOVE THE VOCAL FOLDS?
THE MOVEMENT OF SPEECH ARE
COMPLEX COMPARED TO THE MOVEMENT
OF THE AROUND AND YOU CAN MODEL
THE MOVEMENTS.
THE MUSCLES OF THE LIPS, FOR
EXAMPLE HAVE A TOLDLY DIFFERENT
SET OF PROPERTIES MORE LIKE AN
IT SPHINCTER AND CAN STUDY IT
USING MICROSOUND AND IF YOU LOOK
AT ONE ELECTRODE, THE SITES IN
THE MOTOR CORTEX ARE ENCODING
MOVEMENT TRAJECTORIES AND
ENCODING MOVEMENTS OF OUR VOCAL
TRACT WHEN WE SPEAK.
BY THEMSELVES THEY HAVE NO
MEANING BUT WHEN YOU PUT THEM
TOGETHER AND GIVE RISE TO ALL
CONSON
CONSONANTS.
THIS IS AN EXAMPLE THAT IS VERY
IMPORTANT FOR ENCODING THE DA
SOUND.
THIS PARTICULAR TRAJECTORY WHERE
THE TONGUE TIP GOES TO THE BACK
AND THE JAW GOES UP AND DOWN
THAT ONE PARTICULAR ELECTRODE IS
ENCODED BY A VIDEO AND THE
TONGUE TIP GOES FORWARD AND THE
JAW OPENS AND CLOSES.
IT'S A COORDINATED MOVEMENT
ACROSS ARTICULATORS ALL ENCODED
IN THIS ONE PARTICULAR SPOT.
ALL OF US HAVE THIS.
THIS ENCODING OF THIS SOUND.
THAT'S HOW WE MAKE THE DA SOUND.
IF YOU COMPARE IT TO ANOTHER
ELECTRODE A CENTIMETER AWAY IT'S
IMPORTANT FOR THE LABIAL
TRAJECTORY AND YOU'LL SEE THE
MAIN MOVEMENTS HERE ARE AROUND
CLOSURE OF THE LIPS.
SO WE HAVE THESE DIFFERENT SPOTS
IN THE MOTOR CORTEX THAT ENCODE
DIFFERENT TRAJECTORIES OF
MOVEMENT THAT CONTROL DIFFERENT
PARTS.
AGAIN, BY THEMSELVES THEY HAVE
NO MEANING BUT WHEN YOU SEQUENCE
THEM YOU PUT THEM TOGETHER AND
THEY'RE WORKING IN A COORDINATED
WAY IT GIVES RISE TO ALL
CONSONANTS AND VOWELS WHICH GIVE
RISE TO ALL WORDS AND POSSIBLE
MEANINGS.
THINKING ABOUT THESE IS
FUNDAMENTAL FOR COMMUNICATION IN
BUILDING BLOCKS FOR SPEECH.
ONE THING WE CAN DO IS WE CAN
LOOK AT THE DATA ACROSS
ELECTRODES.
EACH COLUMN CORRESPONDS TO A
DIFFERENT SENSOR AND EACH ROW
CORRESPONDS TO DIFFERENT PARTS
OF THE VOCAL TRACT AND BELOW
THAT ARE THE PHONYMS IN ENGLISH
AND THIS IS ALL THE ELECTRODES
AND HOW THEY'RE ACTIVATED BY
DIFFERENT PRODUCTIONS.
ONE REASON I SHOW THIS IS AS
EARLY AS EIGHT YEARS AGO WE HAD
THIS INTUITION WE COULD ACCESS
THE ENTIRE FRENETIC ALPHABET,
THE ENCODING, A DICTIONARY IN
SOME WAYS OF THE ELECTRICAL
PATTERNS IN ENGLISH AND THAT WAS
IMPORTANT BECAUSE THEN WE
STARTED OF TRANSLATING THIS
TECHNOLOGY FOR PEOPLE WHO ARE
PARALYZED.
I'LL TALK ABOUT WHAT WE'VE DONE
WITH THAT KIND OF INSIGHT TO
BUILD A SPEECH PROSTHETIC.
THIS QUOTE FROM STEVEN HAWKINGS
THOUGH I HAVE TO SPEAK THROUGH A
COMPUTER, IN MY MIND I AM FREE.
THIS ENCAPSULATES PART OF THE
PROBLEM WE'RE TRYING TO SOLVE
FOR PEOPLE WHO ARE LOCKED IN
WHERE ESSENTIAL LY AND HAVE TO
SEND SIGNALS OUT TO THE MAP.
THERE'S WE'VE ENDEAVORED TO
BUILD A DEVICE, A TECHNOLOGY
THAT CAN TRANSLATE THAT CODE
I'VE BEEN DESCRIBING INTO
SOMETHING THAT CAN DIRECTLY READ
OUT AND PRODUCE SPEECH YOU CAN
LEAR.
SO THE BASIC QUESTION FRAMED BY
THIS M.I.T. TECH REVIEW COVERING
THE EARLY WORK IS COULD A PERSON
UNABLE TO SPEAK LIKE STEVEN
HAWKING USE A BRAIN IMPLANT TO
CARRY ON A CONVERSATION?
THAT WAS THE BASIC QUESTION WE
WANTED TO PURSUE.
AND ONE QUESTION IS WHY SPEECH?
WHY NOT HAVE PEOPLE COMMUNICATE
THROUGH HANDWRITING?
BRAIN-INTERFACE BY WHAT PEOPLE
ARE TRYING TO EXPRESS THROUGH
THEIR WRITING OR TYPE OR TEXT?
THOSE ARE ALL EFFECTIVE WAYS.
IN FACT THE WHOLE FIELD WAS
FOCUSSED ON GETTING PATIENTS
PARALYZED ABLE TO COMMUNICATE
THROUGH MOVING A COMPUTER CURSER
QUICKLY TO DIFFERENT PARTS OF
THE KEY BOARD.
WE WANTED TO DO SOMETHING
DIFFERENT BECAUSE SPEECH IS I A
DIFFERENT PART OF COMMUNICATE
AND ALLOWS US TO COMMUNE UP TO
200 WORDS PER MINUTE.
THAT'S SUCH A SPECIAL BEHAVIOR
FOR OUR SPECIES.
IT ALLOWS ME TO TRANSMIT
INFORMATION FROM MY MIND TO
YOURS AND BACK IN A VERY FLUENT
WAY.
IT ALLOWS US TO BUILD
CIVILIZATION.
IT ALLOWS US AS SOCIAL BEINGS TO
TRANSMIT IDEAS TO BUILD, TO
LEARN AND FOUNDATIONAL.
SUCH AN IMPORTANT PART OF BEING
HUMAN.
THIS IS WHY WE WENT STRAIGHT TO
THIS AMBITIOUS IDEA OF TRYING TO
DO A SPEECH PROSTHETIC.
EARLY ON A LOT OF MY CLOSEST
FRIENDS AND COLLEAGUES SAID THIS
IS IMPOSSIBLE.
TOO COMPLEX.
SPEECH IS TOO ABSTRACT.
I KIND OF AGREE WITH THEM BUT AT
THE SAME TIME WE WERE SEEING
THIS ELECTRICAL CODE LAID OUT IN
FRONT OF US SO WE SAID, OKAY,
WHY DON'T WE GIVE IT A TRY?
ONE OF THE FIRST THINGS WE DID
WAS TRY IT IN EPILEPSY PATIENTS
AS PROOF OF PRINCIPLE.
THESE ARE PEOPLE SPEAKING.
NORMALLY SPEAKING.
WE COULD RECORD ABOUT WHAT THEY
ARE WERE SPEAKING AND DECODE
THEIR BRAIN ACTIVITY FROM WHAT
THEY SAID.
A GREAT WAY TO START BECAUSE WE
KNEW WHAT THEY SAID AND TRY TO
BUILD A COMPUTER MODEL TO DECODE
WHAT THEY WERE SAYING FROM THE
BRAIN ACTIVITY.
THAT'S WHAT WE DID IN THE STUDY
LED BY OUR GRADUATE STUDENT AND
POSTDOC.
IN THIS PARTICULAR APPROACH WE
TRANSLATED THE BRAIN ACTIVITY TO
A SIMULATION OF THE VOCAL TRACT
INTO A COMPUTER AND HAD THAT
CREATE WORDS YOU COULD HEAR.
IT WAS A TWO-PART DECODER.
ONE PART IS TRANSLATING BRAIN
ACTIVITY INTO MOVEMENTS.
A SIMULATION OF THE VOCAL TRACT
AND THEN TRANSLATING THOSE
SIMULATED MOVEMENTS INTO SOUND.
AND IT IS POSSIBLE TO GO FROM
BRAIN ACTIVITY TO THE SOUNDS BUT
ONE OF THE REASONS WE CHOSE THIS
METHOD IS BECAUSE WE KNEW THIS
PART OF THE BRAIN IS CONTROLLING
MOVEMENT OF THE VOCAL TRACT.
WE WANTED TO BE SORT OF AS BIO
MIMMENIC AS POSSIBLE AND MATCH
THE MODEL AND THIS IS AN EXAMPLE
OF THE SPEECH.
HERE'S A SPECTROGRAM RECORDED ON
TOP AND WHAT WAS GENERATED BY
OUR FIRST DECODER OF THOSE TWO
SENTENCES JUST FROM THE NEURAL
ACTIVITY.
YOU CAN SEE IT HAS A LOT OF THE
SAME ENERGY CONTOURS AND GROSS
SPECTRAL AND TEMPORAL
INFORMATION THERE WAS STRUCTURE
OF THE SPEECH WAS MISSING.
WE WERE EXCITED TO SEE IT HAD
THIS VISUAL APPEARANCE THAT WAS
CLOSE BUT WHAT WAS EXCITING IS
SOME SENTENCES WERE INTELIGABLE
AND TO PROVE THEY'RE POSSIBLE
WITH THE ELECTRODES NEVER DESIGN
TO DO THIS KIND OF PURPOSE.
AND SEE THE VOCAL TRACT AND THEN
HEAR WHAT THE PERSON SAID TO
COMPARE.
>> THE APPROACH YOU ARE SEEKING
IS NOT AVAILABLE IN BOOKS.
THE PROOF YOU ARE SEEKING IS NOT
AVAILABLE IN BOOKS.
SHIP BUILDING IS A MOST
FASCINATING PROCESS.
>> SO, WHAT WAS EXCITING WAS FOR
THE FIRST TIME HEARING A VOICE
IN THIS SYNTHESIS.
IT'S RASPY, IT'S NOISY.
AND IF YOU HALLUCINATE YOU'LL
HEAR VOICES AT THE BORDERLINE
BUT THAT WAS EXCITING TO HEAR
AND USE THAT WITH THE LEARNING
MODELS.
THIS IS JUST AN EXAMPLE OF WHY
WE DID THIS WITH THIS KINEMATIC
MODEL.
THE MCD IS A COEFFICIENT THAT
EXPRESSES HOW SIMILAR THE
ORIGINAL SPOKEN SPECTROGRAM IS
TO WHAT IS DECODED AND GOING
STRAIGHT FROM BRAIN ACTIVITY
SOUND YOU CAN DO IT WITH ENOUGH
DATA.
BUT BUILDING AN ARTICULARY MODEL
REQUIRES MORE DATA.
WITH BRAIN ACTIVITY TO SOUND HAS
TO DEVELOP A COMPLEX MAPPING.
THE STRATEGY HERE WAS LET'S
REDUCE HOW MUCH THE NETWORK HAS
TO LEARN BY IMPOSING STRUCTURE
WE KNOW TO BE TRUE.
THE VOCAL TRACT MOVEMENTS ARE A
CAUSAL MECHANISM OF SOUND
GENERATION AND PUT THAT IN AND
THAT CAN REDUCE AND REDUCING THE
TRAINING TIME IS VERY IMPORTANT
FOR PEOPLE WHO ARE PARALYZED OF
COURSE BECAUSE YOU CAN'T TAKE
FOREVER.
SO WE BASICALLY CONVINCE
OURSELVES THAT WE COULD TRY
THIS.
IN SOMEONE PARALYZED.
SO WE CREATED THE BRAVO TRIAL AT
UCSF IT STANDS FOR THE BCI
RESTORATION OF ARM AND VOICE.
MY LAB AND A REHABILITATION
NEUROLOGIST STUDYING HOW
CORTICAL REPRESENTATIONS AROUND
THE HAND ARE USED TO CONTROL A
CURSER OR ROBOTIC ARM.
THE PART MY LAB WAS FOCUSSING ON
WAS THE PART OF THE SPEECH.
WE HAVE PARALLEL BUT
COMPLIMENTARY STREAMS AND
THEY'RE FOR PEOPLE WITH SEVERE
PARALYSIS AND LIMB MOVEMENT.
THIS WAS APPROVED WITHIN I.D.E.
THE FIRST PARTICIPANT THAT
UNDERWENT THE STUDY WAS
DESCRIBED IN THIS PAPER IN THE
NEW ENGLAND JOURNAL IT DESCRIBES
THE FIRST SUCCESSFUL DECODING OF
WORDS AND SENTENCES FROM SOMEONE
PARALYZED.
I'LL TELL YOU A LITTLE BIT ABOUT
HIS CASE.
HE WAS THE FIRST IN THE WORLD
AND IMPLANTED ABOUT SIX AND A
HALF YEARS AFTER THE ORIGINAL
IMPLANT.
AND THE WAY HE NORMALLY
COMMUNICATED IS WITH A STITCH
ATTACHED TO HIS HAT AND TYPES
THINGS ON THIS ASSISTIVE DEVICE
MANY PEOPLE USE WITH A
COMMUNICATION DISORDER CALLED
AAC.
THE GOAL IS TO ALLOW PONCHO TO
COMMUNICATE FASTER.
THIS IS WHAT HIS MRI LOOKS LIKE.
THIS IS AN IMPORTANT CONCEPT.
WHERE HIS STROKE WAS HERE IN THE
PART CALLED THE BRAIN STEM AND
THE BRAIN STEM IS WHAT CONNECTS
THE BRAIN TO THE SPINAL CORD AND
CRANIAL NERVES.
SO HE CAN'T GET THE SIGNALS OUT
OF HIS BRAIN BECAUSE HAD A HUGE
STROKE THAT OCCURRED IN THE POS
BUT EVERYTHING ABOVE THAT IS
WORKING NORMALLY.
HIS THOUGHTS, HIS IDEAS ARE ALL
THERE.
HE JUST CAN'T GET THEM OUT
BECAUSE THE COMMUNICATION TO THE
EXTERNAL WORLD HAS BEEN SEVERED.
SO WHAT WE DID WAS WE BUILT AN
ARRAY -- WE TOOK THE ARRAYS WE
WERE USING FOR EPILEPSY AND
WORKED WITH SOME OF THE
COMPANIES, IN PARTICULAR,
BLACKROCK THAT HAD THIS PORT
APPROVED WITH MICROELECTRODE
ARRAY WE SAID WHY NOT ATTACH IT
TO AN ECOG BECAUSE I WANT TO
JUST RECORD FROM THE BRAIN
SURFACE AND WE WANTED TO DO THAT
AND THE FDA WAS OKAY WITH THAT
BECAUSE AFTER TESTING WE COULD
PROVE IT HAD THE SAME SAFETY AS
OTHER FORMS IT WITH WAS ALREADY
APPROVED FOR AND ADDITIONAL
SAFETY TO PROVE THE NEW FORM
WOULD BE OKAY.
THIS IS THE FIRST ECOG AND
RECORDS FROM HIS BRAIN AND
SIGNALS ARE TAKEN OUT AND
TRANSLATED AND DIGITIZED FROM
ANALOG BRAIN WAVES INTO DIGITAL
NEURAL SIGNALS.
THE COMPUTER IS DOING SIGNAL
PROCESSING AT THE FIRST STEP.
PICKING OUT THE PARTS OF THE E
COG MOST INFORMATIVE.
THE SECOND IS AN A.I. ALGORITHM
DETECTING WHEN HE'S TRYING TO
SPEAK.
THE THIRD PART IS DETERMINING
WHAT WORD HE'S TRYING TO SAY AND
IN THE FIRST EXPERIMENT WAS 50
WORDS AND RANKED PROBABILITY OF
WORDS AND DISPLAYING THE FIRST
ON THE SCREEN.
SO IT'S THE FIRST ATTEMPT AT
THIS.
IT'S FOR PEOPLE TO UNDERSTAND
WHAT THIS IS ABOUT.
IT'S ABOUT A PERSON WHO HAS
VOLUNTEERED TO BE PART OF AN
EXPERIMENTAL STUDY.
THE FIRST TIME IT WAS EVER DONE
AND IS NOW LIVING WITH THIS PORT
ON HIS HEAD AND THIS IS WHAT
THAT CONNECTOR LOOKS LIKE.
YOU MIGHT BE ABLE TO APPRECIATE
IT BUT IT HAS 128 LITTLE GOD --
GOLD CONTACT AND EACH HAS A WIRE
GOING TO A DIFFERENT PART OF HIS
BRAIN.
SO WE'RE LITERALLY TAPPING INTO
PON
PONCHO'S BRAIN USING THIS PORT
AND HE'S HAD THIS FOR QUITE SOME
TIME BUT WE'RE MOVING TOWARDS
WIRELESS.
THE SURGERY TAKES ABOUT FOUR
HOURS TO DO THIS BUT WE SPENT
HOURS, DAYS, MONTHS, YEARS
TRYING TO UNDERSTAND THE SIGNAL
GENERATED FROM IT.
THE CHALLENGE IS TO TRANSLATE
THIS KIND OF SIGNAL INTO WORDS.
IT'S IN THERE BUT ANY OF US WHO
STUDY THE BRAIN IN NEURAL CODE
IS ESSENTIALLY WHAT WE ARE
TRYING TO DO WHICH IS TRANSLATE
THE ELECTRICAL CODE INTO
SOMETHING INTERPRETABLE.
THIS IS AN EXAMPLE OF WHICH
ELECTRODES WERE FOR TRYING TO
SPEECH AND THE AREA SHADED IS
THE ENTIRE MOTOR CORTEX HERE.
THEN THE AREAS THAT WERE
IMPORTANT FOR CLASSIFICATION
WERE DOWN HERE IN THE CENTRAL
PART OF THIS MOTOR CORTEX IN
THIS AREA IN THE MIDDLE.
THE FIRST 50 WORDS WE CHOSE TO
WORK ON SHOWED DECENT NOT
PERFECT, FAR FROM PERFECT
DECODING ACCURACY BUT IT
BASICALLY WORKED.
WE COULD DISCRIMINATE DIFFERENT
WORDS FROM ONE ANOTHER AND NOT
SURPRISINGLY THE WORDS THAT ARE
SIMILAR LIKE WHAT AND IS ARE
CONFUSABLE AND THOSE DISSIMILAR
WERE -- THAT SHOULD MAKE SENSE
BASED ON THE PREVIOUS WORK WE
SHOWED HOW THE INFORMATION IS
CODED IN THAT PART OF THE BRAIN.
WE WERE ABLE TO SEE OVER WEEKS,
MONTHS, NOW YEARS A STABLE
ACTIVATION OF THIS OVER TIME
BECAUSE THIS IS WHY WE CHOSE IT.
SOMETHING CLINICALLY STABLE OVER
SOME TIME TO BE ABLE TO SHOW
IT'S SOMETHING SOMEONE CAN USE
OVER A LONG TIME PERIOD.
THIS IS THE DEMO VIDEO FROM
PONCHO.
SOMEONE WILL WRITE TO HIM AND
SEE WHAT'S DECODED FROM HIS
BRAIN ACTIVITY.
THIS IS THE PORT WITH THE HDMI
CABLE.
THE BREAKING DOTS MEANS THIS
BLINKING DOTS HERE CORRESPONDS
SO THE ALGORITHMS DETECTED.
HE'S TRYING TO SAY SOMETHING.
THAT'S THE SPEECH DETECTION.
THE REST OF THE TIME HE'S TRYING
TO FIGURE OUT WHAT WORD HE SAID
AND ONCE IT FIGURES OUT THE
PLOT.
NOW, IT'S TRIED TO RANK ALL 50
WORDS.
IT'S JUST GOING TO SHOW THE TOP
RANK BUT KEEPS TRACK OF ALL 50.
SO THIS IS DOING WORD BY WORD
ANALYSIS.
CHUNK IT BY WORDS AND THE LAST
ONE I INCLUDED BECAUSE IT'S GOT
A COUPLE ERRORS BUT USES THE
AUTOCORRECT LANGUAGE MODEL.
BECAUSE IT KEEPS TRACT OF THE
RANKED PROBABILITY IT CAN USE
THE CONTEXT OF THE WORDS BEFORE
TO UPDATE EITHER THE SPELLING OR
SO WE'RE USING A LOT OF THESE
THINGS PEOPLE ARE FAMILIAR WITH
FROM TEXTING AND ING LIKE
AUTOCORRECT TO ACCELERATE CALLED
LANGUAGE MODEL.
AND THERE'S SIMILAR SITUATION
BRAIN STEM STROKE EIGHT YEARS
PRIOR AND IS THE COULD NOT MOVE
HER LIMBS AND COULD NOT SPEAK.
>> TELL ME WHERE YOUR LIVING
RIGHT NOW.
>> ANN USES EYE TRACKING
TECHNOLOGY IN ORDER FOR HER TO
COMMUNICATE.
SHE'S AN INCREDIBLE PERSON AND
RAISED TWO INCREDIBLE DAUGHTERS
THIS WHOLE TIME SHE WAS
PARALYZED.
AN INCREDIBLE STORY.
AND BRAIN STEM STROKE BY PART OF
THE BRAIN AND WE WENT BACK TO
THE FDA AND ASKED IF WE COULD
INCREASE THE -- DOUBLE IT FROM
128 TO 153 CHANNELS.
THIS IS REPORTING FROM THE BRAIN
AND WE SAW BETTER PERFORMANCE
WITH ANN.
ONE OF OUR INTERNAL METRICS IS
HAVING PEOPLE SPEAK NATO CODE
WORDS.
IT'S A STANDARD TASK WE CAN
QUANTIFY AND COMPARE ACROSS
INDIVIDUALS.
AND ANN WAS ABLE TO INCREASE
ACCURACY.
WE FROZE THE CLASSIFY IER
ANALYZING THE DATA TWO WEEKS
AFTER STARTING THIS AND USED THE
SAME ALGORITHM FOR WEEKS AND
MONTHS LATER AND STILL COULD GET
GOOD PERFORMANCE.
IT MEANS THE DECODER IS STABLE
OVER TIME AND THEREFORE COULD BE
PRACTICAL FOR SOMEONE TO DO
WITHOUT HAVING TO TRAIN EVERY
DAY.
AND WE HAD SEPARATE PARALLEL
A.I. TRANSLATING HER BRAIN
ACTIVITY INTO WORDS YOU COULD
HERE CALLED SPEECH SYNTHESIS.
THEN ANOTHER ONE THAT IS
DECODING FACIAL MOVEMENTS
ASSOCIATED WITH SPEECH THROUGH
AN AVATAR.
ONE WAY IS TEXT, ONE IS SPEECH
SYNTHESIS AND THE THIRD IS
EXPRESSIVE AND WHAT IT LOOKS
LIKE.
WE'RE INTERESTED IN HOW HAVING
EMBODIMENT OF SPEECH CAN GIVE
FEEDBACK TO GIVE ARTICULATION.
WHAT'S IT LIKE TO HAVE AN AVATAR
OF YOURSELF AND SEE AND
UNDERSTAND WHAT'S WORKING OR NOT
WORKING.
WHAT WE LEARNED FROM THEM IS IT
GOES BEYOND THAT.
BEING ABLE TO CHOOSE THE EYE
COLOR OR HAIR COLOR, FEATURES OF
THEIR AVATAR WAS RESTORING
IDENTITY.
PEOPLE WHO HAVE LOST ABILITY TO
COMMUNICATE HAVE NOT JUST LOST
THEIR VOICE BUT IDENTITY AND
THIS IS THE MODEL WE BUILT AND
HAVE A NETWORK FROM BRAIN
ACTIVITY TO CODING THE DIFFERENT
PHONETIC ELEMENTS.
WE ARRANGED THE PHONETIC
ELEMENTS INTO WORD FORMS AND
THEN WE PROCESS THIS INTO A TEXT
AND GO FROM BRAIN ACTIVITY TO
TEXT.
WE'RE GIVING A COUNT DOWN AND
ASKING HER TO ARTICULATE THE
WORDS AND SENTENCES.
BECAUSE WITH HAVE MORE DATA AND
INFORMATION WE ASKED HER SPEAK
AND THE ANALYSIS IS NOW
ANALYZING THE WHOLE PHRASE IN
THE SENTENCE NOT WORD BY WORD.
YOU CAN SEE IT'S A LOT FASTER
WITH 75% ACCURACY.
FASTER WITH VOCABULARY AND 1500
WORDS WHICH CAN EXPAND TO
SEVERAL THOUSAND.
245 WAS SUPER EXCITING TO SEE.
WHAT ABOUT THE SPEECH SYNTHESIS?
WE TRANSLATED TO UNITS OF
SPEECH.
LATE ENT UNITS LEARNED FROM A
SPEECH RECOGNITION MODEL
PRODUCED BY META.
WE WERE ABLE TO TRANSLATE THOSE
TO SPEECH YOU CAN HEAR.
THIS PARTICULAR CASE WE USED
ANN'S ORIGINAL VOICE FROM HER
WEDDING WE HAD ACCESS TO AND
QUASI-FIT IT TO HER AND USING AN
AVATAR MACHINE LEARNING METHOD.
>> YOU ARE WONDERFUL.
>> SO IT'S DECODING TEXT AND
MOVEMENTS SIMULTANEOUSLY.
UNCANNY, RIGHT?
KIND OF STRANGE TO SEE SOMEONE
COMMUNICATING IN THAT WAY.
VERY SURREAL TOO.
BECAUSE WE BUILT AN AVATAR I
ASKED THE TEAM LET'S USE FACIAL
EXPRESSIONS.
THAT WORKS WELL.
YOU CAN CLASSIFY EMOTIONAL
EXPRESSIONS.
HERE'S EXAMPLES OF ANN
EXPRESSING A FAKE SMILE.
>> ALL THESE LINK BACK.
AS SHE IS MAKING THESE ATTEMPTED
SPEECH MOVEMENTS WE CAN GO BACK
TO THE FIRST PRINCIPLES WE
DISCOVERED 10, 15 YEARS AGO
ABOUT THIS.
THE CLEARANCE, THE HAND AREA,
THE LIPS, TONGUE, ETCETERA.
IT'S ALL RELATED.
THE SAME PRINCIPLES INVOLVED AND
WHEN PONCHO HAD HIS ACCIDENT HE
WAS A SPANISH SPEAKER AND WE
BUILT A BILINGUAL DECODER AND
HAS A LANGUAGE MODEL SPECIFIC TO
ENGLISH OR SPANISH AND ONLY
PRESENTS THE ONE THAT DETERMINES
ENGLISH OR SPANISH.
SO IT'S THE BRAIN AREA BUT
DETECTING IF THE SEQUENCE OF
ELEMENTS CORRESPONDS AND
DISTINGUISHES.
SO THAT'S AN EXAMPLE OF ENGLISH.
I'LL GIVE ANOTHER IN SPANISH.
WE DON'T TELL THE DECODER.
IT HAS TO MOVE FORWARD WITH
THESE KINDS OF SEQUENCES IN
PHONETIC ELEMENTS IN THE BRAIN
ACTIVITY.
OKAY.
ANOTHER THING NOT PUBLISHED YET
BUT ALREADY IN PRESS FROM TWO
GRADUATE STUDENTS BUT HERE WE'RE
TRYING TO MAKE THE SYNTHESIS
FASTER.
IF YOU NOTICE IN THIS EXAMPLE
THERE'S A LONG LATENCY.
WE'VE TRIED TO REDUCE THAT TO A
SECOND AND WE'RE DOING REAL-TIME
SYNTHESIS.
HERE'S AN EXAMPLE OF THAT.
>> I HAVE AN APPOINTMENT NOW.
>> LET ME TELL YOU --
>> THAT'S WHAT WE CALL EXTREMING
SYNTHESIS.
WE CAN TAKE IT OUT EVERY MONTH,
WEEK, WHATEVER.
THIS FIELD IS EXPLODING AND
THERE'S INCREDIBLE RESULTS A.
OTHER GROUPS AS WELL THE NEW
ENGLAND JOURNAL ASKED ME TO
WRITE A REVIEW AND I PUT FORWARD
SOME CRITERIA WHAT TO LOOK FOR.
WE WANT IT TO BE HIGH
PERFORMANCE, FAST, ACCURATE,
SOMETHING EASY TO USE.
WE WANT IT ROBUST.
WE DON'T WANT TO SPEND HOURS
CALIBRATING.
WE WANT IT STABLE AND USE
DIFFERENT SETTINGS AND DISEASE
STAGES.
WE WANT IT TO BE HIGH LATENCY.
WE WANT IT TO BE FULLY
IMPLEMENTABLE AND WIRELESS AND
SURGICALLY AS EASY AS POSSIBLE
NOT ONLY TO IMPLANT BUT ALSO TO
REMOVE.
SO THIS IS THE WISH LIST.
HERE'S SOME OPTIMIZATIONS.
WE CAN PREDICT HOW TO GET HIGH
ACCURACY FROM PAPERS AND OTHERS
AND PROJECTIONS ARE VERY SIMILAR
THE NUMBER OF SENSORS MATTERS
AND DENSITY MATTERS.
WE NOW HAVE THE DATA TO MAKE
THOSE KINDS OF PROJECTIONS AND
MAYBE OPTIMIZATION.
I'LL SUMMARIZE HERE AND GET TO
BIG PICTURE.
SO WHAT I'VE DESCRIBED HERE IS A
PATH THAT WE'VE TAKEN FROM THE
BASIC SCIENCE OF THE CORTICAL
PHYSIOLOGY OF THE WAY WE SPEAK.
IT GOES DEEP.
I'VE ONLY SHOWN SOME HIGHLIGHTS
RELATED TO THIS MAP OF AUDITORY
MOVEMENTS AND THE LARYNGEAL
CORTEX THAT WAS UNDER
APPRECIATED.
HOW MOVEMENTS ARE ENCODED BY
CORTICAL ACTIVITY IN THIS PART
OF OUR BRAIN.
WHEN HAVE YOU A STROKE OR
NORTH -- INJURY YOU HAVE THE
LOSS OF THE ABILITY TO TRANSLATE
AND THIS IS THE ERA OF A.I. AND
WE CAN EMBODY FORMS FOR VERBAL
AND NON-VERBAL COMMUNICATION.
WE NOW BASIC PARAMETERS TO
OPTIMIZE.
IT'S NOW A QUESTION OF HOW GOOD
CAN WE MAKE IT.
IT'S NOT ABOUT IF IT'S POSSIBLE
AND WE'RE VERY FOCUSSED ON THAT.
SO, BIG PICTURE, I WANTED -- I
THINK IT'S VERY IMPORTANT TO ME
PERSONALLY TO BE HERE TODAY
BECAUSE NONE OF THIS WOULD HAVE
BEEN POSSIBLE WITHOUT THE
SUPPORT OF THE NIH.
I FEEL EXTREMELY LUCKY TO BE A
BENEFACTOR TO COLLABORATE WITH
NIH AT LARGE AND NICHD AND ONE
OF THE LUCKIEST PEOPLE IN THE
WORLD TO DO THIS WORK THAT'S
BEEN SO FULFILLING AND
PRODUCTIVE.
I DO WANT TO CALL OUT ROGER
MILLER WHO IS HERE.
THERE WAS A NIDC WORKSHOP AND I
REMEMBER READING THE SUMMARY AND
I WASN'T ELIGIBLE TO ATTEND BUT
I REMEMBER READING THE ARCHIVE
SUMMARY AND THINKING, WOW, THIS
IS SUPER EXCITING THIS IS EVEN
POSSIBLE AND IT PLANTED A SEED
IN MY MIND LIKE I WANT TO WORK
ON SOMETHING RELATE TO THAT AND
IT TOOK A WHILE BUT IF YOU
NOTICE THIS IS 2006.
THAT'S ALMOST 20 YEARS AGO.
SO THANK YOU FOR THE VISION AND
FORESIGHT.
A DECADE AND A HALF LATER WE
WERE ABLE TO GET SUPPORTED BY A
CLINICAL TRIAL PROGRAM THROUGH
NIDCD AND SUPPORT OUR PROGRAM.
IT WAS THE FIRST PILOT CLINICAL
TRIAL FOR SPEECH PROSTHESIS.
WHAT I JUST SHOWED YOU IN THE
TRANSLATION WAS REALLY SPONSORED
BY THE WORK OF NIDCD AND NIH.
SO THIS TIME LINE IS MY
REFLECTION OF WHAT MY CAREER AND
OUR WORK HAS DONE AND THERE'S
MANY OTHER LABS THAT HAVE THIS
PARALLEL KIND OF TRAJECTORY.
THE FIRST DESCRIPTION OF THE
SPEECH AND MOTOR CORTEX WAS IN
2013 SPONSORED BY AN INNOVATOR
AWARD FROM THE DIRECTOR'S OFFICE
CALLED FUNCTIONAL ARCHITECTURE
OF SPEECH MOTOR CORTEX AND US
WITH ABLE TO DOUBLE DOWN ON THAT
WITH THE NIH BRAIN INITIATIVE
AND GAVE US CODING FOR CONSO
CONSONANTS AND VOWELS AND WERE
ABLE TO APPLY MORE A.I. AND THEN
ABLE TO BUILD THE EVIDENCE AND
CASE FOR MOVING TO A PATIENT
AREA AND TAKING IT OUT OF
EPILEPSY MONITORING AND MOVING
IT TO PEOPLE WHO ARE SUFFERING
AND GET SYNTHESIS SUPPORTED AND
NOW A NEW UGA-3 SUPPORTING
DEVELOPMENT OF THE WILL LAFRD
WEAR.
A TIRELESS SYSTEM SO WE DON'T
HAVE TO DEAL WITH THIS CRAZY
PORT.
SUPER IMPORTANT TO DO THE
INITIAL DERISKING BUT NOW WE
HAVE TO USE BETTER TECHNOLOGY.
WE'RE HOPING FOR THE FIRST IN
HUMAN DEVICE IN THE NEXT TWO
YEARS.
KNOWLEDGE OF THIS WOULD HAVE
BEEN POSSIBLE WITHOUT THE
SUPPORT OF THE NIH OR NIDCD AND
BRAIN INITIATIVE.
THE LAST THING I WANT TO
DESCRIBE IS WHERE WE'RE GOING
WITH THE SCIENCE, FULL CIRCLE.
TALKING ABOUT SCIENCE TO
TRANSLATION BACK TO SCIENCE.
WE'RE USING A NEW TOOL CALLED
NEURAL PIXELS A WAY TO STUDY
NEURONAL ACTIVITY AND A FRIEND
IS A PIONEER IN RECORDING THIS
IN HUMAN USING A SLIGHTLY
DIFFERENT TECHNOLOGY ALONG THE
LINEAR ARRAY.
THIS AREA WE DESCRIBED A LOT IN
THE LAST DAY AND A HALF USING
ELECTROCARDIOGRAPHY FOR RARE
CASES WHERE THERE'S ACCESS BUT
OVER THE YEARS WE'VE LAD RAY
COUPLE.
WHAT YOU'RE LOOKING AT IS
UNPUBLISHED WORK WHERE FROM A
SINGLE NEURAL PIXEL WE CAN PULL
NEURONS AND THIS IS WHAT THEY'RE
DOING IN A TASK WHERE PEOPLE
HAEFR A WO -- HEAR A WORD AND TN
GIVEN A PROMPT TO STREAK 37 IN
THE CORTEX IT'S NOT JUST
ACTIVATED DURING SPEAKING BUT A
SUBSET OF THE CELLS ARE
PRIMARILY RESPONDING WHEN THEY
HEAR THE WORK.
SO THIS IS NOW THE CELLULAR
LEVEL SO THERE ARE CELLULAR SUB
POPULATIONS THAT ARE VERY MUCH
RESPONSIVE WHEN YOU HEAR A WORD
AND OTHER POPULATIONS PREPARING
TO SPEAK THE WORD AND THEN
EXECUTING.
YOU CAN LOOK AT OTHER PARTS OF
THE GYRUS AND SEE SOMETHING
SIMILAR.
IF YOU LOOK AT THIS AREA, IT
DOESN'T LOOK LIKE THERE'S MUCH
INTERESTING GOING ON THERE AT
ALL.
LOOKING AT THE AUDITORY CORTEX
YOU HEAR RESPONSES TO HEARING
THE WORDS AND STRONG RESPONSES
TO HEARING THE WORD.
THE CELLULAR LEVEL HAS DOZENS OF
NEURONS RECORDED SIMULTANEOUSLY.
IT'S INCREDIBLE TECHNOLOGY.
AND IF YOU LOOK AT THIS OVER A
MULTIPLE DATA SETS WE'RE TALKING
ABOUT SINGLE NEURON AT SCALE.
FOR EXAMPLE, THIS IS AN EXAMPLE
OF A COUPLE OF DATA SETS
TOGETHER, 500 UNITS ALONG THE
DIFFERENT PHASES ACROSS THE
DIFFERENT BRAIN AREAS.
SO WE'RE NOW STARTING TO ENTER
THIS NEW PHASE OF HUMAN
NEUROSCIENCE WHERE WE CANNOT
ONLY DO THE SAME BUT AT SCALE.
WE CAN GET A LARGE POPULATION TO
UNDERSTAND MORE HOLISTICALLY HOW
THE POPULATION REPRESENTS
INFORMATION.
ONE THING SUPER EXCITING IS NOT
JUST IN THE BUT POPULATION BUT
DIFFERENT EXAMPLES AND NEURONS
IN THE GYRUS AND WE'VE BEEN
DESCRIBING THE DECODING AND IT'S
MORE DIVERSE AT THE CELLULAR
LEVEL.
ONE NEURON IS ONLY ACTIVE IN
LISTENING TO THIS AND HERE'S
ANOTHER NEURON ACTIVATED WITH
THE SPEAKING.
THIS IS ANOTHER SUPER
INTERESTING ONE WHERE IT'S
ACTIVE DURING BOTH LISTENING AND
SPEAKING.
BUT FOR THE SAME WORD.
SO IT'S ACTIVATED WHEN YOU HEAR
A WORD AND REACTIVATED AND THEN
ANOTHER SET ACTIVATE THE WHEN
THEY GET THE GO WITH QUEUE AND
ANOTHER ACTIVATED WHEN THEY HEAR
A WORD AND LISTEN AND STAYS ACT
VA
-- ACTIVATED UNTIL YOU SAY IT.
IT'S ALL HAPPENING IN THE
CELLULAR POPULATION.
THESE ARE WHAT WE CALL THE
MIRROR NEURONS AND FIGURE OUT
WHAT WORD THEY'RE ENCODING.
HERE THEY'RE RESPONDING AND HERE
RESPONDING TO THE PRODUCTION OF
THE WORD.
HERE'S AN EXAMPLE OF WHAT WE
CALL BRIDGE NEURONS.
THEY'RE ACTIVATED BY HEARING
WORD AND THEY STAY ACTIVATED
UNTIL YOU SAY THE WORD.
REALLY INTERESTING PHENOMENON
AND PROBABLY HAS A LOT OF
IMPLICATIONS FOR UNDERSTANDING
HOW WE LEARN TO SPEAK AND OTHER
PHENOMENON.
INTEGRATION HOW WE HEAR AND
SPEAK AT A CELLULAR LEVEL AT THE
PREMOTOR CORTEX.
I JUST WANT TO FINISH HERE AND
THANK THE PEOPLE IN MY LAB AND
THANK ALL OF YOU AT NIH.
IT'S BEEN AN AMAZING
COLLABORATION WITH THE INSTITUTE
AND BRAIN INITIATIVE AND NIDCD.
THANK YOU SO MUCH FOR ENABLING
ALL THIS WORK.
HAPPY TO TAKE SOME QUESTIONS.
WE HAVE A FEW ONLINE.
>> HI, THANK YOU FOR THE TALK.
I HAVE TWO QUESTIONS IF THAT'S
OKAY.
THE FIRST ONE IS LIKE WHEN I'M
TRYING TO SAY SOMETHING TO
SOMEBODY BUT HAVE PASSING
THOUGHTS I DON'T MEAN TO
VOCALIZE LIKE I'M TIRED OR
SOMETHING IF SOMEONE ASKS HOW
I'M DOING WELL I'LL SAY WELL BUT
I'M SLEEPY.
WITH THESE PATIENTS SINCE
THEY'RE NOT TRYING TO VOCALIZE
THAT THOUGHT WOULD IT NOT COME
OUT.
>> YEAH, WE'VE ACTUALLY TRIED
THAT.
WE TRIED TO DECODE WHAT THINGS
PEOPLE ARE THINKING ABOUT TO SEE
IF THEY ARE COULD AND FOR THAT
WE NEED TO RECORD FROM DIFFERENT
BRAIN AREAS.
>> OKAY.
>> THIS IS NOT REALLY ABOUT
DECODING THE THOUGHT IT'S ABOUT
DECODING WHAT SOMEONE IS TRYING
TO VOCALIZE.
>> WHERE WOULD THOSE THOUGHTS BE
STORED IN THE BRAIN?
>> THAT'S THE BILLION DOLLAR
QUESTION.
ACTUALLY THROUGHOUT THE BRAIN
REALLY DISTRIBUTED.
THERE'S NOT A LOT OF PLACES WE
CAN DO SURGERY ON THAT
FUNDAMENTALLY CHANGE SOMEONE'S
ABILITY TO THINK.
WE THINK THIS KIND OF
INFORMATION IS DISTRIBUTED.
>> THANK YOU.
SECOND QUESTION IS SIMILAR.
THE A.I. HAPPINESS, SADNESS THAT
COMES OUT -- I'M ASSUMING THAT'S
ALSO FROM A DIFFERENT BRAIN
REGION FROM THE MOTOR SPEAK SO
HOW IS THAT GENERATED?
>> THAT'S THE OPPOSITE IN WHICH
ALL THESE THINGS OPERATE AT THE
MUSCLES OF OUR FACE OR FACIAL
STRUCTURES SO OUR BRAIN HAS ONE
MODE OF MOVING THOSE TO CREATE
THE WORDS AND SOUNDS AND THE
OTHER MODE IS JUST FOR
EXPRESSION.
WE DON'T JUST USE THE VOCAL
TRACT TO SPEAK, WE USE IT FOR
DIFFERENT THINGS IT'S THE SAME
BRAIN AREA BUT DIFFERENT FOR
THOSE GESTURES.
>> OKAY.
THANK YOU VERY MUCH.
>> EDDIE, THAT WAS A FANTASTIC
TALK.
I HAVE A LOT OF QUESTIONS BUT
I'LL ONLY ASK ONE.
THE ONE I THINK ABOUT WHEN YOU
TALK ABOUT IMPLANTING PEOPLE SO
LONG AFTER THEY'VE SPOKEN YOU
MIGHT EXPECT THERE'S DEGRADATION
OF THE BRAIN'S ABILITY TO
ACTUALLY PRODUCE THE NEURAL
ACTIVITY BECAUSE THEY HAVEN'T
SPOKEN.
I WONDER IF ONE THING THAT
MAINTAINS THAT FUNCTION IS THE
LAST COUPLE OF SLIDES OF THE
INTEGRATION OF THE HEARING AND
SPEECH.
WE THINK OF COCHLEAR
IMPLANTATION AND HAVE PLASTICITY
PREVENTS PEOPLE LONG DEAFENED
FROM BENEFITING BUT YOU'RE NOT
SEEING THAT HERE SO WONDERING
YOUR THOUGHTS.
>> THANK YOU.
THE QUESTION IS AFTER NOT
SPEAKING FOR OVER A DECADE, IS
THAT PART OF TO THE BRAIN STILL
WORKING?
THAT WAS THE BIGGEST QUESTION IN
MY MIND BEFORE WE STARTED THIS
IS THAT PART OF THE BRAIN DOES
IT BECOME DORMANT.
IT'S REALLY CLEAR THAT'S NOT THE
CASE.
LIKE YOU SAID, PEOPLE HEARING
WORDS AND UNDERSTANDING WORDS
AND I DIDN'T SHOW THAT THERE'S
READING RESPONSES IN THAT PART
WE THOUGHT WAS MOTOR CONTROL HAS
MORE SENSORY PROCESSING
INCLUDING BREATHING AND
LISTENING.
I THINK WHEN PEOPLE INTERNALLY
HAVE INNER SPEECH OR TRYING TO
EXPRESS THEMSELVES WITH THE
POINTER OR WHAT NOT, THEY'RE
USING THEIR INTERNAL SPEECH TO
GENERATE WORDS.
IT GOES BEYOND THE MOTOR
CONTROL.
I THINK THERE'S A LOT OF HOPE
THOSE CIRCUITS ARE STILL
FUNCTIONAL AND WE CAN TAP INTO
THEM AT THE RIGHT RESOLUTION WE
CAN UNLOCK WHAT EXPRESSION IS
WITH THEM.
>> MY QUESTION IS SIMILAR TO
WHAT WAS JUST ASKED.
I WAS CURIOUS IF THERE IS ANY
DIFFERENCE IN THE DATA YOU GET
FROM PEOPLE WHO ARE ABLE TO
UTILIZE THEIR ARTICULATORY
KINEMATICS IS THERE MORE NOISE
IN THOSE NOT ABLE TO FORM THE
WORDS WITH THEIR MOUTH OR DO YOU
SEE ANYTHING LIKE THAT?
>> THEY'RE DECODING FROM SOMEONE
WHO HAS SEVERE PARALYSIS IS
NOWHERE CLOSE TO SOMEONE WHO
SPEAKS FLUENT.
SO, WHILE IT'S THERE AND LATENT
IT'S NOT AS CLEAR AS THOSE WHO
ARE SPEAKING EVERY DAY ALL THE
TIME.
ONE MUCH THE THINGS WE'VE SEEN
ANECDOTALLY IS THE SIGNALLING
CHANGES OVER THE WEEKS, DAYS AND
MONTHS WE DO THE TRAINING.
THE SIGNALS BECOME CLEARER.
THE ANALYSIS BECOMES A LITTLE
BIT MORE STRAIGHTFORWARD HOWEVER
IT'S NOT JUST THAT.
WE SEE PEOPLE ARE HAVING
INCREASE IN THE MUSCLE TONE AND
THEY STILL CAN'T SPEAK BUT FOR
THE FIRST TIME THEY'RE TRYING TO
SPEAK FOR HOURS THROUGH
PROSTHETIC BUT SEEING THE
BYPRODUCT.
>> TWO MORE.
>> HI.
MY QUESTION WAS REGARDING THE
SECOND PATIENT WHERE THEY'D GO
AHEAD AND READ THE QUESTION BACK
THE ENTIRE SENTENCE.
>>
>> THERE'S AN IMPROVEMENT IN
MOTOR FUNCTION.
>> THERE'S FIBERS REGROWING IN
THE SPINAL CORD.
MAYBE THE DEGREE OF PARALYSIS
DIMINISHES MUCH
>> WE OBSERVED THE STRENGTHENING
OF MUSCLE TONE AND MOVEMENT.
I DON'T KNOW IF IT'S BECAUSE OF
NEW CELLS
>> PROBABLY SYNAPTICALLY THE
CIRCUITS ARE BEING STRENGTHENED.
IN THE WHERE IT'S FIXED IS WHERE
IT'S DECLARED.
WHERE WE'RE STUDYING IS IN
COMPLICATED REGIME WHERE THE
PARALYSIS IS DEGENERATION.
AT THE SAME TIME WE'RE TRYING TO
GET THE DECODER TO WORK.
IT'S A LITTLE BIT OF MOVING
TARGET AND DEGENERATION OVER
TIME.
IT'S QUITE IMPORTANT.
>> IMPORTANT.
>> ALL RIGHT.
THANK YOU SO MUCH AGAIN.
THAT WAS ABSOLUTELY WONDERFUL.
EVERYONE, PLEASE JOIN US FOR A
RECEPTION IN THE BACK AND GIVE
DR. CHANG ANOTHER ROUND OF
APPLAUDS.