I'M TOYIN AJISAFE, PROGRAM OFFICER AT THE NATIONAL CENTER FOR REHAB RESEARCH AT NIH. I'M A MIDDLE AGED MALE WITH DARK BROWN SKIN AND BLACK HAIR, WEARING A COLLARED SHIRT WITH ALTERNATING WHITE AND BLUE STRIPES. ON BEHALF OF OUR DIRECTOR, WHO WILL UNAVOIDABLY HAVE TO BE AWAY TODAY, I'M THRILLED TO WELCOME EVERYONE INCLUDING OUR SPEAKERS AND MODERATOR. IN TERMS OF STRUCTURE, THERE WILL BE TWO TALKS, EACH WILL LAST 15 TO 20 MINUTES, AND THEN WE'LL HAVE AN ENSUING Q&A SESSION FOR 8 TO 10 MINUTES, AND EVENT MODERATOR WILL HAVE FIVE MINUTES TO PROVIDE CLOSING REMARKS. GROUND RULES, WE ASK ALL REMAIN MUTED AND OFF CAMERA UNTIL THE Q&A SESSION. IF YOU HAVE A QUESTION, PLEASE FEEL FREE TO POST IN THE CHAT, USING THE HAND RAISE FEATURE OR UNMUTE TO POSE YOUR QUESTION DIRECTLY. OUR FIRST SPEAKER DR. JACOB GEORGE ASSISTANT PROFESSOR IN DEPARTMENT OF COMPUTER ENGINEERING AND DEPARTMENT OF PHYSICAL MEDICINE AND REHABILITATION AT THE UNIVERSITY OF UTAH, DIRECTS THE UTAH NEURAL ROBOTICS LAB, FOUNDATIONAL RESEARCHER FOR CRAIG NELSON REHAB HOSPITAL. SEEKING TO AUGMENT NEURAL NETWORKS WITH ARTIFICIAL NEURAL NETWORKS AND BIONIC DEVICES TO TREAT NEUROLOGICAL DISORDERS AND FURTHER UNDERSTANDING OF NEURAL PROCESSING, INTERSECTION OF A.I., ROBOTICS, NEUROSCIENCE, LAB IS DEVELOPING A.I. AND BRAIN-MACHINE INTERFACE TO RESTORE AND/OR ENHANCE HUMAN FUNCTION. SECOND SPEAKER DR. DEANNA GATES, ASSOCIATE PROFESSOR OF MOVEMENT SCIENCE, UNIVERSITY OF MICHIGAN, SOCIAL PROFESSOR IN COLLEGE OF ENGINEERING MEDICAL SCHOOL AT THE UNIVERSITY. DR. GATES DIRECTS THE REHABILITATION BIOMECHANICS LAB, RESEARCH FOCUSES ON BIOMECHANICS, REHABILITATION, PROSTHETICS, ORTHOTICS, CONTROL OF MOVEMENT, MODERATOR IS DR. SUNIL AGRIWAL, PROFESSOR AT COLUMBIA UNIVERSITY, NEW YORK. HIS INTERESTS REHABILITATION ROBOTICS, DYNAMIC CONTROL, DIFFERENTIALAND PARTNERS, WORK FOCUSES ON HOW NOVEL TRAINING ROBOTSIN CAN HELP HUMANS RELEARN, RESTORE, OR IMPROVE FUNCTIONAL MOVEMENT. HE COLLABORATES ACROSS DISCIPLINES INCLUDING NEUROLOGY, REHABILITATION MEDICINE, PEDIATRICS, AND GERIATRICS. DR. JACOB GEORGE, YOU HAVE THE FLOOR. I'M PROFESSOR JACOB GEORGE, UNIVERSITY OF UTAH, ASSISTANT PROFESSOR, AND ALSO LOCATED IN NELSON REHABILITATION HOSPITAL, DIRECTOR OF NEUROROBOTICS LAB. SO, AS A BRIEF BACKGROUND, DURING MY Ph.D. I STARTED WORKING WITH IMPLANTED PERIPHERAL NERVE INTERFACES FOR BIDIRECTIONAL SENSORIMOTOR CONNECTION FOR MULTI-ARTICULATE PROSTHETIC HANDS. THEN DURING MY POSTDOC I STARTED WORKING ON CONTROLLING A LOWER LIMB EXOSKELETON, AND NOW WITH SUPPORT FROM NIH MY LAB IS WORKING ON DEXTEROUS CONTROL FOR STROKE PATIENTS USING SURFACE MYOGRAPHY, MY LAB OPENED TWO YEARS AGO, SERVING STROKE, SPINAL CORD INJURY, TRAUMATIC BRAIN INJURY, AND LIMB LOSS. WE ALSO SERVE THE LARGEST PORTION OF U.S. BY LAND MASS, PEOPLE COMING FROM ALL OVER THE INTERMOUNTAIN WEST INCLUDING EASTERN WASHINGTON AND EVEN MONTANA. AND WE ADMIT ROUGHLY 200 NO STROKE PATIENTS PER YEAR IN THIS FACILITY. ONE OF MY FAVORITE THINGS ABOUT THE HOSPITAL IS INTEGRATION OF RESEARCH AND CLINICAL SPACE INTO A SINGLE BUILDING, SO ALONGSIDE CLINICAL FACILITIES WE HAVE AUGMENTED REALITY AND RESEARCH GROUPS. THE ENTIRE HOSPITAL ALSO HAS INTEGRATED SMART HOME TECHNOLOGY SO EVERYTHING IN THE HOSPITAL CAN BE CONTROLLED WITH ADAPTIVE INTERFACES. PEOPLE WITH HIGH LEVEL SPINAL CORD INJURY WITH USE SIP AND PUFF TO OPEN A DOOR, ADJUST THE DOOR, CALL FOR A NURSE, INTERESTING PLATFORM TO WORK WITH NEW AND INTUITIVE CONTROL STRATEGIES. WITHIN THE HOSPITAL MY LAB WORKS AT THE INTERSECTION OF THREE AREAS, FIRST IS REHABILITATION ROBOTICS. WE FOCUS PRIMARILY ON UPPER LIMB PROSTHESES AND ORTHOSES, AND PERIPHERAL NERVE INTERFACE, AND ARTIFICIAL INTELLIGENCE WHICH IS THE LINK BETWEEN THE PERIPHERAL NERVES AND ROBOTIC DEVICES, TO INTERPRET BIOLOGICAL NEURAL NETWORKS PROVIDING THE OPPORTUNITY TO EXPLORE AND DEVELOP BIO-INSPIRED A.I. ALGORITHMS. SO MY LAB COVERS A BROAD RANGE OF RESEARCH BUT TODAY I'LL FOCUS SPECIFICALLY ON THE WORK THAT WE'RE DOING AS PART OF THE NIH DIRECTOR'S EARLY INDEPENDENCE AWARD. THIS IS ESSENTIALLY AN R01 THAT PROVIDES A PATHWAY FOR YOUNG SCIENTISTS TO RAPIDLY TRANSITION INTO AN INDEPENDENT RESEARCH POSITION, SO IN MY CASE I FINISHED MY Ph.D. JUST A LITTLE OVER TWO YEARS AGO, SPRING OF 2020, DID A SHORT THREE-MONTH POSTDOC OVER THE SUMMER, AND STARTED MY OWN INDEPENDENT LAB IN FALL OF 2020. SO THAT MEANS I'M A VERY NEW FACULTY MEMBER, AND I'M ALSO WORKING WITH A PATIENT POPULATION THAT'S NEW TO ME AS WELL. I'D LOVE FEEDBACK ON THE RESEARCH WE'RE PRESENTING TODAY, AND I'M ALWAYS OPEN TO COLLABORATIONS IF YOU HAVE EXCITING OPPORTUNITIES. SO THE GOAL OF THE NIH GRANT IS TO PROVIDE PATIENT-CENTERED REHABILITATION THROUGH A DEXTEROUS AND ADAPTIVE ASSISTIVE UPPER LIMB ORTHOSIS, USING THE EXOSKELETON MADE BY MYOMOM, USED BY STROKE PATIENTS AS ASSISTIVE DEVICE, AND USING THIS ASSISTIVE DEVICE DAILY HAS BEEN SHOWN TO HELP REHABILITATE UPPER LIMB FUNCTION AFTER STROKE. OUR GOAL IS TO LEVERAGE HIGH DENSITY EMG AND MACHINE LEARNING TO IMPROVE FUNCTIONALITY OF THIS TYPE OF DEVICE. AND WE HOPE THAT WITH THE IMPROVED FUNCTIONALITY AND DEXTERITY WE CAN ALSO IMPROVE REHABILITATIVE EFFECTS WITH THE WEARABLE SYSTEM LIKE THIS. SO, THE CURRENT MYOPRO DEVICE HAS ELBOW FLEXION/EXTENSION, AND THREE-FINGER PINCH GRIP, BOTH CONTROLLED USING EMG, SO JUST A SIMPLE EXAMPLE WHEN THE FLEXORS ARE ACTIVE, HANDS WILL CLOSE, ENSURERS OPEN THE HAND WILL OPEN. THE CONTROL ALGORITHM IS SIMPLE. THEY PLACE A SINGLE EMG ELECTRODE ON THE LEFT OVER THE MUSCLE, THEN YOU RECTIFY THE EMG SIGNAL USING MEAN ABSOLUTE VALUE, APPLY SIMPLE BINARY THRESHOLD, YOU CAN USE THAT AS CONTROL SIGNAL, BINARY CONTROL SIGNAL. AND THERE'S TWO MAJOR LIMITATIONS TO THE CONTROL STRATEGY THAT'S BEEN OUTLINED HERE. THE FIRST IS THAT ARE THE OUTPUT IS BINARY, WHICH MEANS THAT THE USER CANNOT ACTUALLY REGULATE THEIR FORCE OUTPUT, AND REGULATING FORCE OUTPUT IS NECESSARY FOR VARIETY OF ACTIVITIES OF DAILY LIVING SO, FOR EXAMPLE, IF YOU'RE TRYING TO MANIPULATE A FRAGILE OBJECT, YOU WOULD NEED TO REGULATE YOUR FORCE TO NOT BREAK THAT OBJECT, BUT STILL HAVE IT STRONG ENOUGH TO BE ABLE TO PICK IT UP. AND ANOTHER EXAMPLE IS YOU WOULDN'T WANT TO PICK UP AN EGG OR SHAKING SOMEONE'S HAND WITH MAXIMUM OUTPUT FORCE THIS CAN PROVIDE. THE SECOND LIMITATION WITH THIS CONTROL STRATEGY IS THAT IT DOES NOT SCALE BEYOND A SINGLE MOVEMENT. SO IT WORKS WELL FOR THE BICEP AND CONTROLLING THE ELBOW WHERE THERE'S A SINGLE LARGE MUSCLE TO CONTROL THE SINGLE MOVEMENT, BUT WHEN YOU GET DOWN TO THE HAND IT DOESN'T SCALE WELL BECAUSE IN FOREARM YOU HAVE DENSELY PACKED MUSCLES CONTROLLING THE HAND, AND AS A RESULT WITH THIS CURRENT CONTROL STRATEGY, PEOPLE CAN ONLY DO ONE MOVEMENT OF THE HAND WHICH IS THREE-FINGERED PINCH GRIP, SO THEY CAN'T SIMULTANEOUSLY CONTROL THEIR WRIST AS WELL AS MOVE THEIR FINGERS INDEPENDENTLY. OUR GOAL IS TO PROVIDE MORE DEXTERITY AND BETTER CONTROL BEYOND THAT CURRENT APPROACH, AND SO WE USE HIGH DENSITY EMG TO PROVIDE SIMULTANEOUS AND PROPORTIONAL CONTROL OF MULTIPLE HAND MOVEMENTS, SO WE'RE DEVELOPING A WRIST BAND CAPABLE OF RECORDING HIGH DENSITY EMG FROM A WRIST IN THIS ELEGANT FORM FACTOR THAT RESEMBLES A SMART WATCH. SO WE COUPLE THESE HIGH DENSITY RECORDINGS WITH MACHINE LEARNING TO MAKE THIS POSSIBLE, HIGHER LEVEL THREE STAGES TO THE MACHINE LEARNING ALGORITHM. THE FIRST STAGE THAT THE USER MIMICS THE COMPUTER, EXOSKELETON IS PASSIVELY MOVING THE USER'S ARM, THE USER IS ACTIVELY ATTEMPTING TO FOLLOW ALONG. THEN A SECOND STAGE, THE COMPUTER LEARNS WHAT SIGNALS ARE CORRELATED TO WHAT MOVEMENTS, SO FOR EXAMPLE OUR ALGORITHM MIGHT IDENTIFY THAT PATTERN OF EMG ACTIVITY, PATTERN A, FOR EXAMPLE, IS ASSOCIATED WITH HAND GRASP. AND PATTERN B MIGHT BE ASSOCIATED WITH WRIST ROTATION. AND THEN A THIRD STAGE, THE COMPUTER IS SOLVING THIS INVERSE PROBLEM, SO IN REAL TIME THE COMPUTER IS LOOKING AT THESE PATTERNS OF EMG ACTIVITY COMING IN. IF IT SEES PATTERN A IT WILL INFER THE USER IS TRYING TO PERFORM THAT HAND GRASP MOTION. SO, THE ALGORITHM THAT WE USE CURRENTLY IS A DEEP CONVOLUTIONAL NEURAL NETWORK, COMPUTER VISION, THE ALGORITHMS USED BEHIND SELF-DRIVING CARS, ALL TYPES OF THINGS. THE IMAGES ARE ACTUALLY SPACE OR TEMPORAL PROFILES OF EMG ACTIVITY. SO WE'RE CONVERTING THESE COMPLEX EMG ACTIVATION PROFILES INTO COMPLEX KINEMATIC MOVEMENTS. ON THE LEFT HERE WE CAN SEE EXAMPLE OF EMG IMAGE. IF YOU GO FROM THE TOP DOWN TO THE BOTTOM, ON THIS IMAGE ON THE LEFT, THE Y-AXIS IS REPRESENTING THE ELECTRODE NUMBER, IN THIS CASE 32 ELECTRODES, LEFT TO RIGHT ACROSS THE X AXIS, SEEING TIME, GOING FROM RELAXED RESTING STATE TO MIDDLE FINGER EXTENSION. HOT COLORS ARE MAGNITUDE OF EMG ACTIVITY. YELLOW REPRESENTS GREATER ACTIVITY. IN THIS IMAGE YOU CAN SEE THIS UNIQUE COMBINATION OF MUSCLE ACTIVITY WHERE CERTAIN ELECTRODES ARE SLOWLY INCREASING AS THEY ARE EXTENDING THEIR MIDDLE FINGER, DEEP NEURAL NETWORK GOAL TO UNDERSTAND SPATIAL TEMPORAL RELATIONSHIPS BETWEEN IMAGES AND USE THOSE TO PREDICT COMPLEX MOVEMENT PATTERNS. SO, THE REALLY FUN THING IS AFTER ABOUT TWO OR THREE MINUTES OF COLLECTING DATA, AND TRAINING THE ALGORITHM, SO IT'S RELATIVELY A QUICK PROCESS, THE PERSON IS ABLE TO CONTROL A VIRTUAL HAND. WHAT'S INTERESTING HERE IS THAT EVEN THOUGH THE INDIVIDUAL'S HAND CANNOT PHYSICALLY MOVE IN THIS CASE THEY ARE ABLE TO ACTIVELY CONTROL THE VIRTUAL HAND. >>THAT LOOKS AWESOME. >>YEAH. >>AS YOU CAN IMAGINE BEING ABLE TO SEE YOUR HAND MOVE AGAIN AFTER HAVING IN THIS CASE A STROKE CAN BE PRETTY INSPIRING. SO THERE ARE A FEW WORDS FROM THE PARTICIPANT. >>THIS IS WHAT I WAS BEFORE! YEAH! SO I SHOULD DO MORE, MORE EFFORT, TO HAVE BETTER. IT MAKE ME HAPPY. BECAUSE THERE'S SO MANY REASONS, EVEN I WANT I DO THERAPY, AND I REALLY DO TO GET BETTER BUT SOMETIMES THE PROGRESS IN YOUR BRAIN IS SLOWER THAN WHAT YOU CAN DO. >>YEAH, DEFINITELY. >>ALL THESE THINGS HELP YOU TO MOTIVATE YOU AND HELP YOU TO GET BETTER. >>SO, WHILE OUR IMMEDIATE GOAL OF THIS TYPE OF SYSTEM IS TO RESTORE FUNCTIONALITY, IN ASSISTIVE CAPACITY, THIS PATIENT RIGHTLY POINTS OUT THE VISUAL FEEDBACK BY A SYSTEM LIKE THIS IS GREAT MOTIVATING FACTOR FOR PATIENTS TO CONTINUE THERAPY AND REHABILITATION. SO, USING THIS VIRTUAL BIONIC HAND IS A TEST BED TO BEGIN WITH. WE'VE DEMONSTRATED THAT PATIENTS ACTUALLY HAVE SIMILAR CONTROL WITH THEIR PARETIC HAND AS CONTRALATERAL LIMB. WE USE A TARGET-TOUCHING TASK. A PARTICIPANT IS ACTIVELY CONTROLLING THIS VIRTUAL HAND, TRYING TO MATCH A DESIRED HAND POSITION. SO SPHERE IS TURNING RED, PARTICIPANT MUST MOVE INTO THE SPHERE TO MAKE THE SPHERE TURN GREEN. DATA FROM THIS TASK WOULD LOOK LIKE THIS, WHERE YOU HAVE A TARGET PLOTTED IN BLACK, AND WE HAVE THIS KIND OF AIR WINDOW AROUND IT IN GREEN. ON THE Y-AXIS IS THEIR PERCENT OF FLEXION, HOW MUCH THEY ARE FLEXING THEIR HAND. AND THEN ON THE X-AXIS IS TIME. AND SO AS WE WOULD EXPECT, PARTICIPANTS CAN DO THE TASK WELL WITH THE HEALTHY ARM, SHOWN IN THE GREEN TARGET WINDOW. BUT WHAT'S REALLY IMPRESSIVE IS HOW WELL THEY CAN DO THE SAME TASK WITH THEIR PARETIC LIMB. SO OF PARTICULAR IMPORTANCE HERE THE CONTROL THEY ARE PROVIDING IS PROPORTIONAL, NOT JUST A BINARY OPEN AND CLOSE SIGNAL BUT THE PATIENT'S MODULATING THE POSITION OR FORCE OUTPUT OF THE HAND IN REAL TIME. AND WE CAN QUANTIFY PERFORMANCE, LOWER ARM C VALUE IS REPRESENTATIVE OF LESS AIR, THEREFORE BETTER PERFORMANCE, WE'RE LOOKING AT 7 PARTICIPANTS IN THIS CASE, WE CAN SEE PERFORMANCE IS, YOU KNOW, QUITE SIMILAR BETWEEN GRASPING MOTIONS, BUT AS YOU START TO LOOK TOWARDS EXTENSION MOTIONS SUCH AS OPENING YOUR HAND, THE PERFORMANCE TRENDS TOWARDS BEING WORSE WITH PARETIC SIDE. THIS NATURALLY MAKES SENSE. IF YOU WORKED WITH PATIENTS WITH SPASTICITY THEY HAVE A HARDER TIME OPENING THEIR HAND THAN THEY DO FLEXING THEIR HAND. SO, AS AN EXTENSION TO THIS WORK, OUTSIDE OF OUR NIH RESEARCH THAT WE'RE WORKING ON NOW WE'VE BEEN PARTNERING WITH META, FORMERLY KNOWN AS FACEBOOK, TO HELP DEVELOP INCLUSIVE META VERSE WHERE ALL INDIVIDUALS CAN INTERACT WITH THE META VERSE SEAMLESSLY REGARDs WILL THE OF PHYSICAL DISABILITY, SUCH AS HEMIPARESIS, THEY ARE RECORDING EMG FROM WRIST OR FOREARM TO CONTROL A VIRTUAL HAND, YOU CAN INTERACT WITH OUR OBJECTS, OUR GOAL IS TO TEST WITH MUSCULAR DISABILITIES AND FIND WAYS WE CAN STRATEGICALLY BUILD OUT THE META VERSE SO IT'S INCLUSIVE TO PATIENTS. AND I THINK THERE'S A LOT OF POSSIBILITIES AND FUTURE EXPLORATION IN THIS VIRTUAL REALITY SPACE FOR BOTH ASSISTIVE TECHNOLOGY AS WELL AS REHABILITATIVE THERAPY. MOVING BEYOND VIRTUAL REALITY, WE'RE WORKING WITH MYOMO TO IMPLEMENT PROPORTIONAL CONTROL OF POWERED EXOSKELETON DEVICE. WE HAVEN'T TESTED THIS PART YET WITH PATIENTS, BUT WE HAVE BEEN ABLE TO MODIFY THE FIRMWARE ON THE DEVICE, NOW WE CAN PRECISELY CONTROL THE DEVICE AND REGULATE FORCE OUTPUT, SHOWN IN THIS VIDEO IS ONE OF MY STUDENTS USING EMG CONTROL FROM THEIR ARM WHICH IS ON THE LEFT, TO CONTROL ANOTHER INDIVIDUAL'S ARM, SHOWN ON THE RIGHT. SO ULTIMATELY WE HOPE THAT THE IMPROVEMENTS IN CONTROL WE CAN PROVIDE WILL HELP PATIENTS COMPLETE MORE TASKS IN THEIR DAILY LIVES. AND THE FACT THAT PATIENTS ARE USING THIS DEVICE DAILY ALSO PROVIDES THE UNIQUE OPPORTUNITY FOR US TO ANALYZE LARGE DATASETS OF EMG OVER TIME. SO IF WE GO BACK TO THE ORIGINAL IDEA OF MACHINE LEARNING ALGORITHM, TYPICALLY WOULD COLLECT DATA, TRAIN A COMPUTER, AND THEN CONTROL THE ARM. AND YOU MIGHT GO THROUGH AND RECALIBRATE THIS, YOU KNOW, PATTERN RECOGNITION SYSTEM EVERY ONCE IN A WHILE TO GET THE BEST PERFORMANCE. BUT IF PATIENTS ARE USING THIS DAILY THAT RAISES THE QUESTION INSTEAD OF RECALIBRATING AND STARTING OVERING WITH NEW DATA CAN WE SIMPLY ADD MORE DATA IN THE SYSTEM OVER TIME AND BY CONTINUALLY ADDING DATA CAN THE COMPUTER IMPROVE AND ALSO CAN THE USER LEARN TOGETHER WITH THE COMPUTER TO IMPROVE? OUR APPROACH TO DO THIS IS LOOKING AT MORE BIG DATA AND DEEPER LEARNING SYSTEMS. SO FAR WHAT WE'VE SEEN IS THAT THE ERROR ASSOCIATED WITH MACHINE LEARNING ALGORITHM WE USE DECREASES AS WE COLLECT MORE DATA. WE'RE LOOKING AT THE SQUARE ERROR ON THE X-AXIS, IT DOESN'T MATTER WHAT ALGORITHM WE USE, AS THE NUMBER OF TRAINING DATASETS INCREASES YOU SEE THE ERROR DECREASES AND SO IN OTHER WORDS MORE DATA IS STARTING TO LEAD TO BETTER AND BETTER ALGORITHM PERFORMANCE. BUT IT GOES BOTH WAYS. NOT ONLY DOES THE MACHINE IMPROVE BUT THE HUMAN CAN IMPROVE IN THIS CASE, SO IN A CLOSED LOOP TASK THE PARTICIPANTS IS ACTIVELY MODULATING INPUT TO THE ALGORITHM, EMG ACTIVITY, CHANGING THE OUTPUT. WE'VE SEEN THE INDIVIDUALS CAN GET BETTER AND BETTER SO LOOKING AT THIS VIRTUAL TARGET TOUCHING TASK, LOOKING AT THE METRIC, THE WHOLE TIME, HOW LONG CAN THEY STAY WITHIN THE GREEN SPHERE AND WE'VE SEEN THAT PARTICIPANTS PERFORMANCE CAN INCREASE DRASTICALLY OVER TIME. AS THIS PERSON IS USING THIS DEVICE DAILY, LEARNING TO CONTROL MUSCLE ACTIVITY BETTER, LEARNING TO CONTROL THE DEVICE BETTER EACH DAY. ULTIMATELY, THIS DEVICE IS SERVING AS A BIOFEEDBACK SYSTEM TO PROMOTE ACTIVE USE OF THE ARM EACH DAY, AND THIS RAISES THE IDEA OF MONITORING EMG OVER TIME, THROUGHOUT STROKE RECOVERY, TO PROVIDE A QUANTITATIVE MEASURE OF THEIR IMPROVEMENT AND THEIR RECOVERY. SO CURRENTLY THE MODIFIED SCALE IS PRIMARY INCLUSION CRITERIA FOR ASSISTIVE EXOSKELETON DEVICE, MOST OF YOU ARE FAMILIAR WITH THE MODIFIED SCALE AND LIMITATIONS, BUT THE DEVICE REQUIRES AN MAS FOR 2 OR LOWER AT THE HAND, 3 OR LOWER AT THE ELBOW TO QUALITY FOR USING THIS. OUR GOAL IS TO SEE IF WE CAN REPLACE THIS SIMPLE DISCRETE QUALITATIVE SCALE WITH CONTINUOUS AND QUANTITATIVE MEASURE DERIVED FROM EMG WHICH ULTIMATELY IS THE CONTROL SIGNAL. SO WE ASK PARTICIPANTS TO GRASP AND HOLD FOR THREE SECONDS WHILE WE RECORD EMG FROM FOREARM, SAME TASK WHERE THEY WERE EXTENDING THE HAND, AND THEN WE STARTED LOOKING AT THE EMG PROFILES AS THEY WERE DOING THIS TASK, AND WE SAW A SHARP INCREASE IMMEDIATELY AFTER STARTING CUE. YOU SEE A LONG DAY BEFORE THE EMG RETURNS TO BASELINE. IT TAKES THE SPASTIC PARETIC HAND LONGER TIME TO RELAX THAN IT WOULD THE CONTRALATERAL IN THE HEALTHY HAND. SO WE STARTED CALCULATING TIME CONSTANT, WE DETERMINED ROUGHLY 36.8% OF DECREASE, TIME IT TAKES TO DROP TO THAT POINT. AND IF WE LOOK AT AGGREGATE DATA, IN THIS CASE FOUR PARTICIPANTS SO FAR, WE FOUND ALREADY SIGNIFICANT DIFFERENCE BETWEEN TIME CONSTANT IN PARETIC AND HEALTHY HANDS, REEMPHASIZING PARETIC ARMS TAKE LONGER TO RELAX EMG BACK TO BASELINE. IF WE LOOK AT EACH PATIENT INDIVIDUALLY WE START TO SEE THIS INCREASING TREND WHERE THE HIGHEST TIME CONSTANT OCCURS AT HIGHEST MAS SCORE, WE HAVE TIME CONSTANT ON THE Y-AXIS, EACH PARTICIPANT IS ON THE X-AXIS, ARRANGED SO THE PATIENT ON THE LEFT HAS THE LOWEST SPASTICITY, ON THE RIGHT THE HIGHEST SPASTICITY, BASED OFF OF MAS SCALE. SO, THE DIFFERENCE IN TIME CONSTANT BETWEEN THE HEALTHY AND PARETIC ARMS IS PRESENT AS SOON AS YOU GET ABOVE MAS SCORE OF 1, AND TIME CONSTANTS INCREASE WITH INCREASING MAS. AND INTERESTINGLY WE ALSO SEE THIS BILATERAL EFFECT TOO, WHERE HEALTHY CONTRALATERAL ARM HAS A LONGER TIME CONSTANT THAT INCREASES WITH SPASTICITY ON THEIR CONTRALATERAL SIDE. I THINK THERE'S TWO EXPLANATIONS, ONE, THERE'S EVIDENCE THAT SAYS CONTRALATERAL SIDE IS NOT HEALTHY AFTER A STROKE, OTHER STIMULUS -- SYSTEMS LEVEL THING LIKE COGNITION OR REACTION TIME THAT WE'RE LOOKING AT AS WELL. WE'RE ENVISIONING THESE WEARABLE MYOELECTIVE DEVICES LIKE POWERED ORTHOSIS AS A PLATFORM FOR PATIENT CENTERED REHABILITATION, FITTED AFTER LEAVING THE CLINIC, IMMEDIATELY GETTING BACK TO ACTIVITIES OF DAILY LIVING, AND DEVICE WOULD BE PROMOTING USAGE AND STRENGTHENING MUSCLES, PROVIDING REALTIME BIOFEEDBACK SO THE PATIENT AND MACHINE WOULD BE LEARNING TOGETHER AS THE PATIENT RECOVERS AND WE WOULD HAVE THESE BUILT-IN DIAGNOSTIC MEASURES THAT WOULD HELP ASSESS A PATIENT'S RECOVERY OVER TIME AND GUIDE THEIR THERAPY, SO WE HAVE A LONG WAY TO GO FOR THIS VISION BUT WE'RE EXCITED ABOUT THE POTENTIAL OF THESE BIONIC DEVICE WAS BIOLOGICAL SIGNALS AS THEIR CONTROL SIGNAL TO ASSIST IN REHABILITATION OF PATIENTS. AND I'D LIKE TO THANK EVERYONE WHO HAS MADE THIS RESEARCH POSSIBLE INCLUDING FUNDING FROM NIH AND OTHER FUNDING AGENCIES, UNIVERSITY COLLABORATORS AND OUR INDUSTRY SPONSORS. THANK YOU ALL FOR LISTENING. >>THANK YOU, DR. GEORGE. WE'LL HOLD QUESTIONS AND GO STRAIGHT TO DR. GATES' TALK. >>THANK YOU FOR THE INTRODUCTION. MY LAB AT THE UNIVERSITY OF MICHIGAN FOCUSES ON UNDERSTANDING AND IMPROVING HUMAN FUNCTION WITH ASSISTIVE TECHNOLOGY. SO WE LOOK AT THIS FROM A VARIETY OF DIFFERENT DIRECTIONS, IN TERMS OF THE DESIGN OF THE DEVICE, CHARACTERISTICS OF THE INDIVIDUAL, HOW IT'S CONTROLLED AND HOW IT INTERFACES WITH THAT INDIVIDUAL. I'M GOING TO FOCUS ON ONE SERIES OF STUDIES, PART OF AN NIH-FUNDED R03 THROUGH NCMR. SO, LET ME SEE, CAN YOU ADVANCE THEM? I CAN'T DO THAT EITHER. YEAH, SO, THE ANIMATIONS ARE NOT GOING WORK. MAYBE JUST CLICK THROUGH. YEAH. I WONDER IF I COULD SHARE ON NON-PRESENTATION MODE. I DON'T KNOW IF THAT WOULD WORK AT ALL. BUT IT'S GOING TO BE A LITTLE CHALLENGING TO DO IT -- >>I JUST TRIED TO GIVE YOU CONTROL, DOES THAT WORK? >>LET'S SEE IF I CAN DO ANYTHING. OKAY. IT WILL ADVANCE THEM. GREAT. OKAY. SO, THERE IS A NUMBER OF DIFFERENT ASSISTIVE TECHNOLOGIES THAT CAN IMPROVE WALKING FOR PEOPLE WITH LIMB LOSS. IN PARTICULAR, I'M JUST GOING TO SKIP OVER SOME OF THESE. THERE'S ONE POWERED PROSTHESIS THAT WE'VE STUDIED EXTENSIVELY IN MY LAB CALLED THE BIOM, NOW THE OTTOBOCK EMPOWER, PROVIDES ACTIVE WORK TO THE INDIVIDUAL AS THEY MOVE. SO, MOST DEVICES ARE PASSIVE SUCH THAT THEY COMPRESS WHEN THE FOOT IS THE GROUND, RELEASE THE ENERGY, BUT DON'T GIVE ACTIVE WORK. BY DOING THIS, THIS DEVICE HAS ENABLED PEOPLE TO WALK WITH LOWER METABOLIC COST AND INCREASE PREFERRED SPEED. SOME WORK OUT OF MIGHT HAVE LAB HAS SHOWN PEOPLE HAVE INCONSISTENT FINDINGS, WE SEE SOME PEOPLE WHO IMPROVE WITH THIS, IN TERMS OF REDUCING METABOLIC COST BUT MANY, MANY OTHERS WHO DON'T. ONE OF THE OPEN QUESTIONS IN THIS AREA IS REALLY HOW DO WE TUNE A DEVICE TO AN INDIVIDUAL? AND THERE'S MANY DEVICES, NOT JUST THIS ONE, THAT HAVE TO BE ADJUSTED TO THE PERSON. AND AS YOU CAN IMAGINE, THE MORE COMPLEX THE DEVICE, THE MORE BUTTONS YOU HAVE TO TWIST AND KNOBS TO MAKE IT FUNCTION FOR THAT PERSON. HOW DO WE ADJUST THESE PARAMETERS? WHICH PARAMETERS MATTER? IF WE FOCUS ON THIS DEVICE, YOU CAN SEE THIS IS A PICTURE OF OUR PATIENT WHO IS WALKING ACROSS THE LAB, HE'S IN THE CLINIC, HE'S WEARING THE PROSTHESIS, WE HAVE OUR PROSTHETIST WITH A TABLET. ON THAT TABLET HE SEES THIS GRAPH. SO WHAT HE'S DOING IS CHANGING THE DIAL TO CHANGE PARAMETERS, SEES A GREEN DOT ASSOCIATED WITH EVERY STEP THE PERSON TAKES, IN THIS CASE THESE DASHED LINES ARE 95% CONFIDENCE INTERVALS, GREEN IS GOOD RANGE, DOING SOMETHING THAT'S SIMILAR TO WHAT A HEALTHY ANKLE DOES. RED DOTS INDICATE POOR PERFORMANCE, TOO MUCH POWER. IN THIS CASE THOSE ARE TURN STEPS, WHEN THE PERSON TURNS YOU GET MORE POWER THAN EXPECTED FOR THE SLOW SPEED. THIS IS ONE PARAMETER. HE CAN SAY HOW MUCH POWER DO I GIVE? BUT THERE ARE A MULTITUDE OF OTHER PARAMETERS THAT CAN BE CHANGED, AND IT'S NOT REALLY CLEAR WHICH OF THE PARAMETERS MATTER, WHICH SHOULD BE CHANGING AND HOW DO WE IMPLEMENT THOSE CHANGES? AND SO WE DID A STUDY WHERE WE FOCUSED SPECIFICALLY JUST ON THE POWER SETTING. YOU CAN GO FROM ZERO TO 100%, THAT'S YOUR GAIN FACTOR. THAT'S SHOWN HERE. THE PC REPRESENTS THE VALUE THAT THE PROSTHETIST WOULD CHOOSE BASED ON VISUAL GUIDELINE THEY GET. SO THEY ARE JUST WATCHING THE PATIENT WALK, MANIPULATING, LOOKING FOR GREENISH DOTS TO SAY THIS IS RELATIVELY NORMAL. AND SO WHAT WE FOUND IN THIS STUDY IS THAT IF WE INCREASE THE AMOUNT OF POWER THAT'S DELIVERED TO THE INDIVIDUAL, WE CAN REDUCE THE METABOLIC EFFORT ASSOCIATED WITH THEIR WALKING. NOW, WHAT WE SEE ON THE INDIVIDUAL BASIS IS THAT THE WEARER THAT MINIMUM OCCURS IS NOT THE SAME FOR ALL INDIVIDUALS. SO FOR SOME PEOPLE, HAVING, YOU KNOW, POWER AT THE 75% LEVEL IS THE BEST. AND ONCE YOU GET BEYOND THAT, IT ACTUALLY IS HARDER FOR THEM TO WALK. WHEREAS FOR OTHERS, THEY HAVE THIS PATTERN CONTINUALLY GOING DOWN SO YOU WANT TO MAX OUT POWER AND THAT'S GOING TO HELP THEM MINIMIZE THEIR EFFORT. AND SO IT'S CLEAR IT DOES MATTER, THERE IS SOME SETTING FOR THIS YOU WITH DO BETTER THAN WHAT THE PROSTHETIST WOULD DO BASED ON CRITERIA THEY HAVE AVAILABLE TO THEM. WE CAN BE DOING BETTER. WHAT THAT BETTER LOOKS LIKE IS INDIVIDUAL SPECIFIC. SO, ONE OF THE COMPLICATIONS IS, OKAY, THIS IS GREAT, YOU CAN DO THIS, FIGURE OUT WHAT THE VALUE IS FOR EACH PERSON. HOWEVER, IT'S A TIME INTENSIVE PROCESS. WE'RE HAVING A PERSON WALK AT A CONSTANT SPEED, OFTEN ON A TREADMILL, FOR FIVE MINUTES UNTIL THEY GET TO STEADY STATE, AND YOU HAVE ANOTHER THREE MINUTES OF WALKING AT THAT STEADY STATE VALUE. AND THAT'S HOW YOU'RE GETTING THE ONE PLEASURE. AND SO WHEN YOU LOOK AT EVERY ONE OF THOSE MEASURES FROM THAT STUDY, THAT WAS 48 MINUTES OF WALKING. AND SO YOU CAN IMAGINE THAT FOR PEOPLE WHO HAVE A DISABILITY, THIS IS PRETTY LIMITING IN TERMS OF WHO YOU CAN ACTUALLY PUT IN THESE STUDIES AND HOW WOULD YOU IMPLEMENT THIS CLINICALLY. AND SO WE LOOKED AT -- THIS IS WORK WITH DAVID REMY, HOW DO WE DO THIS MORE QUICKLY? WE USED AN APPROACH CALLED HUMAN-IN-THE-LOOP OPTIMIZATION, MEASURE METABOLIC COST, MEASURING AMOUNT OF OXYGEN THEY ARE USING WITH A MASK. IF WE JUST LOOK AT THIS DATA DID IT MAKE IT BETTER OR WORSE? LOOKING AT THE GRADIENT. AND THEN HOW CONFIDENT ARE YOU IN THAT. SO PRETTY CONFIDENT THIS MADE THIS WORSE SO LET ME CHANGE THE PARAMETER IN THE OPPOSITE DIRECTION. OR I'M PRETTY CONFIDENT IT MADE IT BETTER, LET ME CHANGE IN THE SAME DIRECTION AND SEE WHAT IT DOES. THIS ALGORITHM IS CONTINUALLY CHANGING AS THE PERSON MOVES. AND WE FIND WITH THIS WE CAN ACTUALLY PREDICT THESE MINIMUM VALUES, FASTER. AND SO YOU CAN SEE THE SOLID DOTS REPRESENT ALL OF THOSE INDIVIDUAL PARAMETERS, FROM THE PRIOR STUDY. SO THIS IS ONE INDIVIDUAL IN WHICH WE CHANGED THE TIMING, SO WHEN THE POWER WAS DELIVERED, AND THEN THE AMOUNT OF POWER THAT THEY WERE GETTING. AND SO YOU CAN SEE AT THESE -- AT THE SHORTER TIMING, IT DIDN'T REALLY MATTER, WHY THE LINE IS SO LONG IS WE DON'T HAVE A LOT OF CONFERENCE. PARAMETER DATA SUPPORTS THAT TOO. YOU CAN PRETTY MUCH -- AS LONG AS YOU DO IT EARLY ENOUGH, YOU'LL BE BENEFITING FROM THE POWER. IF YOU DO IT TOO LATE YOU DON'T GET AT THE POWER. IN TERMS OF THE AMOUNT OF POWER, YOU CAN SEE IT HAS A SMALLER CONFIDENCE BAND, AND IT REALLY ACCURATELY PREDICTED FOR THIS INDIVIDUAL, 100% POWER IS THE BEST SETTING. SO, THE IMPORTANT THING HERE IS IT TAKES A THIRD OF THE TIME SO 13 MINUTES TO FIND THESE SETTINGS VERSUS IT WOULD TAKE 48 FOR EACH CONDITION BEFORE TO FIND THOSE VALUES. SO, ONE OF THE QUESTIONS WE HAVE IS REALLY HOW DO WE IMPLEMENT THIS APPROACH CLINICALLY? IF WE WANTED FOR INSTANCE THIS DEVICE NOT ONLY TO SET THESE SETTINGS INITIALLY BUT MAYBE WE WANT THIS TO SEND THEM HOME WITH IT, HAVE THEM ADAPT TO THE INDIVIDUAL AS THEY GAIN, THEY LOSE WEIGHT, DO DIFFERENT TASKS, IN ORDER TO DO THAT WE WOULD NEED A WAY TO MEASURE THE OUTCOME. SO, METABOLIC COST OR VO2 IS THE COMMON OUTCOME MEASURE BECAUSE IT REPRESENTS SOMETHING THAT'S WHOLE BODY RELATED, HOW EASY IS IT TO WALK WITH THIS DEVICE. THE PROBLEM IS AS YOU CAN IMAGINE THIS SYSTEM IS WHILE PORTABLE, PRETTY BULKY AND REALLY IMPRACTICAL FOR DAILY USE BECAUSE YOU HAVE A MASK THAT GOES OVER YOUR NOSE AND MOUTH, AND SO TAKING THAT HOME AND IMPLEMENTING IT IS NOT REALLY FEASIBLE. THE QUESTION WE HAD IS CAN WE ESTIMATE OUR METABOLIC COST USING ONLY WEARABLE SENSORS? AND SO THIS IS AN EXPERIMENT BY KIM INGRAHAM, TEN HEALTHY INDIVIDUALS WALKING WITH A VARIETY OF SENSORS ON DIFFERENT BODY PARTS SO YOU CAN SEE THIS CONSISTED OF VO 1 MASS TO GET THE GROUND TRUTH, HEART RATE, SKIN TEMPERATURE, EMG ACTIVITY, ACCELERATIONS, AND THEN THE INDIVIDUAL'S PERFORMING OF A VARIETY OF TASK AT MULTIPLE LEVELS GOING FROM STANDING TO RUNNING TO HIGH-SPEED INCLINE WALKING. SO THOSE TASKS REQUIRE VARYING AMOUNTS OF METABOLIC EFFORT. WE SPLIT THE DATA INTO SENSOR SETS, AND SO WE USE THE DATA THAT WAS ONLY FROM LOCAL SENSORS THAT FROM -- WE'RE CALLING LOCAL SENSORS THINGS ON DIFFERENT SEGMENTS, LIKE EMG OR ACCELEROMETERS. GLOBAL, THINGS THAT RELATE TO THE WHOLE BODY, SO SKIN TEMPERATURE, HEART RATE. LOCAL AND GLOBAL, WHICH IS ALL OF THE SENSORS. AND THEN SOMETHING CALLED THE HEXO SUIT THAT MEASURES BREATH VENTILATION AND ACCELERATION BECAUSE IF YOU'RE TRYING TO IMELEMENT CLINICALLY WE WANT TO SEE OF THE COMMERCIAL PRODUCTS IS THERE ANYTHING WE COULD USE TO DO THIS. WE PERFORMED SIGNAL PROCESSING AND THEN USE LINEAR REGRESSION WITH THE SENSOR DATA AS INPUT AND PREDICTED METABOLIC COST AS THE OUTPUT. SOME EARLY INSIGHTS, WE LOOKED AT THE RAW SIGNALS AND SO WE HAVE IN THIS GRAPH BLUE IS RAW DATA, AND FILTERED DATA. FOR ACCELERATION AND EMG FILTERING MADE A BIG DIFFERENCE, IF WE JUST LOOK AT ONE SIGNAL HOW WELL DOES IT CORRELATE WITH METABOLIC EFFORT. IN TERMS OF GLOBAL SIGNALS IT DIDN'T MAKE OF DIFFERENCE WHETHER YOU DID FILTERING OR NOT, LIKE THE HEART HEART RATE AND BREATH VENTILATION. WE SEE NOT SHOWN HERE METABOLIC COSTS CAN BE PREDICTED USING DATA FROM ONLY A FEW SENSORS OF DIFFERENT TYPES. SO IN TERMS OF WHICH FEATURES WERE MOST IMPORTANT, MINUTE VENTILATION HAD THE LARGEST CONTRIBUTING FACTOR, SO IF YOU BUILT A MODEL USING ONLY THAT WOULD GET PRETTY LOW AIR, IF YOU ADDED UP TO THREE MORE FEATURES COULD MINIMIZE AIR BEYOND THAT, DIDN'T MAKE MUCH OF A DIFFERENCE. IN THE BOTTOM PLOT YOU SEE THE ERROR AND OVER TIME, SO WHAT THIS IS SAYING IS HOW QUICKLY CAN WE PREDICT THE METABOLIC COST, AND MINIMIZE -- WHILE TRYING TO MINIMIZE THAT ERROR. SO THE SOLID LINES THERE REPRESENT THE DATA OF THE DIFFERENT SIGNAL SETS, SO LOCAL GLOBAL COMBINE, THE DASHED REPRESENTS WHEN YOU INCLUDE THE DERIVATIVES OF THOSE SIGNALS. WHAT THIS SHOWS IS IF YOU INCLUDE RATE OF CHANGE OF THOSE SIGNALS, YOU CAN ACTUALLY REDUCE THE ERROR FASTER SO WE CAN GET A MORE ACCURATE PREDICTION MORE QUICKLY. SO THAT SEEMS PRETTY GOOD. IN THIS CASE ERROR RATE ON THE ORDER OF 1 WATT PER KILOGRAM, ABOUT 10% OF THE RANGE OF THE METABOLIC COST WE MEASURED. SO THEN WE WANTED TO SEE IF MACHINE LEARNING COULD HELP US IMPROVE THESE PREDICTIONS, AND SO WHAT WE DID WAS USE THE SAME FOUR SIGNAL SETS AS USED PREVIOUSLY, BUT THEN COMBINED A BUNCH OF DIFFERENT APPROACHES. WE THEN DID LINEAR REGRESSION AGAIN, BUT ALSO USED FIVE DIFFERENT MACHINE LEARNING APPROACHES WITH THE SAME UNDERLINING DATA IN EACH OF THE DIFFERENT MODELS. SO TO TEST HOW WELL THESE MODELS PERFORMED WE DID THREE DIFFERENT TYPES OF CROSS-VALIDATION. SO THE FIRST ONE IS CALLED A K-FOLD. BASICALLY THIS IS WHERE YOU TAKE 80% OF YOUR DATA, TRAIN IT, AND THEN YOU TEST IT ON THE OTHER 20%. AND SO THIS WOULD BE A CASE WHERE IF WE WERE GOING TO IMPLEMENT THIS YOU BRING A PARTICIPANT INTO THE LAB, GET SOME TRAINING DATA FOR THEM, BUILD THE MODEL, SEND THEM HOME, SEE HOW THIS WOULD PERFORM. NEXT APPROACH WOULD BE LEAVE ONE SUBJECT OUT, THIS WOULD BE THE CASE WHERE ESSENTIALLY YOU BRING SOMEONE INTO THE CLINIC, GIVE THEM A SYSTEM TRAINED ON OTHER PEOPLE, SEE HOW IT WORKS FOR THEM WITHOUT ANY OF THEIR DATA BEING IN THE MODEL, SIMILARLY LEAVE THE ACTIVITY OUT, YOU'VE TRAINED THE DATA ON THIS INDIVIDUAL BUT YOU SEND THEM HOME AND THEY DO SOMETHING THAT YOU DIDN'T TRAIN ON THIS. WE DIDN'T DO EXHAUSTIVE LISTS OF TASKS SO THEY TRIED TO DO SOMETHING ELSE, HOW WELL DOES THE MODEL PERFORM FOR THEM. AND SO THIS ONE SHOWS HOW WELL IT PERFORMS FOR THREE TYPES OF CROSS-VALIDATION. THE TOP ONE IS OUR K-FOLD WHERE SOME DATA FROM THAT SUBJECT AND THAT ACTIVITY IS CONTAINED IN THE MODEL. AND SO YOU CAN SEE THERE ARE TREE MODELS, DECISION TREE AND RANDOM FOREST, REALLY GOOD, MUCH BETTER THAN LINEAR REGRESSION PREDICTING STEADY STATE METABOLIC EFFORT, REGARDLESS OF THE SENSOR SET. WE THEN GET INTO THE DATA WHERE WE DON'T HAVE DATA FOR THAT SUBJECT, OR DON'T HAVE DATA FOR THAT ACTIVITY, AND YOU THEN CAN SEE THAT BUILDING IN THESE MORE COMPLEX MODELS ACTUALLY DOESN'T IMPROVE THE PREDICTED OUTCOME, ROOT MEAN DOESN'T DECREASE RELATIVE TO PURPLE LINEAR REGRESSION, SIMPLE LINEAR REGRESSION IS PROBABLY THE APPROACH TO USE. SO THIS IS JUST ANOTHER ILLUSTRATION OF THAT. BUT YOU CAN SEE ESSENTIALLY THE BLACK LINE REPRESENTS THE STEADY STATE METABOLIC EFFORT FOR THESE DIFFERENT ACTIVITIES. THEY WEREN'T PERFORMED SEQUENTIALLY, I WILL SAY THEY ARE SPLICED TOGETHER BUT WEREN'T PERFORM THAT WAY BECAUSE EACH PIECE OF EQUIPMENT WASN'T IN THE SAME ROOM. BUT YOU CAN SEE THE BLUE REPRESENTS WHAT WE'RE PREDICTING. AND THE BLACK IS GROUND TRUTH. AND SO FOR THIS ONE PERSON, WHO IS IN THIS LEAVE ONE SUBJECT OUT VALIDATION, DATA IS NOT INCLUDED IN THE MODEL, LINEAR REGRESSION DOES PRETTY WELL. SIMILARLY OTHER APPROACHES. DECISION TREE AND RANDOM FOREST ARE POOR AT PREDICTING THAT INDIVIDUAL'S METABOLIC EFFORT. SO IN CONCLUSION WE FOUND BASICALLY MODIFYING DEVICE PARAMETERS CAN AFFECT AND INDIVIDUAL'S PERFORMANCE WITH THAT DEVICE. IT IS IMPORTANT TO MANIPULATE OR LIKE CHANGE THESE PARAMETERS FOR AN INDIVIDUAL IF YOU WANT TO MAXIMIZE THEIR PERFORMANCE. IN TERMS OF FUTURE WORK COULD USE FUTURE ENGINEERING TO IMPROVE PERFORMANCE OF THE MACHINE LEARNING APPROACHES, STILL AN OPEN QUESTION, WHAT IS A GOOD ENOUGH MEASURE, IS ONE WATT PER KILOGRAM SUFFICIENT? THE MORE DATA YOU ARE GET, THE MORE YOU SHOULD EXPECT THE ERROR WILL GO DOWN. I WOULD SAY THE LAST OPEN QUESTION IS REALLY WHILE WE DO CARE ABOUT METABOLIC COST, IS IT NECESSARILY THE BEST/ONLY FEATURE WE SHOULD BE OPTIMIZING FOR. AND SO THERE IS INSTEAD ESPECIALLY IN POPULATION LIKE THIS WHO HAS AN IMPAIRMENT, THEY MAY BE PRIORITIZING OTHER FACTORS, SO THAT THEY ARE PRIORITIZING REDUCING COSTS TO MAKE WALKING EITHER BUT ALSO FOR INSTANCE STABILITY, AND THOSE THINGS MAY HAVE DIFFERENT WEIGHT FOR DIFFERENT TASKS, AND SO IN THE FUTURE WE'VE COULD LOOK AT MULTI-OBJECTIVE COST FUNCTION INSTEAD. SO COMBINING SOME OF THESE FEATURES IN ORDER TO OPTIMIZE THE HOLISTIC VIEW OF THEIR PERFORMANCE. THIS SUGGESTS METABOLIC COST IS NOT NECESSARILY THE ONLY THING INDIVIDUALS CARE ABOUT, THIS WAS A TAKEHOME STUDY ABOUT PREFERENCE WITH DIFFERENT -- POWERED VERSUS UNPOWERED PROSTHESIS. THERE'S NOT AN OBVIOUS CORRELATION THAT THEY LIKE THE DEVICE ANY MORE, SO MORE FACTORS GO INTO THAT DECISION. I WANT TO THANK ALL OF MY COLLABORATORS, IN PARTICULAR DAVID REMY, CO-P.I. ON THESE GRANTS I TALKED ABOUT TODAY. AND THEN OF COURSE NIH FOR FUNDING THE WORK, AND KIM INGRAHAM WHO DID A MAJORITY OF THE WORK I TALKED ABOUT TODAY. THANK YOU. >>THANK YOU, DR. GATES, FOR A LOVELY TALK. WE'LL NOW GO TO THE Q&A SESSION MODERATED BY DR. AGRIWAL. >>FIRST OF ALL, THANK YOU FOR JOINING THIS WORKSHOP. WE HAD TWO FANTASTIC PRESENTATIONS WITH DIFFERENT POINTS OF VIEWS, BUT IN SOME SENSE VERY MUCH IN COORDINATION WITH EACH OTHER. IF I COULD HAVE ANY -- PEOPLE IN THE AUDIENCE, YOU CAN UNMUTE YOUR SECH AND -- UNMUTE YOURSELF AND ASK A QUESTION OR PUT A QUESTION IN THE CHAT. WE'LL TRY TO MONITOR THAT AND BRING IT UP TO THE SPEAKERS. WHILE WE'RE WAITING FOR A QUESTION, LET ME SEE IF THERE'S ANYBODY WHO HAS A QUESTION FROM THE AUDIENCE. SO, PERHAPS MAYBE I COULD START THE QUESTIONS HERE. FIRST OF ALL, DEANNA, VERY NICE PRESENTATION. AND THE WAY YOU WERE TUNING THE PARAMETERS FOR THE DIFFERENT ACTIVITIES. IN I WAS TO GO MORE PHILOSOPHICAL, IF I THINK OF AN ACTIVITY, LET'S SAY YOUR AMPUTEE WANTS TO USE IT FOR A VARIETY OF TASKS, RIGHT? THEY MAY NOT -- A DAILY TASK IS NOT NECESSARILY WALKING UP STAIRS OR WALKING ON THE FLAT FLOOR OR CLIMBING DOWN THE STAIRS. A MIX OF THESE ACTIVITIES, RIGHT? IN DAILY LIFE YOU DO MANY OF THESE ACTIVITIES IN COORDINATION AND SO FORTH. SO WHEN YOU TUNE YOUR SYSTEM, LET'S SAY WITH ONE OR SOME OF THESE ACTIVITIES, WOULD YOU CONSIDER THOSE PARAMETERS AS STATIC PARAMETERS OR DO YOU HAVE TO SORT OF TUNE THE PARAMETERS DYNAMICALLY AS A PERSON IS DOING THE TASK? >>YEAH, I MEAN, SO THE HOPE HERE WOULD BE IT WASN'T CLASSIFICATION OF ACTIVITIES BUT IT WOULD BE LIKE JUST MEASURING METABOLIC EFFORT,ING -- NO, SIR -- AGNOSTIC TO THE ACTIVITY. THE HOPE IS THIS THING IS REQUIRING I USE MORE ENERGY, I SHOULD MODIFY PARAMETERS BASED ON THAT. IT TOTALLY DEPENDS ON WHAT THE DEVICE IS. IF YOU USE ENERGY BECAUSE YOU'RE ON A BICYCLE WHERE YOU DON'T WANT A LOT OF ANKLE POWER, THAT MATTERS. THERE HAS TO BE A COMBINATION OF FACTORS THAT GO INTO THE DECISION. >>THANK YOU. AND READING THE FIRST QUESTION IN THE CHAT FROM MIRIAM. DID YOU USE THE RAW DATA FOR ALL IN YOUR MACHINE LEARNING MODELS. >>IT WAS NOT RAW DATA. IT WAS ALL PROCESSED. ALL THE DATA THEN WAS FILTERED. AND THEN IT WAS DOWN-SAMPLED TO BE AT THE SAME SAMPLING RATE, THE BEST WE COULD DO WAS 1 HERTZ, EVEN THOUGH A LOT OF INITIAL SIGNALS WERE AT A FASTER RATE THAN THAT. >>UH-HUH. THANK YOU. MAYBE WHILE WE'RE WAITING, THERE'S A SECOND QUESTION HERE. NO, OKAY. PERHAPS MAYBE I CAN HAVE A QUESTION FOR DR. GEORGE. THANK YOU SO MUCH FOR THIS PRESENTATION THAT YOU DID. NOW, VERY INTERESTING TO SEE -- I MEAN, YOU HAD THIS VISUAL OF THE VIRTUAL ARM, VIRTUAL HAND, AND WHILE THE -- SO WERE THE PATIENTS ABLE TO MOVE THEIR HAND AS WELL OR WERE THEY JUST VISUAL OF THE VIRTUAL HAND THEY WERE ABLE TO SEE AND EXECUTE? >>YEAH, THANKS FOR THE QUESTION. IT DEPENDS ON THE INDIVIDUAL PATIENT. IN THIS CASE WE WERE WORKING WITH A PATIENT WITH, YOU KNOW, MAS OF AROUND 3, MAYBE EVEN LIKE CLOSE TO 4. AND IN THAT CASE THEY REALLY COULDN'T MOVE THEIR HAND MUCH AT ALL. THE INTERESTING THING WE COULD STILL PICK UP, YOU KNOW, TRACE EMG ACTIVITY OF THEM ATTEMPTING AND COULD DISTINGUISH FROM WHEN THEY WEREN'T TRYING TO MOVE OR WHEN THEY WERE TRYING TO DO DIFFERENT TYPES OF MOVEMENTS. FOR THOSE PATIENTS, YOU KNOW, THERE'S GOING TO BE STILL THIS, YOU KNOW, PASSIVE RESISTANCE OF THEIR HAND, KIND OF THE SPASTICITY COMPONENT THAT WOULD MAYBE MAKE IT HARD FOR THEM TO USE AN EXOSKELETON, BUT WITH THE VIRTUAL REALITY THEY CAN STILL SEE THEIR HAND MOVING, THAT CAN STILL PROVIDE, YOU KNOW, A CONTROL SIGNAL FOR, YOU KNOW, OTHER ASSISTIVE DEVICES. BUT, YEAH, WITH OTHER PATIENTS THAT HAVE LOWER MAS SCORES, WE OF THEN SEE THEY CAN SLIGHTLY MOVE THEIR HAND. THEY JUST DON'T HAVE ENOUGH STRENGTH TO REALLY HOLD OBJECTS FOR EXTENDED PERIODS OF TIME AND DO MORE DEXTEROUS TASKS. >>YEAH, THANK YOU SO MUCH. MAYBE WHILE WE'RE WAITING FOR MORE QUESTIONS FROM THE GROUP, I HAVE ANOTHER QUESTION HERE. SO, MOST DEVICES THAT WE BUILD FOR LET'S SAY ASSISTANCE OR REHABILITATION, I MEAN, THE DEVICE IS THE SAME BUT THE GOALS ARE DIFFERENT AND WE WANT TO SORT OF USE THIS -- I MEAN, THE WAY I SEE REHABILITATION, YOU WOULD BE ABLE TO TRANSLATE TO THE ACTIVITY, SAY, WITHOUT THE DEVICE TO BE USED LATER ON. SO, WHEN YOU ARE GOING TO USE THIS DEVICE FOR TRAINING OR REHABILITATION WHAT WOULD YOU CHANGE IN ORDER TO BE ABLE TO DO THAT? >>SORRY, CAN YOU SAY THAT ONE MORE TIME? >>SO WOULD THE TRAINING -- SO WITH THE REHAB, WHICH WOULD BE PERHAPS OVER MUCH OF A TRAINING SESSION, HOW WOULD THOSE SESSIONS LOOK DIFFERENT FROM WHAT YOU'RE DOING RIGHT NOW? >>YEAH, IT'S A REALLY INTERESTING QUESTION. SO, THERE'S KIND OF ONE BIG CONCERN WITH USING MACHINE LEARNING, AND THIS IS BECAUSE AS A PERSON IS RECOVERING, THEY ARE CHANGING THEIR NEUROMUSCULAR ACTIVATIONS AS THEY ARE RECOVERING FROM A STROKE IN PARTICULAR. IF THERE'S A CONCERN IF YOU START EARLY ON USING, YOU KNOW, A LOT OF MACHINE LEARNING, AND REALLY KIND OF FOCUS IN ON LIKE THIS IS THE PATTERN THAT ACTIVATES THIS TYPE OF THING, THEN YOU COULD END UP ENCOURAGING A PERSON TO PROMOTE BAD HABITS, RIGHT? SO IF THEY ARE DOING THIS WEIRD CO-CONTRACTION THAT CAUSES THE HAND TO MOVE THEY COULD CONTINUE TO PRACTICE THAT, AND THAT WOULD NOT BE EFFECTIVE FOR REHABILITATION APPROACH. SO, THAT'S ACTUALLY ONE OF THE AIMS THAT WE'RE STILL WORKING ON IN OUR GRANT BUT I DID NOT TALK ABOUT HERE. BUT THE BASIC IDEA IS WE ENVISION DOING A SHARED CONTROL STRATEGY WHERE YOU COULD KIND OF BE SWITCHING BETWEEN REHABILITATIVE MODES AND TITRATING SO THE END GOAL IS MAYBE, YOU KNOW, ASSISTIVE MODE WOULD BE TO PROMOTE THE PERSON TO DO THE TASK EFFECTIVELY, IN REHABILITATION TO PROMOTE CORRECT USAGE, CREATING MODELS OF WHAT HEALTHY HAND ACTIVATION PATTERN LOOKS LIKE AND SLOWLY GETTING THE PERSON TO FOLLOW INTO THOSE PATTERNS. BEYOND THAT, JUST THE FACT THAT THEY ARE USING THEIR ARM AGAIN AND TRYING TO SEE THIS AND HAVING THIS VISUAL FEEDBACK CAN ALSO BE A REHABILITATIVE EFFECT. >>THANK YOU. THE NEXT FROM THE CHAT, FROM TOYIN, ANYTHING THAT -- HAVE YOU USED SOME OF THE FREQUENCY DEMAIN ATTRIBUTES WITHIN -- AND HOW DO THOSE CORRELATE WITH THE RESULTS THAT YOU'RE SEEING? THIS IS A QUESTION FOR BOTH OF YOU, EITHER, IF YOU CAN ANSWER, OR BOTH OF YOU CAN ANSWER. >>WE DID NOT IN WHAT I PRESENTED, LOOKING AT FUTURE ENGINEERING, IT DOES SEEM TO MATTER FOR EMG, SOMETIMES THE FREQUENCY INFORMATION IS MORE USEFUL. I DON'T HAVE ANY OF THAT IN HERE BUT YEAH, IT DEFINITELY SEEMS TO CHANGE HOW MUCH THAT CONTRIBUTES. >>AND SAME IDEA HERE, WE'VE DONE SOME FREQUENCY ANALYSIS ON EMG AS WELL. THERE'S BEEN SOME INTERESTING STUDIES THAT HAVE SHOWN AS A PERSON FATIGUES, THE FREQUENCY, THE POWER DISTRIBUTION, MEDIAN FREQUENCY FOR THAT POWER DISTRIBUTION STARTS TO SHIFT AND DECREASE AS A PERSON FATIGUES. AND WE'VE BEEN REALLY INTERESTED IN SEEING IF WE CAN INCORPORATE THAT INTO OUR CONTROL ALGORITHMS BECAUSE WE OFTEN SEE THAT AS A PERSON FATIGUES THEIR ACTIVATION PATTERNS CHANGE YOUR MACHINE LEARNING STARTS TO FAIL, IT'S NOT AS ROBUST OR WORKING AS ACCURATE. IF WE CAN INCORPORATE THAT AS A FEATURE THAT MIGHT ALLOW US TO CAPTURE SOME OF THAT INTO MACHINE LEARNING MODELS, AN AREA WE'RE ACTIVELY EXPLORING. >>THIS IDEA OF FATIGUE IS REALLY IMPORTANT. I THINK THE FREQUENCY CHANGES ARE NICELY CORRELATED WITH SOME FATIGUING MODELS. NEXT QUESTION, FROM DR. TAKAHASI, FOR DEANNA. WHEN YOU INCREASE YOUR POWER, I MEAN THAT CORRELATED WITH A BETTER METABOLIC COST, AND SO WHAT YOU OBSERVE IS THAT MANY OF YOUR SUBJECTS HAD 100% POWER. IS THAT -- SO IS THE HIGHER THE POWER THE BEST FROM METABOLIC COST PERSPECTIVE? >>YEAH, FOR CERTAIN PEOPLE, THEY DEFINITELY HAD THE POWER, IF YOU MAXED OUT THE POWER, THEY DID THE BEST. I'D SAY SOMETHING ABOUT IF YOU COULD INCREASE EVEN FURTHER WOULD IT FURTHER REDUCE IT, I HAVE TO BELIEVE YES, BUT SINCE THERE'S A GAIN FACTOR THAT'S ALL WE CAN DO IS GET TO THAT POINT. BUT ANECDOTALLY, THE PEOPLE AT 100 AND NOT, WHETHER YOU'RE WILLING TO SORT OF RIDE IT OUT. THIS IS THE THING YOU CAN MOSTLY DO ON A TREADMILL, IF YOU LET THE FOOT DO WHAT IT WANTS TO DO, THAT'S WHAT SOME SAID, OTHERS FOUGHT IT AT A CERTAIN LEVEL, OH, THIS IS JARRING, AND SO IF YOU HAVE ANY CONCERNS ABOUT STABILITY I DON'T THINK YOU WANT TO LIKE RIDE THIS THING OUT AND LET -- THERE HAS TO BE SOME KIND OF TRADEOFF. METABOLICALLY YOU CAN MINIMIZE YOUR EFFORT IF YOU REALLY RIDE UP THE POWER FOR BUT FOR SOME PEOPLE THAT'S NOT A COMFORTABLE THING. >>THANK YOU, DEANNA. WE'RE GETTING TO THE TOP OF THE HOUR. I WANT TO REALLY THANK THE SPEAKERS, DEANNA AND JACOB, FOR THE WONDERFUL PRESENTATIONS. I THINK ALL OF US IN THE COMMUNITY KNOW THAT THE MACHINE LEARNING IS A VERY NICE APPROACH TO BE THINKING MORE ABOUT, THERE'S A LOT OF PARAMETERS THAT THE MACHINE HAS, THE HUMAN HAS, AND ALL OF THESE HAVE TO BE OBSERVED IN REFERENCE TO THE PATIENT AND THEIR ABILITY TO PERFORM FUNCTION. OF COURSE THE HUMAN MIND CAN ONLY DO SO MUCH. AND SUPPLEMENT OF THAT WITH ACTIVE MACHINE LEARNING APPROACHES CONVOLUTION AND OTHER NETWORKS ALLOW US TO OBTAIN THOSE INFORMATION AND HOPEFULLY THAT CAN BE TRANSLATED TO THESE DIFFERENT REHABILITATION APPROACHES. AT COLUMBIA UNIVERSITY WE HAVE BEEN SORT OF USING A LOT OF MACHINE LEARNING TO ACTIVELY PREDICT THE STATES OF THE HUMAN BODY, AS WE ARE TRYING TO SORT OF CREATE ALGORITHMS OR APPROACHES TO REHABILITATION. SO THANK YOU SO MUCH FOR JOINING THIS WORKSHOP, AND I WANT TO THANK TOYIN AND NIH FOR BRINGING US ALL TOGETHER TO HAVE THIS WONDERFUL TIME. LET'S THANK THE SPEAKERS AND GIVE THEM A ROUND OF APPLAUSE FOR THIS WONDERFUL PRESENTATION. [APPLAUSE] >>THANK YOU SO MUCH. THAT CONCLUDES THIS SESSION OF OUR SPEAKER SERIES, AND WE'LL SEE EVERYONE AT SOME POINT IN THE SPRING. TAKE CARE.