>> GOOD MORNING, EVERYONE.
WELCOME TO THE THIRD TALK IN THE
BIOWULF SERIES.
THIS YEAR, WE ARE CELEBRATING 25
YEARS OF PROVIDING SCIENTIFIC
COMPUTING TO THE NIH SCIENTIFIC
COMMUNITY.
TODAY, WE ARE HONORED TO HAVE
DR. SAMEER ANTANI AS OUR
SPEAKER.
DR. ANTANI IS A PRINCIPAL
INVESTIGATOR IN THE NATIONAL
LIBRARY OF MEDICINE.
HE EARNED HIS PhD AND MASTER OF
ENGINEERING COMPUTER SCIENCE AND
HIS B.E. IN COMPUTER
ENGINEERING.
DR. ANTANI IS A FELLOW OF THE
AMERICAN INSTITUTE FOR MEDICAL
AND BIOLOGICAL ENGINEERING,
FELLOW OF THE INSTITUTE MUCH
ELECTRICAL AND ELECTRONICS
ENGINEER, IEEE, AND A SENIOR
MEMBER OF THE SPIE.
HIS RESEARCH FOCUSES A NOVEL
ALGORITHMS IN MEDICAL IMAGING
AND MULTIMODAL MACHINE LEARNING
AND AI.
AND NOW FOR GLOBAL HEALTH
SETTING, CHARACTERIZING DATA AND
INNOVATIONS IN MACHINE LEARNING
AND ALGORITHM DESIGN FOR
RELIABLE AI PREDICTIONS WITH
APPLICATIONS IN THE SCREENING,
DIAGNOSTIC, RISK PREDICTION AND
TREATMENT.
DR. ANTANI'S RESEARCH ON BIOWULF
HAS IN THE PAST FOCUSED ON DEEP
LEARNING FOR MEDICAL IMAGE
IMPLEMENTATION AND RESEARCH
RELATED TO VARIOUS DISEASES SUCH
AS CERVICAL CANCER AND CARDIAC
REJECTION FRACTION.
ON BIOWULF, HE HAS MADE -- OTHER
RELATED FIREWORKS.
DR. SAMEER ANTANI'S TALK TODAY
IS TITLED EXPLORATIONS IN USE OF
SYNTHETIC IMAGES FOR BIOMEDICAL
MONTICELLO LEARNING AND AI
RESEARCH.
WELCOME.
>> WELL, THANK YOU.
THANK YOU VERY MUCH, AND
WELCOME, EVERYBODY.
I AM GOING TO BE SPEAKING WITH
SYNTHESIZING IMAGES AND USING
THEM IN SOME APPLICATIONS
RELATED TO DISEASES THAT WE HAVE
ONGOING, AND THIS IS A
STANDARD -- I'LL GO OVER A BRIEF
FEW MINUTES ON WHAT KINDS OF
THINGS I'M WORKING ON AND THAT
WILL ESTABLISH THE CONTEXT
WITHIN WHICH SOME OF THESE
EFFORTS WERE EXPENDED.
AND TRY TO LAY THE FOUNDATION OF
HOW WE THINK ABOUT SYNTHESIZING
IMAGES AND TALK ABOUT A BIT OF
BACKGROUND IN GENERATIVE
ADVERSARIAL NETWORKS, LATENT
DIFFUSION MODELS IN PARTICULAR,
AND SOME OF OUR SUBPOENASES OVER
THE PAST SIX OR SEVEN YEARS IN
TRYING THIS TECHNOLOGY AS SUCH.
AND SOME CONFOUNDING RESULTS AND
SEASONAL VERY GOOD RESULTS.
SO THAT SHOULD HOPEFULLY LAY THE
GROUNDWORK THAT IT IS PROMISING
AND VIABLE, BUT IT HAS SOME --
IT LEAVES SOMETHING FOR THE USER
OF THE TECHNOLOGY TO THINK ABOUT
WHEN THEY ARE -- WHEN THEY'RE
ADOPTING THIS.
A BIG THANKS TO MY RESEARCH TEAM
AT THIS MOMENT.
I HAVE SOME HERE IN THE
AUDIENCE, A SCIENTIST IN OUR
GROUP.
A POST-DOCTORAL FELLOW, ALSO A
POST-DOCTOR FELLOW AND JAYLENE
WHO IS NOT HERE IN THE ROOM WITH
US TODAY, BUT SHE'S MY STAFF
SCIENTIST AND OVER THE YEARS,
WE'VE HAD A NUMBER OF PEOPLE
CONTRIBUTE WHO ARE NOT PICTURED
HERE.
BROADLY AS WITH AN INTRODUCTION,
I CONDUCT RESEARCH IN RELIABLE
AND INTERPRETABLE MACHINE
LEARNING AND AI AND THE KEYWORD
HERE IS RELIABLE.
WE'VE OBSERVED AS THE DATASETS
CHANGE, EVEN WITHIN THE SAME
TOPIC AREA, WE EXPAND TO OTHER
ACQUISITION MODELS OR
ACQUISITION METHODS, THE OUTCOME
IS NOT NECESSARILY ANALOGOUS TO
WHAT WE SAW IN THE EXPERIMENT
TAM SECTION, BUT I BEGAN WITH.
OR WE CHANGE THE BALANCE OF
CONTROLS AND WE SEE A DIFFERENT
BEHAVIOR.
WE CHANGE THE CONDITIONS ON
IMAGES WITH THE EXPOSURE OR
CONTRAST ISSUE, THE BEHAVIOR
CHANGES AGAIN.
SO NOW ONE COULD SAY THAT YOU
JUST HAVE TO DO FEWER, MAYBE
THAT IS TRUE, BUT THERE ARE SOME
OBVIOUS THINGS THAT WE BELIEVE
THE METHOD SHOULD NOT MISS, AND
YET THEY DO.
AND WITH REGARDS TO
INTERPRETATION, WHAT I'D LIKE TO
ASPIRE TO DO, AND THIS IS
SOMETHING THAT WE ARE -- THE
BASIC STUFF WE DO IS
HIGHLIGHTING THE PARTS OF THE
IMAGE THAT MOST ATTENTION WAS
GIVEN BY THE ALGORITHMS.
THAT TO ME IS THE BABY FIRST
STEP.
I'D LIKE TO GET TO A POINT WHERE
THEHINES ARE ABLE TO USE,
PARTICULARLY IN TODAY'S
ITERATION, OF MULTIMODAL AI, USE
THE IDEA OF TRYING TO EXPLAIN AS
TO WHAT THE FEATURES MEAN OR
WHAT THE IMAGE FEATURES MEAN,
WHAT DOES IT TRANSLATE INTO AND
USE THE CLINICAL CONTEXT TO BE
ABLE TO INTERPRET WHAT THE
OUTCOME REALLY MEANS, WHAT THE
PREDICTION REALLY MEANS.
SO THAT'S WHERE I'D LIKE TO GO
AND THAT RELATES TO THE
AUGMENTATION OF THE CLINICAL AND
RESEARCH APPLICATIONS BECAUSE
MUCH OF MY WORK IS IN
COLLABORATION WITH VARIOUS
INSTITUTES, HEART AND BLOOD,
NCI, OTHER CANCER, WITH UCSF ON
SARCOMA.
IN ALL THESE -- THE ENDS POINT
ARE THEY'RE TRYING TO EITHER
DERIVE ON DECISION OF WHAT THE
BEST PLACE TO BIOPSY, SHOULD WE
BIOPSY ON ONE END AND ON THE
OTHER END, TRY IT IN THE FIELD
AND MINIMIZE THE NUMBER OF
ERRORS THAT POSSIBLY MIGHT BE
MAKING.
SO THERE IS THIS FACTOR OF
TRANSLATING THE OUTCOMES OF
MACHINE LEARNING ALGORITHMS INTO
THE FIELD AS WELL.
AND THAT'S A CONTEXT FOR GLOBAL
HEALTH THAT I'VE MENTIONED
BELOW.
SO WITH THAT, WE NEED TO START
THINKING ABOUT, WELL, IF YOU
HAVE A METHOD X THAT DOES VERY
WELL, THEN THE VARIABLE THAT
WE'RE PLAYING WITH THE MAINLY
DATA, AND THE DATA IS HAVING
SOME EFFECT, OBVIOUSLY, IN THE
LABORATORY IN -- IN THE
RELIABILITY AND GENERALIZATION
OF ALGORITHMS AND WE'VE BROKEN
THEM DOWN TO VARIOUS FACTORS
DEPENDING ON WHAT WE GET FROM
THESE INSTITUTES AND
COLLABORATORS AND SITES
WORLDWIDE, IS THE CONCEPT OF
QUALITY.
THERE'S EXPOSURE, THERE'S BLUR,
THERE IS SHADE, THERE'S CONTACT
ISSUES, BUT THE FRAMING OF THE
ORGAN OF INTEREST, REGION OF
INTEREST WAS NOT ACCURATE, AND
SO ON.
SO THERE'S A SPECTRUM OF THINGS.
THERE ARE ALSO OTHER MUNDANE
ISSUES LIKE YOU HAVE THE
PHOTO -- IF YOUR PATIENT'S BY
THE FOLDER AND YOU HAVE THE
WRONG IMAGE IN THE WRONG FOLDER.
IT SOUNDS SILLY, BUT THERE'S --
SO THE QUALITY SPECTRUM IS VERY
BROAD.
IT GOES INTO ACTUAL PROCESSING
QUALITY TO DATA QUALITY IN
GENERAL.
THE VOLUME IS WELL UNDERSTOOD
ISSUE.
THAT'S NOT ALWAYS TRUE AS SOME
OF THE OUTCOMES SHOW BECAUSE WE
TRIED TO COLLECT A NUMBER OF
IMAGES -- THE NUMBER OF UNITS OF
DATA THAT THE ALGORITHM
PROCESSING WITH.
AND THAT'S NOT ALWAYS TRUE.
MORE IS NOT BETTER.
MAYBE DIVERSE IS BETTER AND THAT
CAN LEAD TO VARIETY, BUT WHAT DO
YOU MEAN BY DIVERSITY?
IS IT JUST A CASE OF THE NUMBER
OF CASES YOU'VE SEEN, WHAT KIND
OF REGIONS ARE YOU SEEING IN THE
IMAGE, ARE WE USING ANATOMICAL
KNOWLEDGE OR PATHOLOGICAL
KNOWLEDGE TO INFORM THE MACHINE
LEARNING ALGORITHMS?
DO WE HAVE ENOUGH VARIETY IN THE
PATHOLOGY TYPES OR IF THERE ARE
THINGS THAT ARE PRESENT, THE
SEVERITY OF DISEASE AND WHAT
POINT YOU CATCH IT.
PARTICULARLY YOU GET INTO
LONGITUDINAL DATASETS AND THIS
PREDICTION AND THIS MATTERS A
LOT.
HOW DO YOU HANDLE IMBALANCE AND
WHAT FACTOR IT DOES PLAY?
THEN YOU COME TO THE HUMAN
ELEMENT OF THIS.
IN SUPERVISED LEARNING, WE ASK
EXPERTS TO ANOTE TATE THE IMAGES
FOR US AS WHERE THEY BELIEVE THE
DISEASE IS WHAN THE MACHINE
SHOULD LEARN.
WE'VE FOUND GREAT VARIETY IN THE
EXPERTS THEMSELVES.
HAT THEY'RE WRONG, IT'S
THAT THEIR PERSPECTIVE ON THE
IMMEDIACY OF CARE, AND
THEREFORE, RATING THE PATIENT AS
HIGH GRADE OR MEDIUM, MODEST OR
LOW GRADE, VARIES BASED ON WHAT
THEY PERCEIVE OR THEIR
EXPERIENCE TELLS THEM SHOULD BE
SOMETHING THAT THE MACHINE
SHOULD PAY ATTENTION TO.
AND A SIMPLE EXAMPLE WE HAVE
FROM COVID WAS WE HAD,
IMMEDIATELY AFTER AND IN THE
MIDST OF COVID, A BUNCH OF
IMPRASE THAT CAME AROUND AND WE
ASKED TWO RADIOLOGISTS OF RENOWN
TO TRY AND ANNOTATE THEM FOR US.
ONE WORKED IN A FIELD HOSPITAL
AND THE OTHER WAS A RESEARCHER.
THE RESEARCHER WAS FAR MORE
METICULOUS, FAR MORE SELECTIVE,
VERY CONSERVATIVE BECAUSE HE --
I GUESS THE MINDSET WAS, I WANT
TO BE MORE ACCURATE, WHILE THE
FIELD GUY WAS -- THIS REGION IS
FINE.
PICK THE ENTIRE REGION AND GO
WITH IT.
SO YOU HAVE THESE -- THESE
FIGURES ARE COMPLETELY OFF,
TRYING TO ARRIVE AT SOME KIND OF
CONSENSUS AND THAT IS A FACTOR.
HOW DOES LONGITUDINAL DATA PLAY
A ROLE AND IF YOU HAVE SAME
PATIENT OVER TIME, ACCOUNTING
AND ACCOMMODATING FOR THE AMOUNT
OF MEMORY THAT THE MODEL HAS,
HOW YOU COMPENSATE FOR THAT.
AND USE OF TABULAR DATA OR ANY
OTHER TEXT THAT SURROUNDS THE
IMAGES THAT COULD BE VALUABLE
FOR PROVIDING LABELS.
AND WHAT OF UNCERTAINTY?
WE CLASSIFY CONFIDENCE AS A
PROBABILITY OF THE ELEMENT
BELONGING TO A PARTICULAR CLASS,
BUT IS IT TRULY PROBABILITY?
IS IT PROBABILITY THAT WITHIN
THAT YOU ARE EXPOSED TO, WHICH
COMES BACK TO THE VOLUME OF DATA
YOU'RE TALKING GOOD, OR IS IT IN
GENERAL THAT NO MATTER WHAT,
THIS IS THE INCIDENCE RATE AND
HOW DOES IT RELATE TO THE EP
PEOPLOLOGY PART OF IT.
-- EPIDEMIOLOGY PART OF IT.
BIAS, LOOKING AT DARK SKINNED
VERSUS LIGHT SKINNED IMAGES.
MOST OF THE IMAGES WE HAVE ARE
OF LIGHTER SKIN TONES AND IF
YOU'RE PARTICULARLY LOOKING FOR
DARK SKIN TONES DISEASE
PATTERNS, SARCOMA BEING A BROWN
PATCH OF PRESENTATION TYPES ON
VERY DARK, UGANDAN SKIN OR SOUTH
AFRICAN SKIN, THEN YOU DON'T
HAVE THE DATA THAT WOULD
ACTUALLY BUILD A FOUNDATION FL.
THOSE ARE FACTORS THAT PLAY INTO
THE ROLE OF MY WORK AND THIS IS
JUST A QUICK SCREEN SHOT OF SOME
OF THE THINGS WE TALKED ABOUT,
SO I WON'T BELABOR THAT TOO
MUCH.
SO BASICALLY, THAT SORT OF
BUILDS THE CASE FOR WHY
SYNTHESIZING DATA.
WE ARE COLLECTING FOR KAPOSI AND
REDUCING VISCOSITY BYSI REDUCIN
THE VOLUME OF NUMBER OF SAMPLES.
HOPING TO INCREASE THE VARIETY,
AT LEAST IN TERMS OF CASES.
IMAGE REGION IS IN THERE BECAUSE
WE NOTE WHEN WE COLLECTED DATA
TO CERVICAL CANCER FROM VARIOUS
SITES, TO THE HUMAN EYE, IT HADE
NO DIFFERENCE.
TO THE MACHINE, WE COULD CLUSTER
THEM IN SEPARATE CLUSTERS, WHICH
WAS BIZARRE.
SO THE QUESTION, IF WE ARE DID
GO TO TRY AND ROLL THIS OUT IN
THE FIELD, THEN DOES IT MAKE
SENSE TO TRY AND ARRIVE AT
MOVING IMAGES FROM ONE CLUSTER
TO ANOTHER SO THAT YOUR MODEL
ACTUALLY STARTS BEHAVING BETTER?
BIAS WE TALKED ABOUT AND
IMBALANCE AS WELL IN THE PAST.
SO THERE ARE DIFFERENT KINDS
OF -- AS MOST PEOPLE WHO ARE IN
THE FIELD MAY KNOW AND THIS IS
NOT SUPPOSED TO BE EXHAUSTIVE,
NOR IS IT SUPPOSED TO BE
COMPLETELY A TALK THAT TELLS YOU
HOW TO DO THESE THINGS, BUT
ABOUT THE KINDS OF -- THEY ALL
BEGAN VARIATIONAL ENCODERS THAT
CAN BE USED TO MAKE NEW DATA AND
TRY TO DIVERSE THE FEATURESSEOU
HAVE COMPRESSED INTO A SMALLER
SPACE.
AND AN EXAMPLE IS SHOWN IN THE
FIGURE WHERE YOU HAVE, DID YOU
HAVE AN ENCODER THAT COMPRESSES
THE FEATURES AND DECODER
RECREATES X, VERY CLOSE TO X
ITSELF.
AND THEREFORE, YOU HAVE THOSE
FEATURES D, YOU CAN HOPEFULLY
CREATE NEW VARIATIONS.
THAT WAS THE ORIGINAL IDEA.
YOU GET INTO AUTO REGRESSIVE
MODELS, THE FOUNDATIONAL ELEMENT
OF LARGE LANGUAGE MODELS TODAY
IN GPT-3 AND THEY ARE
ESSENTIALLY TRANSFORM MERS.
THEY ARE MORE SEQUENTIAL
PREDICTORS OF TOKENS APPEARING
IN THE CONTEXT OF A LARGER
FRAGMENT OF DATA, AND IN THE
IMAGING YOU CAN THINK OF THEM AS
SEQUENTIAL IMAGES.
YOU CAN ALSO THINK OF THEM AS
PART OF THE IMAGES THAT ARE
RELATED TO EACH OTHER THAT
CO-OCCUR AND AS YOU'RE WALKING
THROUGH AN IMAGE, YOU WILL
ESSENTIALLY CREATE A MODEL THAT
WOULD USE THAT TO YOUR
ADVANTAGE.
I MOVED THEM DOWN HERE, THEY'RE
ACTUALLY IN TIME POINT RIGHT
AFTER ENCODERS, WHERE QUOUF TWO
COMPETING NETWORKS, ONE THAT IS
SYNTHESIZING POSHLY FOR
RANDOM -- POTENTIALLY FOR RANDOM
NOISE, BUT YOU CAN REPLACE THE
RANDOM NOISE BLOCK -- OOPS,
SORRY, BACK AGAIN ONE.
YOU HAVE THAT RANDOMIZED BLOCK
RIGHT THERE, YOU CAN CHANGE IT
WITH ANOTHER MODEL THAT IS
TRAINED OFF OF AN IMAGE THAT YOU
HAVE SITTING WITH YOU, AND YOU
CAN START FROM A POINT THAT'S
CLOSER TO WHAT YOU WANT THE
DISCRIMINATOR TO DISCRIMINATE
AGAINST.
AND THIS IS ANOTHER MODEL THAT
IS JUST LEARNING OVER TIME TO
DECIDE IF THE SYNTHESIZED MODEL
IS CLOSE ENOUGH TO WHAT IT WOULD
CONSIDER AS BEING IN CLASS
VERSUS OUT OF CLASS.
AND SO THE PROPERTY OF GANNS,
THEREFORE, UNLIKE THE DIFFUSION
MODELS THAT COMES LATER IS
CREATING LOOK-ALIKES THAT BELONG
WITH SMALL VARIATIONS THAT ARE
PER CHANCE USUALLY.
I'M GOING TO SKIP THE FLOW-BASED
MODEL FOR COMPLETENESS, BUT THE
LATENT DIFFUSION MODELS ARE
STILL -- IT'S A BRANDED PRODUCT.
BASICALLY TRIES TO GO ONE STEP
BEYOND THAT WHERE THE DIFFUSION
PROCESS ADDS MORE NOISE AT
VARYING SCALES AND VARYING
DEGREES OF NOISINESS TO AN
IMAGE, THAT THE MODEL TRIES TO
LEARN AND RECOVER THOSE IMAGES
FROM.
AND THE IDEA THEREFORE IS THAT
IF YOU HAVE AN IMAGE GOING IN
WITH ALL THE NOISE IN THERE AND
IT'S TRYING TO UNDO THE NOISE,
IT ALSO IS IMPLICITLY LEARNING
THE IMAGE FEATURES THAT ARE
TRULY BELONGING TO THE IMAGE
BECAUSE EACH TIME THERE'S A
LOSS, IT HAS LEARNED IT MADE A
MISTAKE AND WHAT WAS ACHIEVED
THROUGH NOISE AND WHAT WAS
ACHIEVED THROUGH CONTENT.
IT BECOMES A LITTLE MORE
INTERESTING WHEN YOU START
ADDING TEXT.
OKAY.
I'LL GET TO THAT.
ADDING TEXT TO THAT WHERE YOU
NOW HAVE TEXT LABELS GOING ALONG
WITH THE IMAGE AND THE CONTEXT
OF THOSE PARTS OF THE IMAGE THAT
IT'S LEARNING AS TRUE IMAGE AND
NOT BEING NOISE ARE NOW BETTER
ASSOCIATED AND YOU HAVE A
PAIR-WISE CONTRAST OF LEARNING
GOING ON WHERE YOU HAVE A TEXT
LABEL OR TEXT CONCEPT COMBINED
WITH IMAGE FEATURES ALONG WITH
NOISE AT VARIOUS SCALES, AND
ENOUGH DATA, IT TENDS TO NOW
SYNTHESIZE WHAT YOU WANT BASED
ON THE CONCEPT THAT YOU
REQUESTED, ASSUMING THAT THE
CONCEPT WAS THERE IN THE FIRST
PLACE IN THE TRAINING MODEL, SO
THAT CONDITION STILL REMAINS.
SO I'VE ALREADY DESCRIBED THIS
GOING THROUGH THE GAN ELEMENT,
AND THE KIND OF GAN, WHAT YOU
SAW IN THE PREVIOUS SLIDE THERE,
BASICALLY AGAIN, YOU HAVE NOISE
AND YOUR DISCRIMINATOR AND IT'S
BEEN TRAINED TO SOMETHING AND
KEEP ON LEARNING UNTIL IT FITS
THE LAST FUNCTION.
BUT THERE ARE VARIATIONS IN
THIS.
YOU CAN DECIDE TO SCALE IMAGES
UP AND DOWN AND IT WILL LEARN
MORE AND CAN USE THE INDIVIDUAL
LAYERS NOW TO LEARN MORE ABOUT
THE FEATURES THEMSELVES.
AND SO YOUR CONTEXT IS NOW
ESTABLISHED BETTER BY
ACCOMMODATING FOR IMAGES
APPEARINGAPG OR LOWER
RESOLUTION THAT MAY BE -- SO
AGAIN, SELECTING WHICH KIND OF
GAN YOU WANT FOR YOUR
APPLICATION BECOMES AN IMP A
FACTOR.
CONDITION GAN NOW APPLIES --
USES LABELS TO SAY CERTAIN
IMAGES, CERTAIN IMAGE FEATURES
CAN ONLY CO-OCCUR IF SOMETHING
ELSE HAS HAPPENED AND YOU BUILT
THAT INTO THE GAN ITSELF.
GAN IS LOT LIKE THE VANILLA GAN,
EXCEPT THERE ARE DIFFERENT LOSS
FUNCTIONS FOR IMAGE APPEARANCE
REGIONS.
YOU WANT SOMETHING TO REALLY
LOOK LIKE SOMETHING AND WE HAVE
AN EXAMPLE IN THE SLIDES COMING
AHEAD WHERE WE USE THAT TO
REMOVE FOCAL BLUR IN IMAGES.
FOR IMAGE -- YOU HAVE CYCLE GAN
THAT IS A PAIR-WISE MATCHING
BETWEEN TEXT CONCEPTS WHERE TWO
IDEAS AND THEN IT WILL TRANSLATE
FROM ONE TO THE OTHER, AND STYLE
IS CONTROLLING, AS SUGGESTS, THE
APPEARANCE OF AN IMAGE FOR A
PARTICULAR SCALE -- IF YOU WERE
TO SYNTHESIZE A PAINTING IN THE
STYLE OF PICASSO, PERHAPS, AND
YOU'RE NOT LOOKING A LATENT
DIFFUSION MODEL, YOU COULD
EXTRACT T WAY CERTAIN PATTERNS
CO-OCCUR ON THE IMAGES AND YOU
HAVE A LABEL THAT'S ASSIGNED TO
THAT.
SO THESE ARE DIFFERENT KINDS OF
GANs THAT ONE SHOULD THINK ABOUT
AND THERE ARE A FEW OTHERS THAT
YOU CAN THINK ABOUT IN TERMS OF
WHAT YOUR APPLICATION REALLY
DEMANDS.
THAT'S WHERE THE NUANCE STARTS
CREEPING IN TO THIS DISCUSSION.
I WENT OVER THIS IN THE PAST,
AND GOING TO REMOVE NOISE IN A
STEP-BY-STEP FASHION IN EACH
ITERATION AS WE REMOVE THE NOISE
ARE IT LEARNS SOMETHING MORE
ABOUT THE IMAGE.
THE MORE FEATURES IN EAC S
OF NOISE REMOVAL IS ALSO
EXTRACTING FEATURES AND IF YOU
COMBINE THEM WITH
TEXT, YOU GET
A BETTER MATCHING.
AND, OF COURSE, GAN, WHILE THESE
ITERATIONS AT DIFFERENT SCALES,
IT'S GOT MORE POWER AND MORE
FLEXIBILITY TH FOR THE
KINDS OF DATA THAT TR NOT
NECESSARILY A SINGLE FRAME
IMAGE, BUT TEMPORAL DATA OR
ASSOCIATED FRAMES OR SLICES IN A
CT THAT ARE CORRELATED WILL
START MAKING A LOT MORE SENSE TO
THE MODEL ITSELF AS OPPOSED TO A
SINGLE X-RAY THAT'S GOING
THROUGH.
SO BOTH -- SO IT OFFERS MUCH
MORE POWER AND CONSEQUENTIAL
DEMANDS MUCH MORE COMPUTE, SO
THAT'S THE OTHER COST C THAT C
HAVE TO PAY.
AND THIS IS IN THE CLIP PART,
WHICH YOU'RE ADDING TEXT AND
IMAGE TOGETHER, AND FOR IMAGE
CAPTURING REASONS OR REVERSE,
REQUESTING IMAGES THAT BELONG TO
A PARTICULAR CLASS BY HAVING A
TEXT CAPTION THAT GOES WITH IT,
IDENTIFYING OBJECTS AND IMAGES,
ASSUME YOU HAVE A LARGE ENOUGH
SET AND YOU HAVE THE WORD
ASSOCIATED WITH THE CONTEXT OF
IMAGES IN THE ASSOCIATED TEXT
THAT GOES WITH THAT IMAGE, THEN
FINDING THOSE OBJECTS IS NOW
THEREFORE DERIVABLE AND IT IS
ESSENTIALLY REPETITION OF A
PATTERN OVER TIME THAT THE MAGIC
REALLY HAPPENS.
AND TEXT TO IMAGE, WE'RE DOING
SOME TEXT-TO-IMAGE WORK AS WELL
AND IMAGE CLASSIFICATION WORK AS
WELL IN SOME OF OUR MORE RECENT
APPROACHES.
SO A VARIANT OF MODELS, WHICH IS
ABOUT CLASSIFYING PRESERVATION,
WHERE BECAUSE YOU'RE BEING IN
THE MEDICAL DOMAIN, WE ARE
STARTING WITH PRETRAINED MODELS
THAT HAVE BEEN BUILT OFF OF
STOCK PHOTOGRAPHY.
NOW, WHILE THEY WORK, IT'S NOT
OPTIMAL AND ESPECIALLY IN --
YOU'LL SEE SOME EXAMPLES AHEAD
WHERE YOU SEE SOME -- I MEAN
TODAY, IF YOU WENT TO DAL I OR
GEMINI OR OPENING EYES VERSION
OF THAT, YOU HAVE AN X-RAY AND
YOU SEE GLOSSY NON-LIFE LIKE
SILHOUETTES WHICH IS OBVIOUSLY
WRONG, AND IT'S BECAUSE IT
DOESN'T HAVE THE DATA, ALL THE
IMAGE PIXEL EXPRESSION TO
SYNTHESIZE THOSE KIND OF IMAGES.
SO A CLASS SPECIFIC FEATURE
PRESERVATION ALLOWS YOU TO
PRODUCE YOUR OWN IMAGES ON TOP
AND ADD YOUR OWN LABELS THAT
MAKE SORE SENSE, BUT STILL BUILD
OFF OF THE MODEL.
YOU'RE IN THE BUILDING A
FOUNDATIONAL MODEL, SO WE DON'T
NEED TO HAVE TWO MILLION OR
THREE MILLION X-RAYS TO START
DOING AN X-RAY MODEL.
YOU CAN START WITH THOUSANDS,
BUT YOU ARE SORT OF FINE TUNING
OR SHAPING IT WITH YOUR CLASSES
THAT ARE IMPORTANT TO YOUR NEEDS
AND YOUR APPLICATIONS.
TO RECONSTRUCT THE IMAGE AS HIGH
QUALITY.
NOW, WHILE THIS ALL SOUNDS VERY
LIKE THAT IT WORKS LIKE MAGIC,
THERE ARE LOTS OF CAUTIONS HERE
BECAUSE IT DOESN'T UNDERSTAND
DISEASE, IT DOESN'T UNDERSTAND
ANATOMY, IT DOESN'T -- OBSERVING
WHAT WE'VE BEEN DOING, I GO BACK
TO THE END OF MY TALK WITH --
LITTLE KIDS ARE BEING X-RAYED,
THEIR MOMS ARE THERE, THEY SE,
AROUND AND SO THERE'S NO
CONSISTENCY.
SO IT BECOMES A -- TO US, WE NOW
START LOCATING THE REGIONS OF
INTEREST THAT WE WANTED TO FOCUS
ON AND CORRECT FOR THE
ORIENTATION, AND THEN IT WORKS
IN THAT SUBCONTEXT.
SO THERE'S -- HOW MUCH OF THIS
IS ART, HOW MUCH OF THIS IS
ENGINEERING, AND HOURCH OF THIS
IS RECOGNIZING THE CLINICAL
CONTEXT ARE VERY IMPORTANT
FACTORS IN THE WAY YOU DESIGN
AND IT'S USUALLY NOT ONE
MONOLITH, IT IS A PROCESS THAT
YOU HAVE TO SEQUENCE THROUGH.
THAT'S ANOTHER FACTOR THAT'S
IMPORTANTS ABOUT THIS IS SOME
WORK WE'VE DONE RECENTLY WHERE
WE'RE GOING TO SYNTHESIZE X-RAY
IMAGES.
I THINK THIS IS SANDY FROM
OUR -- WE ARE PARTNERING IN AN
ACQUISITION CHALLENGE ORGANIZED
WHERE THEY WANT TO GO BEYOND
STATE OF THE ART, NOT TO SAY YOU
HAVE PNEUMONIA, YOU DON'T HAVE
KNEW MOAN YEAH, BUT TO ACTUALLY
SAY WHICH PART OF THE LUNG
ACTUALLY HAS IT AND HOW MUCH IS
THERE.
SO THEY'RE TALKING ABOUT MORE OF
SPECIFIC ABOUT THE EXPRESSION OF
DISEASE.
AND AS USUAL, WE DON'T HAVE
ENOUGH SAMPLES THAT ARE OF THAT
KIND, SO WE WANTED TO START WITH
EXISTING IMAGES OUT IN THE OPEN
AND USE THEM TO BUILD MORE
SPECIFIC MEASUREMENTS THAT
ARE -- NOW, FID AND KID IN THE
COLUMNS THERE REFER TO THE
PERCEPTION DISTANCE AND I'LL
LEAVE THAT TO YOU TO LOOK UP,
BUT BASICALLY LOWER IS BETTER.
HOW FAR IS THE SYNTHESIZED IMAGE
FROM THE ORIGINAL IMAGE IN TERMS
OF FUTURE SPACE AND THERE ARE
TWO DIFFERENT WAYS OF LOOKING AT
IT.
AND AS YOU CAN SEE WITH PRIOR
PRESERVATION, IT'S LOWER
COMPARED TO A IMPAN AND OTHER
DIFFUSION MODELS THAT TRIES
TO -- AND FORGIVE ME FOR THE
FLIP OF THE BEFORE AND AFTER
TUNING, BUT THIS IS WHAT CAME
OFF JUST PLAIN VANILLA OUT OF
THE BASIC MODEL AND YOU CAN SEE
IT'S NOTHING LIKE AN X-RAY THAT
YOU WOULD BELIEVE IS AN X-RAY,
AND THIS IS WHAT WE ENDED UP
WITH.
WHILE ON THIS SIDE, THIS IS WHAT
WE END UP WITH, WHICH IS MUCH
LIKE WHAT WE WANT AND THIS IS
NOT EVEN CT.
I DON'T KNOW WHAT THIS IS.
SO CLEARLY, WHAT THIS TELLS YOU
IS THAT IF YOU DOWNLOAD
SOMETHING, BEWARE AND THEN BE
AWARE ABOUT WHAT TO -- OR HOW TO
SHAPE IT TO YOUR SPECIFIC NEEDS.
THESE ARE MORE EXAMPLES WITH
DISEASES OF DIFFERENT KINDS,
WITH COVID, WITHOUT COVID, AND A
DIFFUSION MODEL ON THE FIRST ROW
WHICH GIVES MORE CONTRAST
IMAGES.
SO IN THIS, THE ARGUMENT CAN BE
MADE THAT IT'S UP TO THE EXPERT
EYE TO SAY THAT THIS IS NOTHING
LIKE HERE, OR THERE ARE MISTAKES
HERE.
TO THE NAIVE EYE, THEY ALL LOOK
PRETTY GOOD, BUT THEN WHAT IS
GOOD AND MAYBE COME BACK TO THE
QUALITY QUESTION WE COVERED, AND
IT IIMES BEYOND THE SCOPE
OF THE FIELD IN THE LAB TO BE
ABLE TO DETERMINE THAT, BUT THEN
RELY ON A CLASSIFIER THAT HAS
BEEN TRAINED TO REAL DATA TO SAY
HOW WRONG OR RIGHT WE WERE WITH
THE SYNTHESIZED IMAGES BEING A
SEPARATE TEST OBJECT.
SOME EXAMPLES WITH X-RAYS, AN
INTERESTING CASE HERE, WHICH IS
A DISEASE OF SOME KIND AND -- SO
THERE WAS AN INTERESTING TALK
YESTERDAY OR TWO DAYS AGO WHERE
THE SPEAKER WAS TRYING TO USE
SPECIES A SIGNALS ON SYNTHESIZED
IMAGES TO SEE HOW DIFFERENT THEY
ARE AND THIS IS AN EXAMPLE
WHERE, BECAUSE YOU'RE USING LDM
DIFFUSION MODELS, YOU'RE NOT
GOING TO CREATE CLONES.
YOU'RE TRYING TO CREATE IMAGES
THAT BELONG TO THE SAME CLASS AS
UNDERSTOOD BY AN EXPERT, A HUMAN
EXPERT.
AND THEREIN LIES THE RUB IS
WHERE -- WHAT IS THE LOSS
FUNCTION AS RELATES TO A
MATHEMATICAL MODEL AND WHAT IS
THE LOSS OF THE ERROR AS RELATES
TO THE CONTEXT AS ESTABLISHED BY
THE HUMAN JUDGE BECAUSE THE
IMAGE THAT YOU'RE EXPECTING THE
AI MODEL TO GIVE A
MEANINGFULNESS AT THE END, AND
THAT'S NOT ALWAYS THE CASE AND
SOME ARE JUST TERRIBLE IN
CONTRAST.
SOME ARE MORE BELIEVABLE.
THESE ARE NOT -- THIS IS NOT AN
EXHAUSTIVE LIST, BUT I PUT DOWN
SOME KEY ONES.
THERE'S A FEW MISSING BETWEEN 21
AND 22 WORK WK THE GROUP AT PENN
STATE ON CREATING HISSO
PATHOLOGY IMAGES AND THOSE ARE
MUCH MORE TRACTABLE BECAUSE
THERE'S A VERY PRONOUNCED
STRUCTURE.
THE CELLES FROM THE BIOPSY
AND THE LAYERS THERE MATTER.
THE CELL POSITION MATTERS AND
THOSE STRUCTURES ARE REPEATABLE,
SOT DID A MUCH BETTER JOB AND
THEY WEREERISHED, SO I DIDN'T
INCLUDE THEM HERE.
I PICK THE ONES THAT ARE
HIGHLIGHTED IN BOLD HERE TO MAKE
A POINT ABOUT THE SUCCESSES AND
SOMETHING THAT MADE OUR HEADS
CLINK AND GO, REALLY?
WHAT'S GOING ON HERE?
AND WE DON'T HAVE GOOD ANSWERS.
WE HAVE HYPOTHESES, BUT NOT GOOD
ANSWERS FOR THAT YET.
SO T
FIRST EXAMPLE WITH DEEP
LEARNING IMAGES, THE BACKGROUND
IS THE CERVICAL CANCER WORK.
THE IDEA IS THAT A WOMAN GOES TO
A HAVE IT, SHE'S TREATED WITH
THE 5% ACETIC ACID AND IF SHE
HAS AN HPV INFECTION, THE CERVIX
WOULD LIGHTEN.
AND WHAT ARE THE LESIONS?
AND WE WANT THE GUIDANCE FOR THE
CLINICIAN, IS THIS WOMAN -- ARE
THESE LESIONS CONSISTENT WITH
THOSE THAT ARE SEEN ON
PRECANCEROUS WOMEN, AND TWO,
WHICH ONE SHOULD WE BIOPSY AND
FOR A WOMAN THAT IS POSITIVE, IS
SHE ELIGIBLE FOR ABLATION.
THAT'S ALL THE END GOAL.
NONE OF THIS IS BEING DESCRIBED
HERE.
IN U.S. AND MOST OF THE FIRST
WORLD, IF I MAY USE THAT TERM,
IT'S A CAMERA THAT'S MOUNTED ON
A TRIPOD OR HEAVY WHEELS, HEAVY
BASE, NICE THREE ALREADY JOINTED
ARM AND YOU GET ROCK SOLID
IMAGES SO THAT PROBLEM DOESN'T
EXIST.
MUCH OF THE PART OF THE WORLD
WHERE THIS DISEASE IS HIGHLY
PREFER VENT AND MORE PROBLEM --
PREVALENT AND MORE PROBLEMATIC
AND NURSES ARE GOING INTO THE
VILLAGES USING HANDHELD MOBILE
PHONES, AND THIS DEVICE, THEY
HAVE A PROBLEM OF SHAKE,
ILLUMINATION, ETCETERA.
SO WE BEGAN TACKLING BOTH
ILLUMINATION AND FOCUS AND
MOTION BLUR AT THE SAME TIME
BECAUSE MOBILE PHONES WITH LOW
ILLUMINATION WILL TEND TO OPEN
THE SHUTTER FOR LONGER AND YOU
GET A LONGER EXPOSED IMAGE AND
YOU'RE NOT LOOKING AT FRAMING AT
THIS POINT, OR WE WERE NOT AT
THAT POINT IN THIS WORK.
THE IDEA WAS SIMPLE IN SOME
SENSE.
YOU HAVE PAIR-WISE IMAGES, YOU
TRAIN IT TO KNOW WHAT IS GOOD
AND WHAT IS BAD AND HAVE IT
CORRECT FOR WHAT IS -- SO THE
OUTPUT OF THIS, THERE WERE, IS
GOING TO BE VERY -- A GAN IS THE
BEST IDEA HERE BECAUSE YOU'RE
CREATING SOMETHING THAT'S
EXACTLY THE SAME, JUST SHORTSTOP
R SHARPENED, SO YOU'RE IN
ESSENCE DEBLURRING.
SO FOR THAT WE USE TWO KINDS OF
GANs, CONDITIONED ON THE
APPEARANCE THE IMAGE.
WE WANTED THAT SHARPNESS OF THE
PAIR TO GET INTO THE FEATURE
EXTRACTION PROCESS AND A
DISTANCE MEASURE WAS USED FOR
THE DISCRIMINATION PROCESS, SO
WE WANTED IT TO HAVE THE BEST
FIT TO THE CLASS THAT YOU WERE
SYNTHESIZING THE IMAGES FOR FROM
THE GENERATOR.
AND THE LOSS IS NOT ANYTHING TO
SPEAK ABOUT.
AND WE GOT A SET OF 2400 SHARP
AND 1200 BLURRY IMAGES FOR THIS
WORK.
NOW, THE QUESTION BECOMES, IF
THE BLURRY AND THE SHARP AREN'T
A PAIR, YOU REALLY CAN'T DO THIS
WORK.
SO WE HAVE A BLUR THE SHOT ONCE
AND THE KINDS OF THINGS WE WERE
SEEING AND THEN LEARN TO UNBLUR,
THE EXACT SAME THING THAT LDM
WOULD DO TODAY IS JUST NOW --
THIS IS PRE-LDM DAYS, SO NOW
THAT WE LOOK AT 2019, OH YEAH,
THEY WERE USING THE SAME STUFF.
AND WE HAVE AN 85/15 SPLIT, AND
THE 85 INCLUDES THE SUBSET THAT
ARE NOT MENTIONED HERE.
YOU END UP WITH -- YOU WANT TO
MEASURE SOMETHING THAT IS OF
GOOD QUALITY, FIX THE NOISE
RATIO, WHICH IS LOWER IN THE
CASE OF THESE IMAGES THAN THE
OTHER, AND SUCH A SIMILARITY
MEASURE WHICH, AGAIN, LOWER ISS.
DOESN'T MEAN MUCH, BUT IT'S A
VE FIT TO THE ORIGINAL
IMAGE.
THE POINT OF THE GOVERNMENT ROLE
WAS USE THE DEBLURRED IMAGES IN
THE CLASS FIRE TRAINED ON THE
NONBLURRY IMAGES.
AND WE SAW A BOOST.
WE SAID THERE WAS A MIX SENT
THROUGH WHICH HADRO PERFORM
OF 20% LESS THAN WHAT WE WERE --
I DON'T HAVE THE DETAILS IN MY
SLIDE HERE, BUT THE PAPER HAS
THE DETAILS.
AND YOU SAW A BOOST.
THE POINT WAS THAT THE
DEBLURRING THROUGH A GAN
MECHANISM DOES WORK AND DOES
MAKE A VIABLE OPTION FOR
IMPROVING FOR THE APPLICATION
THAT YOU'RE INTERESTED IN.
SO THERE ARE LOTS OF FOLLOW-ONS
HERE.
THEN WE WERE THINKING ABOUT HOW
TO ASSESS -- BOTH SPECIFIC
COLLECTIONS DONE FOR SPECIFIC
STUDIES ARE -- YOU GET IN
HUNDREDS LOW HUNDREDS IF YOU'RE
LUCKY.
ALSO, THE INCIDENCE RATE OF
DISEASE MATTERS.
SO IF YOUR INCIDENCE RATE IS
1.5% OR 1.3% AND YOU COLLECT A
THOUSAND PEOPLE, YOU'RE GOING TO
GET 150 SAMPLES.
SO HIGH END BALANCE, YOU HAVE
VERY FEW SAMPLES, SO QUESTION
ABOUT CAN WE CREATE LOOK-ALIKES,
MUCH LIKE THE EXAMPLES I SHOWED
WITH THE CT SCAN AND THAT'S
WHERE OUR THINKING IS RIGHT NOW.
THE SECOND SUBSET THERE, DO THE
IMAGES -- CONVINCELY REAL
IMAGES?
THIS IS A RECENT BLANK.
WE DON'T HAVE AN ANSWER FOR
THIS.
I WAS AT THE AMERICAN IMAGE
COMPUTING CONFERENCE JUST A WEEK
AGO AND SOME GROUPS ARE TRYING
THIS IDEA WHERE THEY'RE HAVING
TWO CLASSIFIER PATHWAYS.
ONE IS WHERE THE REAL IMAGE IS
MYLY I AM BALANCED, YOU BALANCE
IT OUT, USE TRADITIONAL
AUGMENTATION AND GET A
PERFORMANCE X.
THE OTHER PATHWAY,OU RUN THE
LDM IN ONE CASE OR GANs, WHAT
HAVE YOU, AND YOU DON'T WORRY
ABOUT IT BEING A GOOD IMPRESSION
OR NOT.
YOU JUST -- YOU ASK FOR A LABEL,
YOU GET WHAT IT IS AND THAT'S
THE LABEL YOU'RE GOING TO USE.
SO NOW INCREASIN THE DIVER
OF THE APPEARANCE WITH NO
VARIATION THAT THE SYNTHESIZED
IMAGES HAD ANYTHING TO DO WITH
THE DISEASE, BUT SOMEHOW, AN
EMSEM BELL OF THE TWO TEND BE TO
BE BETTER THAN JUST USE A SMALL
SET.
NOW, I NEED TO DWELL MORE ON
THIS, BUT THAT'S AN INTERESTING
THOUGHT.
HERE SO FAR, WE'VE BEEN THINKING
ABOUT SYNTHESIZE SOMETHING THAT
LOOKS LIKE IT, REPRESENTS THE
IMAGE THE WAY THE DISEASE AND
SEVERITY AS WE XN IT TO BE, SO
WE IN -- WE EXPECT IT TO BE SO
WE INCREASE THE SAMPLE SIZE
BELONGING TO THAT CLASS.
IT'S A MIXTURE OF AUGMENTED AND
NON-AUGMENTED IMAGES.
SO THAT IS AN OPEN THOUGHT AND I
DON'T HAVE A GOOD ANSWER, AND
WE'RE LOOKING AT BOTH GENERATIVE
AUGMENTATION AND TRADITIONAL
AUGMENTATION FOR NOT ONLY
TRADITIONAL X-RAYS, COMPARED
AGAINST NO CONDITION WHATSOEVER.
THIS IS WORK WE'LL TALK ABOUT
NEXT, BUTNE THAT BULLET IS VERY
RECENT TALK THAT I'VE INSERTED
IN THERE BECAUSE OTHERS HAVE HIT
THE SAME WALL THAT I'M GOING TO
SHOW YOU NOW AND ARE BYPASSING
IT SOMEWHAT IN WORK BY
JUST NOT CARING.
AND I DON'T KNOW IF NOT CARING
IS A GOOD IDEA OR NOT.
THEY'RE NOT DOING DISEASE
EXPLANATION OR HEAT MAPS BECAUSE
THEY CAN'T.
THEY DON'T HAVE THAT
INFORMATION.
SO THAT IS DEFINITELY A
NEGATIVE.
AND THE IDEA HERE IS WE'RE USING
PROGRESSIVELY GROGAN WHICH
ESSENTIALLY SAMPLES THE IMAGES
AT DIFFERENT SIZES OF
RESOLUTION, AND YOU HAVE NO
OBLIGATION, YOU HAVE TRADITIONAL
AUGMENTATION TECHNIQUES AS SHOWN
IN THE BOTTOM ABOUT PATHWAY AND
GAN-PRODUCED IMAGES AND SEE IF
THE CLASSIFIER PERFORMANCE
ACTUALLY IMPROVES OR NOT.
IN THIS EXAMPLE, THE TOP RIGHT
SHOWS HOW DIFFERENT RESOLUTION
IMAGES ARE CREATED BYRE FG-GAN
AND YOU CAN FORCE THE PG-GAN TO
I WANT ONLY A FOURY FOUR
IMAGE, AN EIGHT BY TEN IMAGE, SO
YOU SYNTHESIZE AN IMAGE THAT IS
A LOOK-ALIKED CONDITIONED ON
THAT RESOLUTION THAT YOU WANT.
YOU SEE W
ESOLUTION AND
SCALE WAS AUGUST THE BEST
AUGMENTATION TO THE END POINT AS
THE CLASSIFIER.
THAT'S WHAT WE'RE INTERESTED IN.
WE'RE NOT TRYING TO SYNTHESIZE
PICTURES REALLY IN THIS EXAMPLE.
AND WE HAVE A GOOD NUMBER OF
IMAGES, SO WE ACTUALLY USE THAT
IN DIFFERENT RESOLUTIONS AND
WHAT I'M GOING TO SHOW YOU HERE
DID BETAT THE GANs. AUGMENTAT
SO REALLY, ALL THE EXTRA WORK
PUT IN DID NOT MAKE AN IOTA OF
DIFFERENCE IN THE PERFORMANCE.
AND THIS BEGS THE QUESTION, ARE
WE -- BY
ADDING MORE IMAGES, ARE
WE MOVING THE NEEDLE AT ALL?
AND IN THIS CASE, WE'VE TAKEN A
WHACK A LITTLE.
HANG ON, WE DID ALL THIS WORK,
THE PICTURES LOOK GOOD, AND YET
WE'RE MAKING NO DIFFERENCE.
BECAUSE THE GANs ARE CREATING
IMAGES THAT LOOK LIKE AND BELONG
TO THE SAME CLASS THAT THE
DISCRIMINATOR WAS ASKED TO
DISCRIMINATE AGAINST, THE
FEATURE SPACE THAT'S OFF THE
IMAGES THAT ARE BEING -- THEY'RE
USING FOR EXTRACTION BELONGS TO
THE SAME CLUSTER, INCREASING THE
MORE TYPES OF, YOU'RE NOT
SPREADING THE POINTS OUT, YOU'RE
NOT HAVING MORE DIVERSITY IN THE
PATTERNS, SO YOU'RE GETTING A
PERHAPS MORE ELABORATE, BUT NOT
BETTER RESULTING -- THIS
REPEATS AGAIN, AND SO THAT'S
SOMETHING THAT WE WERE -- AND
WORKING, AGAIN, THAT COMES TO
MIND WITH THE OTHER COMMENT I
MADE ABOUT SOME OTHER GROUPS
TRYING TO BYPASS THIS WHOLE
QUESTION.
BUT THEY'RE NOT DOING THE
EXPLAINABILITY PART OF IT, SO
THAT IS A COST.
THAT'S A PRICE YOU PAY.
MAYBE WORK FOR SOME
APPLICATIONS.
SIMILAR WORK, IN THIS CASE
WORKING WITH AI TO TRY TO
DETERMINE IF ONE OR ANOTHER
LESION IS PRECANCEROUS, AND TWO,
AFTER LESIONS ARE THERE, SHOULD
IT BE BIOPSIED OR NOT.
THE CURRENT PROBLEM IS IT'S 100.
THE PATIENT NOTICES A LESION OR
THE DENTIST NOTICES A LESION,
SENDS YOU TO A CLING, THEY SEE A
LESION, THEY BIOPSY WHICH IS
QUITE DISCOMFORT FOR PATIENTS
ARE PRONE TO BE NEGATIVE AND FOR
THREE WEEKS, THEY'RE ON A
DIFFERENT DIET ALSO ALTOGETHER
AND SINGS LIKE THAT.
SO WE NEED -- AND THINGS LIKE
THAT.
SO WE NEEDED TO HAVE MOVE
EXAMPLES, SO WE HAVE AN
IMBALANCE AGAIN WHERE THERE ARE
CANCERS AND PRECANCERS AND WITHI
A NUMBER OF LESIONS, SO THE
LESION NUMBER IS DIFFERENT FROM
THE NUMBER OF PATIENTS THAT HAVE
CANCER.
AND IN THIS CASE, WE USED
TEXT-IMAGE SYNTHESIZER WHERE YOU
HAVE THE MNL THAT CONTAINS THE
EXISTING ORAL CANCER AND THE
PROMPT IS ORAL A COLD FRONT WITH
CANCEROUS LESION AND THAT'S JUST
ADDING WORDS TO THE CLIP MODEL
FOR IT TO LEARN FROM AND IT
CONTAINS A CANCER TO PRECANCER
SER RATIO OF NINE TO ONE AND YOU
GIVE IT THE IMAGES AND TRY TO
SYNTHESIZE, AND THESE ARE SOME
OF THE SYNTHESIZED IMAGES,
ACTUALLY.
PICTURES.ATES PRETTYRE CONVINCI
NOW, I THAT SOMEBODY COULD
COUNT THE TEETH AND SAY I
DISAGREE WITH THE NUMBER OF
TEETH, I DON'T SEE CANINE TEETH,
SO THAT SHOULD HAVE BEEN CATCH,
BUT THE LESION PART IS PRETTY
GOOD.
SO AGAIN, THE REGION OF
INTEREST, IS THE IMAGE LOOKING
ACCURATE OR NOT?
THAT'S SOMETHING TO THINK ABOUT.
THEN WE HAD DEVELOPED A SEPARATE
LESION SEGMENTATION MODEL THAT
FOUND THE LESIONS FOR US, APPLY
IT TO SYNTHESIZED IMAGES TO
EXTRACT THE LESION.
SO AGAIN, IT'S FINDING THE
LESIONS AND THE LESIONS ARE
ACTUALLY REAL BECAUSE IT HAS
BEEN TRAINED ON REAL IMAGES, SO
IT FINDS THIS T AND WE CROP THE
OUT.
SO YOU FOCUS ON AN IMAGE OF THE
MOUTH, YOU HAVE LOTS OF LESION
PATCHES NOW ON WHICH YOU WERE
TRAINED.
AND IGNORE THE TEXT.
IT WILL MAKE A DIFFERENCE.
SO SOMETHING THAT IS NOW
FOCUSED, REAL, ANOTHER
CLASSIFIER BELIEVES THIS LOOKS
LIKE A LESION THAT'S NOT RELATED
TO THE TRAINING CLASSIFIER AND
WOE HIT THE SAME WALL AGAIN OF
DIFFERENT NATURE, IMAGES WHICH
ARE SHARP AND PRETTY LOOKING AND
FINE.
SO ARE WE ESSENTIALLY OBSERVING
A DISTRIBUTION SHIFT HERE OR ARE
WE NOTICING THAT WE DON'T HAVE
DIVERSITY DECISION AS I SAID
EARLIER.
I DON'T KNOW.
THIS IS QUESTIONS THAT -- OF
SOMETHING TO QUAUN DER ABOUT AND
WE DON'T HAVE GOOD ANSWERS FOR.
ON THE FLIP SIDE, WE'RE GOING TO
HAVE SOME GOOD NEWS NOW, WHICH
IS USING LATON MODELS
IN PEDIATRIC CHEST IN, WHICH
IS WHAT I WAS ALLUDING TO
EARLIER.
AGAIN, THERE'S THIS CLASS
IMBALANCE WHERE YOU HAVE FAR
MORE -- FAR MORE NORMALS THAN
DISEASE POSITIVE, WHICH IS, AS A
WORD, WE'RE HEALTHY.
WE'RE
SAMPLING THINGS WITH FAR
FEWER PEOPLE WHO ARE EXPOSED TO
RADIATION FOR THE FUN OF IT.
AND THE ARGUMENT HERE, IS
TRADITIONAL AUGMENT SUFFICIENT
AND IS IT SUFFICIENT MAINLY
BECAUSE OF THE POSE OF THE KID
WHO IS IMAGED.
BECAUSE IF YOU HAVE -- FOR ADULT
X-RAY, IT'S A LOT MORE
STRAIGHTFORWARD, BUT THE PERSON
IS AGAINST PLATE, THEY'RE
FRAMED WELL, ENSURED BY THE
RADIOING ON -- RADIOGRAPHER AND
YOU REPEATABLE IMAGES.
KIDS ALL OVER THE PLACE, AGE
MATTERS A LOT.
LUNG SIZE IS DIFFERENT.
KIDS BELOW 5 YEARS OF AGE,
SMALLER, BOYS ARE LATER IN TAT T
LIVES BY, LOOKING MORE
LIKE ADULTS, SO THERE'S THAT
TRAIN THAT IS THERE IN THE SHAPE
OF THE OBJECT OF INTEREST AS
WELL, REGION OF INTEREST.
AND WE HAVE 9,000 IMAGES WE CAN
PULL PUBLICLY FROM REFERENCE
NUMBER ONE AT THE BOTTOM HERE
AND THEY HAVE TWO KINDS,
PNEUMONIA AND WE DO THE CLASSIC
COMPARISON.
THIS IS PNEUMONIA, BRONGO
PNEUMONIA AND THERE'S A 20% FOR
HOLD-OUT TESTING, THE CLASSIC
BREAK YOU KNOW.
AND WE USE ULV5 TO FIND THE
REGION OF INTEREST, SO NORTH
WHAT THE POSE IS -- NO MATTER
WHAT THE POSE IS, WE LOCK DOWN
ON THE REGION OF INTEREST AND
WHAT WE WANT TO FOCUS ON.
IT LOOKS LIKE A LOT LIKE THE
IMAGE YOU'RE SEEING IN THE
EXAMPLE ON TOP, KIDS OF
DIFFERENT AGES THERE.
NOW YOU HAVE A DECISION WITH AN
INCIDENCE THAT SAYS CHEST X-RAY
SHOWING PNEUMONIA AND IT --
RESOLUTION GOING ON AND YOU
HAVED PEDIATRIC X-RAY
WHICH IS NORMAL, AND IT LEARNS
TOTO SYNTHESIZE IMAGES THAT ARE
WHAT YOU WANT AND IT IS GOING TO
BE GENERATING IMAGES WITH A
PIECE OF TEXT AND YOUNT TO
SYNTHESIZE IMAGES THAT LOOK LIKE
THAT IMAGE.
IN YOUR ASK, YOU'RE NOT USING A
TEXT PROMPT TO GIVE YOU IMAGES,
YOU'RE USING TEXT PLUS PROMPT.
HERE'S AN EXAMPLE IMAGE, WE NEED
ALL SAMPLES THAT BELONG THO THIS
IMAGE TYPE.
THERE'S A BIT OF A SHIFT GOING
ON HERE FROM THE GAN-BASED
APPROACHES WHERE THEY'RE PUTTING
THE DECISION MODEL INTO --
TAKING ADVANTAGE OF THE VARIETY
THAT EXISTS.
WHEN YOU HAVE THOUSANDS OF
X-RAYS TO BEGIN WITH.
SO THERE'S SOME BENEFIT TO THIS
APPROACH THAT'S DIFFERENT FROM
THE PAST APPROACHES.
AND YOU NOW HAVE THE EXAMPLE OF
PEDIATRIC CHEST X-RAYS SHOWING
KNEW MOAHO THE NORMAL, AND
THE SYNTHESIZED, HOW IT
TRANSLATES ONE TO THE OTHER, AND
YOU HAVE BRONCHO PNEUMONIA GOING
FROM ONE TO THE OTHER.
SO VISUALLY, IT IS DOING
BELIEVABLE THINGS WHEN YOU LOOK
AT NUMBER OF SAMPLES, GOING FROM
NORMAL TO ABNORMAL, AND
THEREFORE INCREASING THE NUMBER
OF ABNORMAL THAT WE HAVE IN THE
SET SO THAT THE CLASSIFIER
THAT'S GOING TO BE TRAINED ON
IMAGES IS SEEING A BALANCE
ON THE OTHER SIDE.
SO YOU HAVE AN IMAGE THAT IS
TRAINED BETWEEN TWO MODELS,
TRAINED ON A CLASS IMBALANCE
TEST FIRST, THAT'S YOUR BASELINE
FOR NORMAL VERSUS PNEUMONIA
YOU HAVE ADD TO ITHA TO IT
OUT AND YOU HAVE A NEW SET THAT
INCLUDES A MIX OF SYNTHETIC AND
NONSYNTHETIC IMAGES, BUT YOU'RE
TESTING ONLY ON THE SYNTHETIC,
ON THE -- I BEG YOUR PARDON, ON
THE REAL IMAGES.
YOU CAN'T TEST ON SYNTHETIC, YOU
DON'T KNOW WHAT IT IS.
AND YOU TEST FOR A VARIETY OF
METRICS.
THE IMPORTANT PART, IF Y F
TO TAKE IT INTO THE FIELD, YOU
USE THE INDEX WHICH TELLS --
YOU KNOW, WHAT IS THE BALANCE OF
INTENSITY AND SPECIFICITY OF A
TEST AS IT GOES INTO THE FIELD
AND YOU SEE IT'S A LOT -- THE
COEFFICIENT IN TERMS OF
COMPARING HOW MANY OF THEM ALONG
THE DIAGONAL AND HOW MUCH
AGREEMENT THERE IS.
THE COEFFICIENT IS USING MACHINE
LEARNING, COEFFICIENT IN A
THE PEARSON'S TEST VERSION OF NN
IT, SO DIFFERENT NAMES FOR THE
SAME KIND OF TEST BASICALLY.
AND YOU'RE SEEING SLIGHTLY
BETTER RESULTS FOR THE AUGMENTED
ONES.
BASELINE VERSUS AUGMENTED,
SPECIFICITY WISE, BASELINE STILL
BETTER AND MAYBE THE NORMALS ARE
DON'T KNOW.PECIAL ABOUT THEM,
WE HAVEN'T VERIFIED THIS IN
TERMS OF HOW WHY IS THE
SPECIFICITY TEST IMPROVING AT
THE COST OF A FEWSAMPLES, AND
YOU SEE THE TWO TABLES BELOW
WHERE YOU SEE HERE, THE
AUGMENTED CLASS WENT DOWN BY 7
IN THIS CASE AND -- BUT YOU
GAINED MORE TRUE POSITIVES AT
THE COST OF SOME FALSE NEGATIVES
AS WELL -- FALSE POSITIVES AS
WELL.
SO THE FALSE NEGATIVES ARE
NOT -- SO THAT IS THE PRICE
PAYING.
ON THE WHOLE, THIS IS TRAINING
TOWARD AT LEAST CONFIRMING YOUR
HYPOTHESIS THAT GAN-BASED
REPLICATION IS GOOD FOR CHANGING
THE APPEARANCE OF AN IMAGE, BUT
NOTSARILYTT ADDING
STATISTICAL VALUE TO THE FEATURE
DISTRIBUTION.
LDM-BASED DOES BETTER.
NOW, THIS PARTICULAR SET, WHICH
IS A GIGANTIC CONDITION HERE,
HELPED POINT US IN THE RIGHT
DIRECTION.
CAN WE ACTUALLY STILL USE HEAT
MAPS TO TELL WHERE YOU'RE
LOOKING, FOR KIDS ABOVE THIS --
THE REASON THEY'RE WORKING ON
THIS IS WE ARE HEADING TOWARD A
CHEST X-RAYS FOR KIDS WITHOUT
HI HAVE TB BECAUSE THEY
DON'T -- EXPECTORATE, SO IT'S
OFTEN MISDIAGNOSED AND IT'S A
MORTALITY FOR KIDS WHO ARE
IMMUNE COMPROMISED.
SO WHAT YOU WANT TO DO IS TELL
YOU IN MOAN YEAH FROM TB.
-- KNEW MONA FROM TB.
YOU CAN TRAIN AN OBSERVER WITHIN
A DAY TO LOOK AT THESE PATTERNS,
BUT FOR KIDS, THESE PATTERNS
DON'T EXIST.
THEIR LYMPH NODES EXPAND AND
THEY GET OTHER EXPRESSIONS ON
THE LUNG THAT LOOKS A LOT LIKE
PNEUMONIA, BUT IS NOT, IT'S TB,
AND THERE'S A MAGICAL
COMBINATION OF THIS THAT IS
EMBEDDED IN THE DISEASE THAT
WE'RE TRYING TO EXTRAPOLATE OUT
BY THEIR LABELS AND WE GOT LABEL
JUST COMPLETING ON A SET OF A
THOUSAND IMAGES OF 200 OF WHICH
ARE NORMALS AND THE AEMAINING
800 AND CHANGE ARE ABNORMALS.
THE CHALLENGE IS GOING TO
PRESENT ITSELF BECAUPR THESE ARE
SELECTED FROM 15 SITES
WORLDWIDE, SO WE DON'T GET THAT
EXPOSURE VARIATION THAT
HAPPENS -- WE GET THAT EXPOSURE
VARIATION THAT HAPPENS BECAUSE
OF THE PROTOCOL AT DIFFERENT
CLINICS AND WE HAVE TO
COMPENSATE FOR THAT, SO MAYBE
THERE WILL BE SOME GANs PLAYING
A ROLE, NOT SURE YET, BUT THAT'S
WHAT WE ARE -- WE KNOW WE'RE ON
THAT TREADMILL HEADING TOWARD
THAT, AND SO THIS IS ALL
BUILDING OUR BACK KNOWLEDGE FOR
ANSWERING THE QUESTION, AND IT'S
HEARTENING TO KNOW THAT LDM
ACTUALLY IS BETTER THAN GANs IN
BALANCING THE SLA ANDND
PRODUCING HIGH QUALITY IMAGESGE
AND GIVING CLASSIFIER WORKS THAT
ARE SOMEWHAT WERN THE
PREVIOUS ONE, SIGNIFICANTLY
BETTER IN THE INDEX FOR SURE,
BUT NOT ENOUGH TO GO OUT TO THE
FIELD BECAUSE THE INDEX IS NOT
WORTH LOOKING AT FROM A FIELD
TESTING POINT OF VIEW.
AND THAT'S BECAUSE IT IS HEAVILY
SKEWED TOWARD SPECIFICITY AS IS
OBJECT IN THE SCORE.
SO THE MODEL IS ONLY AS GOOD AS
THE DATA AND THE PROBLEM AT
HAND, WHICH IS NOT MENTIONED
HERE, BUT IS IMPLIED, AND
THEREFORE, I PROPERLY SHOWED YOU
THAT THE IMAGE SYNTHESIS USING
GANs AND LDMs ARE A VIABLE
APPROACH, BUT CHALLENGING.
YOU HAVE TO KNOW YOUR PROBLEM,
KNOW THE DATA, KNOW THE
DISTRIBUTION AND THE QUALITY OF
THE IMAGES AND THE KINDS OF
THINGS YOU'RE LOOKING FOR BEFORE
YOU CAN START USING A PARTICULAR
APPROACH OR A COMBINATION OF
APPROACHES.
CONTEXT IS KEY, AND CONTINUOUS
EVALUATION AT DIFFERENT STAGES
IS CRITICAL.
WITH THAT, I I THANK YOU I Y FO
ATTENTION AND IF YOU HAVE ANY
QUESTIONS.
[ APPLAUSE ]
>> SO THANK YOU FOR THE TALK.
SO BEFORE WE START THE Q&A
SESSION HERE, FOR THE PEOPLE WHO
ARE WATCHING US ON -- FOR THE
PEOPLE WATCHING US ON VIDEO
CAST, PLEASE SEND YOUR QUESTIONS
VIA -- THERE'S A BUTTON LABELED
LIVE FEEDBACK, AND YOU CAN SEND
US YOUR QUESTIONS THROUGH THERE.
I'LL KEEP AN EYE ON THOSE
QUESTIONS.
SO FOR THE PEOPLE HERE IN THE
ROOM, ANY QUESTIONS?
LET ME HAND YOU THE MICROPHONE.
>> I'M NOT FAMILIAR WITH THE
MODEL THAT YOU MENTIONED,
DEBLURRING -- DEBLUR GAN.
IN A CLASSICAL GAN, YOU TRAIN
THE MODEL ON GOOD DATA AND THEN
TRY TO GENERATE AS GOOD AS THE
DATA IMAGES FROM SCRATCH, FROM
NOISE.
IN THE BLUR GAN, I'M ASSUMING --
DEBLUR GAN, I ASSUME INSTEAD OF
NOISE, YOU'RE USING A REAL
IMAGE.
>> A PAIR OF IMAGES.
ONE THAT IS THE GOOD IMAGE AND
ONE THAT IS THE NOISY IMAGE.
THAT'S GOING TO DISCRIMINATOR TO
TELL APART ONE FROM THE OTHER
AND THE BLURRY IMAGE IS BEING
FED TO THE GENERATOR.
>> YOU TRY TO MOVE FROM BLURRED
TO IDEAL IMAGE.
>> CORRECT.
>> AND KIND OF STOP AT SOME
POINT WHERE --
>> WHEN THE LAST FUNCTION, WHICH
TRIES TO -- WHICH TRIES TO
MINIMIZE THE GAP BETWEEN
BLURRY -- I'M SORRY, A DEBLURED
IMAGE AND THE AVERAGE DEBLURRED
IMAGE, IF YOU WILL.
>> OKAY.
>> AND SO IT IS USING A
DIFFERENT LOSS FUNCTION FOR THE
SCRIM NATOR AND A DIFFERENT
FUNCTION FOR THE GENERATOR, A
CONDITIONAL GAN.
IT'S CONDITIONED ON THE IDEA
THAT YOU HAVE GLUR RI AND
NONBLURRY IMAGES.
SO IT IS A DIFFERENT CONSTRUCT
FOR -- BECAUSE IF YOU'RE TRYING
TO TRANSLATE A BLURRY IMAGE INTO
A NONEXISTENT DEBLURRED IMAGE
FROM THE REAL WORLD, SO WE DON'T
WANT TO CHANGE THE IMAGE.
YOU WANT TO KEEP THE SAME
CONTENT OR CONTEXT AS IT MIGHT
HAVE BEEN HAD IT BEEN DEBLURRED
AND THAT'S WHY YOU HAVE THE
NOISE RATIO COMPARISON BECAUSE
WE ACTUALLY BLURRED THE IMAGES
OURSELVES, WE KNOW WHAT THAT IS,
SO HOW FAR IS THE DEBLURRED
VERSION FROM THE ORIGINAL.
THAT'S ONE THING ON THE VALUE
AND THE OTHER IS THE INDEX WHICH
TELLS YOU IF YOU WERE TO
ESSENTIALLY OVERLAND THE TWO,
HOW MANY PIXELS WOULD BE OFF.
AND THAT NUMBER WAS LESS THAN
ONE PIXEL WAS OFF.
SO IN THE NORMALIZED SCALE.
>> OKAY, OKAY.
I'VE GOT THE IDEA, THANK YOU.
>> YEAH, SURE.
>> ACTUALLY, ONE MORE QUESTION.
AND AT THE BEGINNING OF THE
TALK, YOU LISTED A SET OF
REASONS WHY ONE MAY NEED
SYNTHETIC IMAGES.
>> RIGHT.
>> YOU MENTIONED INCREASING
NOISE, INCREASE IN THE VARIETY
OF YOUR DATA.
>> CORRECT.
>> REDUCING THE DATA IMBALANCE.
ARE THERE ANY CONDITIONS WHERE
YOU WOULD LIKE TO GENERATE
SYNTHETIC IMAGE BECAUSE YOU
COULD NOT PRODUCE IT
EXPERIMENTALLY FOR SOME
TECHNICAL REASONS?
>> THAT IS A VERY VALID POINT.
IT HASN'T FALLEN WITHIN MY
PURVIEW BECAUSE IN MY CASE, IT'S
USUALLY DIAGNOSTIC, SO I'M
TRYING TO DETECT DISEASE
PATTERNS, BUT YES, IF YOU WANT
TO JUST SYNTHESIZE SOMETHING, I
MEAN, BASICALLY FORGET MEDICINE
FOR A SECOND.
IF YOU GO TO GEMINI OR STABLE
DIFFUSION OF THE DALLY TREE, IT
IS DOING JUST THAT.
IT SYNTHESIZING IMAGES OUT OF
CONTEXT.
THE ASSUMPTION YOU'RE MAKING IS
THAT THE MODEL FROM WHICH YOU'RE
SYNTHESIZING HAS THE WORDS
EMBEDDED IN THERE THAT BELONG TO
THE -- TO WHAT YOU WANT TO SEE
OUT OF IT.
AND, TWO, THAT YOU HAVE A SCRIM
NATOR OF SOME KIND, WHETHER
HUMAN OR MACHINE, THAT WILL
DETERMINE THAT WHAT HAS BEEN
SYNTHESIZED BELONGS TO SOMETHING
YOU WOULD LIKE TO SEE AN IMAGE
OF.
FROM A VISUALIZATION
PERSPECTIVE, IT'S FANTASTIC, BUT
THAT FALLS MORE IN THE ART
CATEGORY, BUT FROM A -- FROM THE
APPLICATION SPACE THAT I WAS
TALKING ABOUT WHERE YOU ARE
TRYING TO DETECT DISEASE AND
CORRECT FOR THE IMBALANCE, IT
DOESN'T FIT VERY WELL IN THAT
IDEA.
BUT AS AN IDEA, THAT EXISTS.
THE CHALLENGE BECOMES ARE THE
WORDS THAT YOU'RE GOING TO USE
TO SYNTHESIZE SOMETHING FROM
NOTHING, DO THEY BELONG TO THE
TEXT VOCABULARY THAT THE
DIFFUSION MODEL/LLM WAS EXPOSED
TO WITH THE RIGHT KIND OF
APPEARANCES.
YOU HAVE TO DESIGN SOME SIGNAL
FROM SOMEWHERE TO SYNTHESIZE.
IT HAS TO BE EXPOSED, MAYBE NOT
SPECIFICALLY TO A DATASET LIKE
IN OUR CASE, BUT IN THE BROAD
WORLD, IT WOULD JUST -- WE'VE
SEEN SOME EXAMPLES WHICH WOULD
RELATE TO THE WORDS THAT YOU'RE
USING FOR CORE SYNTHESIZING MAP
THE APPEARANCE THAT YOU EXPECT
TO SEE.
NOW, IF IT'S JUST PRETTY GRAPHS
AND THINGS LIKE THAT, I MEAN,
THERE ARE SITES THAT WILL CREATE
CARTOONS BASED ON A PIECE OF
TEXT.
I MEAN, IT'S A STYLE GAN
UNDERNEATH SOMEWHERE WITH AN LDM
MODEL WHICH WOULD SAY I WANT A
CARTOON IN THE STYLE OF DISNEY
FOR EXAMPLE AND IT CREATES A
CARTOON IN THE RECONCILE OF
DISNEY CHARACTERS, BUT THE
CONTENT OF THE CARTOON IS
COMPLETELY DRIVEN BY YOU AND THE
FUNCTION OF THE WORDS IN THE
CONTEXT.
I TRIED THIS YESTERDAY.
I FORGET WHAT SITE IT WAS, JUST
TO SEE SOME EXAMPLES, AND IT
FAILED MISERABLY.
THERE IS FOR WORD SCIENTIST,
THERE IS NO WORD MEDICINE, THERE
IS NO WORD X-RAY, SO THE
VOCABULARY IS PRETTY -- PRETTY
CHILD CARTOON LIKE, BALLOONS AND
ALL THAT CAME THROUGH JUST FINE.
SO IT'S -- IT'S WHAT -- LIKE MY
CLOSING SLIDE, WHAT IS THE
CONTEXT IN WHICH YOU'RE TRYING
TO SYNTHESIZE AND IT DOES MAKE
SENSE IN THAT CONTEXT.
>> OKAY, THANK YOU.
>> SURE.
>> OKAY, WE HAVE A QUESTION FROM
THE VIDEO CAST.
THE QUESTION SAYS, ALTHOUGH I
BELIEVE YOU ONLY USE THE
GENERATOR FROM A GAN,
EXAMINATIONING THE DISCRIMINATOR
MAY HELP YOU UNDERSTAND ITS
LIMITATIONS.
DO YOU KNOW, FOR EXAMPLE,
WHETHER THE DISCRIMINATOR
RECORRECTS EXAMPLES THAT WERE
NOT IN THE TRAINING -- REJECTS
EXAMPLES THAT WERE NOT IN THE
TRAINING SET?
>> THAT'S A GOOD QUESTIONS ABOUT
WE HAVEN'T EXPLORED THAT.
I THINK WHAT THE QUESTION IS
TRYING TO POINT AT IS WAS THE
LOSS FUNCTION ADEQUATE IN -- WAS
IT TOO STRICT IN REJECTING GOOD
EXAMPLES THAT DIDN'T FIT THE
DISCRIMINATOR MODEL.
IT'S TIED TO THE REAL DATA IT
HAS SEEN TO TRAIN IT, AND I'M
NOT SO SURE IF -- WE HAVEN'T
DONE, THIS OF COURSE, BUT IT'S
AN INTERESTING POINTER TO SEE
WHAT ARE THE REJECTS OR THE
PATHWAY FROM THE DISCRIMINATOR
IN TERMS OF WHAT WOULD BE A --
WHAT COULD IT HAVE BEEN, SEE
THAT POOL AND SEE IF THERE WERE
ANY CANDIDATES BECAUSE, AGAIN,
RELYING ON MACHINE THRESHOLDS TO
DETERMINE THE LOSS THRESHOLD TO
DETERMINE IF AN IMAGE PASSES THE
DISCRIMINATOR STAGE INTO THE
OUTPUT, AND IF THAT THRESHOLD IS
TOO TIGHT, YOU'RE GOING TO GET
THINGS THAT ARE GOOD FIT, BUT
MAY BE COSTING US, THE VARIETY.
BUT THEN THE CHALLENGE BECOMES,
WHICH IS A GOOD THRESHOLD IN THE
FIRST PLACE SOIRNTION A BIT OF A
CHICKEN AND EGG -- SO IT'S A BIT
OF A CHICKEN AND EGG, BUT FROM A
STUDY PERSPECTIVE, AN
INTERESTING THING TO SEE.
HOW DOES ONE ARRIVE AT A
MEANINGFUL LOSS THRESHOLD AT THE
DISCRIMINATOR SO YOU CAN INDUCE
THE VARIETY YOU WANT WITHOUT
COMPENSATING FOR QUALITY.
>> ALL RIGHT.
THANK YOU.
ANYMORE QUESTIONS HERE?
OKAY.
I THINK I JUST HAVE A FINAL
QUESTION.
I BELIEVE IN ONE OF THE SLIDES,
YOU WERE COMPARING GAN VERSUS
JUST TRADITIONAL AUGMENTATION,
AND CAN YOU ELABORATE A LITTLE
BIT MORE ON THE METHOD THAT YOU
USED FOR COMPARING BETWEEN THOSE
TWO?
I MEAN, HOW DO YOU KNOW WHICH
ONE IS PERFORMING BETTER?
>> THE OUTPUT OF A CLASSIFIER.
SO YOUR TRADITIONAL
AUGMENTATION, YOU HAVE
ESSENTIALLY A DATASET THAT IS
TRADITIONAL AUGMENTED AND THAT
IS THE BLOCK HERE, SO YOU'RE
TRAINING ONE AND YOU'RE
COMPARING AGAINST -- YOU'RE
TRAINING AGAINST A TRADITIONAL
AUGMENTATION IMAGE DATASET,
WHICH IS -- A GAN-BASED DATASET
AND SEPARATE DATASET AND THE
ORIGINAL UNTOUCHED DATASET, SO
YOU HAVE THREE THINGS COMPARED
AGAINST EACH OTHER WHICH BECOME
THE COMPARISON ON THIS TABLE
WHERE THE FIRST ROW IS NO
AUGMENTATION OF ANY KIND AND
THAT'S THE BASELINE PERFORMANCE.
THIS IS WHAT SOMEBODY MIGHT JUST
DO OFF THE BAT, TRADITIONAL
AUGMENTATION AND ALL THE FINE
STUFF.
AND YOU GET NUMBERS THAT ARE
OBVIOUSLY BETTER -- IN THE GA,
WE DID NOT DO ANY AUGMENTATION
THAT IS TRADITIONALLY DONE.
WE JUST INCREASED THE TEST SIZE.
SO GA IN THIS ROW IS A LOT LIKE
NO AUGMENTATION, EXCEPT THE
NUMBER OF IMAGES HAVE GONE UP
AND BAM OUT.
SO YOU BALANCE -- IN THIS CASE,
IT'S A HIGHLY I AM BALANCED SET,
THE FIRST ROW.
IN THE THIRD ROW, IT'S A
BALANCED SET, BUT WITH MORE
SAMPLES THAT WERE SYNTHESIZED
AND THE MIDDLE ONE IS IN PROCESS
AUGMENTATION THAT IS DONE TO
CORRECT FOR THE IMBALANCE
THAT -- THE TRADITIONAL
TECHNIQUE AS ANYBODY ELSE MIGHT
DO.
SO WE WANTED TO SEE THE EFFECT
OF ADDING JUST MORE IMAGES FOR
THE AUGMENTATION PART.
OUR AUGMENTATION STEP IS
SYNTHESIS.
NOW, ONE CAN GO FURTHER.
TAKE THE GA IMAGES AND DO
TRADITIONAL AUGMENTATION ON ALL
OF THEM.
NOTHING WRONG IN DOING THAT, AND
MAYBE THE NUMBER WOULD BE
SLIGHTLY BETTER.
WHO KNOWS, BUT THE POINT WAS TO
ACTUALLY SEE WHAT DID GENERATIVE
AI BRING ANY VALUE, PLAIN
VANILLA, AND IT JUST -- IT SAYS
IT DOESN'T BY LOOKING AT THE
DATASET.
BUT IF YOU GOT THE TWO EXAMPLES
AND YOU HAVE SEEN A SIMILAR
TREND LINE WHERE WE ARE NOT
SEEING A BUMP THAT WE HOPE TO
SEE BY CORRECTING IMBALANCES..B
FACTOR.
IT'S SOMETHING MORE THAN
IMBALANCE IS THE FACTOR.
SO IMBALANCE -- IF YOU LOOK AT
JUST AUGMENTATION AND NO
AUGMENTATION, IMBALANCE PLAYS A
ROLE, NO DOUBT.
NOT A BIG ONE IN THIS, NOT SO
AMPLIFIED IN CASE, BUT IT DOES
PLAY A .. GENERATIVE AI IS NOT Y
PATHWAY.
IT MAY BE ONE OF THE PATHWAYS
COMBINED WITH SOMETHING ELSE AND
THAT'S THE WHOLE POINT OF THIS
EXPERIMENT WAT TO INDUCE THAT
PART.
JUST BECAUSE YOU HAVE A
MECHANISM TO SYNTHESIZE NEW DATA
DOESN'T MEAN -- OF COURSE,
THERE'S A WHOLE SEPARATE FIELD
THAT IS DOING SYNTHESIZING DATA
ON TABULAR DATA WHERE YOU'RE
MISSING FIELDS, MISSING VALUES,
AND YOU GET INTO SIMILAR
PROBLEMS, BU DO THE PERSON'S
SIZE AND WEIGHT MAKE A
DIFFERENCE, DO THE BLOOD VALUES
MAKE SENSE OR NOT, ETCETERA.
SO THE POINT IS TO GET YOUR FEET
WET AND RECOGNIZE THAT IT'S A
VIABLE TECHNOLOGY, MEANINGFUL
WORK USING -- GIVEN THE
REALITIES THAT WE FACE WITH
CHALLENGES.
>> GREAT.
THANK YOU FOR THE CLARIFICATION.
AND THANK YOU FOR A GREAT TALK.
[ APPLAUSE ]