Abstract: Optical coherence tomography (OCT) has now become a standard of care, impacting the treatment of millions of people every year. There is tremendous clinical and preclinical OCT progress in diagnosing cancers and disorders in ophthalmology, cardiology, neurology, dermatology, gastroenterology, etc. In this talk, deep learning algorithms for detecting/segmenting the crucial cell/tissue/lesion features, such as nuclei, the dermal-epidermal junction of human skin, and tumor boundaries, will be addressed. The performance can be explained by visualizing the neural network’s feature activations in response to the cell-like structure of human tissues. Histopathological stained images are considered the gold standard for clinical cancer diagnosis. However, the staining processing time is long, especially when surgery progresses. There is an unmet need to build an image translation model to convert the grey-level OCT images to mimic the stained images. Both semi-supervised and unsupervised approaches toward virtue histopathology will be addressed. Leveraging the ever-escalating techniques in applying deep learning algorithms to medical image analysis could accelerate the acceptance of deep learning applications among clinicians and patients.
Bio: Dr. Sheng-Lung Huang received his Ph.D. from the Department of Electrical Engineering, University of Maryland, College Park, in 1993. He is a Distinguished Professor at the Graduate Institute of Photonics and Optoelectronics (GIPO) and the Department of Electrical Engineering at National Taiwan University. He served as the Chairman of GIPO from 2007 to 2010. He was also a guest professor at the Abbe School of Photonics, Friedrich-Schiller University of Jena, Germany, in 2014. Dr. Huang is a Fellow of the Optica. He pioneers the development of cellular- resolution optical coherence tomography and has used it clinically in the early-stage diagnosis of cancers and diseases. In 2014, he co-founded Apollo Medical Optics and was the Chief Technology Officer. Dr. Huang served as Chairman of IEEE/LEOS (now IEEE/PS) Taipei Chapter from 2005 to 2006. He was a steering board member of the European Master of Science in Photonics (EMSP). Dr. Huang served as an Associate Editor of the IEEE Photonics Journal and was a Topical Editor, Optics Letters, 2005–2011.
Dr. Huang has received the Outstanding Research Award from the Ministry of Science and Technology and the University/Industry Cooperation Award from the Ministry of Education, Taiwan. He has also received Chimei Innovation Excellence Award and Optical Communications Elite Award.
All right hi everyone um today we are honored to have an ENT phonics distinguished lecturer Professor Shang long hon present his talk titled deep learning empowered Optical coherence tomography Dr Wong received his PhD in electrical engineering at the University of Maryland he is a distinguished professor at The Graduate Institute of
Photonics and op electronics and the department of electrical engineering at Taiwan University Dr Juan’s research focuses on developing cellular resolution Optical coherence tomography for clinical diagnosis of disease such as cancer today we will learn uh how deep learning can enable this Imaging to improve un steady the art techniques for
Better better diagnosis and treatment of disease without further Ado please join me in welcoming Professor [Applause] shangan okay so uh good morning uh ladies and gentlemen thank you for joining uh this session uh first of all I think I just thank Kevin for the introduction and for organizing this
Program yeah it’s very nice to know that even though here you have very active artics King here yeah and for this talk I I think I should also thank I for society uh for sponsoring my trip here yeah and so so this talk is about Optical coherence tomography uh it was
Uh has been invented for more than 30 years so right now it is U widely deployed worldwide in hospitals and medical centers and the um at the Eno here you have very strong ocp program and made a very significant contribution and Innovations so it’s a very uh it’s
Really my pleasure to come here uh to share with you our recent work and uh to exchange our ideas for the future directions and for the audience here you are not quite familiar with occt I will also give a very brief introduction and uh uh tell you the reason U the present
Status and the future Trend and uh share uh with you myself how we apply the AI to uh Empower ocp for clinical diagnosis so uh I’m sh and uh these are my uh collaborators I have a collaboration with our University hospital interor hospital and this is M
Memorial strong cing Cancer Center uh no this is M Memorial Hospital this is memorial strong cing Cancer Center in New York and uh because for clinical trial uh you need a system that satisfy all the regulations and the qualities so I established a company uh N9 years ago
So they made a protype and so we can in the hospital and this AI n is our University Center it’s very resourceful for the high uh high performance computation so for the uh de learing kind of algorithms okay and so for the audience if you are not familiar with Taiwan I
Will give you a very brief introduction about the photonics in Taiwan so this is apparently this is the island and the number here represent uh the number of universities in each of uh uh cities so overall uh in Taiwan right now we have 160 uh universities um so among this
Uh 160 universities um we have 35 of them that have dedicated bonics the college or department or graduate insute so this is kind of quite unusual why I think one reason is that in Taiwan right now we have a very active photonics industry such as display uh solid state lighting and some
Optical Communications so right now so according to I got this table from Optica last year so this is shows the worldwide Optics and the phonics production market share so even though Taiwan is a small island so we cover more 10% of the worldwide market share
Yeah I think uh one of Reon because uh uh I think these days people know that in Taiwan we have tsmc okay we actually we have more than tsmc for example the company Lan here maybe you you never heard about the company Lan but I believe many of your cell phone you have
A very compact lens for the camera for for this camera there are maybe six to eight lenses uh I think for the Lo down Market sh I believe is more than 50% so high your cell phone use l and also Acer Asus and W this the company for making the notebooks so
Several other companies so from the biomedical components devices to mobile health care and even to Smart Hospital Solutions so in case you are interested in forming a kind of star accompan you can find you know people to uh the found service or um make the prototype for you in Tai it’s very
Convenient okay so let me go back to my topic so this is the outline of my talk I will first give a very brief overview about the how AI was used on Imaging and then I will show you how we use the AI algorithms our OCD imagings and finally
This is kind of a uh in my personal perspective one of the Holy Gra of OC because uh right now World why in case you have certain tumor you need a exisal biopsy but for OC because this noninvasive uh so in case the in the future in case OCD Imaging is good
Enough for the Physicians or clinicians to make a diagnosis then probably you can reduce the number of exisal BS in the future okay so apparently AI is not something new because starting from 50 70 years ago people talk about uh AI but uh very recently uh people uh start to
Talk about deep learning uh the reason is that if you look at this uh figure on the rental axis is the amount of data so apparently in case you have a a larger data then you should have a better performance but in case you use the machine learning algorithm so when the
Data is very big then the the performance kind of saturate in case you use a deep learning algorithm can keep on uh improving the performance linear Trend so this is something very good for biomedical Imaging because usually for biomedical Imaging modality you can acquire data very fast for example for
OC uh it’s very easily you can acquire more than one gigabyte of data in one second so it’s very really a very big data so it’s a kind of a perfect match withing because with a larger data you can have a linear Improvement performance so these days worldwide so
In all the biomedical uh areas breast cancer abdomen or many cardiovascular ucation if you look at the number of applications all all them kind of exponential growth that means the Deep learning really works for the DI Medical Imaging and specifically if you look look at the Publications uh people main
Focus on phology and the Brain the related to nerve the central nerve system so and for pathology usually people care about the resolution you want to see the detail of human tissues the nuclei the the density of the nuclei the shape of nuclei and whether the nuclei is have smooth boundary or very
Rough boundary you need a high resolution okay so to uh uh using the AI on biomedical Imaging right now at least there’s one very successful case uh p m is a company established in United Kingdom but it was acquired by Google I think almost 10 years ago so
Right now they do have kind of OC integrated AI system that pass which is approved by FDA so right now it’s really on the market you can see here this is the RO OCD Imaging black y so this is the Physicians anotation this is the AI predictions you can see that the
Segmentation and prediction is quite accurate and using the AI algorithm it can also help you to find out this kind of where is the Hotpot for the Physicians we should focus on yeah so already there’s a successful story to integrate OC with AI okay so for the
Audience that are not familiar with the OCD I give you a very quick introduction so basically the idea is very simple for example this is your device on the test this is scheme okay L you try to detect back skate life okay the back skate life
On the surface of the sample actually is very strong but the back SC from deep into the tissue actually is very weak so you need a very uh sensitive detetion skin so the detaching skin here actually I think for student here you should be very familiar with this kind of skin
This is just a typical micos micro in the parameter uh the only difference is that uh the light source usually in front use n b laser but OC actually prefer B and L and the broader the BWI the better the uh the resolution you can have so U remember I mentioned because
For the signal de into the tissue the back SC is very weak and so this kind of micro freter actually in a sense it’s just like homo detection in Optical communication system so you know that for homo detction very sensitive yeah so uh so in case whenever the light
Penetrate into the device on the test you will see a kind of index change you’ll see a spike okay so in case you have a onedimensional scan on the sample or you scan your beam you can see a two dimensional Imaging or in case you have
A two dimensional skin on Sample or you use two dimensional camera as the detector it have three dimensional Imaging so this human he follicle the video the time represent the depth of the he fle so so because this OCD technology is very fast so in the one hand you can acquire this kind
Of 3D imaging just maybe 30 seconds you can have this P you can also use this kind of the temporal response to add a color on on your Imaging for example for the 2D Imaging if you uh uh use the for example for transform you can find out
Was the the camper response okay then you can use the frequency range to add colors you have a Ando color on the 2D images okay so right now uh the OC system Technologies is Advanced very fast I believe right now worldwide uh one years ago I heard that there about
100 OCD companies but I believe R now should be much more because uh I just visit India uh two months ago people told me in India there are about 25 OCD companies just in India okay so so roughly about 80% of OCD company right now they working onology okay for
Example you can see this is the human retina yeah about 10 layers the average layer thickness is about 20 Micron so you don’t need a very high resolution you can resolve all the layers of the retina so people can use for diaba okay also uh for age related mular
Generation yeah so uh basically this is so these days people even try to use a longer wave lens so canate deeper than the retina to the Coro l so there are vessels here yeah so a lot of company working onology another very popular education area is the Cardiology okay so
You need a kind of f Pro so to P Into Your Vessel for example the coronary vessel so you can have this kind of 3D imagings okay be used to detect the vulnerable PL or in case you have you have install St within the the coronary vessel you can use the OCD to
Check whether the St is well positioned yeah okay so I menion OCD was invented years ago Professor James kimoto at MIT so uh the application area starting from Theology and the card uh card cardiovascular Imaging so in the early days people mainly care about the Imaging Ste because for biomedical
Education your patient they will move your IM May blink so need a very fast Imaging speed recently people not just care about the iming speed of OCD start to care about the resolution because you want to see the details for examp want to see the cell you want to see the
Nuclei because for many diseases for example for cancer you more detail information about the cell to make diagnosis so in either theology you can see the the number of application is growing very fast in card vascular ucation and the neurology uh for and dermatology and the gal anology
Okay so in all the area so if you look at the the mark by application uh so apparently theology occupy the largest p and the second one is cardiovascular application so this means that our publication really works right if you have more publication you have kind of
Larger market size okay all right so this is a pretty much present status of the ocp now let me show you some of our recent Works about U cellular resolution ocp so this one video shows the N Nas means that all this two dimensional Imaging is kind of in parallel to the
Surface so at different time Pro into the different deps so this is a processional that means at different time you have different lateral uh movement okay so you see those black holes actually they are the nuclei of the within the skin you can see the layer structure and because OCD
Is something very fast so you can see even the this flowing in the microvascular and one nice very nice thing for ocp is that not just that it can give you a very high resolution also the power actually is very low it’s typically just less than stre one you
Know for this red Point typically the the CW power for red point is more than 5 m so it’s even higher than OCD so you can imagine in general OCD is a technology that’s very safe yeah okay so you can on the human skin apparently you can also apply on the
Conia okay so this is invo rapid Imaging this is The Superficial layer of the cona this a wing layer this is the basal layer and these are the bman layer and you can see I don’t know whether you can see some dtic structure that’s the we
Call sural nerve okay so from the ocp you can see the layer structure ner structure actually these days there many people that uh the density of the nerve the tortuosity of the nerve are closely related to many diseases for example the Parkinson diseases type two diabetes if you have this kind
Of quantitative information from this nerve is very helpful to have early diagnosis of the diseases and for OC you can give you a high resolution structural information you can also have a very broken information from OCD remember the Life Source for OCD is a proant Life Source so so at different uh
Tests of the SLE you can you can detect the best SC Spectrum so from this spectral uh you can Pro for example there are many pigment uh in in our human body like such as like melanine uh hotex melanine and uh uh and several other uh pigments in the
Human steam so you can also get this kind of information from the and remember I mentioned for our OC is kind of cellular resolution so we have a number of experiment to verify so this is a kind of indidual study so for the same Cale we use ocp to scan so you can
See the image at different TS then we use the confocal microscopy to do the same same cell so you can compare so know that how to interpretate this kind of black y OCD imings and we also have legal experiment verify so on the left hand side you see these two are kindur
The H Scan Imaging for all the Physicians and clinicians they are very famia with this kind of uh uh uh purple and the and uh pink colors because for the purple usually you represent the nuclei but for the OC even though you have high resolution but this is the
Black and white so usually it’s not quite easy to be recognized by uh Physicians but if you compare the details for example here is a ha parle by the H this o this a f cell okay so you can see that really for the OC system there’s a kind of cellular
Resolution so once you you make sure you have the cell information so you can use the apply the AI so you can segment out the the nuclei so I think a few years ago we used the machine learning technique algorithm to segment out the the new fromal human schem but the only
Issue is that because we use the machine learning so it’s very slow to acquire uh to segment this image it took us about 10 hours so very recently try to Ed a de deep learning algorithm hopefully we can do this in real time to seg out this
Okay so in the future uh I think um because right now um Health expenditure worldwide is escalating very fast so this is the data in the United States you can see that uh over the years the the uh the National Health EXP uh the ratio of the GDP is kind of
Increasing so the uh you see the the cost is 4.6 trillion US Dollars kind of infinity I think U for OC is I believe in the future is a the technology that could be able to uh to redu to to solve this problem because really it cost too
Much money worldwide for the human uh Health expenditure because uh for many diseases in case uh you can detect it in the early stage it may be very easy to to be cure but in case in the late stage even though you spend a lot of money it
May not be helpful for example many of the audience here you wear the glasses Okay the reason you wear glasses is because one day you feel that you cannot see something clear so you go to the hospital and you the physician suggest you need to wear a glass but in case you
Can know that there’s something wrong in your eye in early stage probably you just uh more rest or when you’re reading something you have a stronger light you don’t need to wear glasses because uh for many diseases uh the stability of disease uh diseases dises for the functional of um the structure
Information actually is a more a better kind of indicator for functional abnormal okay because of a functional change is kind of a br change okay but for structure change usually it’s kind of a more linear because C you can detect a structure change so even for the function it seems like still
Something working well but structurally speaking start to change so OCD has high chance to do things in the early stage okay so uh as I mentioned because for OCD with a high resolution you can know that for example whether a cell is is process or process status or what is the
Cellular resolution or what what is the cellular interaction by the high resolution OC remember ocp uh we have a very Bren light sources so you can also have a spectroscopy information because ocp is very fast so you can also have Dynamic uh information so from this information
You may be able to find out what’s the activation paway or the system coordination so in the future if you combine this structure and the dynamic and the Spectros scoping information with the AI we could be able to have a early disease diagnosis so we can reduce the health health care uh
Expenditure okay so that’s a very brief overview now let me show you how we use the Deep blending on the C resolution we start from animal model okay so we collaborate with our University Hospital we just apply some chemicals on the back skin on the back of the mouse so after
One week or two week you can see that something change on the on the on the back of the mouse we call this aasia after maybe six or seven weeks you start to grow the uh SC cell carcinoma okay see these tumors it’s kind of tumor okay
So for each of the Imaging the sample we use the ocp to acquire the Imaging and we also have a BC so we can have this kind of h h& e stand Imaging this is surface ground truths okay this kind of annotation of the ground truth so you
Can see that we can have a very big dat so for this one we have accumulate 1.45 terab of data actually right now we already uh upload the data to an uh image uh repository idea so in case you’re interesting to play with the data you can just download from from this
Okay so basically with this big data we try to the first thing we want to do is that we want to classify whether using OC you can classify whether the Imaging is associated with normal SCH or with SCH or SEC scheme okay so as you can see
Here we EAS we can Achi more than 90% of accuracy okay then we when we discuss with the physician so the The Physician say well whether such a high accuracy is related to high resolution he don’t think that high resolution can help so we purposely try to uh reduce the
Resolution of our Imaging so what we did was to do a down sampling so when when you do the down sampling on your Imaging that means you reduce uh the resolution you can see that the accuracy just start to decrease okay that then the physician they are convinced that we do need a
High resolution so you can have high accuracy okay then another issue for the AI is that even though the AI can give you a pretty good result but people still feel that the AI is kind of black bu you don’t know what what’s going on within this those matric and datas okay
Fortunately these days many people they try to develop some tools so you can visualize the the AI algorithm so for this case we use the algorithm called gr you can see here remember we have three different kind of IM classes one is a normal skin a displasia skin this SE skin okay and
For for this colorful Imaging this is the result by the hand algorithm these are the he okay so the brighter the spot means that for those pixels they play a they play a larger role for the AI to make a decision okay so you can see here
The he one is the AI is focused on some smaller features for Heap 24 is for the larger features okay so within a human skin the small feature actually is related to the nuclei so as you can see here all the bright spot here match very
Well with the black hole here the black hole here is is the nule so which means that this cellular information plays an important role for the AI to make a decision whether the Imaging is a normal or displasia or SEC and for larger features you can see here this right
Band here actually is correspond to the thermal epidermal Junction okay so this is thermal epidermal Junction here and this one this is Stan okay so either the small feature or the large feature actually the the logic for the AI to make a decision is very similar to a
Physician because when we ask a phys how you try to make a decision whether this is a normal or SEC they’s start from looking at try to find out where is the DJ the thermal idal Junction then they try to find out look at the nuclear to see where’s the nuclear nuclear density
The size the ratio the shape okay so it’s so for the AI is very similar they care about the DJ care about the nuclear okay so this give physician some confidence but AI not just give you a good result also kind of explainable okay then later on we have a
Collaboration with mskcc those uh in this Cas there are many hist pathologist over there so they use our system to scan for the uh skin cancer patient so for each of the patient they make a scan on the OC and also uh because this skin cancer patient
They also have a biopsy so they have a stain Imaging so they can compare the stain Imaging with so you can try to learn how to interpretate this kind of black and white OCD because right now mskcc there are they have a lot of skin cancer patient so right now in the
United States there’s a surger call M surgery okay this something quite famous in the United States U because uh in the states each year there I think roughly there about 2.5 million new instance on skin cancer okay so the when you have a skin cancer in the one hand if the Physicians need
To do a exis biopsy you want you can clearly remove all the maligant cells but in case the the tumor is on the face so you also want to reduce the amount of exgen biopsy so the the best solution is that if you can accurately determine where’s the boundary where the boundary
Of the the maligant tumor and the benine tumor okay using you can try to see whe can uh determine where’s the boundary so for the most uh surgery the idea is that because you want to reduce the amount of uh the tissue removal removal from your
Uh skin so they just each time they just make a cup of s slice okay so after the slice and the patient will just uh lay down on the bench and waiting for because for each of the slid you need to the and do the stand usually take about
About one hour so the patient will sit on the bench for one hour waiting for the the result whether the the whole maligant tumor is removed in case yes then he can go back home in case not just have a second slides okay so in
Case using OC you can you don’t need to wait for one hour after each slide you can immediately make a diagnosis will be very beneficial for the patient and for the physician and for the hospital okay so this is the idea of the clinical trial so for each of the sample we have
Both the ocp and this stain Imaging so the uh the pathologist they can help to to segment out you know the Bas cell carcinoma boundary the hair body the spaceous gra and even the mus also this Imaging you can learn how to segment of
An OC so we can train train the uh the AI algorithm okay because very expensive to do clinical trial in the United States so so for this uh experiment we only have 23 imagings okay so these are the kind of annotation I call this brown Tru okay so different colors represent
The different tissues uh so for each of this patient actually the image is pretty large this is about 2 cm by 2 cm is a one M resolution okay so still a pretty not a very small data set such a PTI patient uh trial as you can see this
Is in know test okay so these are the input OCD imagings and these are The annotation and this is AI result you can see that yellow color is true positive okay so you can see the CNN result is compared to The annotation is reasonably well and if you want to have a
Quantitive assessment you can see for the cell detection the sensitivity is % theity is a little low is more than 80% but remember I mentioned for deep learning the larger the data set you can expect the better the performance because for this Tri it’s only 20
Patient so in case you have 100 1,000 uh supposedly you should be able to have a better result in the future but for the most surgery still it’s a kind of isal education so the true uh I think Advantage for OC is invo because uh for
The amount of optical power uses is very low so it’s very safe so actually uh so the company I uh established 10 years ago right now they have a invo system so you can just acquire the skin data you can find out where is the thermal Junction where’s where’s
Gros and the micab small vessel for this kind of black and whiteing even for physicians in case they are not familiar with OCD they cannot tell okay only when they look at this kind of Imaging for long they know how to how to uh uh uh
How to recognize how to read this kind of black and Y imagings okay so right now for this system was approved by FDA and also in Taiwan we also have a Taiwan FDA okay yeah so um so right now we start to have some clinical trials uh in
Taiwan right now we have collaboration with uh about 10 uh hospitals and the uh medical centers and worldwide uh right now we have collaboration with the first one is mskcc in New York also we have one system in Berlin and we have two system in Beijing right now for the
Clinical trial so because we start this kind of uh inal ucation so remember even for uh Physicians because for Physicians usually when they are when they woric uh student they are very very familiar with the AG St okay but OCD is kind of a new technology so we try to
Create uh something like a textbook okay we call this appus okay so for each of the patient will recruit we we take four Images one is the h stand one is o one is the typical clinical Imaging which just just like use cell form to take a
Picture of a legion and this is thermoscope thermoscope is very similar to clinical image just uh you can see a little bit deeper with higher resolution okay and you can uh so that one this is uh for the cancer actually we have thisal trial we recruit about 200
Patients in Taiwan only about five to 10 patients that really have Bas is nor so in Taiwan this is really not something popular and for this uh empd is also a medicant tumor so we have several patients like this kind of data okay and we also have some collaboration with
Aestic medicine doctors example uh these days um you heard about people use the to Second laser F second laser on on your face to for example you can remove the the navi the neas or you can try to stimulate the collagen when you use a p laser laser usually the physician they don’t
Know you can generate host INE your skin we call this a laser induced Optical breakdown okay so without the OC you have no idea what’s the dash of of this Hol because it depends on the the wave length you use the dash of this is different okay and depends on the power
You use the size of this uh breakdown is different so with the OC you can identify what’s the best wave length what’s the best optimiz uh Optical power to generate this kind of Las breakdown yeah so right now we remember we we have a pical trial for about 200 patients so
We think this 200 patients there are about 20 different diseases so we make an Aras so that for each of the patient we collect all the four imagings now so actually we have a PDF file for this upas so in case you interesting OC I can
Send you a copy so that you know how all these different skin related diseases how the What is the is ocp what is clinical Imaging and gcop okay yeah so based on this uh clinical trial data so we try to apply the Deep learning to extract more informations so remember
About four to five years ago we use the machine learning we can sement out the nuclei but it’s very slow so right now we try to use the Deep learning algorithm to see whe can uh segment out the nuclear faster so for this algorithm actually we use a very popular uh
Network called un is a very successful segmentation algorithm but the the the differ thing here is that because for our OCD imagings remember within this OCD Imaging we have some feature that’s very small for example the nuclear we also have some feature which is very big for example the thermal epithermal Junction
So when we train the UN we develop so called a deep feature sheding which means that we have a new loss function for this loss function you care about both the small features and large features so using the this algorithm to R out segment of the the BJ the certain Conan
And the nuclei so as you can see here these are some typical input of the ocp so The annotation is here The annotation by The Physicians and this is a prediction by the CNN if you compare these two qualitatively it matches reasonably well can ative assessment of this uh
Segmentation so for the new Pi uh we use the I as match is intersection over okay so the UPI is about 72% for the BJ is almost 90% yeah so so the P you can apply this the this uh Network for 3D segmentation so once you have this information so you each
Imaging you can have a lot of usually roughly about 100 nucle so you can have a lot of different kind of statist statistics about uh the density and the shape of the for example the nule sometimes it look like a sphere sometimes more elliptical so you can have all these kind of statistical
Information okay also at different human locations you can see that all the parameters are different okay about the second part of my PO and finally I want to show you a recent work um how we uh try to translate the OC into the colorful H St
Imaging this virtual B so which all you can see this Optical B which means that there’s no really uh exisal bsy okay because of H is is something really very important because uh for many diseases for example for cancer the Imaging is kind of the go standard okay yeah and
Because for the Imaging you have the color specif different color represent different things but for ooc even though you have high resolution but this is black and white so how can you so in case you can translate this kind of black and Y Imaging into this colorful
Imaging that be very helpful for the Physicians to to read this kind of black and Y images fortunately right now worldwide there are many uh algorithms that people have developed to translate Imaging for example you can translate a painting with one style to another style
Okay or for example for photo this is a photo of a winter you translate into a summer okay but the the difference is that for this kind of image translation it’s kind of artwork okay so you don’t care about accuracy okay just you translate the winter to sum right you
Don’t care about accuracy but if you want to translate OC into H this something really care about accuracy if the accuracy is low it may cause humans life okay so try to uh So based on this existing algorithm we try to add in Physicians knowledge so that when you do
The translation you can have a better accuracy okay so right now uh for the Eng translation there’s a very famous algorithm called CYO I don’t heard about cyle again so basically you have two data set this is one data set OCD this is another data set H okay so these two
Data set actually they are they are independent they are not pair data because it’s almost impossible you can have a pair data of OC and the H okay so this so the the nice thing for cycle gain is that it’s not a pair data okay
So start from the OC you can you can translate OC into a call is a converted H apparently start from beginning this is a wrong H okay but you can but you can conver H to OCD again we call this reconstructing OCD so in case all the
Conversion is correct so you can imagine this o reconstruct OCD should be the same as the original OCD similarly for HG you can convert HG to OCD from this OC you convert to this reconstru HD so if everything is correct then this HD should be the same as this original okay
So this is the typical cycle game on this cyle G we try to add in Physicians knowled for example remember we can segment out the the thermal epithermal Junction and we can sign out the nuclei so we ask the Physicians help us to label the nuclei and the DJ okay so make
A l function of this OC and this convered H so in case this conversion is is close to a true they got this two got to have the same DJ and got to have the same nuclear allocation simil H okay so let me show the result so these are the
Input back y OC this is translated okay so if you look at the DJ which I used the green line the DJ matches very well and also the nucle location match very well um we have a quantitative uh assessment so if you only use the cycle G the arrow usually
You can see that so this pretty large using our algorithm is the pH ination so the uh for example for the DJ the AR is only about 1.6 micr which is much smaller than one cell okay so this is a Reon good and this is IOU is about 90%
We also is a PSN coent we can see that so it seem to me this kind of English translation at least for the DJ and for the nuclear it seems works okay but within the human skin you can imagine there so many different things right so
Not just the DJ nuclei you have you have melanine you could have microcirculation you have all kinds of different cells and eventually we hope that uh this kind of uh English translation can be used for uh the tumor boundary uh segmentation so I think start from
Beginning right now if we are adding The Physician knowledge on this so in the future in case we’re adding more information so I believe this kind of image translation is really there’s a possibility to make this work um but the only issue is that remember for for deep
Learning we need a big data and for this algorithm you need a Physicians to label for you so in case you have a collaborator with a physician you asking well I have 20 images can you label for me most likely they will say yes but if
You telling I have 1,000 Imaging can you label for me well probably it won’t be agree okay so um so so when we develop this kind of AI algorithm you got to think about how you can reduce the amount of time that the physician need to help you okay so very recently we
Also tried to develop an un supervised OCD to H translation so for this algorithm you don’t need any Physicians knowledge okay so I think this algorithm can Ser as a kind of pre-training okay so once you use this kind of supervise to pre-rain your network then later you
Can use a small amount of data with the physician knowledge to find C your networ okay so this translation something very interesting remember we have two data set one is HG data set one is OCD data set okay so right now I don’t have any segmentation
But I will try to add in some noise because for example starting from OC you can see that if you compare the OCD Imaging this H the HG usually look like kind of high quality Imaging but OCD is well known that there a specal noise okay so we you do the image
Translation you don’t want the network to spend a lot of effort to focus on the noise okay so you try to un purposely add in Some Noise Okay so so you can mitigate the OCD Speck noise impact okay then from this H to the OC adding some
Noise because uh for this kind of usually people found that with a small change on the H then the output may change a lot so if you adding some noise you can defend this people visal attack okay also from here to here adding noise because uh see you it’s not it doesn’t
Make sense you translate a high quality Imaging to a low quality Imaging so you can add in the M here so that uh the the network can focus on the important features okay so also for this part we adding Noise Okay so we’re adding four different noises okay so that to force
The network to Lear something important and ignore the noise and this is the outcome okay so this without any Physicians knowledge okay so we have input OCD and this is a translated H okay so you can see that well even though it’s not the accuracy may not be very high
But at least they looks reasonably well so this can serve as kind of pre- training and also for this algorithm is pretty good for by directional translation you can also Translate uh H back into OC you may ask why why people need to do this because for H the
Physicians already can read everything you need why why you want to also translate H back to OC because for within each OC right now we can recognize the DJ the ther Thal Junction we can recognize where is the nucle but actually within this IM there are many other
Things for example many spatial cells for example the Merc cells the gangling cells even nerves okay those information right now even myself we canot recognize but if but those information from the HG St is very easy The Physician can recognize so if you can translate the H
Back into the OCD so maybe it can help us in the future try to uh try to read those spatial cells okay so using all algorithm we have bench mark with many different algorithms we call our algorithm the signal scan okay weuse FID and the kid as the magic so for these
Two magic the smaller the better okay so you can see that using our algorithm we have best result compared to other algorithm yeah uh so we yeah we just published this result this year okay so seem to me uh uh deep learning is something really uh useful for for
Biomedical Imaging we know that these days for AI generate a a lot of public people care about AI people also worry about AI so people ask that whether your AI is kind of trustworthy whether your AI is responsible whether your AI is ethical whether your AI really is
Beneficial okay so you can see that the core idea behind these questions is that whether your AI is fair whether your AI is isable so when your AI is fa it come cares about whether you can protect the privacy of your patient okay whether you can you don’t have any racial
Discrimination but this kind of issue can be solved by the IRB because uh for any clinical trial you don’t need the IRB to review okay so that kind of thing even though it’s very difficult to make a judment but can be solved by the IRB it’s planable AI really is kind of issue
How the data SC data scientist you can Comm is a public to convince people that your AI is really reasonable so fortunately at these days there are many uh kind of algorism that people develop try to help people to understand what this kind of Black Box really doing you
Know to give you results okay so remember I already show you one example I show the Heap right the Heap is kind of only a kind of region show where is hot spot but the many uh many uh algorithm that can help people to explain the result
For example recently we tried the soal shop the shopy parameter so from each of our images we try to extract some uh features for example from the human skin you can find out what is the average cell size this is something important related to diseases what is the standard
Deviation of the nucle area uh what is the average length of your DJ what’s the amount of melanine what’s the sickness of your St Conan and uh yeah so we try to exract many different parameters okay because this parameters may be related to whether a Imaging is a is a cancer or
Not whether it’s some diseases or not okay so right now because in Taiwan remember I mentioned we we have collected about 200 patients among these 200 patients there are only about five patients that have skin cancer so the the the amount of data are too small but
For eima for some reason in Tai we have a lot of patients we have more than 80 patients so we use the exema as an example to see whether we can explain why the AI can give you a correct diagnosis okay so for example uh using
The shot we know that uh in case of patient you have XA okay so apparently you should have a very sick uh epidermis okay and also the uh nuclear uh area the standard deviation should be high okay so so based on the a features I just
Mention for you so we can train the network to see which one are important which one are not okay so a a a second show here there are four example from this four patients okay with eczema the first three Imaging the AI give a correct diagnosis the AI found that well
This IM come from xma patient why give you a correct answer because for the AI it really diagnoses that the patient have a very sick uh acmis okay for all the three patient for the first one even though the patient is still X am the a
Give a wrong uh diagnosis the reason the AI give the wrong diagnosis is because the AI cannot diagnoses the canot the correct ad sickness okay so at least using this approach people can know what are the factors that affect the AI to make a decision okay so including those
Uh the heac okay I hope that in the future so people don’t don’t need to feel about AI they can understand how the AI make a decision okay so with this I would like to come come to my call U so so I hope that I can show you that
With the Ser resolution OC you can R the nuclear mology and some lay structure in real time and uh also using the Deep learning algorithm you can segment out the very important features and also you can translate the gr label ocp into the colorful super color stain Imaging and I
Think for the Deep learning Alm this Ser resoltion OCD is really explainable and can be linked to other IM for diagnosis so uh for my work I need to acknowledge to my collaborators on the ocp life sources on the AI the product of the company and also my collaborators of the
Companies of hospitals yeah uh thank you for your uh attention thank you right so we can have question it’s a beautiful talk I was this might be a thumb question because I’m not from a computer science field but as I think your technology actually and one of of cell actually
Larg volume nucle you try that to actually do take a scan of the tissue and recognize that precisely where the cancer cells depends on nucle shapes or volumes one example that we have a collaboration with mskcc that is an experiment we try to segment out the the uh the boundary of the uh
Boundary humor I think yeah um so these are annotation of the boundary of the M tumor and the yellow is the true positive predicted by the CNN yeah and I think you mention very important point because right now for h stand It’s the Go standard all the
Physicians they know H they believe H but this day they don’t believe OC so I think still take a long time for the Physicians to really accept o um but I think for for h St you know um let me Imaging um for the for the h stand kind of
Imaging even though it’s a very high resolution in the lateral Direction but actually for the tissue there’s a sickness okay yeah so typically the is maybe four Micron 10 Micron for some tissue maybe one or two Micron the typical is four Micron but for OC we have a high resolution in this the
Lateral direction also in the this direction yeah so I think U even though the for the OC C is bre why but we have more information we have 3D usually for HD it’s only just 2D slide you can have 3D slide so I think there’s no U it’s in
The future really there’s a possibility we can persuade the medical field the OCD really can be used for know I don’t want to say replace actually can uh can confident to stand yeah thank the question uh just to remind us uh in O the uh as you shown
You we get depth resolution and that comes from the the the bandwidth of your light source so that’s why you want to Broad Source um but what I trying to remember is as you’re scanning in depth how are you able to extract the the Spectra as a function of depth oh
Because ocp will call this there’s a kind of optical sectioning so for for the uh micro frer if you use a laser you always have interference pattern but for OCD because it’s a Bren light source so only when uh uh when the the the back Reflection from a certain deps match the
Length of your reference down then you have a signal so we have kind of dep section in capability yeah and then what you do then is is then at that point is uh that spectrum is what you’re yeah yes okay and I was going to say the uh I
Think in general what what you’re doing is from a high level is you’re basically capturing lots of images and trying to find trying to see things that uh the computer can pick up uh from that in order to diagnose uh a certain thing so
I was going to ask you in terms of of diagnosing either eczema or skin cancer what is your standard scan is it like a a centimeter in other words you know the image that you use that that you’re uh for the diagnosis is that some standard uh you know sample you know
Coming from someone’s hand or or so right now for the inal uh Imaging we have a prototype I think the so basically uh The Physician they just this is the probe okay so when they look at the your legion so they can decide where to Pro and right now for this
Function actually in the future we are trying to you can also have a lateral scan because for each the few of view for each Imaging is about uh 500 Micron withd by a depth of about 400 micr Al so for some region this maybe too small so
But in case you have some kind of stage regime so you can have skin larger yeah so that depends on kind of deases the physician can make a decision how how big they want to P okay so so so so so this isn’t a question of saying of
Scanning just you know hundreds and hundreds of people and then those that are that have skin cancer then looking for something this is actually looking at the cancerous regions and then uh diagnosing that yeah and try to find out where is the boundary of the and the
Pite tumor where’s the found so then I guess projecting into the future it’s just possibly uh a general sensing technique you know that uh you go to the doctor or annual checkup and we get a oct scan you know I mean is this possible that uh that you could you know
Uh find these images find these things in the images only enough that we could use it for diag prediction that’s our hope yeah but you but you know but for bi medical field it’s kind of very conservative so everything progressing a very slow pace uh for example right now
We have a clinical trial it’s only for 20 patients people don’t believe a result from 20 patients you got to show me a 2,000 1,000 patient then people can start to believe why it works yeah uh so and even though you have a 2,000 patient you have a good result it doesn’t means
That hospital will use it because these days you got to have the insurance company for reimbursement right so starting from you got to have a system that approved by FDA then you got to have a small clinical trial so you can have kind of people call CPD code
Because you want to reimbursement you got to have a code once you have a code it doesn’t means the insurance company will reimburse you got to have a large clinical trial okay the still uh have a good result even that case the insurance company will still they may not
Reimburse this this kind of diagnosis because they want to make sure in case they reimburse for your instrument they can reduce their cost yeah and uh even for the insurance company they can reduce cost the hospital may not want to use because they have the soal I think called HTA Health technology assessment
Whether the hospital can make money yeah so everything is quite slow yeah so uh so actually for my company um 10 years ago when we uh got our first investment people believe that use for most surgery but later on we realized well it cannot happen in 10 years and the the new
Company cannot supply for 10 years so we change to indal system Target for other educations yeah so everything’s very slowly yeah you know for my clinical trial I attend to recruit 2,000 patient in Taiwan so I I write a proposal submit to theit hospital they they review my
Proposal is they spend 18 months for the IB to approve this clinical 18 months year and a half yeah for the our supervis IM translation between OC and HR and hmb um what is the hidden parameters or hidden correlations that this Imaging translation is unsupervised algorithm is actually
Optimized yeah um I’m not sure whether our algorithm is optimized or not um so this is the superv um actually this work start on my my students idea u i first because without without uh without this four adding noise it’s just a typical cycle game okay so in
Literature people use cycle gain they they start to have some a reason reasonable EV translation but with this the four noises we just have a better result and the reason why just a typical cycle gain without any pair data because this HG and this OCD is not pair data uh
I think because U there are some algorithm works for example uh for this H after two trans station this one has to be the same as this one right similarly for OC after uh you translate on original one to this Recon one you have the same then for the cyle game
There’s also a discriminator here because it can charge whether you uh convert this kind of uh uh convert fake H whether this is a really like a true H we can use a discriminator so because of this kind of cycle consistence and this kind of discriminator they start to
Build up the relationship between the H and ocp yeah so I’m not sure whether I know where is the he parameter that I’m not sure so this is really a I was curious if you have enough data yet to look at lesion on specific parts of the body
Thein you’re probably well aware little um does that have any effect on your results can say that yeah like so the skin on your hand is quite different than on your face you have enough data to distinguish these yet or maybe there is no difference no differ lot for example the
Sickness of your straighten cona on your face is very thin on your foot is very sick yeah so I different location is quite different yeah um but right now for the interesting you know for basal cell carinoma it’s quite frequently happen on your face yeah uh but but but
I think you are right in case uh that’s why people need a big data you got to have all from different locations then the you can Trend the algorithm well right yeah segment andarge is The Benchmark and is it publicly available okay right answer your last question first right now only
For the animal model who have a data assignation upload to the IDE everyone can download for the our clinical trial U collaboration with the hospital because I I don’t think I have the right to release the data you got to get a proof by the hospital also right so but
In case people are interested in those data say I think I can give you several some of the data set I cannot give you the whole data set probably it’s okay I think question you ask is about how costly this to oh yeah actually yeah so
For The annotation it really cost a lot of time so usually right now in our case um we have a c they help to to circle out the segment then we give the data to The Physician they they have a uh double check the result yeah to have this kind
Of annotation it really cost a lot of time yeah that’s that’s why I mention you you got to think about how to reduce the time me for the physician it’s very important especially in Taiwan all the Physicians they are very busy because their Sal is linearly proportional the
Number of patient they you know they they take care yeah thank you let you mentioned with the uh h& that you can only get uh 2D images but with ooc you can get 3D can you actually translate a 3D o image to um a 3D quotequote HD
Image exct right now we are working on yes yes yeah um there are some yeah so I told my student because um usually for The annotation of Medical Imaging actually even for physician they can circle out where is the BJ where is the nuclear actually it’s not very accurate
Know because for many reasons okay it’s not accurate but in case you have a 3D imaging if you uh translate uh your OC into H you can imagine so for 3D imaging it’s composed of from from this way or from this way so in case you have a
Correct translation from this way or from this way the result should be the same I think this is a better ground truth than The annotation yeah so that’s something right now looking up yeah we hope we can after