In this program, we address the cardinal points allowing efficient digital technology transfer between academia and medtech industry. Research and development (R&D) strategies for conducting solid translational research. Creation of an environment in which innovation (I) relies on complementary resources and expertise.
Hi everyone I hope you can hear me uh I’m David from the organizing committee I will quickly remove my share and welcome to the next installment in our seminar Series today we we are discussing MRI the topic is MRI and machine learning uh we have with us Ricardo dalbello from University Hospital
Zich and the topic of today is bit uh hoto synthetic CT from MRI images for RT planning radiotherapy planning and um well the question is uh the underlying question is can we get rid of cities for radiotherapy of course he a bit simplified and provocative um before I
Let Ricardo speak just one reminder who needs a certificate of attendance um just leave the email in the chat and then we will send the certificates in the next um days um quick introduction of Ricardo so he’s a medical physicist from Italy he studied in Trieste originally his topic was
Diagnostic he was working with breast Imaging with synchrotron light and then he switched to the other side of medical physics that’s uh radiotherapy so he was in Germany in Heidelberg working with radio therapy and uh ions and then he moved to us to complete the training in
Medical physics and now he has um uh 5050 position half clinical duties half research and his research focus is of course Mr MRI only workflow and uh automation MRI and uh AI so I hope I didn’t forget anything Ricardo oh that’s a lot thanks a lot for the introduction
And thank you um just one comment I’m I hope everybody is seeing the slides if not please uh stand up uh speak up um and then yeah Ricard you can start okay thank you David and welcome everybody to today’s seminar so the topic will be how to use synthetic CS
Generated from Mr Data for what we call Mr only radio therapy and I’m aware that this is a seminar Series in of Diagnostic Imaging so I will also take a little bit of time to introduce the topic of radiotherapy and why we need Imaging in this context so uh this will
Be the outline so as I said let’s go um ahead and talk about the motivation for synthetic CT and radiotherapy I will discuss some commercial solutions for the state of theart part and then we will come also to the uh developments that we have been carrying on here at
The USA and the general topic that we will uh discuss today is represented by the figure here on the right so basically we have an MRA MRI data set and we want to convert it to a CT or synthetic CT data set but we also want to be uh extremely careful with what
Regards the quality of this conversion so let’s get started with the introduction and this is the radiotherapy workflow or as We Know It uh nowadays so the current workflow works as following we always perform a CT scan of the patient and uh we need a
CT scan because we need to use this data to calculate the interaction between radiation and matter and to calculate the dose that we deliver to the patient or that we plan to deliver to the patient uh in many cases the CT scan might be sufficient um but nowadays it’s more and
More common to have additional Imaging available which could be Mr Mr Data or even pet data that our doctors use in order to better Define the target volume and the organs set risk surrounding it and here you see the first problem that the current workflow has that we are
Dealing with diagnostic mrr SC pet scans and we need to register them to um CT scans performed in treatment positions so I will show you some pictures we have some mask systems for example to keep the patient in place and of course if the position is different between the
Two SCS we introduce we start introducing some uncertainties well we perform our planning and then the day of the treatment we perform additional Imaging that could be planner KV or a com beam CT at uh our Linux in order to reposition the patient to the best position possible and also to monitor
The patient during the treatment so this is the current workflow and in what is called Mr only workflow we basically would like to replace all the Imaging modalities with Mr Imaging and this includes both in the treatment planning and in the positioning and monitoring so here I highlighted in
Green what is already available in the market and um and in currently in clinical use and you see that for positioning and monitoring I already put this in green because there are the so-call Mr Linux us is operating one of these uh where instead of using x-rays to position the patient
Every day you have a Mr integrated with a Lino for um for treatment positioning and uh uh and monitoring so the topic of today uh is this uh I hear some background voices maybe uh participants you can mute yourself thank you um so the topic of today is
This red arrow which is the conversion of Mr Data into electron density or uh CT synthetic CT which we need for those calculation and also how to perform quality assurance on this but why do we want to move into uh Mr only radiotherapy the three main motivations that are commonly reported in literature
Are the following to reduce the radiation exposure to the patient during the planning to eliminate the registration uncertainty since we will now only have one uh image modality and to be more cost effective since uh we need to operate only one um one Imaging devices but what I think is also very
Important and often forgot is that we already perform MRI standard of cares for uh many cases so if we can just use this data then uh there is an intrinsic benefit and um yeah so here are some examples of a prostate and head and neck case where you see the superior quality
Of MRI in the defining the target structure and also the surrounding organet risk and here for example on the top right you can see uh in the CT this mask system that we use for fixing the patient to a given position and be sure that also during the treatment um the
Patient stays in this position and ideally we will also like to perform uh MRIs in the in this setup to reduced the uncertainties so the current workflow nowaday is this Mr Plus CT uh which is well established this is current technology and it’s current standard of care however um there are some
Registration uncertainties and is a bit cumbersome from time to time because maybe we need two appointments on two different days the patient has to come twice uh to prepare to start preparing the treatment uh instead we want to switch to this Mr only workflow uh which however um requires some developments in synthetic
City so and these developments so they come from the realm of artificial intelligence and uh and neural networks in particular and I like often to present uh this graph where you can see the hype cycle for artificial intelligence and uh if you check where we were in
2022 uh you can see that synthetic data is the peak of inflated expectations so everybody is expecting that maybe synthetic cities will totally um remove real CS from the radiotherapy planning but uh let’s see how will it go in the next years uh so what I think is to
Important to keep in mind is also the understanding of what’s going on how to and this is important for QA because if we have a lot of data and this is needed to train our networks we can basically live in a world without physics so we have a lot
Of data we train our Network and we get our synthetic CP out of it and uh this is very good for automation so we can perform this process in the background without much human interaction however we also want to perform quality assurance and for this I think it’s very
Important to have an understanding of the single steps of what’s going on and maybe we have a bit less data uh but we can troubleshoot in case we um we see some differences from our expectation so let’s go to the state-ofthe-art and let’s check what are the commercial Solutions available to
Perform this conversion from Mr to synthetic City and uh first a historical picture so the use of neural networks for this task has been proposed in 2017 by a paper by hanal where they firstly investigated the conversion of Mr Data of the brain and you can see
That ever since this first proposal this topic has been expan growing so this data stops two years ago but if you search on pment this this topic is exponentially growing and most of the data comes from high U field images or above 1.5 Tesla and is using this unet or gun architectures
Which require pair data and uh not surprisingly brain is one of the most interesting sites to look at and the first ones that uh have been investigated because due to this Max F Max fixation system um you can have um the same positioning in multiple Imaging
Modalities and also the brain is a regid structure which if you didn’t have the MK system in DMR you can register to your city but also had the neck and pelvis have been hot topics and uh have been investigated and Commercial solution are available we see a fewer number of
Investigations for torax lungs radiations for example and in the abdomen mostly because of the moving anatomy and the lower quality of the Imaging so there are several companies in the market providing solutions for this and uh again here as you can see brain and pelvis are somehow state of
The art and they are clinically available commercially available with uh certified product head and neck there is one vendor which has it available but uh due to data scarcity uh this is also ramping up and there has been there have been quite some developments also in this field and then abdomen brass torax
Are still under investigation so before looking at some of the images and some of the data um I’d like to show how this conversion is performed and to do so I will use as an example one specific Network the cycle gun which we have been using in our
Studies and I think is also pretty easy to understand what’s going what’s going on so the advantage of cycle gun compared to other architecture is that it can be used uh also for ampair data and this is very important in the abdomen where you have an physiological changes and basically you have stomach
And bowel feeling that are different between the Mr and the CT scan so your images are intrinsically not paired images they are showing you different anatomies so in this case we we can use Mr and CTS from same patients but they don’t need to be registered and we have
Several ingredients or components that we need to use to to perform this conversion so the first ones are generators are functions that uh take take one image from one domain and transform it uh as a matrix in an image in another domain and we need two generators one one from transforming Mr
Into CTS and the other one CTS into Mrs then we need discriminators to basically decide or uh evaluate whether the image we are looking at is CT or an MR and then we have to quantify the difference between uh between the real and the
Generated image and to do so we uh do it in a cycle so we perform the following we start with anmr we convert it into a synthetic CT and then we back convert it uh into a synthetic Mr and if everything is done correctly basically going from
One domain and to the other and back to the original domain we should regenerate the same uh Mr that we started with if we don’t we we have a difference between the images and here is where we put our loss so we can uh try to optimize the network by reducing this cycle
Consistency loss and then we do the same starting uh from the CT so generate synthetic Mr and then back to a synthetic CT and so this network is uh robust then of course we train uh so many different components of this network the only element that we are
Using at the end is just this function G to transforms Mr into uh CS very well so um once we have our uh our Network trained and ready to transform data we uh we have to consider how to implement this into a clinical workflow and there has been a paper
Published this year actually proposing a stepwise approach with five faves in order to switch from the classical workflow that I discussed at the beginning to the Mr only workflow and all the intermediate steps are some intermediate workflows done during commissioning evaluation and uh F tuning maybe of of your
Network so the classical workflow is what I’ve discussed early phase two um is basically using mrrs into uh planning position and uh not anymore in the diagnostic setup and this is some something that we have been doing at us since 2019 so we acquired quite some experience with this Mr linu
This hybrid device where patients are imaged and treated in the same device and therefore they also have a fixation system like masks for head at the neck or this vacuum pillows for for other locations and you you can see that um we have treated several cases from brain to
Prostate head and neck even the in the torax region so phase three then um is introducing the in the workflow the synthetic City or here in this image also called C so what you do in phase three you do your planning Mr in the treatment setup with a flat bed stop and
The fixation system you still acquire a planning City and you fuse this two so as you will normally do and you perform your planning on the uh on the planning City however you also generate a synthetic City out of the Mr Data and you can retrospectively evaluate uh how
Well your synthetic city is compared to the real City and this is what we have also been doing in several studies with networks that we have developed here at usad uh in order retrospectively evaluate how well um your network performs and in general you can see here
Some reports from Eran and from the Harvard Medical School normally those deviations of the relevant targets and organs at risk are within uh 1% so at the end we are interested not only in having good images but also having those calculated accurately on such images phase four is then what is ongoing right
Now at the at the usad in our department we are performing our planning Mr we are generating our synthetic CT and we are performing the planning on the synthetic City not on the uh on the real City but still in the background we are acquiring
Um a CT in order to compare the true results and be sure that our synthetic CT is accurate enough for our needs and accurate enough for us has two meanings the first one is for dose calculation we want to make sure that if we calculate those on the synthetic CT we get the
Same dose that we will get on the CT and we can quantify this also with some other simplified approaches what I mentioned before with a bit less data and more physics into the into the context and I will come back to this later but also for us is very important
Also the matching so then when we go and perform the treatment we perform our kbkb or comim CT to um to position the patient and we want to make sure that the registration MR2 CT or Mr to synthetic CT um and then including also the com CT in the in the in this
Triangle delivers accurate result so there is a two-fold evaluation on those and uh on the matching quality and then phase five once everything works uh uh properly one can go into Mr only planning and remove the uh CT from the treatment planning workflow and this is what we have
Already done for um male pelvis so we are nowadays under certain inclusion criteria treating prostate um with without performing a CT acquisition before um before the yeah before the treatment delivery so we perform only mrr Imaging and we have observed that the the dmetry uh is good is within 1%
And also the preliminary evaluation of the matching uh is within our our requirements and that’s why we went into this uh we will do it also for female pelvis and for brain the invest iation is still ongoing so what do we do practically so we have a solution by
Seens and what we do is that when the patient comes we ac acquire um yeah the diagnostic images that are required for contouring and then on top of this we also acquire a T1 Dixon which is which has a large field of view has Distortion correction and this is used to generate
The synthetic cities of course one can only add more sequences into the protocol in order to support contouring and then the results that we got are the following for brain on the left for pelvis on the right you see the Dixon and then uh we have the generated
Synthetic CT where you can see the classic contrast that one will get also from the CT however with a bit more blurring of the bones since this is generated from m data and the time requirements at the 1.5 Tesla scanner that we have in our department uh range from two to four
Minutes so the additional time required to do to acquire this uh Dixon sequence in the order of three minutes so as a summary for the uh clinical part or let’s say the state of the art uh there are site specific Solutions that are available and certified the time
Requirements at DMR scanner is increased by two to three minutes with the patient already in the scanner and uh yeah the do imetric and uh matching performances are good but every Center that wants to implement such a solution has to prospectively evaluate this however quality assurance Solutions are
Currently not provided by the vendors and and therefore we have looked into to uh what one could do and uh the idea is to perform an additional calculation on uh a different electron density map that could be extremely naive so we could just say okay our patient is water let’s
See what is the dose there or there have been some reports here on the right where uh people developed a patient specific Phantom out of the Mr and uh and basically you can get to those deviations smaller than uh than 1% uh of course we want to be somewhere
In between for the clinical workflow and this is also what we will uh discuss later on so now I will uh quickly go over some developments that we have had here at the at the US St and uh this is true fold on one side the generation of synthetic CD4 sides
Which are not currently standard of care such as abdomen or head and neck and the other part is the equality Assurance so first of all for the abdomen uh we use this network the cycle gun which I discussed earlier and out of this Mr Lina um we we acquired retrospectively
Data from almost 200 patients 186 and these are low field images so the Mr that we we have in in our scanner combined with the Lin is a 035 teslam Mr and uh yes and these are the features of the Mr Data that we acquired so some
Images that I hope that speak More Than Words um so on the top you see the Mr that we started with and the and at the bottom you see the CT acquired with the patient lying on the same setup but uh within half an hour usually within half
An now where we acquired the DCT and you can see that the air pockets in the stomach they move quite significantly so these images are uh are not matched to each other they are not pair data sets uh because of this physiological movements and uh that’s why we use this
Cycle gun architecture because it can compensate for this and as a matter of fact when you then generate your synthetic CT out of the Mr Data you you can see that the air pockets are reproduced correctly and I’m not discussing air pockets just because they are nicely visible but also because they
Are highly relevant for us for those calculation because uh the radiation that we deliver is not attenuated through hair and therefore if there is or if there is not air this plays a difference in the do deliver to the Target and to the organs at risk
Therefore we have to take this um great into into consideration so here another case where you can see the city in the lung window also the lung tissue is nicely reproduced for this treatment of the liver but at the end we are not interested uh only in image quality but
Also in the differences of those delivered and calculated on the two images so the synthetic City or the planning City and here I reported U the dose different the relative dose difference for several structures so PTV and gtvs are our targets and then some organs at risk in the region and you can
See that the mean those differences are well within half a percent with Max deviations limited to 2% so this was uh the case for the abdomen then for head and neck we decided to use a slightly different network uh this residual V visual Transformer and this has been also
Recently published I think the paper was accepted only last week so this is also a preview and also in this case we used the low um field images and we trained the network in a pelo 3D approach where we didn’t feed all the 3D Volume in one go but we basically fed
The three orthogonal planes uh as three different input channel of the network uh and to then generate uh our our output and also here you can see the differences between the original Mr and the synthetic CT here on the right and the they are very similar so also in this case for those
Calculations we hardly see any differences and therefore we also wanted to look at something else uh which is how to Contour or automat IC Al Contour uh structures in on such images and the reason is that we also use neural networks in order to smooth and speed up
Our workflow such such that we don’t not have to um manually Contour every single structure that we have in our treatment plan but we start with some automatic contouring performed by a software in this case it’s a commercial solution mime where uh we can feed a CT into the
Uh into the into the software and we get out uh the uh the Contour of some structure such as oral cavity salivary glands pards and so on and the idea is that if we go into an MR only workflow all these autoc contouring softwares they have been developed to be used on
CT data but if we feed synthetic CTS which have a bit lower quality are we getting there uh are are we achieving the same quality of the Contour that we get out and the answer in this case was yes so you can see here for the oral cavity for example
The differences between the autoc Contour on CT and synthetic CTS are very similar if then um are compared to what our doctors accept at the end and uh and if we compare synthetic CT to uh CT itself we hardly see any difference so here is the dice score coefficient for
Example then the last topic that we uh investigated is the quality assurance how to perform quality assurance of this uh synthetic City that we generate and uh I would like to start with saying that there are at the moment no National and international guidelines on on this topic so there are several working
Groups at uh European level at a Swiss level in order to yeah start with the stateof the- art surveys and so on as you can imagine in order to go towards guidelines but these are incremental steps and we are not there yet uh what happens is that in the review papers
Or novel papers with new science you see basically a zoo of parameters that one can use to evaluate the image similarities so we have over 15 to 20 parameters that one can evaluate but then for clinical use one would like to simplify all of
This and uh the idea is that once we go into M Mr only workflow we do not have anymore this CT in order to retrospectively evaluate uh what’s going on and and evaluate the quality so one can has to find different solutions and one elegant solution I think is that
When we comim C for positioning the patient one can recalculate the dose on a corrected comim C because at the end this is CT and um it has been shown that for prostate one can achieve those deviations uh within 1% so this is one possibility the other possibility is more important when you
Go into Mr Lin treatments where you do not have thect and uh as I said earlier you you have a range of different complexities that you can use in order to uh to test your synthetic CT and what we tried to do we tried to start it from a very naive
Representation of the patient with water to go into bulk densities that you can automatically or manually Contour so you say okay there are five uh tissue classes which I know the density of Let’s uh have a model of the patient based on this or you can have for example a second independent neural
Network trained on a different set of patients and of course as you increase this complexity you also expect a better Precision of your quality assurance and this is also what we observed and we investigated this for standard cases cases where there are this massive air pockets that I showed you earlier where
There are lungs uh in the along the beam path or even with implants and you can see that if you use this independent neural network so the um the orange bars here uh if you span over all different cases uh you always within one one to two%
Deviation outliers never go out of the 2% bars so I think I will skip this because we are in interest of time and uh I will just show you a extreme case of what happens when we generate synthetic CTS uh of patients with implants uh what
Happens is that most of the time these networks are trained on data that does not have uh artificial implants within the training data set so as you can imagine in DMR we do not see the in the implant it is not present in our training data and therefore in the
Synthetic CT this implant just disappears and instead if if we perform a CT simulation of the patient the implant is clearly there and in this case it was even within the target structure and uh if you compute the do differences you can see that you can underdose or overdose this local region
And this is one of the reasons why for um enrolling a patient into Mr only workflow is very important to screen the patients and be within the limits of the training data set of of your network otherwise you have cases such as this one which are clearly an exclusion
Criteria and a C is needed so to conclude um I showed you that there are site specific commercial solutions that are available and can be implemented in the clinic however guidelines are not available yet on how to do this and therefore yeah some some work is ongoing on defining all the
Quality Insurance criteria synthetic cities the image quality is a bit lower compared to the um normal planning cities however they are suitable for dmetry and for matching and patient positioning and quality assurance is a is one of the most important topics that I think needs to be investigated since
N2n testic is not possible uh we always need CT of the patient to perform this endtoend testing with software solution we can be within 1 to 2% and we have to be sure that we do not increase the total uncertainty budget of our radiotherapy workflow uh by using the
Synthetic CT and then of course the the complete chain of quality assurance includes also other factors such as Imaging QA data transfer and so on which I didn’t discuss in this presentation so with this uh I’m done and thank you for the attention and I’m happy to take questions thank you very much
Ricardo uh very very interesting um yeah we have nine minutes for questions so please just unmute yourself and speak up or in the meantime Ricardo I can give you a a comment so I was a bit surprised that bones seem not to be a problem so I I
Would have expected otherwise you said that there is some blurriness on the Bon structures and of course because the you start from an MRI image but in the end you have the Symmetry with sufficient Precision so yes so this is a very relevant comment actually so if we here are some
Images generated by our now developed or trained Network and you can see also for example in the coronal images here how the ribs are a bit blurred whereas on the CT they are nice and sharp so for radiation attenuation what matters is the total water equivalent path length so radiation usually comes
Orthogonal to the axial plane so is the total water equivalent path length until the point where you want to calculate radiation and we usually deliver radiation from a 360° uh or from 360 degree angles around the patient and therefore if the bone the ribs in particular are not so
Nicely reproduced or they are a bit blurred what this does not yeah compromise much the the dosimetric accuracy of of the synthetic C so for those calculations even if you have unsharp bones but they have the correct oun field unit values along the VM puff this is totally fine however for
The matching this is more problematic because uh when we do the matching with our com beam City our com beam city is a bit more noisy is a bit more uh uh as a smaller field of view and we really rely on the bones for matching so if you have
To match your comim C to a synthetic city which has blurred bones then you have a problem and uh um and then the matching quality uh decreases however this problem is somehow removed at the Mr lak because we do Mr to Mr matching and the synthetic
City is in the background only for those calculation at the normal lak yes this can be a problem and has to be evaluated so the the blurriness of the bones is more relevant for for patient positioning rather than those calculation because with photons we are pred bust against such small differences
If one uses protons or electrons then for as your radiation then then this is a different story but yeah this was not the topic of our investigation but yes good point thanks for clarifying that um we have room for other questions hi Ricardo thank you for the
Great talk um I just have a kind of a question or I’d like to ask your comment on kind of the amount of data that you used for the training I think you mentioned somewhere it was like less than 200 which seems quite low in nowadays in in machine learning so is
There like a minimum amount of data that that you use or yeah can you just comment a bit more on how much data ideally you would ultimately want for this yes yeah that’s that’s a good point so these networks are designed to be used with millions of images as training
Data this of course is something that we do not have available in uh uh yeah in our retrospective databases so for the abdomen we use this cycle gun which uh is unpaired training and this is a bit more demanding and that’s why we we use all the patients that we had order of
200 patients but is many more images because every patient has a 3D um volume that is imaged and yeah the field of view is in the order of 40 cm and we have a slice thickness of two to three millimet so every patient delivers more
Than one image so you have to scale up uh this however if we go to the head and neck case you see that we only had 31 patients available because we do not have many patients in this study and that’s why we use the different network which use paired images so you basically
Remove from the network this need of taking care of having unpaired images but you know that every voxel in the CT corresponds to a voxel in DMR this can create some problems if the images are not perfectly matched um however this compensates the fact that we not have so much data available and
Uh there have been some studies reporting that um yeah of course there is a a trend as you increase your amount of data you also uh improve the output but it saturates pretty quickly so if you are already in the order of 20 to 30 patients with a 3D uh volume available
And good quality data this is also important you not get much better than results than having order of 200 patients with bad quality data so it’s a bit more important uh filtering your data at the beginning and be sure that the images are correctly matched you do
Not have implants you do not have contrast media and so on and then you can also work with such numbers 30 patients or so uh yeah but in principle the the more the better yeah cool thank you okay two more minutes anybody has questions uh if not before closing I have maybe
One comment for Ricardo you mentioned head neck and pelvis what next yes if you can yeah yeah um so if I go back to the live at the beginning here I think one should also check what is clinically so there have been some studies for breast you can generate
Synthetic cities for breast but we don’t use Mr Data for yeah for contouring the breast so I think abdomen is the next side where this could be extremely useful because you can see the lymph noes very well for for them preparing the um the radiotherapy plan torax uh yeah for lung
IR radiations honestly uh CT for D CTS are performing extremely well so I don’t see torax coming anytime soon as a Hot Topic I think the next one uh will be abdomen that that will go into the uh C certification and the an introduction in the in the clinical
Workflow okay um we’re looking forward to that maybe next year we might have an update on yes okay and uh instead um final remark before going for lunch um next month we have on the 17th of August another talk on Vision language models in thoracic Imaging so now the the body region was
Kind of connecting uh last question to the next topic so looking forward to to have all of you next month connected and uh small anticipation we plan to make all the recordings available at some point um on the on the some platform probably the the hospital
Website so I will tell you more next time okay thank again Ricardo thank you all for participating and talk to you soon