Professor Andreas Riener, Professor at Ingolstadt Technical University, Germany
This speech explores the intersection of human factors, human-computer interaction (HCI), and evaluation of automated driving systems. Based on exemplary research projects, we will delve into simulations as a powerful tool for studying automated driving. We will discuss study setup in driving simulators and mixed reality environments, addressing the sim-real domain gap. Wizard-of-Oz studies and the explainability of automated driving systems are introduced, along with the crucial role of driver-vehicle cooperation. We also explore confidence and trust in these systems, emphasizing their impact on user experience.
Related links:
https://dl.acm.org/doi/10.1145/3543174.3546976
https://dl.acm.org/doi/abs/10.1145/3349263.3349602
https://journals.sagepub.com/doi/abs/10.1177/00187208211051435
https://dl.acm.org/doi/abs/10.1145/3473682.3481878
https://dl.acm.org/doi/abs/10.1145/3473856.3474034
https://ieeexplore.ieee.org/document/9986669
https://ieeexplore.ieee.org/document/9737417
My name is Linda Bole and uh I’m in civil I’m also your Vice te so but it is my pleasure to introduce my colleague Andreas ler he’s from TU English do he’s here with his PhD student um Shantal who is sitting here and she will be giving a
Talk on Thursday and um so I just want to let you know me and Andreas we’ve known each other for many many years we both sit on the steering committee for the automotive user interface conference and there are papers that are due coming up so if any of you want to go it’s
Going to be in California and silicon Val this year at Tu English he is a professor and he also is the academic director for the user experience design so thank you very much for coming and look forward to your talk so good afternoon Mike is working for the online I think so yeah
Okay so what I prepared for you this afternoon is uh a little bit of insight and journey through the world of simulation onroad tests mixed reality just would like to provide you some insights um of of studies that we have done and also looking a little bit towards automated driving and how it
Might also change the way of testing and the way of uh involving the uh in the future before going into the topic uh presentation how it should work uh some acknowledgments of my team so this is typically for for us as professors we just pretending what our team is doing
We are steering we are guiding but all the hard work is done by the students so I’d like to acknowledge my team so this is part of my students CH is there which is also one of my PhD Dems putting all her life all ler time into research the
This this team and I’m just showing you couple of things that we have done in the recent uh in the recent years so just to maybe give you a bit perspective what we doing uh we are un computer interaction group our research is most likely in in the mobility domain or in
The future Mobility but not only so we also do some stuff this HCI in general we are running the US experience design programs which means we have students that that are on the interface between um computer science and design which also allows us to have some cool visualizations which you might
Not see my presentation but at least we have students that are also in this design area and we are trying to make the best out of the two gos so we Comm of human IND action as Lena mentioned we come from Eng some of you might know Eng is home of
Audi of AI and also BW where chalis is working at is very close by so it means automatically a lot of Mobility research in eng where in This Heart of Bavaria area and um also our University is lot into this uh we have basically we are founded from
Um requests by the mobility industry that’s why years doing a lot of Mobility research not only mro but at the ENT University have programs or many programs Associated to Mobility to some degree it’s not only cars it’s also um aviation industry it’s also sustainability these days but our
Research and this was the main reason why I’m there is that our University founded the haris research institute some 10 years ago um Charisma cent of Automotive Research um so this is the first three letters of Charisma and there the rest is safety measurement and uh you have a lot of
Facilities um and you see just a few highlighted these are the ones that we using in our research group we are owning a this is not working on kind of screens we’re owning a movement based Dynamic simulator uh we have here a large testing facility um with an
Indoor test track kind of uh you will see a video or some some images later on there the auto test track so we can combine simulation with real driving in also in uh safe and secured area and we even have a built a a mockup in a
Trailer which means we can use this trailer and directly go to our Des participants being it school kids Orly people that have no chance to come to our lab so you just put there and do the studies um at their whatever residential homes or like okay but now coming uh in into the
The topic of today’s talk um and I just wanted to motivate it with a problem that we all know at least if we are driving our cars own Cars R cars there are a lot of assistance systems integrated or install in the cars but um for the normal
Customer it’s often not really easy to find out how the systems actually works how how they configured what functionality functionalities comes with this functions and also what the reliability like this is and starting with a very old example adaptive cruise control which is around more than 20
Years all of you or in all of your cars in all of our cars um ECC systems are in operation we know that they’re looking differently so not only the inter face but also how they are computed from different brands different makes different models um so the configuration
Means the Gap settings that they allow um and also the configuration possibilities possibilities that you as a driver the thing is as every system it comes with limitations the limitations also change over time this different sensors that are used for the ACC systems and also other systems but there
Is some limitation still evidence today so for example sensor cannot look around the cor the corner um if you have a curvy roads the system might fail and also still stationary objects are problem but does the driver does the user actually know about this problem and just looked up some slides that
Linda sent me a couple years ago so I have research from Linda which seed also 15 years ago asking ACC users and nonusers if they are aware of the limitations of the systems and the interesting fact is that back that time and that’s quite sure that it’s not Chang that much uh about
50% of people are not aware of the limitation of the system this of course is not ideal if you’re using a system but you’re not aware of the limitations then this might cause problems why are so if the non users are not aware okay they have not uh looked
Into the systems but if the users are not aware of the limitations this is really a problem why could this be the case they might not have experienced the situation um where the system just uh was not buil for do you have a question yes uh so this data looks like it’s
About more than a decade old we have inclinations as to whether or not this number has changed has it gone up has it gone down roughly the same yeah so this is we have not done similar so we have not looked into this data but just on
The next slid uh I show you a study that we have done on the a system automa emercy breaking system which is quite a newer system and also the research that we have done is uh quite new so so we don’t have any update on this maybe
Linda has but also I just I don’t want to whatever to stick to the numbers just saying there is a problem in in any system that that we might use that we are not aware of limitations potentially and if if this goes down to 10% still
It’s too much we should not use systems U whose configuration is is not known to us or where we don’t know where it might f okay so ACC re old system re old study uh let’s come to something uh more up to date an automated emergency braking assistant which is set to
Increase Road Safety a lot uh the road safety Ty is measured amongst other things with a star rating in for example Euro end cap or us cap end capap so vehicle gets a star rating and if you now you’re about to purchase a new vehicle you might look for a five
Star vehicle because this is something that at least induce um a high chance of safety it saves you it saves other Road users around you but still also for a five star reading it’s very difficult to say how safe or how reliable assistant systems on a fstar vehicle actually are because there’s
Also a lot of different configurations there’s a lot of different situations um the Sens of Technology changes from generation to the next and there’s a lot of situations that not even have been tested um in the test procedures to achieve that St so one thing that we were interested
In is for example uh if you look at um situations scenarios that are that you’re confronted with when you drive real road traffic you have different times different day times there’s maybe reflection of sun there is rain there is fog there is snow there is Reflections
From the from the road surface and so on um this is still only partially included the test procedures um so we were interested in this how this also might change the star rating or the reliability of these functions um and the second thing is also that uh the
Star rating or the the an procedures they are um yeah they are renewed or updated every every second year every two years which means for vehicles that are homologated 2 years ago they are Des tested by the NCAP 2022 procedure for new makes and models that are homologated this year they might be
Tested or they have to be tested this the 2024 procedures and the procedures over time change based on changes also in the um in the road infrastructure and also for new categories of um Road users so for example a couple years ago these cters came up they were not included in older
Test procedures but now included for sure in the newer test procedures also the systems are updated over time and find a wayes so let’s uh let’s have a look in a study that we run for the um for the road safety board in Austria and switzland and Germany was also included
Here but did not pay so it was paid by Austrian with road safety board this study um and what we were looking into was um yeah how does actually the star rating um how can we interpret the star rating which is an idealized test procedure to some degree as I said
There’s no real rain testing there’s no nighttime testing um in the star in the the test procedures and also the test procedure had there straightforward in how they how the Des are conducted so for example one of the discs that uh that we tried to replicate is the
So-called cpnc 50 CEST near Child near side child 50% and I’ve just and try to visualize how it’s actually conducted so we have here an we have an e vehicle here number B driving that direction we have a pedestrian that is walking here from left to right
Um partly uded by this car there’s two cars that are just parked on the on the road side and the 50% says pedestrian and E vehicle meet here at 50% of the width of the of the V tests are conducted in a way that um
The E vehicle B is moved uh with a speed of 20 to 60 km per hour so in a 5 km hour increment the pest here moves with fixed K 5 km per hour if it’s a child 8 km if it’s an adult and 15 km per hour if it’s a
Bike and then you just count how often time the vehicle hits The Pedestrian or the D or the Target or how often it applies the um automated emergency break system or assistant and saves um and saves the the time and then there’s some whatever some tables you can look up how whatever
Which fraction of the desk might fail in which specific Kil hour range and so on and this there’s some complicated whatever formula behind that gives you finally the star reading but it’s all transparent you can read it out yes quick question on the pedestrians of it The Pedestrian is
Always in constant speed yes pedan is always in constant speed also the car is in constant speed but uh the speed changes so there’s first test 20 km per hour second test 25 30 and okay so this is how it looks like then in our in our test scenario and
What we did is so we had just replicated the standard test and then VAR um the speed a little bit so we were interested also how much speed variation just a little bit outside the specified range might have an influence on the results and then we introduced some rain so we
Build up an artificial rain facility for our test track um and then we conducted this test also indoor where we have a rain and also a fog faciliity so we can generate fog artificial fog which has the same characteristics than real fog um just to see how much it affects the results
So here’s some some images of the tests we conducted and we also used different vehicles uh vehicles are one manufactured in 2016 one in 2018 one in 2020 so we have three different Vehicles also uh premium uh Brands and cheaper Brands German based and outside Germany produced Vehicles so different
Different models and then so this is how it looks like we did all the recording the measurement and so on and maybe I show you just a few seconds of the video to see to show you how it how it look like just few insights into the longer videos it’s not an
Interesting we had here for example a robot Movement platform and a Target um which was equipped with the dummy and then we generated some rain some fog and tested the the function um and just measured if the system actually CED emergency break if it hits the target which happened quite
Often um or if it did not even apply the systems uh based on the configuration that they put into the system you see those many targets were hit so the um the study was only done like one time right like like they didn’t do it over like the No No we
Did so we repeat that many times but the but the but participant only like hit somebody once you see what I’m saying so what we what we tried is we r haed one experiment um so many times that at least three time the target was not hit okay
Okay and then you did also some nighttime tests also with rain so we see also here sometimes just hard yeah I the reason why I’m asking this question because I’m curious how people like usually like when we do studies once they hit somebody even if it’s fake their their behavior changes
After that so that’s why I was wondering if that same person does it again yeah so that you mean the person in the car operating the vehicle yeah so this one was only a whatever fall bag operator did not even do anything I see we just looked into how uh the automated
Emergency breaking system okay it’s really focus on on the system itself got it we only tested the system not not the driver okay okay the automated emergency braking assistant should apply the break when whenever it sees a dummy that you recognize as as an object um and that’s
Why we had just had the fallback safety operator to stop the vehicle in case the assistant did not break and this was a Str driver so we did we only had one person doing all all the driving test and we conducted more than 200 test Comm uh okay
So few results just very uh general what we saw is that so star rating is one thing but if you write the the setting the configuration just a little bit then you might achieve worse result than the one that is just uh done in the official test what we also found is that
Bicyclist and also newer Generation Um or new newer forms of vulnerable Road users like Scooters or ebikes they were quite well recognized in particular by the new vehicles so it means that this is all very good implemented and the object detection algorithms also recognize objects but on what what we wanted to
Say or what my main message behind is also that okay you cannot surely you cannot just trust uh a star rating say okay I have a safe vehicle and there’s no need for me to watch traffic or so the vehicle is doing it for for me this is
Not what going to happen because the one side the test procedure is limited it’s just very narrow very narrow part of the entire possible situations that might accur in wheel traffic and also the test conditions are idealized in a way that um yeah it has to be reusable of course
But in reality many more influencing factors play a role that that never can be considered by such a test um what also is evident is that when we as humans arrive then we know that nighttime drive and nighttime plus rain plus you’re getting older also nighttime rain for elderly people is really not
Easy to drive but uh in particular in these situations assistance systems should help but right now in most of the test procedures this situation not included so you know you have a five star rating vehicle but it only achieves the five stars during idealized daytime dry road condition which is also not really
Ideal um then another thing that might whatever goes into detail already that for one specific um brand and model let’s say uh as I said AI time let say Audi A6 um model 2020 with a specific whatever sensor setup even for this very specific um vehicle you cannot say that it always
Achieves the same rating because whenever you go to the C dealer they do some software updates so they update the algorithm and nobody knows which algorithms runs on your actual vehicle it’s really hard and so this is also what we what we saw if you bring the same
Vehicle um to the test track it might result in different um it might give you different results and the least thing here also when it comes to the interpretation of road safety or the interpretation of star rating of course we don’t have to trust or should trust only Technical Systems behind we
Know that when we are when we have black clothes or dark clothes in the night then we as a humans might have problems in identifying these objects and sensor systems have the same problems depending on the setup for example so it’s always important also to be cautious and to be aware of
Okay so this is this is systems that are now on the car um pretty old systems and coming closer now to the um the automation meaning higher levels of automation we have seen ACC and ab systems which are all level one systems a very low level of automation the
Driver has to uh drive and be aware and be active all the time question is now when it comes to higher levels of automation level two level three systems are also um on the roads already how to test and evalate functions on these levels and we have uh some solutions or
We have done some um different types of testing so one is driving simulation driving simulation studies this is something that uh is very popular so you have a driving simulator you implement some specific scenario then you find out um how well Automation and humans maybe work together and the next thing will be
Just to try to make an assumption how well this might be reusable in real in whe traffic so here’s an here’s an example of a system that we implement it back some time just to see how well system and humans work together with the lower level of automation we have not so
Fensive systems we still have lot of situations that are not so well handled to be handled by by sensors for example human behavior so for human behavior um there’s there’s a good chance that a driver that sitting in the car that still has to monitor and supervise the
Situation which is the case for lower levels of automation might have an an immediate um assumption what what a pedestrian maybe would do and the sensor and the algorithms running on the vehicle might have a hard problem in in getting the same so we often see such kind of Cooperative
Cooperative systems where human and system on a car do some mutual decision making processes is that up a us study I don’t need to go into much detail about the number of participants and the exact setting but what we did was so the situation was um a pedestrian crossing a
Street no walk just whenever uh just arbitary space and we had a we had a sensor setup on the car that just detected a pedest which was a baseline condition so it showed pedest detected no explanation what actually happened and why um the system thought that was actually
Detected and then the person on the car had to decide whether to continue driving uh or stopping based on the situation that was shown on the scen and then we had two um soal confidence uh interfaces or systems again the detected but there was another level of detail here namely an
Explanation uh what the system thinks that U is going to happen the Des is crossing the street uh in addition a confidence level was shown on the uh on the HMI and the second configuration was detected um plus the information crossing the street was assumed by the
System and instead of showing a concrete level of confidence was like a progress bar type of visualization um and we just were interested in if how long it takes for the human cart to intergate so defect the situation intergate the situation um value with his own um
Decision making things and then come to a joint decision with the system yeah how do you get confidence how we get confidence yeah so this was kind of a fake system it was not a real system so it was just programmed um I’ll show you later on we had uh three different
Confidence values 20% confidence 50% confidence and 80% confidence we were just interested how much it plays the role how confident the system is on a certain situation um but there was no real whatever there’s no real algorithm behind it was just a variation of um of the conditions
Here so the experiment was builded up like this um simulated drive at a certain position the scenario was detected the disant disant value of the situation come came to a decision a joint decision making uh with the information shown on the display set a reaction and then the
Situation was cleared so the maneuver was executed either stopped the vehicle or contined to drive and then it was presented if the decision was correct or was was wrong this is how it looked like in the simulator so we have here kind of a head up display The Pedestrian here we see
50% confidence that The Pedestrian is going to stop not cross the street um you see the decision of the the discip to stop in this case go on continue and then the situation was second time so parti press the button on the steering whe either left or right
To to enter the decision in this case the result was um so the Baseline situation that we have no explanation compared to both confidence hmis with an explanation and the confidence level is that this is nothing unexpected decision time when with no explanation was shorter compared to the competence HMI
Of course you have your own interpretation of the situation and then you have to Value how confident the system was and you have to come to Joint decision uh so it takes longer time this is nothing unexpected excuse me when we look into the concrete yeah sorry quick
Question interaction time is is time to press anybody yes so interaction time was the time to come to decision when we look at the the different confidence levels so we have here light blue is the 20% confidence so the system has a 20% confidence in in this rating 50% or an 80%
Then we see of course the lower the higher the confidence value is the shorter the time you come to decision for 20% confidence you see a large variation on the one side and a very long decision time which is also not really unexpected but the interesting
Thing is when we compare uh the the confidence bar versus the confidence perents that people did not like the concrete with percentage values so they took longer with a com with a percent value this is also something that we we might show um like a a progress pathing
Not a concrete percentage values even if you if you are possible um or even if it’s possible to compute such a value it’s better to keep it abstract for the 20% confidence value there was this large variation of of course many participants need it very long to decide okay I rate it
Differently the system is not very confident what it’s what it’s projecting so there’s also some back and forth think so um better not to show at all if you have a very low confidence about the decision or about the situation so it’s only makes really sense if the
Competence level is high to support uh the the human driver and this is also something that keeps going when the system are learning to maybe AI algorithms which are quite popular these days to improve on the competence value so thetion plus the confidence information that it’s coherent then the decision time can be
Um bit lower than in the case of in the situation with uh with no information at all but the the main thing is also that we um just wanted to see how systems and humans operate together and if you think such a such a such an experiment in a
Real traffic environment it’s dangerous because you have pedestrians that might have to step on the road that might be hit by by the vehicle um the question is always still how much can you replicate the results for for reality right so there’s it’s always simulation there’s
No real harm for for nobody um and it’s also questionable how you come to the confidence level which was also a question for yeah uh yeah I’m just curious if you ask people about their sentiments did you ask them how they felt when the system had very low confidence for example yeah
So people did not like uh this this information with a very low confidence value this is that’s also the consequence then later on it’s better to not show it all because if you’re not if you cannot be sure that the system um gives you the right supporting um
Information then it’s of no no value it might even whatever have a negative impact if you trust the system still if the the competence level very very low but still some people might follow the decision that the system was giving them which result then in in a bad situation so um
That’s why I’m saying if the competence level or the competence of the system is is very low and I’m not saying 20% only but maybe even below 60% or so so Department station is something that we’re working on then it’s better to not show it all and just ask the person um
Of so it’s in a cooperation situation uh what they want like or just be on the safe side a break but we also know that wrong uh system Behavior might lead into switching it off this is something that we have also seen in systems that are in operation like the a system the
Automated emergency breaking system this is always or yes a lot of situations in the in the earlier times of um system availability that these systems just apply the Break um without any particular dangerous object and then people tend to switch the system off if you have an assistant systems
That is switch that it’s switched off and it does not count either so it’s very hard to whatever have this paration trust over trust mistrust um experience of situations that um that are not really explainable for the human driver so that’s uh that’s nowadays something that is really um a
Lot of research is put into this particular when it comes to a move from a traditional deterministic system Behavior to an non-deterministic platform Behavior which you have ai systems that so careful parametrization is important and also parametrization for the individual is important because some people so you have a lot of driving
Experience who might rate the situation different than than an noice driver that you know as also not the capabilities for for situation awareness for rating every um whatever traffic object correctly and so on for having no concrete or wellever less experience in in rating distances and potential behavior of of persons in the
Situation so this is very important but driving simulation is still an important uh component in the entire ecosystem of checking out system capabilities and even in the introduction of new system um driving simulation is can be done at a very very early uh point in the development just
To see also if people accept such systems or not and what should be configuration what should be aration before bring it further towards uh production C so driving simulation you saw that simulation and this is something that is known also for a long time in in research simulation
Never replicates reality so we thought okay let’s go one step further and try to do some driving tests on a test track uh which is feeling um more natural for the test participants by but still by conserving safety for every one that is involved this this test and actually um we were looking
Into level three system so little bit higher in the automation level three systems are already on um on our road roads so this is nothing that is far in the future they’re already on the roads but still we need to improve the systems and we need to see how well they are
Perceived and by the participants and how good um the systems actually are configured different scenarios so I don’t know how much you are into this levels of automation or Auto driving at all there is five levels or five levels plus level zero level zero means no automation or just manual
Driving level one is ACC AB systems and level two is a combination of longal lateral control of vle at this lower levels 01 and two the human driver is still um or still has to monitor to drive the time and is responsible for every decision that is made by theic
Plus pluss driver in the higher levels starting with level three um there are situations where the driver can do different things so called non- driving PL tasks and DD um but still the vehicle is not capable of doing the entire drive so there might be situations coming up
Where the vehicle ask the driver to take over this called take over requests the vle says there’s a problem I can’t solve it driver pleas help me take over and try it better so this is the case for level three systems um in the level three system so
They take over or actually being capable to take over on a request by the vehicle is um the situation or the point in the drive which is mostly researched so far um there different timings that are required to be capable of taking over of course it depends on the graphic
Situation it depends on what the driver was doing before was he reading a book or working on his iPad or handheld device or maybe just looking out of the window dat dreaming so this is all affecting the take over time take over reaction time when we look into research then we find timings
That start by about 3 seconds so from the point where the take over request is issued until the driver has the hands on the wheel and being capable of driving manually still this take over requ so this take over time of 3 seconds does not NE necessarily include be capable of
Having full awareness of the situation so situation awareness is something that might take six so five up to 10 seconds in addition but still you’re on the you’re on the wheel which means the um the responsibility is already transferred back from the vehicle to the
Driver and this is the the only value or the only time uh the manufacturer and the legal U um stuff is is interesting so take over time and then post post Take Over Control this is what I mean this might take up to 40 seconds to gain really full awareness of the
Situations being capable of soing everything knowing what’s going on what has been going on was what what was the problem of the vehicle but um this upper boundaries typically it’s not not considered in research um because of the responsibility which is the main the main problem also we know that all of this
Research or most of this research was done in simulator studies so far again we have a driving simulator in a static one or dyamic one there was some artificial scenarios um and then time was measured until the take over was issued until the take over was completed or hands on the
Wheel or whatever uh p ps so this was mainly obtain in driv simulator studies also in our team we run different driving simulator studies uh to um to measure take over timing but then I said before when you try to integrate this for a real vehicle that
Uh has permit to drive on the road like the spe Drive pilot or other systems kinding might change and what what typically is done you like 3 the time four the time to be on the safe side but um drivers might not feel comfortable with a long uh takeover
Request um time ahead the actual critical situation so it needs to be parameterized also in a way that it’s accepted by the triers If you experience some situation where the take over request comes very early the actual critical situation in the next take over request you you might not take over on
Time because you think you still have a lot of time to go before the actual particular situation might happen and then it could lead into a situation so what we did is we set up a an experiment on the test track we used the car just a regular car with
Any particular assistance system we installed a so called driving robot on that car which you see here so there there’s hydraulic activators activating the pedals and there was another um robot system moving the steering wheel so this way any normal production car can be turned into kind of automated
Vehicle we put the PED distance on the driver’s seat um they were fronted with a driving situation and the non- driving related task on the smartphone we put them on a on the test track they uh had drive or they driven basically on the close loop drive with
Some curves at a maximum speed of 30 km hour and we had here a platform that moving a bon Target on either of these two lanes and then whenever the participant completed the next round at around this position here uh take over request was issued 7 Seconds least
Time we implemented and then the person participants had to steer the car manually in eight of the two lanes to avoid crushing into this tet this was combined with the non- driving related task which was um simple um reading span task so there was a there was a sentence presented on the
Smartphone and the semantic correctness of this sentence had to R this was the one condition was visual motoric just putting a pressing a button um and the same task was also presented as an auditory task with voice interaction just agree or reject um and then we had in
Addition the timing for the Takeover request this was our second independent variable with one aspect was okay the take over request is issued whenever it is required to issue it meaning also in between the current task that was that had to be solved and the second variant the second level was issuing the
Take over only between tasks so meaning after one task was completed so maybe you can wait half second more before issuing the Takeover request the task was completed which means also less cognitive demand for the participant for person because uh in this first level you’re midst in a task then the
Takeover request come you take over you drive the car and you get back to the task which induces a So-Cal resumption lag so you have to get aware what was what I’m doing before kind of which causes a bit of cognitive stress which is also something that we would like to
Avoid of course in a real situation again here a couple of uh seconds of the video how it was actually implemented in the car automated drive then here we come close to the last curve take over this requested on on a display here in the center area of the
Car participants have to grab the steering wheel and have to steer the vehicle to avoid crashing into the Tet we did all the measurements and recruited 22 participants here um run into a randomized experiment with all the four conditions so meaning the two tour conditions in between and after the task and that
Two um levels of presenting the the actual task on visual this this physical or motoric feedback and the other one auditory this voice feedback what we found uh is that in on average or take over time was just about 1 second 1.5 second which is compar to the
Simulator studies 3 seconds so it’s way lower uh the second thing is can drive us in this test track setting which is also somehow artificial compared to real traffic because there’s also no real danger involved but it feel more dangerous than a simulator can they respond to the tour safely so the one
Thing is okay the one thing is that the Takeover request was really handled very quickly uh so it means that people were more aware of the situation and of the problem maybe also getting hurt than in a simulator uh and on the other side still 10% uh of test cases or participants
Produced an accident and in about 30% of all cases there were Lane exceedances so it means so they crack the be rather quickly but then we’re not able to complete the task really safe but this is maybe shown here on this graph see here the two lines blue is kind of the
Safe um the safe path we see here one exceeding quite quite a bit and many others also had the compass here so it means um that is way better to do it in a in such a configuration than the simulator but still the question is are test track experiments beneficial compared to real
Driving and I would say yeah they’re more beneficial than just the simulator studies but it’s also more efforts to put into to run such a study but in terms of configuring ow ding real function of the vehicle definitely um I’m definitely in favor of doing such experiments on a test trank compared to
Simulation the next thing is so we have simulation we have test track and nowadays uh also when it comes to more and more situations and also to avoid safety issues um nowadays mixed reality environments um gain more um attractiveness and this is also something that we did because what was
The Logical Next Step then so we had first it’s pure simulation we had the test experim test TR experiments with a road safety for we had our own take over request experiments on a test track and we saw that it’s highly complex and high effort required to run the experiments to get
Kind of close to reality simulation close to reality exper results sorry um and now with the capability of virtual reality and also the capabilities um that we had in our in our Research Center we tried to bring together The Best of Both Worlds in such a mix Duality environment so we had on
The one side we had the the 100% real grouping ground testing with all the equipment brought into uh the Des faity on the other end of the spectrum we have pure simulation not even involving desk participants maybe um just using algorithms and functions implemented in software um and then we can bring both
Worlds together having physical objects physical car maybe here in the test environment but bringing in um simulated objects in one uh in one and the same environment so to be able to test systems so for example a real sensor or a real car against a situation that is just simulated in
Software and this is how it um how it might feel so we have here a real vehicle that is driving on our outo test track for example it’s sending position data to a simulated vehicle that is driving in a virtual reality environment or like color or Unity environment the the virtual vehicle is
Equipped with a sensor a virtual sensor sensor model and once the virtual sensor model detects an object in a virtual environment um it exchanges data with the real vehicle and the real vehicle applies break or moves the steel wheel for example so this way we could replicate
Tests um run often time um in a very safe an efficient manner and this is maybe the last thing that I’m showing here’s the configuration but I’m also some insights um how it finally so what we can finally do with this mixity environment see a bit forward yeah so this is maybe showing
So we have a a person wearing a a motion capturing system and the V reality headset uh we have a real vehicle driving on the test track sending over its position data to our um Central simulation so we have the real vehicle replicated in our virtual environment we
Have um the sensor system and this is the camera of the of the virtual vehicle seen here a person maybe starting to walk um with across the street this person was actually Mimi or a replication of The Pedestrian that is walking here in the virtual reality environment and once the
Sensor in the vual vehicle detects The Pedestrian object that lies the break in the real vehic so this is kind of how it’s um how it was implemented and I just show you once again how it looks like so this is also a way of reproducing tests um in an efficient
Manner without causing safety problems for any of the partici for any of the involved parties and just uh trying to find out how well functions work so like close the loop to the endcap test that we had before um but now in a very efficient more efficient manner but also using
Either software sensor models or or using the real sensor on a vehicle um is a vehicle in the loop ination yeah I think I stop here at this point next the next the next level will be a digital twin approach um but U yeah I think this that’s that’s all so yeah
5:00 I know some of you may actually have um some classes but and and there’s been a lot of questions that were asked throughout the presentation but there’s any additional you have time I mean I’m happy to stay so if there’s any other questions that you might have you know raise your
Hand if there’s online participants they can also ask a questions in the chat just a question here oh one of your findings I thought was really interesting was that different Vehicles so from the first part of research uh recreate the safety testing we found that different models of the same
Vehicle could have different performance results based upon the configuration sensors or even the software which updates you know semi regularly and that is essentially Black Box us researchers how much level of standardization and replicability is necessary in order for these Technologies to achieve widespread trust from society andad adoption yeah
So this is a a question not so easy to answer um because it has been so the thing is the variation might not be um so it might easily be replicated in the study that so like the one that we did to see what really the difference is um if you
Are participating in a real world environment as a as a pest for example you might not see uh what really the the variance is um when we now think a little bit further in the future with automated systems with um AI algorithms then this the whole thing gets even more worse than it
Is now because when you have an automated system like think about all all vehicles are automated to some degree then not only the pedestrians or the vulnerable Road users need to have trust in the systems or need to know how they kind of work but also all the rest of the um the
Traffic objects need to know how the opposite is going to react on a specific situation and this is something that is really um not so easy to maintain because on the one side every person has its whatever kind of personalized driving style now in manal driving we all might want have to
Might want that our vehicles in the future drive also like we want to be driven like more conservative or more aggressive in order to whatever accept technology and use it finally but as long or as soon as uh there is a parametrization or individualization of driving functions the
Opposite it’s it’s hard for the opposite to know how a vehicle might behave so you might be able to exchange data you might be able to communicate your parametrization do the opposite but it’s lot of data uh processing involved then so the ideal world would be having one
Function or having a function configured and then all the vehicles have it have it installed or implemented the same way across makes across models across countries whatever is not not what we really would like to see and also not what we will see maybe one more question and then this
Way they have to go I don’t know on the chat it looks like question there’s two there’s two yeah so this is Elizabeth Elizabeth just okay okay all right comments that people can ask I know that some people have to go I don’t want to yeah did you have question I
Just have a more basic version of that question which is how many more if we’re already seeing this much trouble with level one automation how many more kinds of tests do we need to make sure level five so for so formally we had this endurance tests so and maybe she knows
Way better because she’s working atw which means that for every function before being uh approved uh you have to provide whatever some million test kilometers to make sure that it’s we Ding and every potential situation still then you only you don’t get the really critical situations you don’t get Ed Cas
Nowadays yeah now so so the one thing is that EUR enap has also realized this not only Euro but also the the different assessment protocol and what they are doing um they introduce from 2030 on they introduce instead of this very Qui uh concrete test points and CES they introduce called range tests
Um so this is one way towards solving this problem and the other way is we cannot do we cannot really do all these test in reality there simulation edge cases you mentioned so identification of edge cases and testing these in in um a simulated environment this might so this is the solution that
Is that is currently follow and of course if you have standardization in the sense of every vehicle behaves the same you might argue you’re in a level five scenario you’re not longer driving so it’s like a taxi or so um you can do whatever you want so time travel time or
Speed doesn’t really matter so this is this is an argument still not sure if this everybody would like to to hear this might be the solution and it also come that that’s also something which is called a positive risk balance at least BMW is using that so primarily we’re
Looking at how many accidents are happening without the driver assist systems and we’re having a positive risk balance if less accidents are happening with the system and for us that’s a really standard that’s a real world measurement though right yes okay so you would only find that out after you deploy level
Five oh no before a system enters the market we have internal tests and every system or every final vehicle set up will run a million test kilom before being exposed to okay but at least BMW does that can of Promise about I don’t know then you can think about that
Because she’s gonna be giving a talk on Thursday oh so you can ask her some more detailed questions about the automotive industry by the way there is a question here it’s basic so Jason ding he wants to know if he can get maybe share your paper um titled arvr pedestrian
Simulation in the real world off interested maybe other people may be interested sure to share I can I can share the slides and I can also share the that I mentioned maybe just send it to Elizabeth just to say the video I think discuss with you we usually post
Video on YouTube app you can even post some links in the comments oh nice that’s great so this is going be on YouTube if if you’re okay with that yeah I think it was recorded so yeah good all right thank you so much everybody for coming and for your great
Question I know we went over but there was so many questions in between that was just it was great I’m still around so more question you can