Andreas Weichslgartner (CARIAD SE, DE), Joyabrata Ghosh (CARIAD SE, DE), Vineeth Bharadwaj (CARIAD SE, DE)
Mr. Andreas Weichslgartner is currently working as a Senior Technical Security Engineer at CARIAD SE in the product security department.
Joining the Volkswagen Group in 2017, he since then has been developing an embedded intrusion detection system, evaluating security testing technologies, managing vulnerabilities, enabling crypto agility, and working with machine learning in the area of security.
Before, he had been a researcher at the Department of Computer Science, Friedrich-Alexander University Erlangen-Nürnberg (FAU), Germany, from 2010 to 2017. He received his diploma degree (Dipl.-Ing.) in Information and Communication Technology and his Ph.D. (Dr.-Ing.) in Computer Science from the FAU, Germany, in 2010 and 2017, respectively.
Mr. Joyabrata Ghosh is presently working as a Connectivity products security owner at CARIAD SE. Before that, he was responsible for security and legal technical manager for the Elektrobit Automotive Linux platform for the series production of several automotive OEMs. He started his automotive journey with Direct HMI development for BMW ID7 platform. Over a decade ago, his development journey began in embedded and telecom security domains across many OEMs. He supports EO-14028 CISA SBOM working groups and contributed publications: Types of SBOM, Minimum Requirements for VEX. He contributes to nvd@nist.gov and cpe_dictionary@nist.gov for open-source triage. He was co-presenter of Cybersecurity Expectations in Automotive World, 2021, in ELISA Linux safety workgroup. Likewise, he is open-source enthusiast. He has Master’s Degree from the Illinois Institute of Technology in Computer Science, a BS in Computer Science from RCCIIT.
Mr. Vineeth Bharadwaj Prasanna is currently working as a Senior Technical Security Engineer at CARIAD SE in the product security department.
Vineeth joined the Volkswagen Group in 2018, as a security engineer for Audi AG. Since 2020, he has been a member of the offensive security team and has also been working on building up the vulnerability management system, end-to-end security engineering for China GB-T homologation project for the new PPE/PPC platform for the new Audi and Porsche cars at CARIAD SE.
Vineeth received his Master’s degree in Simulation Science from RWTH Aachen University in 2019 with special focus on optimization, and artificial intelligence.
—
As the automotive industry undergoes a paradigm shift towards software-defined vehicles, the imperative for robust software security becomes obvious. This talk explores the nuanced landscape of identifying, managing, and preventing vulnerabilities from the perspective of an OEM software company.
Starting with an exploration of the escalating role of software in modern vehicles, the talk illuminates the complex software ecosystems that underpin contemporary automobiles. A thorough analysis follows, unraveling the primary sources of vulnerabilities and their potential ramifications on vehicle safety and security.
Central to the discussion is the challenge of handling vulnerabilities within the complex supply chains inherent to the automotive industry. The talk elucidates the difficulties in navigating this multifaceted network of suppliers, emphasizing the necessity for collaborative approaches and effective risk management strategies.
We detail best practices for developing automotive software along the software development life cycle along with various regulation requirements. Especially we highlight the significance of SBOMs in fostering transparency and traceability across the supply chain. The talk delves into how SBOMs can fortify cybersecurity measures by providing a comprehensive understanding of the software components integrated into automotive systems.
Looking forward, the presentation anticipates future challenges and outlines viable solutions confronting the automotive industry, including crypto agility and the incorporation of cryptographic bills of materials (CBOMs).
[Music] all right greetings everyone thanks for the beginning of the afternoon session of vanan 2024 uh I have the distinct pleasure of introducing Andreas joob brata and Venice and the topic in this track today is going to be finding managing and preventing vulnerabilities and Automotive perspective uh our presenters are remote so if anybody has any challenges or issues with uh audio or visual please do shout out we have a here in the room and we will handle anything that uh that we need to handle so a couple of announcements before we start uh Please be aware that session surveys are available in the conference mobile app via QR codes found on taable tents in this room and elsewhere these are really important to us especially as this is our inaugural vcon we are really relying on uh on honest feedback from our participants so please tell us what you are thinking and uh then we will continue to adjust as we go forward cell phones obviously it’s a courtesy right uh make sure your cell phone or laptop or other devices aren’t making sounds um please note that all sessions this one also is obviously live streamed right um and we also have remote participants outside this room so if you ask a question you don’t have a mic no one’s going to hear it so um um will be around with microphones during Q&A session uh most of the time we don’t really take questions uh during the talk but if there’s Q&A um I’ll be around with a with a microphone but just know if you don’t have a mic uh no one can hear you especially our remote speakers today um all right with that we will kick off to our presenters okay thanks for the introduction so we will talk about Automotive vulnerabilities so so um next slide okay so who are we a short introduction I’m Andreas uh I work in carot since 2020 uh before there before that I worked at Audi and before I did academic research uh PhD in the university and my current occupation is I work in vulnerability management I help with fing cic testing and I also was a a developer I developed an IDS system for our current car generation Josh Joy sorry all right yeah so myself um I recently joined kard and before that I had um around six years on Automotive electr bit on the clinex operating system development for multiple oems before and there I also worked on esom security and legal aspect before that I worked on embedded and open source security aspects over 10 years and then give the and to Vin here my colleague yeah yeah hello everyone uh I’m vinit or vin in short uh yeah even I was part of the or have been part of the VW group like since 2018 started at AI in the same team as Andreas and then uh transitioned into uh carad and yeah currently like have been part of the offensive security team uh doing some investigations uh on our cars and also uh yeah part of the vulnerability management team trying to yeah manage and find vulnerabilities yeah and in also like uh in uh yeah as another role like I’m also partly looking into the China gbt regulations uh that are affecting our cars and maybe we’ll also speak a little on this later so yeah thank you now back to yes yeah so now you know who we are and now just a short introduction to our company um so we are at Carat and carat like we provide the software and also part of the hardware of the Volkswagen group currently there are like nine different brands from Porsche Audi to Bentley Lamborghini Volkswagen seat scod CA and I think also recently like there’s a new brand in US called Scout and and yeah so we have like a lot of cars on the street uh getting the data for the Advanced Driver assistance uh system and the the the goal is that we have like one software platform which is used in all the cars of the Volkswagen group so that we can use the the scale effects of this huge group and we currently uh 6,000 employees we have a lot of nationalities we are mainly based in Europe but also we are have departments in US India and China so now you know our company so this is the agenda for today I will first introduce um some Automotive specialities so what is the architecture we are using then we will dive into some problems or weaknesses vulnerabilities we have in the automotive space then Choy will tell us something about the automotive supply chain the whole esom bdrx uh thing then Vinnie will tell us something about the vulnerability management we do there and the Lessons Learned but we experienced the past years so a short disclaimer the content and the vulnerabilities to present in this talks are all publicly known so no secrets in here and maybe if we like have some opinions they may be our own and not of our employer so just keep this in mind so what is the uh Automotive AR architecture we started uh with the canbas it was invented some years ago in 1986 by Bosch it’s a very simple field bus two Twisted wires uh it’s cheap and simple and it’s priority based so on the right you see the actual code struck in the Linux kernel so we have an can ID which is overall 32 bits out of this 32 bits 11 bits are for the address and it’s priority based so the ID zero has the highest priority um if all 11 bits are to one this is the lowest priority then we have a length field from Z to eight bytes then from pading in the end there’s eight bytes of data and yeah so everyone who is on the bus can receive the messages and can read it and there is no security implemented by Design and there’s only some CC check for detecting bit flips um on the bus but no security yeah so the the Classic Automotive architecture was there was one Central Gateway then you have like buses like can bus or Lin bus then you have your ECU so ECU is like the electrical control unit and so overall it’s a very distributed systems in some systems you have like hundred different ecus which are connected over different buses and they communicate with each other this is the the classic one so they also so you buy several ecus from different vendors and in the end the OEM blocks them together in the car and put it to the street then a more modern approach is that you already have like domain controllers which are a bit more powerful for example the infotainment system Advanced Driver assistance system is like one domain controller and then under this domain controller you have another different ecus and the domain controller itself are connected over a service oriented Gateway they already you have like something like Automotive EET some IP so in the service oriented way you don’t always send the signals if they don’t change so only you can subscribe to a service and once uh there is a new event available all the subscriber gets the event sent to them this is what we mainly have now on our street and in the future or like some um some architect is already like this is it’s more Zone architecture where you have one Central component where all the different domains are not mapped to one dedicated controller but it’s more spatially distributed so you have in the corners of your cars I have so-called Zone controllers which deal with the the centor actors in the area of this car and then the central compute is done in one uh big uh Computing platform somewhere in the car and this reduces the wiring harness and also it enables that you can easily update functionality and have like yeah the update ability is easier and you can add more functions easy so this is the future what like every car manufacturer is going to and also it’s known in as the software defined vehicle which is like a Hot Topic at the moment in the automotive space but it doesn’t stop in the architecture in the car itself you have a lot of environmental stuff there you have like a own diagnosis uh yeah system you have Smart Keys you interact with other vehicles you interact with infrastructure of course there are end to end Services you have something in your backend systems and so on and but we will focus mainly in this talk on the car stuff itself this is also what we do in our day drops we focus on the product security of our car uh the backend stuff and our office clients and so on it’s not our department oh my next slide is not working okay so um the operating systems what we use normally um in the past it was more like dedicated operating systems for cars so stuff like aosa which are adapted and classic different flavors this is more a runtime environment and uh yeah middleware also there’s like real real time uh proprietary uh systems like Q andx or VX works but also at the moment there is coming more more Linux based systems and the infotainment you have like Android automotive so there’s more common operating systems but you have also in the in other worlds like mobile and also the the the trend is that it’s not only closed Source you buy from A supplier but also there are several uh open source initiatives which different vendors and also Tech players like AWS Microsoft arm redhead they you can collaborate on different open source projects and also work on non differentiating um software components so that not everyone has to invent the wheel there again and again and have the same vulnerabilities um every time there’s a new car generation so um let’s di into the vulnerabilities um I think last year there was a a pawn to own they also have Automotive track and uh in one track like some competitors they run doom on the uh infotainment system so your infotainment system definitely can run doom and they found two bux and could exploit it to run Doom there on the infotainment system um also going back the first thing car hacking start with like a paper from 2010 where some researchers showed that the canbas is vulnerable to a lot of attacks and that they can display on the instrument cluster arbitrary speeds and also can control the mileage stuff there then one of the most famous car hacks like that was done in 2015 so there was a black hack talk and also I think in in Wired Magazine the some researchers Miller and V sector showed that they could remotely steer uh cheap so this is every car hack talk has to has this specific hack in it so here it is and yeah so coming more to the current thing so the can bus thing is still a thing so this was from last year where there were some thiefs they um they unplugged some headlights connected to the canbas and then they could unlock the car and steal the car in UK mainly and there are devices you can buy on the internet which make it very easy for uh the thief so they just need to find the two wires connect to them press a button and then you can um steal the car and this is blamed to the insecurity of the canb um coming also to some vulnerabilities from our group there was like in the Ia 3 and VW car there was some problem with rest API where you which was not authenticated ated and then you can send like requests there and then the user could not change the volume anymore uh another example is also classic car stuff is that there’s some hardcoded um credentials in the firmware so the UDS Port here that’s an uh diagnostic port then was not yeah if you can read the hardcoded credentials you can connect to the UDS port and can do then there uh on applicated access and of course we also use Curl last year was also this announcement that yeah there’s a high security um yeah vulnerability in the curl project and later then the maintainer had a very nice block cost where he detailed what the root cause was and the thing was a buffer overflow I think and he stated that like 41% of the security vulnerabilities in curl um are related to memory safety issues so if they would have written it in Rust we wouldn’t have this 41% of the issues but of course on the other side you cannot rewrite curl on the weekend in Rust so this is also not going to work another um prominent vulnerability was a an issue in the chipc um again another um buffer overflow um with environment variables yeah it was also last year and the researchers who found it said that they could ReDiscover um this this vulnerability with a a state-ofthe-art faster like within seconds um so that’s lead us to the next questions so what is the root cause of some uh Automotive vulnerabilities um here is the top 25 most interest software weaknesses I picked some of which I think from experience are relevant for us so you see that I didn’t use like cross side scripting and stuff like this because there we don’t have such a use um a Tex surface but again we have a lot of memory safety issues um there’s OS command injection which is was part of the gpeg for example and proper input validation and yeah hardcoded credential is also there and this is from the general list and from a specific paper from 2019 and their methodology they had this different categories and for each category they had certain Search terms and then you can search the vulnerability database for these specific Search terms and then see um yeah what CV you can find uh the original um study was done in 2019 I did it last year with yeah with more data from the four years in between and yeah so this is what the outcome was we see that yeah after filtering out a lot of false positives um there are a lot of Bluetooth issues so 66% of the found vulnerabilities I found with this methodology were due to Bluetooth uh there were a lot of problems with the um tips set um firmware for example in the Snapdragon and the exos or is are from two vendors who has uh who who yeah built firmware for uh chips and we see that from the OEM so this is all the brands who produce cars they only make pre so I think a lot of them are not really reporting cves and are not assigning cves so maybe they are not so um open um to report as other Benders in the who produce Bluetooth and chipsets um to drill down a bit here uh when I did all the the filtering and uh tried to detect false positives I had like this uh in front of me so you don’t need to read it but basically it’s always is the same it’s the cwe 1225 and it’s an out of Bounce read and this is a bounce checking in Bluetooth and it seems to be really hard to get this correct if you have this untrusted input and you need to do the bounce checking probably in Fe and then a lot of mistakes can happen there um if we print this over time then we see um that before 2014 there were like almost no vulnerabilities uh for this people key wordss but after uh open uh heart bed from open SSL the lot of researchers started to to do fing and do a lot of research then we see the redish part here it’s mainly to um yeah memory safety issues and only the blue and gray part are about improper authentication and yeah missing authentication and so on so we see the um memory safety issues are um dominant over the years and um here it’s another graph where we sorted it by overall number of cves according to the cwe classes and the most dominant was the 119 the buffer overflow followed by out of bound read and yeah then the non-memory safety issues are the improper input validation and the exposure of sensitive information and then again we have some use of the free and other memory safety issues so if we could handle um memory safety issues and improper input validation like we would have a lot of the cake already um cut down and could focus on the other parts of the vulnerabilities of the weaknesses so I mentioned memory safety a lot so a bit detailed memory safety what is memory safety so memory safety means that all your memory access are def F so in in Practical means are that you have runtime bounce checks you check before you D reference your pointers and you also have lifetime checks in C if you have like uncheck Point arithmetic uh you don’t have any of these um and if you detail it a bit more you have like two parts you have spatial memory safety this is all to bounce access so you need to access the object within bounds uh there is a lot of also work in SE and C++ how to add this uh yeah Dynamic bounce checking and you can have some hardening mechanisms in your compiler that you can check for this and the second part is temporal memory safety this is usually use of the free that you access an object while it’s already freed this is a bit harder to do in C++ so here rust really with its memory model and the lifetime uh would help so there you can already um ensure this Cur compile time that you don’t have temporary me memory safety issues overall um in in this for this studies so take it with a cran of thought the data stuff there is not all the all Automotive vulnerabilities uh Linux kernel for example was not included there but with the cves in this data set it was about half half so half of the cves were due to memory safety issues and if we um also see see the severity um plot some violin plots from the um CVSs core you also see that the memory safety issues tend to have a higher severity so the buffer overflow it’s going there very to the critical um reach while for example the CW 200 exposure of sensitive information it’s widely spread it goes from yeah 2 to 10 so there’s a lot of different variation while while this all this memory safety issues are very high have high severity and to to use data from a different data source um also project zero they maintain some spreadsheet um where they um document exploits in the wild and they also categorize them they don’t use the cwe um categorization but have their own categorization but here you also see that exploited um yeah exploits often use memory corruption and information leaks and use of the free so this makes 75% of the exploit set vulnerabilities are due to um yeah memory safety issues so the attackers uh seem to know how to exploit this kind of memory uh safety issues this is was the the analysis of the root causes so what we can do about it so um there was a recent paper from also Google on me memory safety so you can do three three things you can try to prevent so the easiest prevention is to not use a memory unsafe language like CN C++ like crust for example or you use safe apis for example if you have untrusted input that you um already make it through the API impossible to yeah to to to exercise certain undefined Behavior or border um bounce bounce Behavior here then you can do mitigations you can have a lot of compiler options a lot of hardening techniques uh which helps you to do some bounce checking at run time there’s also going on some Hardware supports like memory tagging where you can with your uh computer chip help that you don’t have this or you can detect this memory uh safety issues during runtime and you can also try to put some critical stuff into a Sandbox to mitigate the results of an exploit here and also what you should do is you should detect as early as possible so that’s why we also try to do the testing and fing in as part of the development because it’s the easiest way to fix if already um don’t let this box go to production this leads us to the um yeah B model so in the automotive industry if we develop our software according to Automotive spice so this is some standard we use and there’s it’s basically a the model but of course in the AED child world we maybe have some more iterations in the lower side of the V but overall we do a lot of trainings to make people developers aware of new tools like to how they can use rust how they need to write H tests we have like basic requirement of course nice compound burn uh coup uh kgas like it’s conern uning software so also Vinnie has I think something on this this is if you want to do business with ew or deliver software you have to follow these guidelines then we do secure risk analy before on the implementation so there we try to see what are the risk what is the Atta model and what mitigation or controls we need there to reduce the risk um for a spe specific component then during the implementation phase of course we do a lot of uh yeah a static code analysis secret scanning we there try to make the s topic which we also have later and then we do fuzzing uh test if all the binary hardening mechanisms are in the product or not if the um debu ports are closed and so on and then uh in the last step we have the continuous fting continuous vulnerability scanning and the whole vulnerability management so this uh let me get to to chy who will tell us uh something about the supply chain yeah so now uh let’s talk about the supply and security and where it going to come back to us so uh typically um uh in manufacturing World um the idea begins at the before of the production to to 5 years ahead so we ask for our suppliers get the quotations of procurement checks and all the of of all the life cycle K management is managed from there and after the the vehicle is on the road we need to maintain certain regulations from Global various regulatory perspective that means the vehicle M um they need to be Road worthy depends on the country and regulations from minimum to 10 or even 20 years down the line that’s give us to the perspective of where it going to come back to us that in each of the countries have their own variations of the procurement guide guidelines as well as Road worthiness guidelines so each country have their own I’m not going to go into the details but again this idea is evolved from the Niti minimum elements of software build up materials why we need and why it is going to the forward as we see from Automotive perspective particularly the unac compliant countries are 155 156 and part of the r 157’s are critically important to get the homologation type approval for a series of manufacturing start of the production as well as us also have their own set of um compliance check so it is very very important that we understand the supply chain complexity and in manufacturing World be more proactive than reactive because the the use cases start from before 3 to 5 years of the production till next 10 years at least or 15 years so around 10 to 20 years ballp Park area so let’s see how we drive from supply chain secur challenge to the as bomb in the solution space so as we know from the the typical definition of es bom it is a uh inventory of component used to build software artifacts uh artifacts such as software application or some products and now the question is why it become over the time uh we all learned from our experience that it become critically important in particularly in supply chain security aspect so at the before the start of the production when we are in the development phase which typically start from the procurement clarification or compliance like kgas and other compliance there we perceive the the possible risk identifications if that pass through then when free development or prototypes aspect we identify the potential software or Hardware risk and when the production begins or we come come with a huge amount of security incidents like my colleague Andreas explained or even license claim like GPL or lgpl or specific BSD or unknown author or copyright license violation claims as we all know about and to understand the whole problem pH we understood that s bomb is is the way forward to find this light at the end of the tannel from there we can derive not only the the cyber security relevant information also to improve the code or design of the architecture to extend or maintain or the con Next Generation architecture to evolve like CW my colleague explained and in the production of next 10 or 20 years this live monitoring can help us to track down and mitigate identify detect dater and mitigate the risk like CSF or other security framework so going about going forward as we see this is um typical cyclone DX 1.5 example of the s bomb having the mandatory fields and bomb format and and and the spe vers others are optional but which is important is to understand the bomb reference and how we derive the license and component information which is also important to understand that the p uh and the package URL and CPE help us to scale the solution space in a automation ready and maintainable for near the future going forward uh the next step we identified there the vdr or vulnerability disclosure report that is a highly potential Direction Where We can scale the vulnerability management that means if we go back to n definition they can and identify the potentially uh exposed or non-exposed vulnerabilities in a various templates for example if we go to an example or if we take the properties of bdr we know that it can affect or cannot affect it can have the impact it can be sign authenticated and verifiable it can be ensuring that it is compliant to the databases and uh stuff like that what is important that the most recent sbom formats like spdx 2.3 or CDX 1.4 onwards they by default support the vdr now as we have let’s go to the one example as was trying to demonstrate so sorry so here we see one sample example like the the CV ID the source and the scoring which is taken from nvd 2.i API and we have the disclosure the impact area here so it affects this person from there to there and this is the reference package signature in the same way uh we can and scale this approach but we have realized that vdr is very much generic we can also focus on the specific goal oriented approach that whether it affects directly or not to make it more deterministic so there we came into the understanding that cisa is working on the vulnerability exploitability exchange format the first definition was released few years earlier 22 and is being extended by the multiple working uh group going forward and we would like to show how it can be more effective so here it is deterministic mean directly we can look at the field of the v v part which is also compatible with CF 2.0 onwards that the state is affected or not affected or under investigations or fixed and stuff like that and also CSF has their own matching table to onwards one of the profiles like affected exploitable and stuff like that so that way the Vex simplify the generic vdr approach to more focused on deterministic approach we can directly ask the supplier for a specific product or component give us a a Vex and we can automatize or Supply can itself automatize the Vex generation and share us uh disclosable uh report in a confidential way that way each supplier in the chain of the full stack they can uh uniquely and confidentially share their disclosure report up to the OEM or manufacturer without leaking or risking any copyright or legal or or vulnerability risk as well as they are complying to the all the regulations in place and making their process from their side as well as to the our side to more effectively optimizable that mean more efficiency less manual work like any classical work like word table PDF or text files and all that can be obsolet going forward when we able to use the right tools at the right problem going forward uh I would like to hand over to my colleague for the most more advanced upcoming Crypt bomb Andreas please thanks joy for the Mi let go to the next slide so the the upcoming things so now if we have would have all implement mented this like s bomb spec bdr the next thing would be crypto build of material so what is this um so as you know um cryptographic algorithms like C DEC can be deprecated for example in the past like we have PES md5 Shon they all are broken and you should not use them anymore in production in the future if there will be Quantum Computing RSA also will be broken we need something different key material doesn’t last forever it can be invalided um it can be you know it it has a an an end of life date so what we need is we also besides normal Asset Management we also need a cryptographic inventory asset management and how do we achieve this um we could start with something like a finer craned as bom for example here we see a Conan file so this is the package manager for plus C and for some um libraries for example lioton which is a crypto Library you can tell what part of the crypto Library you want to use and then only these parts will be uh compiled and configured into your system so you can say I want a uu ID I want a random number generator and I want the sh 256 and you specify this and you could already expressed this in uh for example Pon DX in the data field where you could specify the configuration and then um tell okay this is the configured modules I have but the downside here is that not many tools support this and also a lot of libraries don’t have this fine crate configuration um techniques so we need a more standardized say here and the answer here is Crypt bomb so on the right side you see how this would look like you have your classical um esom component and then you would have like a sub component that you say okay this component offers this uh cryptographic asset this is a hash algorithm and has has this o ID this is for some cryptographic algorithms they have specific IDs you can look look up in some databases and yeah the nice thing is this was proposed by IBM they made some um proposal and now this will be joined in the next cycle on the ex standard which will be standardized I think in the end of March beginning of April so it should be arrived very soon then we can have this data in the standard way to exchange between uh the suppliers and us and um how we can get this C bomb is with of course that we needs more tooling um currently there’s not much Tooling in this area um but what we did for example we can use like static code analysis on the right side you see um this is the query from Cod at GitHub um how you would find insecure um crypto algorithms like Das um and so on so they do a recx search on function called so internally they have the abstract syntax tree and then you can search where this function is called with this regular expressions and if you know your API you can search for specific function calls um this is the the yeah most of the things but some things are not known during um static compile time so for example if you have a runtime configuration or you need to negotiate a cipher you cannot be 100% sure what algorithm will be used during runtime so there you also need maybe some Dynamic analysis so when your system runs you need to determine what uh cryptographic algorithms are used um just to to sketch this up a bit so for example you have like one uh cryptographic Library which offers certain um um yeah search and functions then you can build a c bombb and tell okay this function offers this and that um algorithms and then on the other side you have your um yeah application Library you generate an s bomb then you search for all the calls to the crypto library and then depends uh depending on this what you found there you can then reference the crypto bomb from the your crypto library and of course if everyone uses your one crypto Library this is much easier then everyone pulls in their um own crypto library or the worst thing they would roll their own crypto then it gets very hard so giving now to Vinnie yes so thank you Andreas and joy uh yeah so I will try to just give some brief overview like on the auto on the automotive side of the vulnerability management uh that we are doing and also like some of the challenges uh that we have because uh like yeah the overall Automotive system it’s a very complex one uh and moving forward okay Works um yeah I mean from the vulnerability management perspective like we U divide like the whole ecosystem like into three parts one is the ecus like where the electronic control units and has already uh introduced uh that and then uh we have the functions um yeah which are running on them something like uh it could be any third party application or uh the the climate control or your remote um yeah wakeup or unlock functionalities or any such things and then we have the third part which is uh which are the services like the TLs um or the secure onboard communication or the key management uh which is used uh inside the car and again the whole uh part of the vulnerability management I’m only speaking from the product uh security perspective and we ignore uh yeah what happens with the back end uh function functionalities uh and moving forward normally I mean for us uh the some of the things uh like how it’s placed this we always have bunch of suppliers and sub suppliers uh that are involved like with all of these things like we cannot build everything on our own so some of the ecus like we Source from our suppliers and sometimes like it’s just one supplier and sometimes like we have like the one tier 2 tier three and it goes on and then we have the integrators we sometimes like they are internal like we do it ourself or sometimes even they are contracted uh yeah for example if there somebody develops a function which overarches couple of isus then we have an integrator between these two between the isus and the functions and similarly we have the services uh like also like have integrators like who are trying to uh or you can yeah call them uh something like yeah the system integrators say yeah and for us the other challenge is we do this for the entire ww group so it’s not just one particular car or one particular uh product we’re talking about it’s for the entire group like including Audi Porsche Bentley C VW and yeah all the other brands um and moving on yeah the other part uh we uh realized uh from the auto from the vulnerability management normally I mean people tend to associate this with like what happens um uh like with the release and the response uh sometimes like as in the vulnerabilities show up uh in the product and yeah uh it’s more of um uh to say uh a reactive way whenever a vulnerability is discovered then we trying to handle it but in our case like we are trying to be more uh proactive and try to get involved in the entire uh V model that Andreas showed like we provide trainings like or share our knowledge on the vulnerabilities that we uh discovered uh previously and educate the teams like uh on how to avoid this and we give feedback towards uh the requirements itself like to make the requirements much more stronger and better and also we get involed uh during the design stage um uh as well and provide our feedback and yeah so so we are pretty much involved or try to get involved either in an active way or in a passive way in the entire uh V model so that we uh like yeah this also goes like along with the shift left approach uh that we have like where yeah we are more uh proactive and out of this like we chose like uh yeah five uh complex or like five challenges that we are facing I mean to say there are really plenty of it but uh yeah uh kind of like summarizing it into five things like there always like the challenge we have of the very complex supplier Matrix as I stated earlier with the Tier 1 tier 2 and tier n suppliers it’s always the list goes on and also tracing the vulnerability to the source uh it’s always a challenge I’ll explain it uh more uh later and also prioritizing the vulnerability reports uh that we receive we have multiple sources like from which we receive vulnerabilities like from scanning and from uh bug Bounty programs and from also like the penetration tests and our own uh internal offensive security teams yeah and also there are really nice talks on prioritizing the vulnerabilities uh in uh in this conference already earlier and also and the other challenge is uh ensuring uh that every patch is Thoroughly tested and uh delivered on time and this is uh especially most uh crucial for us because sometimes like the car is already out on the street with the customer and then we need to make sure that nothing breaks and because sometimes it can have serious uh implications like somebody is driving on the autoban uh over here like at 200 or 300 uh kilm per hour and then there is a vulnerability somebody can exploit remotely and it’s not a good case or we have fixed the vulnerability but it’s not tested properly so yeah we have to really consider these factors as well whenever we deliver uh any patches because so many lives depend on it and just as an um yeah overview like the out of some recent studies like the average life of car is around 10 years and yeah and in Germany especially like it’s around 14.8 years like until people scrap their car so we have to provide the support uh until this uh yeah time or at least ensure that there are no major vulnerabilities uh which are affecting so and going on a bit about the uh what does the regulation States uh I mean I only focus on a couple of regulations uh in from the perspective of the Europe uh it is the United Nations economic commission for Europe like they have or in short like unce they have certain regulation requirements uh towards the vulnerability management and in terms of the process itself uh they State like we should have a process in place um as part of our uh csms system or the cyber security management system uh that we monitor detect and respond to uh any kind of cyber uh attacks threats or vulnerability on the vehicles and yeah and it says like yeah we should have and the interesting part is like uh yeah it says it should be mitigated within a reasonable uh time frame so this is for the unce and for the other thing I mean I just go to the China gbd because this is like where I’m also like working on right now uh there are of course like many other regulations from other uh countries as well uh and the China GB uh regulation like it is still in draft the whole vehicle cyber security regulation it will come into place uh sometime like from 20 27 28 like it this is still like our forecast and yeah and over here it states uh like the vulnerabilities with the high risk or AB must be handled uh within uh 6 months of uh time so these are this is like what reg couple of Regulation States but yeah and also yeah uh now going uh into to the uh like how normally like our uh product life cycle look like we start the car project and we have many minor uh software releases like from all our suppliers and then we have we set major uh maturity Milestones uh which we achieve like where we integrate all the things like and test the whole uh system is tested thoroughly and after reaching uh like the maturity milestones we have set then we start the product ction and then it goes on like with the over there updates until the end of uh the life cycle of the car and again coming back to the poity like of the ecus function and services like we Face some really hard challenges uh as in what happens like if the suppliers or the sub suppliers or someone else like goes out of business like after sop as we saw the average life cycle of the car like if you assume it’s around 10 years then we need to provide software support until then for any kind of vulnerability that comes up and because the automotive side is heavily dependent on the suppliers we cannot have everything on our own it’s very uh yeah crucial that we have some some kind of a system in place if they go out of uh business and also should also consider the financial uh model like if a supplier has delivered something and if there is a vulnerability uh over there and which we want it fixed then the suppliers will normally not do it for for fre weeks should also have a robust financial model uh in place to to tackle this and in the end somehow this will reflect back on the customers so we try to come up with a much optimized uh yeah financial model as much as uh possible okay looking at the time have around 6 minutes so I’ll hurry up a bit so to tackle this like uh from the video group like it one of the obvious solution was like we try to shift from A supplier based model into U own in-house development and this is also the reason like why carad was uh formed and we analyze like what makes more sense like to develop in house and what makes sense like to get it done by the supplier uh thoroughly and then we try to build it uh in house so because we cannot completely remove the suppliers we also have some very strong um requirement towards the supplier and one of it is the VW kgas or the KAG and in German it’s concerned gr unan software where we State our basic uh requirements towards all the suppliers and they have to fulfill this in order to do business with uh with a VW group and and yeah and moving on uh yeah the Second Challenge like um yeah tracing the vulnerability to the source uh assuming I mean vulnerabilities can occur anywhere either in Hardware in Services functions uh like assume like if there is the next heart bleed version to uh that occurs then we tend to derive the facts as in uh like now we know like the TLs service is affected out of this and we find out all the ecus in the car which are which have been affected which are using the affected open SL version uh in it and then we list the functions or the applications which are running uh the TLs services and we should get it fixed but again still there are more things uh as in um yeah we should see like where where is the source of this package sometimes in the car like we also have uh lot of third party applications like which are running who package their own op SSL uh pack uh libraries then the biggest challenge for us is to identify who is the owner of this risk if it’s the ECU or the function or the service so on and quickly moving forward um yeah so with this uh we try to set up multi-directional uh dependency tracking uh among uh yeah among the three and also already as spoken we have the software bill of materials uh we try to do this uh in the Cyclone DX 1.4 or higher as also like stated by the BSI uh requirements so it’s not Automotive relevant but we still follow it to be uh at par with the industry and also uh along with the software bill of materials it’s not written here we also have a hardware bill of materials to track all the hardware artifacts that we have and as per the unce r155 we have very specific security roles that are defined for the eus functions and services we have security owners like who take up uh yeah the ownership like whenever a particular risk uh arises out of a vulnerability and then the prioritizing I don’t go too deep into this it was already spoken I guess in the previous talks as well uh and looking at the time uh yeah we have a customized PRI prioritization function that we follow which includes like CVSs epss the Vex and uh yeah and everything uh so and this model is also continuously uh evolving and out of this we derive a prioritized list of vulnerabilities and we try to uh yeah remediate them because yeah of even we cannot fix all vulnerabilities once so we should prioritize them yeah and also maybe the other thing is now there is also a cyber security Assurance level that is being drafted and standardized uh as per the iso 21434 yes and the last uh the testing and delivering we know like typically for in contrast to the mobile phones we are using uh Android and iOS provides around 8 years 3 to 8 years of support and the Linux LTS kernel support is approximately 2 years but for us like we have to provide much more longer support and that means we have to maintain all the tools and all the testing and the vehicles like if it’s a mobile phone then you know like within a single roof like you can have hundreds of mobile phones like running different kinds of Hardwares and different software versions on them and it’s easier but for us it’s cars we are talking about so we should have every variant of the car and every uh yeah variant of the software uh like and we need to test all of them before we provide any kind of uh release and also uh yeah as per the solution we try to get as much of knowledge transfer as much as possible from the supplier side and we invest highly in the test management and we do penetration tests all the time before delivering anything to the customers and we tend to use as much as open source tools as possible because anytime if we don’t feel confident about any particular test we have done we need to reproduce them uh all the time or anytime sorry um and just to summarize on the Lessons Learned um speakers we have we have another group coming in this room in this channel so uh we are going to have to go ahead and and end the talk at this time uh for folks attending please do know these materials will be available on uh YouTube and the first.org site if there’s anything you missed I do want to thank Andreas J abat and ven for the excellent uh content here today let here for our speakers