Tools and Methodologies for Edge-AI Mixed-Signal Inference Accelerators
Maen Mallah
Senior engineer
Fraunhofer Institute for Integrated Circuits IIS
Roland Müller
Senior engineer
Fraunhofer Institute for Integrated Circuits IIS
A major obstacle for the Adoption of NN on the edge and near the sensors due to their high computational requirements. Our approach at Fraunhofer is to develop novel neuromorphic mixed-signal edge-AI accelerators. This approach comes with challenges that require hardware/software co-design and a dedicated workflow. For this purpose, we developed several tools to facilitate design, training and deployment of artificial neural networks in dedicated hardware accelerators. These tools provide hardware-aware training, automatic hardware generation, compilers, estimation of KPIs like energy consumption, and simulation under consideration of the constraints imposed by the targeted hardware implementation and use cases. The development of such a tool chain is a multidisciplinary effort combining neural network algorithm design, software development and integrated circuit design. We show how such a toolchain allows to optimize and verify the hardware design, reach the targeted KPIs, and reduce the time-to-market.
So let’s start hi everybody my name is Danilo Po and is my pleasure to introduce you this great tiny talks from from Mala and Roland Müller fromer Institute the title of their speeches will be is tools and methodologies for Aji mixed signal inference accelerators but before to leave the floor uh to our
Excellent speakers uh let me remind that these events as well as many other organized by tml Foundation would be impossible without the great support of the Strategic partners and let me remind who they are AI zip Arduino arm brain chip Ed impulse embed Ur efficient green waves gravity IMAX image mob infinion
Inan not AI nxp PNG G Pi polling technology Qualcomm Renaissance Schneider Electric sens Sony silicone Labs STD microelectronics synoptics senent and TDK and apologize if I uh not pronounced properly any of these name these great companies let me move forward being part of tinml essentially means to be part of the
Greatest Community a worldwide level that Federate all the experts and the company and research labs and many more entities small medium Enterprise in one Community which is much concerned interested on Tiny machine learning and there are two main um assets which helps the community to stay alive and to uh
Interoperate among different entities one is the meet up uh across groups across count countries 41 countries so that’s the measure of the diffusion at worldwide level and the LinkedIn tml Community uh which is exchanging continuously information and latest progresses on many aspect of the tin ml uh technology um it’s important to remind
That tin Foundation is making happen the availability of the huge amount of content multimedia content through the YouTube channel uh so if you are looking to information specific to tinml uh Community Dimensions Hardware software tools algori uh and many other subject please do refer to the YouTube channel subscribe to it if you
Didn’t do do yet because it’s fundamental aset and to retrieve an information that maybe you missed or you need to refer uh it’s important Ultra important you keep in mind that uh on late April 22 next year uh it will take uh place the tml research Symposium This research
Symposium is unique is unique in many sense first of all it happens in San Francisco where uh you know the Silicon Valley is is close to there where many entities are placed where lot of innovation are happening and also because uh it’s the unique it’s a unique
Event so please do contribute with your outstanding research and is fully focused on Tiny ml uh enabling technology so it’s a unique opportunity for you to contribute please do so and uh spread also the the the news around your community because it’s important to be there and to contribute uh another
Important document that has been produced by the TL Foundation is 2023 Aji technology report is very useful for all practitioners freshmen Engineers experts which are seeking to stay up to dat on Aji Technologies and uh so far has been a very successful and next year there will be a new edition so the
Document will be refreshed but don’t miss the opportunity uh to download it and to have a look at it and to learn from it then so without hesitation let me introduce May Mala uh which is a researcher in the area of Ed AI at The franer Institute for integrated circuits
He achieved The Bachelor in t communication Engineering in 20 14 at the an NAA National University Palestine and he master in electrial Engineering in 2018 from bant University turkey with the thesis titled multiplication free neural networks that’s a very important uh Topic in t in March 2018 he
Joined frer as an expert for embedded Ai and focusing mainly on energy efficient and neon networks main work and focus on implementing optimized neuronet for Edge application designing special tools required for such a task with a special focus on quantization in F we uh let me also introduce ran muler
Ran achieved this bachelor in at the otth Regensburg in 2017 and master in 2019 at fa aagen b electrical engineer in May 2019 19 he joined the Department of integrated circuits and system at crop IIs eragen Germany where he’s working in the field of analog mixed signal design of neuronal accelerator and design
Automation for such circuits currently he is pursuing his PhD uh his main research interest include low power analog mixed signal circuits neuromorphic Computing and electronic uh design automation so thanks a lot to both to be available and for this speech and I you thanks a lot for the introduction uh let’s start our
Presentation as was mentioned earlier me and my colleague Rand will pre presenting but I want to stress out at the beginning that this is a larger team than the two of us we are just doing the presentation we have the names of the people who participate in the work and
There are others helping to support us there’s not enough room to write the names of everyone but we just want to stress that it’s not the work of two people only yeah first let me take you through the outline of this presentation we will have some introduction about FR
Hofer I and our team and then we will talk briefly about the hardware so our mix signal neuromorphic Hardware that we developed here at frover and then we will talk to the we will go to the main topic of the presentation which is the software tools
That we need to support the mix signal neuromorphic Hardware that goes from the training to kpi estimation mapper compiler and H generator and finally we finish with some characterization and demo and end with a Q&A but as mentioned there will be a Q&A in the middle as
Well if anyone has a question in the middle so first frer for who don’t know is community of research in Germany and IAS Institute for integrated is one of these many institutes for research it’s the headquarters is inan and it dist is distributed over 11 cities in the
Southeast of Germany there are currently more than 1100 employees focusing on a huge range of products and projects and research Fields but our topic is neuromorphic Computing and this is mainly you entit a lot of different things but we try to represent them with a few small aspects
We focused on mixed signal and neuromorphic Asic and that’s for both DNN and snn so deep neur networks and spiking neur networks our talk today is mainly about deep neur networks but in the wor we also have spiking neur Network as well um our belief is that to achieve this ultra low energy
Consumption for inference we need analog multiply and accumulate computation and we also want to do that in real time so we want to achieve ultra low power process ultra low processing time per inference so we can do fast computation and high data processing and finally we should achieve all of that by
Go design between hardware and software and having automated tool chain that achieves all of our Target and mainly the accuracy of our neuron networks one once deployed on the accelerator with that let me first introduce briefly our Hardware I will not go into details uh but mainly this
Is the you know mixed signal architecture that we developed here in house in frover it’s co as Adelia generation 2 we support analog and mixing processing cores there apus that communicate using a cyclic bus fabric you can see on the right just a diagram of the Apu of one of them there are
Multiple and each Apu does analog M compute for fast and energy efficient multiplication and accumulate operations each of these apus is f programable to achieve as much flexibility as possible and finally with all of this crossbar array and the analog in memory compute it’s supplemented by low power adcs to
Interface to the digital where we can achieve the maximum flexibility but keep our efficiency and the architecture supports different weight and input precision up to N9 bits for each of these input and weights so that was the the architecture in general in general and one of these
Architectures at least that we made and fabricated was a six core or chip was made in the project andant and it has one kilobyte of instruction memory Network and ship controller and digital and analog test circuits each APU has 256 * 64 synapses so these are all the connections that we
Have each of these have three bit sign Precision of course we can combine multiple of these columns to generate higher Precision if needed each crossbar array has 8K by distributed SRAM 8K by input memory 8K by m memory and 1K by instruction memory and each of them is
Complemented with 32 low power ads in addition to shift and AD log that can do bash normalization so with this brief introduction of our um yeah Hardware uh we want to now talk about mainly the focus of the tool such a hardware we realized early on requires a
Lot of software development that goes into with it to achieve you High um design fast cycles and to be able to optimize and chular network models that run on such a hardware and for this we designed our own tool flow that I will present now briefly but then go into
Detail of each of these block so we start here at the top in blue with the neuron Network Hardware generator this is mainly used to generate layout and schematic for our Hardware using the description that we provide second we move to a hardware Weare training this is the training tool
That we use to to produce a quanti drop neuron Network model that on our hardware and produces the same accuracy that we see in training and accelerator that quanti model is then to a MPP and compiler tool that generates our compiled program and this compiled program can immediately then be deployed
And run on our Hardware using the deployment time API finally all of the workflow is complemented with two tools that do architecture exploration and simulation and kpi estimation both of these generate metrics and kpi that are very important to do optimization of both our Hardware using the system architecture where we update our
Architecture and our neur Network models that are run on the accelerator by updating the neur network model details so let’s now maybe dive a bit deeper into each of these tools first we have the training part where yeah we have our Hardware training that produces the quantize neural network model or at
Least it should be also as well robust it requires a labed data set and a neur network details so if we go deeper and we explain why we need at least this tool first we have the quantization of our training part and this is the first step
To achieve low energy by reducing the Precision we need in our operations so if we reduce the Precision of the weight inputs we can achieve higher calculations with less energy required and Ne n works are robust to quantization to a degree but eventually we need some quanti aing to go as low as
Possible for this we mainly use or extend on Pras and that does the quantization part so here I have on the left the torch layer that has weights input bi and produces an output and this is a generic torch layer what prita team did is they produced this layer that injects
Quantization details into either weights new inputs bias or output and you can put a quantizer or no quaner at all so you can just pass them as floting points but the torch layer itself is the same layer that can still be run on GPU can be still trained as normal of course the
Quantizer has a lot of different parameters that can be set individually for each layer for each parameter in the network for example scale number of bits if you want to delay the quantizer for a few iterations of the training what is the zero point and many others that you can set and configure
Easily now we have used mainly Pras extended it a little bit to suit our needs for our Hardware we needed but mostly this is out of the box from a open source library then we run into the main issue or let’s say one of the challenges that
We have in analog and memory compute which is the noise and inaccuracy variations mismatch that happen on the hardware for example here we show measurements that we took from one of our accelerators where we measured the weight value that it should be stored so when we want to store a value of for
Example zero we have a distribution that actually ends up in the storage units in memory from minus point2 to2 and so on for all the other weight values these mismatches can cause the neuron Network to produce completely wrong values not as was already designed and integrated so for this
Purpose we introduced fault a training where we we take account of this mismatch and fals the one I showed as an example but there are other fults bit error or noise of the actual values of the activations can be introduced in our tool to mimic the behavior of the
Accelerator of the hardware so that when we actually deploy the two the network we get the same results and for this we devis the hardware where training layer it’s similar to the previous layer however now we can introduce normal quantizer or F training quantizer or no quantizer
At all and the torch layer is still the same basic torch layer that runs on the GPU and can be trained can be loaded and can be stored a fault aware training quantizer is an extension of a normal quantizer so we have the quantier in the middle that takes the same parameters
Can be set individually for each parameter in each layer but in addition it has two extra steps to do one is pre quantize operation and the one is post quantized operation so before we quantize the values we can introduce to them any implementation of any variance
We want and at the same after the quantization operation we can introduce any variation we want with any implementation so to show some results from this tool we use voice activity detection use case here we have an input of log spectrogram size 13 * 64 and that is has an output binary
Either Voice or no voice and the network is five convolutional layers followed by two fully connected layers we have 25 K parameters almost and 1.2 million mag operations for such a network the floating Point accuracy we get is 89% in the test data set and here we introduce
Now or at least I would like to introduce two types of Hardware variations that we use in this simulation one is the mismatch of the anic synapsis that’s the example I showed above and here we use relative variation so the error we introduce is relative to the value of the weight
Itself and then we introduce bit error to the values of the ABC in the accumulation and this we introduced as an absolute variation on top of the actual all the activations we have and to show the results here when we train with without any hardware variation so here we have quantization only training
We get 88% accuracy which is 1% lower than the floating Point Network it’s pretty good for um 8bit Network however when we now introduce in our test the hardware variations our accurs to 50% and basically the new network does nothing it’s just randomly getting a label and getting 50% accuracy
And for this we see now if we turn our new network with Hardware variations we lose about 5% in accuracy so from 88 we drop down to 83 because of all of these variations that we introduced that some of them are significant to really lower the accuracy however again we get is now
When we actually deploy it or test it with the hardware variations we get 83% and this we verified in a lot of different stages in sofware and Hardware so in summary I would say the yeah for Hardware training tool is divided into multiple sections one is quantization of training here we allow
To inject quantization into the P Tor graph to produce a low memory foot print and this reduces the energy cons second we have the fa training part which is the Innovation part from our side where we can introduce now any variation into the input weights and
Bias or even output of any network of any layer in any network and this helps to produce a robot Network where the accuracy that we get in the training is the same accuracy get we get on the inference when we run it on the accelerator and finally we have a custom
Onyx export to help export all the quantization details that are needed by our tool chain to run the neur network on the accelerator and the model exchange as well is standard pytorch so we still can save the model as a standard pytorch model uh with all of this quantization parameters so we
Can retrain the model save it load it and Export it to other workflows when needed um second tool or second part of the tool flow I would talk about is the kpi estimation and this is very important to optimize our neuron Network models for our accelerator we found that
If you design the correct neural network model you can get a lot of improvements from the actual accelator you are running on so to to be able to do that fast you want to do some kpi estimation to produce how much not just accuracy you’re getting on the accelerator but as
Well how much energy and latency you’re getting and for for this we have now our current kpi estimator that assumes IDM mapping takes the neuron Network information so how many layers we have and for each layer the number of Mac operations number of parameters and quantization this detail in addition to the architecture
Descriptions so how many cores we have that we showed in the earlier slides how many cores we have how many processing elements are in each core and what are the srams and Global SRAM we have all of this is used to be to calculate using a reference table the actual energy and
Latency that Network would produce however this here assumes ideal mapping which means we assume each layer or each mag operation on the hardware will be fully utilized we can read the data as fast as needed and everything works ideally this is in theory not always uh or in practice not always was
Right and that’s why we have the map yeah mapping aware estimator so in here instead of using the new neuron Network information immediately where we just assume everything runs perfectly we actually take the map neuron Network and the map neur network is produced by the mapper we will explain it later in more
Detail but in essence it will contain how the neuron network is mapped or how the load of the neuron network is mapped into each of the operations and memory into the hardware this is converted into an action counts table so how many actions should be performed for to carry
Out the computation in the map Network this in corporation with the architecture description and the reference table will give us a more accurate estimation of the latency and energy of a neuron Network so using this kpi estimator then we will be able to determine fast at least the energy and
Latency that will be consumed by a neuron Network on our hardware and this I would startop for a Q&A if there are any questions now thanks thanks sorry I don’t see any question in the q& part um maybe I can POS some some questions in the meanwhile someone post
Their their question question uh certainly you shown that the best um achieve achieved performance is with um fault aware uh quantization training um what kind of type of uh quantizer do you support in your environment uh do you are no variability linear nonlinear quantizers or or you have special
Fus quantier in your in your f chain yeah I would say the quantization as I said is mainly part of bitas uh there they support multiple quantizers so any quer that’s supported in is also supported by hardare training and you can it’s at least the most common one they
Use is yeah each weight is quantied by itself but there is no more details so you can quantize yeah any weight or input to the closest range that you define and the range can be flexibly defined either by learn parameters or for a range that you actually Define by
Hand okay thank you very much there are questions appearing the first is if the kpi estimator is available in somewhere in any form unfortunately not yet this is mainly work in progress as the whole thing is work in progress uh we are working currently on a project to
Publish some of these U tools to the at least as open source software at least when that happens we will notify people but currently nothing is available yet for the public okay second question is how your implementation comparing precision and energy consumption to G GPU based line
Solutions do you have any figures any comment about that um we haven’t compared at least Precision uh in with gpus but we are working on benchmarking now but at least it’s not in our road map to compare to gpus we want to compare to other accelerators that exist
In the market but not gpus because the target is very low power Edge devices that run on maybe battery so if you want to imagine it would not be a GPU it would be something run on a battery for a few days minimum that’s applications for our accelerator and then we will
Compare with similar in the same range or slightly higher range yeah uh thanks man there is also another question more or less is in the same direction at least it give you a little bit of feeling maybe on the importance of benchmarking there is another person that is asking the
Benchmarking compared to TPU uh GPU again the tensor processing unit and application Processing Unit so you get the feeling a little bit how the the community is interesting to compare one solution to to the wellknown one even if as you clearly St is your development is for very low power and resource
Constrain devices any comments or yeah would say benchmarking is also very important for us uh we currently are doing the benchmarking itself we’re testing our accelerator versus others we still don’t have yet public numbers that we can release but it’s also very important to us because we want to know
Where we sit and at least what we observed is that the high yeah the high accelerator market so mobile phones level and higher is already at least in our opinion well developed for and that’s why our Target is the lower than that even of mobile phones and we think
There is a gap there but now we are we’ll show that in numbers of course and we are interested in benchmarking and releasing these numbers as soon as possible okay alline uh next question is if I think is in the in the framework of continuous learning if you can retrain
Refine the model once deployer you sorry can you repeat again the last I think the next question is related to the topic of continuous learning so can you refine can you retrain the model once it has been deployed uh currently not we have plans of future updates of
Our hard where we will be able to modify some of the weights at least and integrate yeah increase the this retraining refinement on the hardware but currently the current solution we are working with only does deployment and inference without rainning okay thank you uh another question is about quantized model export
To anix are you exporting the model with quantized operator like Quantom or using Q theq for format uh which one would give ideally uh lower runtime memory footprint I’m not actually sure what Q format is we looked into Quant com and Quant fully connected operations that Onyx supports however we found they are
Not suitable for our case because they only support integer 8 and un integer it and integer it what in our case we support three five and 9 bit Precision for the weights and similarly for the input we support from 1 to 8 bit precision and all of these were not
Possible at least with the Quant con and Quant fully connected operations that onx currently supports we are looking to try to find a common solution with everyone so we can have a Standard Export not our own export but that’s currently we have just custom onx export
We try to keep it as close to possible to onx so that once we reach a common standard with everyone it would be very fast to change at least that’s what we hope okay uh another question is uh what techniques are used to optimize the PPI is there any optimization step or
Anything that helps um currently we are like that’s not part of our research but at least what we find in important is to produce the to be able to produce the kpi estimation and we use out of the box solutions to optimize genetic algorithms your architecture search where we give
It the black boxes we have so this is the estimator this is the details of the network that we can specify but we are not yet working on the neural architecture search part of this optimization okay thank you was there a thermal drift in accuracy position sorry again was there thermal
Drift in accuracy precision not sure so I think this is related to the a actually and how the circuits are designed there’s very few varations to to temperature so on accuracy level it’s like plusus 1% uh um and therefore I would consider that really as a drift so
The circuits all our circuits are designed to work on uh mismatch and uh temperature variations okay say let’s continue with the presentation if there are other questions we can answer uh wait a second M there are other five questions in the Q&A session so please uh uh make sure
That these answers are are uh answer it even later on uh because you know the community is very interested on on your talk so it deserve also the Merit to to get the answers but definitely is time to move to Roland speeches and so please Roland uh take the
FL yes so thanks mine first of all for presenting the first part so up now we have most of the things we need for training it was a kpi estimation but you also have to get to the actual hardware and the first step you already heard about it is mapping the newal
Network to the hardware so in this case um we have the quantize neural network model we get from the hardware rare training tool um and we also have the system architecture so how does our chip actually look like and that contains number of cores that contains um for example the memory
Sizes um all there is Solutions we can we can Implement and that goes then into the mapper and the mapper actually has the job to distribute the workload on through our our chip um such that we get a compiled program which is then executed on the actual
Hardware and where is this within the within the tool chain so we saw we have the new network model here to mention py toor but it could can also be our most likely it’s going to be from our Hardware training tool then we have onx which is a custom onx also containing
The compensation information um and that then goes into mappa and and compiler and there we get the instructions for our custom Hardware so what’s inside mepp then um we have that down here and the first part is just to check okay is the data actually
In the correct form so we get the onx file we read it uh and we walk through it and see is everything correct can that actually be uh theoretically mapped that’s the first check to give the uh the user a clear info or clear indication okay there’s something wrong
Or we can go next step then is uh pausing the information into a graph so the neural network will be represented as a graph um and the P does that by going through the Onyx file uh converting the information into something we can use inside the
Mappa and then it builds a linked gra to represent a new network next step is analyzing uh the the graph we just created um to understand the data flow how things are connected and what we actually need on the hardware and then it gets actually into
Uh the mapping step and what the mapper in there does is creating a map space and the map space basically contains all kinds of different mappings so neuron networks can be mapped in different ways onto our Hardware you see that on the next slide and this is a huge map Stace space
So there might be thousands 10 thousands of possible mappings to uh execute the same neuron Network on our chip and this means out of the map space we then have to select one mapping that suits our use case and our requirements and that’s done by the Optimizer so the optimizer and that
Comes also back to kpi estimation um the optimizer takes the mapping information gives it to the kpi estimation um from where we get the kpi so this might be then might include uh the latency throughput energy consumption and also how much memory is used and then it’s the the decision is
Made according to what is um what is the actual Target should it be energy efficient should it be fast um we select one at least it has to fit onto our Hardware so we have to have enough resources and that’s the optimizer that’s where we can Target the mapping
Of our neural network into certain directions and finally uh the selected mapping then is given to the Schuler which creates the instructions for our custom hardware and the main workload um is actually there in mapping and optimizing because it’s just such a huge uh map
Space and how does this look then um so we got our multicore accelerator here um with with the crossb and we have one mapping example so the different layers are mapped in different cores so we got layer one here Layer Two is then stretched across two cores as
Well as layer three and so on and then there are some layers which also in parallel in the course and as I said this has a huge impact on K and if we for example take Layer Two the green one here in this case it’s mapped such that all the
Weights are already in our accelerator course so we don’t have to move those weights from memory to the core and back and exchange them they stay stationary there and that for example has a huge influence on on energy consumption and also throughput because uh the reloading of this weight is an energy energy
Consuming task so if they are stationary we have less data movement and therefore we are more energy efficient and the second thing is um if we uh reload those weights because we have to exchange them um this also takes time so that reduces our through but the advantage of that
Would be that we might use two cores less so we could build the smaller accelerator and therefore save some silicon area so there you can see um where the impact of mapping actually lies in and so to achieve this different mappings we have to implement different mapping strategies one for keeping
Weight stationary one to optimizing throughput and one also to optimize the um uh the resource us usage or just by to use actually less resources to uh reduce the Silicon area and in this case EX for example the goal was reducing data movement and in parallel increasing the usage of our hardare
Reses so that’s what the what the mppa does and that where we see that it has a huge influence um the next step is then architecture exploration and simulation and that’s even closer to the hardware because here we want to Define how our system should actually look like so what
We have as an input this a system architecture there might be mul variant of this um and we have our compiled uh NE Network and this goes then into architecture exploration and simulation and what are the goals of this tools first of all what we want to do is
Before we actually Implement uh the the circuits on Silicon we want to see what are the best options um what should be actually implemented in that we need a high level architectural simulation where we can see okay what influence uh do the architectural decisions have on the performance of our
Future ACD and that’s divided into two parts the first thing is architecture structure exploration so that’s where we actually check what different combinations of how we can Implement these circuits will influence the behavior and this might include for example um how many cores we want to
Have um should there be Global or local memories do we need multiple hierarchies of cach memories or what kind of onchip communication we want to use so those are the decisions we want to take in uh structural architecture exploration and how we do that and that’s all down to making it faster than
Actually implementing it in in Hardware or in simulating it we keep the functionality where it’s allowed on a very abstract level so on some blocks of our of our circuits um we don’t really need the exact modeling of them we just need a behavioral description and that leads to
A low implementation effort and therefore uh we can do this architecture uh exploration in quite fastly um but still because we also want to get some kpis from the whole tool is uh that we have to where it’s critical we have to uh simulate the timing accurate so some of the like on
Communications are actually modeled uh uh with the accurate timing but not all of them and that also improves the simulation speed and the final point there is we use abstract interfaces so we don’t really Implement implement the interfaces like they would be on Hardware again uh we have we modeled
Them on an abstract level and that enables us to exchange some models and actually uh plug them together and connect them how we want them in the simulation um the other part is then architecture parameter exploration and there we we try to find out what kind of
Dimensions we need how big our cores have to be what should be the memory sizes or what Buss we want to use for the communications and from all of that we again get uh kpis to see actually how the decisions influence the performance of our circuits so our solution then contains um
Hetrogeneous system C model so um as I said some some components are just Loosely timed or where it’s not timing critically others like the interfaces not iny cyly accurate we calculate uh the kpis in the simulations to see what the effect and latency power uh we can uh estimate from the architecture
Decision and everything is driven by setup files like architecture configuration uh instructions input data and Hardware metods to make this soil flexible and all in all we achieved the high simulation speed um by doing that uh behavioral implementation and that enables us uh to do model module level verification
System level verification and we also can verify our map compiler before having the implemented and so we can optimize kpis already on architecture level what are the outcomes uh as I said uh metrics and kpis um we get the performance of the neuron networks we get some
Waveforms uh where the timing is uh uh accurate actually and we get the visualizations on animation of how the whole chip behaves by which helps us to better understand how for example a mapping works then the next step is uh another step closer to the hardware and that’s actually neural network Hardware generator
Tool and there’s the first question why we actually need this why do we need to automate so we want to automate the design of our NE Network accelerators and that’s down to the fact that we build analog mixed signal AI accelerators and normally how these circuits are implemented is
Um by doing a manual layout and schematic design so we have a layout for example here of one of our accelerator cores and normally there’s a layout engineer and schematic engineer and they create the circuits manually but in the case of neuron networks and and accelerators we have more than 10,000
Circuit instances so in this example 16,000 circuit instances so it takes a very long time to do all of this manually and therefore we have a long time to Market so we want to reduce that and therefore we need automation the next thing is um we want
To make our design process a bit more robust um and that’s also down to the size and the complexity of the circuits there are multiple millions of circuit nodes thousands of pins and that leads to the situation that there might be errors already in the schematic design
And there was in the layout design and uh we can’t catch them in Sim because simulating these circuits actually in use case data we estimated uh like 100 years or even more so 100 years simulation time so we can’t catch the errors in simulation on the complete
Chip level so what we do uh we use the automation we create small examples which contain the same structure but just smaller um verify them those as they are smaller we can also simulate and then we can assume if we scale that up it’s still going to be the same circuit
Structure and therefore it can be correct so we reduce the possibility of Errors the final point is uh the design cycle so we develop the neurometric algorithms and the hardware in parallel which means we have to react fast to changes on both sides and therefore we also need the uh the
Automation um how it works is actually we have a set of standard cells this is signups and neuron cells for example we feed them uh into our um synthesis flow where we first of all create multiple configurations um how we could do the schematic end layout they go into a
Performance estimation this is also back to the K estimation but more on an analog circuit detailed level we select the best shoting block for that design the best set suting standard s and that goes then into synthesis and finally we try to at least do some simulations uh
To see if we match uh the requirements and with that what we achieved is that from uh having Aur Network and architecture in configuration in less than half an hour we can actually get the complete circuits and if you compare that to how long it would have taken us to do the
Circuits manually so we estimated it here for three different designs um where we had a run time for The Tool of 30 minutes in the first one and 5 minutes in the other tools and we estimated uh 1 to 3 months of manual design time so we uh greatly reduced the
Time we need uh to get our seits um designed and that helps us in reducing the time to Market final step then um is the runtime AP API um for AC characterization and also demonstrating uh the the Circ the AC our circuits in in a real world
Environment and in this case what this does is actually executing the neuron Network on the chip T so it takes the compiled program um and uh a label data set and that’s then fed into the hardware and we run the new network and we have two different setup for
This first one is for uh characterizing the as6 and that’s uh in the lab for measurement um and here we have a fully automated test setup which first of all configures the AC on Startup and then run some test patterns or the new network and while running the neural network we
Measure the power consumption we lock the input data the output data and also some anlog values and with that we can verify that the complete chip works and also the new network works and why does this have to be automated um to check all the the uh
Features on the chip and all the weights for example there are very it’s a very high number of test patterns so doing that manually wouldn’t be possible that’s why we have to automate that uh to be able to verify complete e and the final stage then is we got everything
Together we get a neur network which is qu and it’s robust uh in regards to the variation we have on on our chips um we got that mapped uh to the to the A6 so we can actually execute it and it performs with good kpis we have the hardware there and then
We can put it together into a demonstrator um that’s shown here on the right side so in this case uh we got our as and we got the rest per p in the display and on that we can run different t we can use live voice so this was
Voice activity detection the use case so in this case we can run live voice activity detection with a microphone input um we can measure the power consumption or we can run our data set um to see if the accuracy on that platform actually match matches with
What we uh saw in measurement and also in simulation and that then finally shows okay with our chips and the complete tool chain we can actually um execute neural networks in a real world environment in a robust way and that’s about it about our tool chain um I got
The acknowledgement here for the project where this work was mainly founded in and then our contact information so that’s my contact information and Mine mine’s contact information we also in included our uh head of integrated ciruits Department l m she’s the main contact person for all the new hardware AC
And with that it’s time for another round of questions uh thanks Roland for your great presentation and great talk as well as the one from M thanks again quick question for you Rolanda have you explored an architecture that is generic to a lot different solution with with a single chip come
Again uh have you explored an architecture that is generic to allow different solutions with a single chip different solutions in in terms of neur networks of or chip implementation I it’s my interpretation I think uh the question is about uh uh how flexible is your solution towards
Maybe yet the different NE networks yeah so actually everything is configurable so we can execute different neural networks with different number of layers also different quantisation values or conation levels so everything is actually programmable uh the only limitation is the oned resources so there is a limitation in how much memory
We have um and how fast we can process but otherwise from that um we can execute quite a few different neuron networks we are a bit Limited at the moment in terms of what layers we can do uh they are convolutional I think dilated convolutions and fully connected
But we are extending that feature set with every generation of that uh thanks in in there is another question which is related to the previous one if does your solution support Reservoir Computing and or spiking networks even as plans um spiking networks um both know at the moment we are still
We have the first chips now now uh spiking actually um with the architecture we shown and the chips we shown uh that’s would be challenging because of the size we decided for the circuits um and that’s needed for DNN but not really for spiking networks but we have another track where we actually
Developed spiking neur Network accelerators um as well but they look quite a bit different and they also have their own tool chain and Reservoir Computing also not at the moment so m uh CNN we can execute um or all kind of nor units okay thank you Um there is a quite long question let me try to understand the core of the question do you think that maybe if you come up with a framework Tyler to your chip might significantly reduce if not avoid some of the challenges you currently face since you no longer have
To tweak torch to work on your hardware and also take into account the torch behavor as you want it to I believe with a framework tailored for your chip all you have to do is to add feature as you go significantly shortening your developing time please enlight me on
This matter or just a thought from my experience in trying to force tens of FL light to work on some some BS I don’t know if it was clear the was quite long if I understand correctly I think the question is asking if we do our own framework from scratch it would
Be it would solve a lot of the problems we are facing I think in essence I would say probably but the effort to come up with a framework from scratch is way more significant than solving the challenges that we mentioned I think having torch as a backbone reduces a lot
Of the effort of reimplementing everything that we need there and we just add the solutions we have on top so I wouldn’t think building a framework from scratch is the solution here yes I agree and on some points uh some some of of the tools you might be which are very
Tailored to the hardware like our Hardware gener Network generator tool that’s completely from scratch um but also using some existing design software on new networks I fully agree we shouldn’t reinvent the wheel here there are very good tools out there um that can be used and that should also be used
Um simple simply also for the reason that the industry is already used to using such tools exactly and so we also have to support them and support the designers out there bringing their solutions to our Hardware okay um another quick question is what is the SES of each
Core um me site I’m not sure what does it mean in Silicon or the internal composition how many multiply accumulate operations are possible for each core I guess so each core at the moment has uh roughly 16,000 multiply and accumulate uh unit so 16,000 weights um but that’s
Also because we have the tools to scale that to create that automatically we can scale it up and down uh we can adapt the number of course um so we are quite free in the one we showed at 16,000 per um per core 16,000 2bit operations any figure in term of real utilizations
Of these ma in some use cases or studies or Benchmark of your preferences um I’m asking how far are you in in some benchmarks in some case study that you tested how much are you far from the peak utilization in in real use cases Um yeah this is this is dependent on how how we do the the mapping um utilization on the hardware um we’re not that great but we also not that bad so we are somewhere in the middle um but the main goal for what we did turn now is
Lowering the energy consumption and that comes along with lowering keeping the weight stationary for example and that uh decreases the utility ation a bit but with some changes um we should get up to like 70% utilization or some somewhere in these numbers okay uh last but not least
Question uh I think is implemented dependent if the intermediate outputs in the circuits in the different stage of the circuits are fible um for testing purposes yes uh they are uh not for the application okay uh very good we are uh eaing toward the end of the of your
Great speech so there are still outstanding question in the chat uh kindly reminding you to take some time to answer to each of them so that uh all will be addressed and um thanks again again for your speech I’m going to request the control in order to conclude uh uh the
Presentation with some general information thanks thanks to you too for organizing and coordinating you should have control now thank you so uh thanks again to our speakers uh as many others event also Emil talks are possible thanks to the great support of the Strategic partners and in particular let’s go
Through all of them executive the executive strategic Partners qualcom AI research syan then the Platinum strategic Partners embedded you are Sony the goal strategic Partners arm Ed impulse infinion Ina renzas s microelectronics synaptics and the silver strategic Partners quickly reminding a Arduino brain chap efficient green waves
Technologies grity IM image m not AI nxp PNG P Pol technology Snider Electric andil silicon labs and so thanks a lot to everybody especially the speakers and all the attendees thanks Olga and uh let’s meet again to the next at Great bran Els bye