DataFramed
DataFramed

Episode 101 · 5 months ago

#100 Embedded Machine Learning on Edge Devices

ABOUT THIS EPISODE

Machine learning models are often thought to be mainly utilized by large tech companies that run large and powerful models to accomplish a wide array of tasks. However, machine learning models are finding an increasing presence in edge devices such as smart watches.

ML engineers are learning how to compress models and fit them into smaller and smaller devices while retaining accuracy, effectiveness, and efficiency. The goal is to empower domain experts in any industry around the world to effectively use machine learning models without having to become experts in the field themselves.

Daniel Situnayake is the Founding TinyML Engineer and Head of Machine Learning at Edge Impulse, a leading development platform for embedded machine learning used by over 3,000 enterprises across more than 85,000 ML projects globally. Dan has over 10 years of experience as a software engineer, which includes companies like Google (where he worked on TensorFlow Lite) and Loopt, and co-founded Tiny Farms America’s first insect farming technology company. He wrote the book, "TinyML," and the forthcoming "AI at the Edge".

Daniel joins the show to talk about his work with EdgeML, the biggest challenges facing the field of embedded machine learning, the potential use cases of machine learning models in edge devices, and the best tips for aspiring machine learning engineers and data science practitioners to get started with embedded machine learning.

You're listening to data framed, a podcast by data camp. In this show you'll hear all the latest trends and insights in data science. Whether you're just getting started in your data career or you're a data leader looking to scale data driven decisions in your organization, join us for in depth discussions with data and analytics leaders at the forefront of the data revolution. Let's dive right in. Hi Everyone. As you may be able to tell from the accent, I'm not Adele, I am Richie, a data evangelist at data camp. This week I'm filling in for a Dell and I'll shortly begin co hosting the data framed podcast. I've been working in data science and data scientif education for nearly two decades now and I've watched the field go from being a niche nerdy thing to well mainstream nerdy thing, and I spent a lot of my career providing data science support biologists and chemists and business people, and my experiences that even incredibly smart people sometimes get terrified by having to analyze data. So I really care about making data feel a little less scary to work with in the coming episodes, as well as revisiting the traditional data frame theme of Helping Organization has become data driven. I'd also like to provide some inspiration to you all about the joys and the possibilities of working with data. Once you get over the fear, there really is a lot of fun to be had. Today we'll be talking about machine learning on low powered devices like bones and internet of things sensors. So there's a really exciting emerging field and I'm very much looking forward to finding out about it from Dan Situny Yuki. Danny's head of machine at edge impulse and he's the author of the Book Tangy M L and the forthcoming ai at the edge. He previously worked at Google on tensorflow light. So all this is to say he's a real expert in the field. Hi Dan, thanks for joining us today. We're gonna be talking about machine learning on edge devices and that terms actually barely unfamiliar to me. So perhaps for a bit of context, you could just start talking to us about well, what actually is an edge device? Yeah, great question, Richie, and I'm very excited to be here today. So an edge device refers to a computer that's at the edge of the network, and so that sort of makes sense if you think of the Internet as this big, sprawling set of interconnected devices, but it hasn't an edge where there are devices that are sort of connected to the grace and network on one side and then on the other side they're connected to the real world, basically through sensors and other sorts of hardware. So a typical edge device might be, for example, an Internet connected camera, or it could be a personal fitness wearable that you wear around your wrist that logs when you're doing exercise and uploads data about it to the cloud. And there are tons of different applications for edge devices from everywhere, through consumer electronics through to industrial systems, the kinds of devices that are monitoring industrial processing factories, and even the ultimate sort of edge device is a spacecraft, the Mars Rovers that are up there driving around on Mars and they're collecting data and uploading it and sending it back to Earth. Those are our furthest edge devices. Yet that's pretty impressive. There's such a broad range of different applications. But can you tell me why do you have to do machine learning on an edge device? So I think the standard thing, if I'm doing machine learning, is that I'm going to be doing something on my laptop, so that's going to be processing the model, or maybe I'm working in the cloud, so it's going to be some server somewhere processing things. Why would you want to do machine learning on a device which hasn't got very much power? If you think about sensor data and data about the real world, we sort of I have this idea that we when we deal with data's data scientists, we're dealing with data sets that have been collected and a lot of the time we're dealing with data sets that are accrued by our big back end systems and and the systems that are generating the data already within our infrastructure, basically. But if we're collecting sensor data, so we're collecting data about the real world, there's essentially like a bottleneck where the real world has an almost infinite amount of...

...data. It's just this endless source of new information. Imagine you have a device out there that's collecting temperature and pressure and all sorts of other stuff that's related to climates or the weather. There's basically like an infinite stream of data out there for this thing to collect, but it might not have a very good connection with which to send that data to another system that can do analysis of it. So typically edge devices, because they're on the edge of the network. They might suffer from things like unreliable Internet connections or low bandwidth, or maybe they need to operate under constrained sort of energy regime, so they're running off a battery or something like that, or solar power, and they can't afford to be burning loads of energy with network communication all the time. If you think about like your mobile phone, the thing that uses the most battery, apart from the screen of the phone, is using the radio to broadcast information. So if you're in a situation where your device is constrained in some way and is unable to send all of that data back at the rate that you're receiving it, then it makes sense to put some of this intelligence on the device rather than depending on having intelligence running on a cloud service somewhere. Yeah, I can certainly see how with a Mars rover, you might not want to be sending all your data across space from Mars to somewhere on earth and then trying to have all the results and send them back. So it feels like a slightly novel field and I'm curious as to how you got into thinking about edge m l. yes, so edge ML and sort of edge ai more broadly, which is the the idea of doing all sorts of artificial intelligence on edge devices, has only existed for a relatively short time and essentially the thing that's propagated it and allowed it to develop is the fact that these edge devices have gotten powerful enough that they're able to run some of these heavy lifting algorithms, like deep learning models on device. But the way that I got involved with this was I was working at Google on the tensor flow team, and tensor flow, as you're you're probably aware, is Google's big deep learning framework and there's a part of tensor flow called tensorflow light, which is basically designed around the idea of deploying models down to mobile phones, because there are like mobile phones are edge devices right there, these little portable computers on the edge of the network with lots of sensors on, and some folks at Google realized that would be really cool to be able to run models on phones to do do all sorts of cool stuff, and then the sort of next logical step from that was to think about like, okay, what kind of devices are even smaller than mobile owns and more more edgy and more possible and lower power than is it possible to run deep learning models on those. And I was lucky enough to be working at Google at the time, where a new team started by a guy called Pete warden was starting to build and investigate this area, and I basically lucked into being around during this awesome new technology being launched and I realized it was the most exciting thing I'd come across in the tech industry so far. Combined my fascination around embedded software engineering and edge devices with my fascination around data science and machine learning. That's exciting. And so with the work you've been doing, like with tensorflow and things like that, how is it different to just doing regular ml? How is it constrained or what makes it hard? It's really interesting actually, because when you're dealing with this type of real time sense today, to that by itself is a very different paradigm to the sort of work that you do on data sets which are more tabular. So, for example, like we're all very familiar with this idea of feature engineering for tabular data. So maybe we're looking at analyzing the business performance of something or analyzing some government data about some kind of phenomena that are going on in our world. But our typical workflow for identifying good features to use from that data uses a set of tools which occupy one paradigm. But when you think about real time, live sensor data, so this is basically like little samples of information that are coming in at very high frequency and essentially are quite challenging to feed raw into a machine learning model, there's this whole extra field that has existed for a long time called digital signals process, a thing which is based around the idea that there are algorithms that...

...can make sense of these kinds of real time streams of sensor data and distill them down and get the most interesting components most visible, so that whatever you're trying to do downstream isn't swamped with all of this raw data. And these algorithms look very different from anything that you do with tabular data, for example with audio data that you basically, at a certain frequency, get along stream of numbers at a very high frequency which represents the amount of sound pressure. Almost that's happening, the amount that your microphones diaphragm is flexing at any given point and to turn that into something that a deep learning model can understand. One of the best ways of doing it is through building something called a Spectrogram, which is basically a representation of that series of signals or series of values in the frequency domain. So instead of having a single column with these amplitude values, you end up with a two D Matrix which is a representation of time on the x axis and frequency on the Y axis. That's bucketed into into the individual buckets and that is essentially like an image and that's what you feed into your deep learning models so you can use a convolutional neural network to understand it. So it's all about transforming signals from their raw form, which often isn't very convenient to work with, into a maybe higher dimensionality but lower frequency form which is more convenient to use these machine learning tools. That's fascinating. So very sort of distant memories of learning about fourier transforms and wavelets and things like that been signal processing from university, so it's not fresh in my mind. I think it's something I've not used in a data science or machine learning context before. So that sort of engineering type stuff is always very separate, and so it's interesting to hear about how that sort of audio engineering and machine learning, the two of these have come together. The thing I love about it is that there is this huge field of embedded engineering and signal processing that's existed for a fairly long time at this point, and people who are absolutely incredible experts in this type of stuff, and what we're finding is that there's this amazing fusion between the things that are coming from the machine learning and deep learning and data science side with this signal processing expertise, and when you get people from both sides together, you can figure out really amazing ways of interpreting signals and training models. It's really exciting. And when you've got a very low powered device and maybe you haven't got a great into net connection and maybe you're limited by battery life or things like that, how does that affect the machine learning side of things? So do you have to choose different models, or do you have to train things differently, or how does it affect the actual models themselves? So one thing that I'd first of all point out is that when we're talking about machine learning on embedded devices, the vast majority of the time we're talking about inference, we're not necessarily talking about training. So there's nothing stopping you from training models on these edge devices, and in some cases this does happen. But one of the biggest constraints of edge devices is that you've got limited memory to work with, and so a lot of the time when we're talking about training models in a supervised approach, that just isn't feasible if you don't have enough space to store data. Maybe you've only got tens of killer bytes, which obviously isn't enough to store much of a data set. So if you're doing some kind of unsupervised approach or you've got some kind of interesting approach to how you find tuning a model on device, that can work sometimes, but a lot of the time we're just talking about running inference. And then when we're talking about running inference, there are really three main constraints. The first constraint is energy and connected with that is performance, like the performance of the device, because if you're trying to get a device to run on a battery, for example, you're not going to have this like two Giga hurts processor, multi core CPU or like a GPU running in there. There are edge devices that are very powerful, but typically you're going to see more constrained clock speeds than you will in server side infrastructure for running machine learning models. So the issue there is it's quite slow to run models, and so that means that you are going to be biased towards training smaller models, like a smaller model is going to be able to run more quickly on advice, use less energy and fits this set of constraints on the...

...edge. And then the other two things are read only memory and Ram. So the distinction there is that read only memory is where you're going to be storing the weights of your model plus the program that runs on your edge device. So obviously there's a range, always in computing hardware, but at the smallest end you might only have tens or hundreds of killer bytes to store your models weights. So if we're thinking about typical deep learning models for vision, even something like a mobile net model that's designed for running on a mobile phone is going to be too big to run on some of these smallest edge devices. So you need to think about how do we make these models smaller? And then the ram parts, the working memory, is just as important because you need someone to store your signal when it comes in and you need to store the forep learning, for example, the intermediate activations of your models layers while you're working, and if you don't have enough ram you can't do that. So these are the three most important constraints and it typically forces you to think about smaller models and models which are designed in a way to be efficient in terms of ram usage. And it also encourages you to do as much signal processing as you can upfront, because there's often this trade off between you can basically extract features through signal processing and use a certain amount of compute and you can throw things at your deep learning model, for example, and make it figure out the feature extraction part two. And for any given device there's going to be like an ideal set point for a particular data set where there's a perfect trade off between those two sections where it's the most efficient. But that's something you have to find on a per device and per data set basis. So there's this whole extra contrastraint that comes in that you don't have to really think about when you're thinking about data science on big, powerful developer workstations or on servers in the back end. And for me. That's what makes it really exciting. It's always nice to have a constraint because it forces you to be creative and think of new ways of doing stuff. That's incredible. And so I get the sense that there's some sort of trade off where you sacrifice maybe a little bit of predictive power because you're using a simpler model, but that's necessary in order to be able to run it at all because you don't have the computing power or the RAM available. Is that about right? Yeah, I mean that can be true. So if we're if we're thinking about larger, more highly dimensional data, so images for example, typically if you're compressing a model to get it to run on an embedded device, that might involve, first of all, just training a smaller model, like coming up with an architecture that has less weights and training that. And so that's one way and and obviously there's a trade off there with the fewer weights, the less representative power the model has, and so potentially you're going to get performance this isn't as good as you would on a larger model. The second thing there that we have at our disposal is this idea of model compression, where you can take a model that was trained in a typical manner as if it was going to run on on any device, and then you can do certain things to it that reduce its precision, reduce its accuracy potentially, but make it smaller. So one of those things is called quantization. So quantization, which you might have come across in the back end world as well. It's a way of getting models to be smaller by reducing the precision of their weights. In a typical deep learning model, for example, the weights are usually trained in float thirty two format. So we at these thirty two bit floating point numbers. They're very expressive, they can represent a very large range of values. But to get them to run on an embedded device, if you reduce the number of bits available to store each weight. So you reduce them that down to eight bits. So you're now storing the numbers that make up your model as eight bit numbers. That's four times smaller model. So if you can do that and get away with it without screwing up the models accuracy too much, that can be great. And they're actually really smart ways of doing that where you can really lose barely any accuracy but reduce the model by four times or eight times, or even the smallest precision models are actually binarized neural networks, B and NS, which have one bit weights. So literally the weights of the model are binary and there's some special regimes you need in order to train those, but they can work really well. So all...

...of that comes with trade offs and it's about managing the trade offs and keeping the amount of performance that you need for your application. I think a lot of people listening will want to know how do I get involved in this? It sounds exciting, so how do you get started with this? There's lots of different angles to come at this topic. Some people come at it from an embedded engineering angle, where they've worked in embedded engineering before, working with micro controllers or embedded Linux blores like raspberry pie, for example, and they've hit a wall where they want to make sense of the world and the sense of data that they're pulling in on their device, but there's only so much you can do with hand coded Algorithms and basic signal processing, so they want to embed a bit more intelligence in devices. And then there are people coming from the other end where maybe they have a background in data science or machine learning and they'd like to see, like how could I take some of this intelligence that we're able to encapsulate in models and push it down onto these tiny devices so that it's available in places it wasn't available before. So your journey might be a little bit different depending on which side you're coming from, because if you're a data scientist, you're going to be a lot more familiar with these data science tools and less familiar with the embedded side, and if you're an embedded engineer, then it will be vice versa. Let's assume that you're a data scientist and you're excited about getting started with this kind of thing. I think a good place to begin you don't necessarily have to have embedded devices to start thinking about how do you train smaller models. It can be interesting to just start building an awareness of the relationship between the size of models and the amount of resources they consume and just start trying to get into that mindset and tinker around with things and start learning like, Oh, if you change this, what's the impact on performance? And you can start to learn about things like model compression and quanteization and begin your journey that way. And the easiest place to start is probably with the larger edge devices, things like a raspberry pie, because you can just run python on there, so you can train a tensorflow model and then run it directly on the device and there's not that much extra work involved. If you're going down to micro controllers, so these are things like the tiny little processes that live inside of household gadgets and really are super pervasive throughout our worlds and are built environments. The workflow for programming with those is a lot more complex, so they generally require you to write code in C or C plus plus and then use a device specific set of compilation tools that allow you to compile your code to run on that particular device. So are very easy to use. Set of devices in that area made by a company called are Doino and they make micro controllers that are designed to be easy to program and easy to use and it can be really good starting point for embedded machine learning development. But it can be easy to get lost because there's so much going on in this field. So what I would recommend? I'm a bit biased because I work for a company called edge impulse and we have this amazing end to end suite of tools which allows you to integrate with devices, collect data sets and experiment with combinations of signal processing and machine learning models and then deploy to devices. And I would recommend personally heading over to our documentation. Just search for EDGE IMPULSE OR GO TO EDGE IMPULSE DOT COM and dig in and we've got loads of tutorials which are designed to make it easy to jump in and start learning about how all of these different moving parts relate to each other and how you can train a model and deploy it. Is there a sort of standard, easy first project, like what's the hello world equivalent for edge M L? So I wrote a book along with Pete Warden from the tensorflow like micro team a few years back, and the Hello World Project that we came up for tiny ml was basically looking at can you train a tiny model just to do some very arbitrary things? So for us it was to predict the y value given the x value for a sine wave function. So trained the simplest possible model and then just go through the process of converting it into a form that will run on an embedded device and then getting it deployed on there and having all of the parts of this process are quite difficult. Training an effective model is difficult, collecting a good data set is difficult, signal processing is difficult, deploying the device is difficult. So it's good to just take something very easy and deploy it to an embedded device and get familiar with the entire flow there. But that's sort of...

...if you're approaching things from scratch and doing everything yourself. But what I'd recommend it really is going to a platform like edge impulse and following the steps to do something that trains a model that runs on a simple time series. So, for example, a lot of micro controller development boards will have a built in accelerometer, and a really good thing to do is to train a model that recognizes different types of activity based on data from an accelerometer. So accelerometer basically returns a time series of three values. The three values are the amount of acceleration on the x axis, the Y axis and the Z axis. So you get this idea of motion in three D space and you can train a model to recognize different types of activity, so like jogging versus doing push ups or something like that. You collect a little data set with data for each of these motions and then train use some signal processing to extract the person and information, train a little model and then deploy it to the device, and then you can like wear this thing on your arm and it will classify your activities, and that's really cool because that's how a smart watch that people wear for fitness works. that can understand when you're running versus swimming or something like that, and you can start to see how this all fits together end to end. The tricky part is if you don't have any embedded experience, it becomes really difficult even to grab that initial data from a device. So that's why these sort of end to end frameworks, end to end platforms, are so powerful, because they come with all the software that you need in order to grab the data from the device. Making your own fitness watch does sound like a pretty cool project, but I can see how there are a lot of different steps where you need to know a bit about a lot of things in order to make that. So maybe you talk about when people are trying to do this, what's a good idea and what's a bad idea. So perhaps can you tell me a bit about what are common mistakes that people make when they're trying to do this sort of thing? One of the things that I often come across is there's a lot bigger of difference between test data and training data and real world data if we're using sensor data versus if we're training something based on stuff out of a database that came out of a piece of cloud software. So the thing with tabular data that's based on, like, I don't know, the behavior of users in a social network or something like that, is that the data gets measured in the same way and that gets collected in the same way during data collection and during production. It's just that we sort of took a snapshot at a certain moment in time and we've got now a bunch of training and testing data and after that time all the real world data is general rated in exactly the same way. But with embedded machine learning you have a bit more of a challenge around when you collect data. Imagine we're going to design a new product. We're designing a new kind of smart watch that's supposed to help, I don't know, climbers and and give them some feedback on their climbing. So this device doesn't actually exist yet and before we get that device built, we need to design and train this machine learning system that can give the climbers their feedback. But because we don't have any devices built yet, we've got to somehow collect a data set without having our real world actual device available. And so to begin with we'll have to come up with some prototype that just collects data from some sensors. Maybe it's sort of roughly the same size and shape that the real thing will be, and have some people wear that device and do some climbing and label the data in some way. Maybe it's like, oh, that was a difficult climb versus an easy climb or something like at and collect this whole data set and once we've done that, we can train a model and we can test it and we can see, okay, this model can classify accurately whether this was an easy climb or a difficult climb. But the problem is when we then get further through our product development process and we've got our finished hardware design, maybe it's a slightly different shape and size of watch, maybe it has a slightly different sensor on it. That's like it's still an accelerometer, but it's a different model of accelerometer because that one was cheaper and we needed to go with a cheaper one to hit our target for how much the device costs. So when it gets deployed into the real world, the device is slightly different and that means the data coming off it is going to be slightly different and that means our model might not necessarily work very well. And the real challenge is that, because all the data is trapped at the edge of the network and the Divi ice is the thing that's interpreting it, we don't necessarily...

...have a way to check whether it was working well or not. Right we can't just compare the real world performance versus the predicted because we don't have that data and we don't have a way to label it. So there's this big risk that you can end up training a model that looks really good before it hits production and then when it hits production, when it gets deployed to a real device, that doesn't work very well. And that's obviously a risk in all of data science, but it's especially amplified here because we've got less feedback loops between development and production. So that's probably the biggest scariest thing that we we had to work around as a field. Oh Wow, yeah, I'm sure that could be a problem, and so how do companies deal with this sort of thing? Yeah, I mean it's it's tricky and it's it's an ongoing thing that we're evolving systems and workflows to deal with. But the most obvious thing is just keeping things as similar as possible between production and development and, wherever possible, making sure you use the same sensors, making sure that you're collecting data in the same way. But obviously that's not always possible. So then you've got to think about how do you set up some ongoing processes to continue testing and making sure that the system still works as the real world changes, because there's also obviously this concept of drift where the real world is always changing and your model represents one frozen moment of that. And that's even more amplified in this field because we're dealing directly with real world data that's captured from sensors. So one of the things that you can do is periodically send people out to capture more data and test your model against that. So that can be a helpful signal. So it's not as good as directly looking at what's happening in the real world, but it's going to give you a periodic snapshot of what's happening and you can see, okay, with this new data I've collected, the models still working. Another thing you can do is look at the output of your model. So presumably these devices are smart watches that people are wearing for climbing. They would be reporting data back into some system. So if they're determined that the climb was challenging, then that gets logged in the users accounts somewhere so they can see it later. So one thing you can do is look at the distributions of the outputs of the model over time, because even if you can't afford to save all of the raw senser data, you can potentially afford to save the output of the model. And if the distributions of the output are changing a lot over time or if they're very different to what they were on your test data that you originally captured, for example, maybe that's a sign that some drift has occurred or that the real world is occupying a different distribution than the initial data set. So that could be a signal that you need to collect some more data and do more evaluation. But just going back to the climbing watch example, that sounds like it might just be a case of you get your climbers back in and get them to climb the same routs before using the production version of the watch rather than the prototype exactly. And maybe you made the mistake of initially collecting data from professional climbers and then that you found out that the people who are buying your watch in the public are just amateurs, and so maybe you need to get some amateurs to come in and collect data from them and that gives you a better representation of what's out there in the real world. So all of this stuff is things you've got to be really careful about in your sort of experiment design. Yeah, the amateurs fall off the rock face. So, just flipping this around, are there any particular best practices for working with edge? It's a big, big field, so there's a lot of different individual places, but I think that the most important thing for me is to try and figure out, because this is such a big area, there are so so many steps in the workflow of going from a data set to a model deployed on a device. Coming up with a good understanding of all of the different moving parts at the very beginning is very important. So understanding what is the hardware we're going to be deploying this too. So much of these things are based around like figuring out a good compromise based on the hardware that you'll be deploying to. And so is that known ahead of time? Do we get to choose the hardware based on what kind of model we end up training, or is it more the other way around, that we've got a device that already exists and we need to fit the model in its spare memory? So getting that pinned down at the beginning is really really important and that will inform the entire rest of the project. The second thing that's really important, I think, is just having an understanding of what are we trying to achieve...

...here. What's the minimum thing we can do that will result in a useful product, because sometimes it's good to aim for that basic improvement rather than saying like we want to do all these different things and make this sort of all singing, all dancing, magical ai product and then trying and failing. It's better to choose one simple, straightforward thing which is going to be helpful to your customers and then deliver on that and then work up from there and gradually add more features and improve things, but try and bite off something simple to begin with, because this can be a really challenging field. I'd like to talk a little bit more about some of the use cases. So you mentioned that really high tech thing with the Mars rover, and then there's a sort of consumer product idea of this climbing watch, and then he talked a bit about centers and things. So can you just give me some more examples of where this might get used? Edge impulse we have customers across pretty much any space you can imagine. It covers everything from consumer products that use a little bit of intelligence to do something smarter in the home, all the way through to industrial systems that are understanding what's going on in a factory, maybe analyzing products for quality after they've been manufactured so you can understand whether there are flaws in your manufacturing process. Healthcare is an interesting area, interpreting bio signals using machine learning to understand how someone's body is functioning or the progression of a disease. Agricultures are really interesting one, understanding how plants are growing or how animal health is or understanding climate. So anywhere where there's rex connection to the real world and tangible things that are going on and we need to know more about it. Any field you can think of there are applications for embedded machine learning. Yeah, it's amazing machine learnings are sort of invading every part of our lives. So, yeah, it's interesting that. That's something I've really find exciting about this is that until now machine learning has been something that's existed in a constrained set of places that it will fit right. We've got these big, huge models running on powerful computers, but now we're able to push machine learning down to smaller and smaller devices so it can fit more neatly into little corners of our lives. It democratizes things a little bit. Rather than needing these one size fits all giant models that are running in the cloud at some big tech companies data center, we can have little, tiny models that are trained by people who are working in little corners of the world old whether it's in some niche field or a place that has particular problems and challenges that we can address as machine learning. And it's a lot more feasible to get good performance out of these smaller models in little constrained contexts rather than trying to create these big giant models that do everything. So lots of tiny, tiny things on tiny machines. So that's cool. This is very much an emerging field and I get the feeling that there's lots of progress being made, but maybe there are some sorts of limitations at the moment. So perhaps you can talk a bit about where the sticking points are, particularly with tiny m l where I imagine you've got even stronger constraints. So one of the biggest things is around tooling and essentially, if you are working and embedded software or hardware design, chances are if you're building something in agriculture, for example, you're eam is going to have a lot of experience and a lot of expertise around agriculture and you're using that to design and build an effective product that works in that domain. So if you are working in that domain and you're you're already a deep expert in your domain and you're a deep expert in embedded engineering, you probably don't have the bandwidth to also become a deep expert in machine learning. If you want to use machine learning as part of a project to begin with, those this huge really difficult learning curve where you've got to become an expert in m l and then apply that knowledge to the problem that you're trying to solve. So the big problem and the big challenges how do we make it so that people who are domain experts in embedded engineering and in whatever field they're working in can use machine learning without having to become a machine learning expert, without having to take years to study field and learn the INS and outs of...

...data science? So that's where there's a huge amount of opportunity in this field is building tools that empower people who are not from machine learning and data science background and obviously to build those tools requires data science and machine learning expertise. So there's there's a huge opportunity and a huge challenge there in adjusting and adapting these tool chains so that people who are experts in one little subject area that's very important can make use of them. That agriculture example really resonates with me. So my father in law was a sheep farmer and he had decades and decades and decades of farming experience. I don't think he ever touched computer in his life and so getting him to think about machine learning would very much be a no go area. But trying to get all that domain knowledge you had into some kind of model or would be a very interesting challenge. So I can see how it would be a difficult thing. Yeah, absolutely, and that's even something we've directly seen. People using edge ai models to recognize when sheep are in different parts of a farm or walking across the road, or devices that are a warn around an animal's neck that keep track of their bio signals and their activity so that you can understand how healthy they are. So how are things progressing at the moment? What are the most important challenges that are being worked on? So right now I think some of the biggest, most important work is in this field of opening up access to people and allowing people to understand the machine learning workflow without being a machine learning expert, basically building tools to understand sense of data and be able to look at how good a data set is, of sense of data and understand quality and give people feedback around like, Oh, you should be collecting more examples of this type of thing. One of the biggest difficulties is that collecting labeled data can be very difficult. If you think about sensor data, essentially it's like big, massive time series of numbers and it can be quite inscrutable. You can't necessarily look at a table of all of these rows, like a thousand rows per second, of some arbitrary value that gets read from a sensor and make any sense of it as a person. So we need to come up with techniques that allow people to take a big data set of like hundreds of hours of unlabeled high frequency sensor data and label it and then be able to use it for training or perhaps be able to do some kind of training that doesn't depend on labels. So that's that's a really big thing. Another big thing is understanding the performance of models on this type of data and explaining how models are performing and where they're working well and where they aren't, across a big data set of this type of difficult to interpret data. One thing that's quite surprising in this field that we found is that the device constraints around memory and latency and so on aren't as big an obstacle as you would think, because it turns out if you use some intelligent signal processing and you train a small model, you can often do a lot with very minimal hardware. We're used to turning things up to the Max and going with the biggest model that you can get away with without overfitting and and all of that type of thing. But if you're thinking about fitting models onto these devices, there's a surprising amount you can do with what little space you have. There are a lot of people in the field looking into compression and how can you fit bigger and bigger models onto smaller and smaller devices, but it's quite rare for us that we have a customer who needs to do something that requires a model that's too big to fit on an embedded device. It's more often that they don't have a good enough data set to be able to train that model in the first place. That seems to be a common thing with machine learning, is that actually the hard part is getting good data in order to run the model, rather than the actual modeling part itself. Absolutely, and especially with sense of data, where, if you're looking at things like natural language or image data, there are huge data sets out there that have been scraped from the Internet that people used to be able to use for creating and benchmarking new types of models. But with sensor data there just isn't that much out there and people who have access to sensor data are quite wary about sharing it because it represents significant intellectual property on their part. If you have a company that man factures some kind...

...of widget and you've got all sorts of sensors on all of your equipment that monitor how this manufacturing is going, that represents valuable data that your company can use to potentially improve its processes and you don't necessarily want your competitors getting ahold of it. So that's a challenge as well. Is like the availability of data sets and the issues around privacy and sharing of data sets. Yeah, I can see how that would be a problem. Maybe if there are any listeners who have sensor data sets that they can publicly share, then I encourage you to do that if you can do it without causing any privacy problems or IP problems. So this whole thing seems like it's at a point where it might be about to explode in popularity. It feels like the toolings coming together and the techniques are coming together and some of the processes. So what do you think is going to drive adoption of Edge Mul over the next few years? Yeah, the way we think about this is sort of there's this massive untapped potential based on sensor data that we've got. This this huge, incredibly valuable pools of data basically, that exist in the real world that are completely discarded right now. Most data that's connected, collected by IOT devices, for example, isn't really used for anything. And we've got all this technology around sensors and embedded devices that can collect lots of data about real world situations, but we're basically not making any use of it yet because of these bandwidth and latency constraints, things like that. So what's going to drive this is that there's huge untapped value in this data throughout industry and consumer devices and everywhere, and suddenly we've got the tools that allow us to make use of it to build better products and do things more intelligently and be more efficient and create higher quality products on a manufacturing line, for example, or to monitor our natural world to understand how ecologies and the environment are doing and how how the climate is changing. So there's a huge need to interpret all of this data so that we can make use of it, and this is the technology that allows us to do that. Exciting Times. So people who want to give us a try that maybe don't necessarily want to go it alone. Are there any good communities for people who want to get started? There's a really awesome group called the tiny ml foundation, which has existed since quite early on in this field, who run a series of meetup groups across the world. There's dozens of them where you can go to local meetups with other people who are interested in this field. So if you go to tiny ml dot org you can find some links to those. And the tiny email foundation also run a conference or cut a few conferences every year in different parts of the world, and we do these weekly or bi weekly talks as well that you can watch. That a broadcast live and they're also on Youtube and so on. There's also a few forums around, so edge impulse forum is a really good place to go. If you go to the EDGE IMPULSE DOT COM website, there's a link there and there's a ton of people hanging out talking about embedded machine learning. Another good thing you can do is there there are a few courses available. So there's a coursera course which was put together by some folks at edge impulse, which basically gives you an introduction to how to start working with this technology. There's also a Harvard Ed x course which goes into the more sort of in depth theory behind things. So if you're excited about the behind the scene part, the more like Matthew stuff, the Harvard course is a really good place to go. If you're excited about the application stuff and how do I, just as quickly as possible, start building things with this technology, then the coursera course is really good. I should probably add that for anyone who's interested in analyzing censor data, we do have a data camp course as well on analyzing the Internet of things data in Python. That sounds like a really good place to start as well. Yeah, it's a very gentle introduction and lots of time series analysis and a bit of machine learning. So before we finish, do you have any final advice for people who want to try this themselves? Yeah, so don't be scared, like this is a big field. There's a lot of moving parts, a lot of complexity, but it's definitely possible to focus on the bit that you care about and sort of ignore the rest of the complexity if you use the right tools. So I wouldn't recommend most people asking from scratch and trying to learn all of these little...

...pieces and all the moving parts at the same time. If you're already interested in data science and machine learning, then find a platform that automates the embedded side of it for you, so it makes it easy to collect some data sets, makes it easy to deploy to advice and then you can focus on the machine learning part. Otherwise, if you're an embedded engineer and you're excited about doing that part and you don't necessarily want to spend years becoming a data science expert, the same sort of platforms will help you focus on the embedded part and building the products you want without having to go back to school. So definitely don't be afraid to use tools to help you and don't try and start everything from scratch, because it's very difficult. Okay, thank you very much for answering all the questions. It's been a real pleasure having you on the podcast. Thank you very much, Dan. Likewise, thank you, Richie. It's been awesome chatting with you today. Ye, you've been listening to data framed, a podcast by data camp. Keep connected with us by subscribing to the show in your favorite podcast player. Please give us a rating, leave a comment and share episodes. You love that helps us keep delivering insights into all things data. Thanks for listening. Until next time,.

In-Stream Audio Search

NEW

Search across all episodes within this podcast

Episodes (121)