Keynote at the Microsoft campus arguing that real-world ML must be understood as an end-to-end system — data ingestion, featurization, modeling, deployment, and feedback — not just algorithms.
“What are machine learning systems? In my view they're living systems, both influencing and reacting to their environment. At best they're valuable, resilient, functioning systems composed of many imperfect parts, with many weak contracts between them, built by fallible individuals with broken communication channels amongst them.” – Joshua Bloom
“If you can't define your loss function you can't optimize, and if you can't optimize then basically you're fishing.” – Joshua Bloom
“Real data is not platonic, it's plutonic… It's ugly, it's a dusty snowball with mountains and warts and geysers, and it's got NaNs and missing quotes and all that stuff. It is incredibly ugly, but at the same time it's incredibly rich.” – Joshua Bloom
“When it comes to ML systems, it takes a village. Or another way to say this is, data science as a team sport. You have to have interdisciplinary teams.” – Joshua Bloom
Hello everyone. So when I talk about machine learning systems, I'm thinking about those that are in production rather than those that are one-off data science projects. Not to say anything bad about those, but this is really the focus of this talk. So what are machine learning systems? In my view they're living systems, both influencing and reacting to their environment. At best they're valuable, resilient, functioning systems composed of many imperfect parts, with many weak contracts between them, built by fallible individuals with broken communication channels amongst them, living in a resource constrained world that's constantly changing, the results of which are consumed by capricious and exacting individuals. In short, this is very hard stuff.
I'm very thankful to be here today. It's a great honor to speak in front of you. I also had the honor of being one of the keynotes at the last PyData, so the organizers have presented a major challenge to me: to essentially give a completely different talk to the same audience in the space of one year. So I'll do my best. You've heard a bit about me already from Fernando. I thought I'd leave you, or at least start off, with a graphic that would maybe shock you out of your lunch stupor. Does anyone have a guess what this is? It's Pluto, yeah. One of the new Pluto images from the New Horizons project run through Google's deep dream. If you stare really carefully, you will freak out.
So the reason why I'm showing this is because in some sense this is the confluence of who I am. I've been doing machine learning in production in the context of academia for well over a decade, and as you heard from Fernando, started a company that's doing machine learning, not quite as a service, in a slightly different way. So this is the marriage of those two worlds visualized for you, again just to freak you out. Of course, for those that actually know me, they'll know that I'm not a solar system expert, nor am I a deep learner expert, but this is the closest I could do in some sense.
That work that I've done in the past, both in academia and in industry, informs some of the things I'll be talking to you today about. But the point of departure for this talk really is this paper that came out by folks at Google called machine learning, the high-interest credit card of technical debt. One of the things that I think is very shocking about that paper, which by the way is probably even more shocking than the image that I just showed you for those that are practitioners, and it's certainly worth a read, is this quote that you see up there: it may be surprising to the academic community to know that only a fraction of the code is actually doing machine learning. A mature system might end up being at most 5% machine learning code and at least 95% glue code. So when we think about machine learning and we think about the innards of machine learning and the algorithms and the theory behind it, when you start thinking about machine learning in practice in real living systems, it tends to be a very small fraction of that. By the way, that word glue code should be a bit of a trigger point for you, because that's where Python starts to come in. Python is a wonderful glue code to all the different projects and implementations at perhaps a lower level programming language, and allows us to expose that to a broader audience.
So I'm not going to go in and summarize and give a talk around that paper, but just some of the highlights for me are the following. The idea that there are complex models that effectively erode the abstraction boundaries between the different components of that model, and that's the data dependencies, which are more expensive than actually the code dependencies, which is somewhat unusual if not completely novel for software engineers. It's a system of spaghetti code that has to interconnect, and again it's interoperating and interacting with the external world. So again, this paper is well worth a read. It's good summer reading. It's going to be on the test, so please do read it.
All right, so what's the agenda for today's talk? It's to discuss in some sense from the inside out, from the algorithms, the software, to the hardware that this software has to live inside of. And then of course, as we start thinking about hardware in the modern era, modern hardware is basically composable and programmable with software, so we've got another layer around that. And then we've got the people that actually build this stuff and care about this stuff and maintain it. They of course are only doing this because it'd be crazy to do it by any other means or for any other means, with a consumer in mind. And so the result of a machine learning project or machine learning system in production must have the end user in mind. And then of course there is societal impact to everything that you wind up doing in machine learning. The society could be at the organizational level of the company that's surfacing it, and it could actually be something that's even larger than that. In some sense we heard from this morning's keynote about some of the systemic issues of dealing with and understanding machine learning and some of the trade-offs that we have to wind up making software in this world. And this worldview is connective tissue between the hardware, and it is the thing that's instantiating all the theory and surfacing it, or at least as part of the original layer surfacing it back out, for all of us to be able to use it.
So I'll go into a few aspects of these different components. I'll present some of the new facilitating tools that I and my group and others have been building to help with these components, and then talk about the impact of how you do problem definition in the context of this and also in the context of teams. So of course many of you will see your favorite algorithm or sets of algorithms on the left hand side. Many of you won't see that and you'll get mad at me, and that's completely fine. This is just examples of the kinds of things that we all know are being used in practice in machine learning systems. And the wonderful thing about where we've arrived as a community in Python is that there are some really great packages that are exposing these amazing algorithms that have these wonderful properties. And I think it should be clear to anybody here that despite the provability of some algorithm being convex on paper, until it gets exposed at the software layer and until it actually gets run in real world hardware, it really is vaporware. And so in some sense software is where the rubber in theory hits the road in practice.
What I wanted to point out though is that as we start thinking about growing out of these layers, from just getting out of theory land, there's a really wonderful blog post that John Langford, who's now in Microsoft research, wrote I think it was in 2007, entitled all models of learning have flaws. Lest we forget that our favorite model or classes of models that we like to use to solve our problem actually have some fundamental flaws with them. And so this is a couple of images that are basically subversive to a well-trained deep learning model, that basically trick it into saying something. The other thing that's obviously too much detail to go into for this talk, I just wanted to point out, is that oftentimes we can convince ourselves that these algorithms seem to work really well on our own data and we've got some beautiful held out testing data, when they actually are then applied, almost, I don't even say transfer learning, you just take an entire model and you apply it to a completely different data set that should work because you've told people that I have bounds on my errors, in the context let's say of images, don't work, or at least they don't work nearly as well as people think they are, because there is no guarantee when you build a model in machine learning, even that you've got testing errors, there's no guarantee that's actually going to work when the distribution of the data actually winds up changing.
So this is a couple of different projects and training sets on which you can build these very large classifiers to predict the word car basically from an image. And what you wind up seeing in that diagonal is what was the reported testing error for those projects. And then if you actually apply that same model to the other sets of images and you look for the word car, you wind up noticing that the testing error is actually much worse, or, that is, these are actually accuracies, the accuracy is much worse. So compare 28.2 to 10.6 and you wind up realizing that something's really wrong for a paper that has basically said
these are our errors and this is how well we do relative to everyone else. By the way, does anyone know what the original adversarial image is for deep learning? I found it actually in the literature. It's from McCulloch in 1929, he wrote this up in ICML. Only a few people got that, that's okay. So really the question is, as you're building out these systems, what are you optimizing for? And if we start again from this inner part of the onion and move ourselves outwards, obviously at the algorithmic level and the model level you want to think about learning rate, you want to think about convexity, you want to think about error bounds, you want to think about scaling. And all those things are really important, and people write papers in the academic literature that say my scaling curve is better than your scaling curve, give me a PhD. And that's okay, but when you're thinking about that, typically those are the things that you wind up caring about and you forget about things in terms of the trade-offs between the accuracy that you're going to wind up obtaining and the memory usage and the disk usage and the CPU needs and the time it takes to learn and the time it takes to predict. These things don't show up very often when you're just thinking about theory, and that's again okay, but it's a smaller view of the things that you have to care about.
And then at the project level, and the people that are minding over not only the construction of these systems but the maintenance of these systems, it's what is the time to actually implement one of these projects? Think about it this way: if I could give you a perfect spam filter, would you take it? And you can't answer that question truthfully because I haven't told you how long it's going to take me. Could take me the age of the universe and I guarantee it's going to be perfect, or maybe something less time, but you get my drift. You can't talk about building something that has perfect properties and not think about all the impact on resources. What is the impact on people costs? What is the impact on their time? What is the reliability of these algorithms? How robust are they to the changing world? Are they maintainable? Can you do experimentations with them easily? And we're not done. Then start thinking about something even broader: what are the things that consumers actually care about from these machine learning systems? They want direct value, they don't care about really what's happening under the hood. They want usability. Many times you actually need explainability in why you got to a decision that you got. You'll hear time and time again from some of these large corporations that have been using machine learning in production for decades that they're using linear algorithms. And the reason why they're using linear algorithms is because the CEO can understand it, and when it doesn't work they can explain to the stockholders, well, this is a plus b times x, and that's it. Try to get that into production when you have an incredibly complex model with thousands of dimensions, you have some nonlinear mapping over that, and that just gets harder and harder and harder. Explainability for real consumers in the real world turns out to be one of the most important things for machine learning in production. And then of course at the societal level, what are the direct impacts of the existence of this machine learning system in production?
I think the big view here is that if you have a narrow view of just one part of this component, it can be very very costly when you actually start putting it into practice farther up the stack. So I'm going to pick apart a couple of things around, say, accuracy, still staying close to the algorithmic level, and then also start moving more and more in towards the hardware. When it comes to optimization, we have to ask this question: what is the metric that I really care about? So this is something that we did within my company. It's a 100 dimensional space that we've done a regression problem, we're basically predicting a certain number. And it looks pretty good if you look at the R-squared, it's 0.91. You can come up with some other metrics, root mean square error, etc., and all this stuff looks really good, right? So I'm done, right? In fact, this is the best that I could do. But what I haven't told you is what I actually really care about. Maybe I want to minimize the scatter about this line. Maybe I care a lot about the bias in the numbers that are predicted to be below zero, and maybe that's where all the value is for me. And maybe I don't want to ever be wrong about outliers. So until I tell you what it is that matters, you can't just throw your data into an algorithm, get an answer, and move on.
I think throughout this conference you've been exposed to these so-called ROC curves, receiver operator characteristic curves, where you're plotting mis-detection rate versus false positive rate. It's basically type one versus type two error. And typically what people want to do is minimize all the stuff to the bottom left to get a very good classifier, right? And for some reason, well, I know why, but it's kind of weird, this is called area under the curve and you want to maximize that. So the larger the number, the closer to one, the better, you go all the way down to the bottom left with these curves. So who can do the integral in their head and figure out what's got the smallest AUC? Who thinks red? Who thinks blue? Green? Okay, everyone's sleeping. It's red. So red is the best one, you want to basically minimize your false positives while you also minimize your false negatives. So which one do you choose? Which one do you put in production? This isn't quite fair, it's a bit cheeky, I haven't told you what the question is yet, so you can't answer that my AUC is better than your AUC. But here's the reason why I would choose a blue curve over a red curve, or a green curve over a blue curve.
Let's take two extreme examples. So if I have a product that works in the intensive care unit and I want to make sure that the people that are connected to this thing, basically when they have some problem with something in their cranium and you get a lot of expansion, you want to make sure the nurses are alerted to some extra pressure in the brain. This is actually a real world problem. And so you can build these systems that basically come up with alerts, and of course you want to minimize the number of false alerts, but you don't want to miss too many of these actual alerts, right? So in that case where do I want to be on the curve? Well, I don't want to have any mis-detections because then the patient dies, and for that I'm willing to give up on a whole lot of that AUC and I have to choose the blue curve. So with the blue curve my false positive rate can be very very high but I have zero mis-detection rates. Likewise, if I'm setting up some, let's say, drone system, and I want to shoot down enemy planes and not my own drones, I probably want to be on the other curve. And the other curve is one where I'm willing to take a large mis-detection rate just so I don't blow up my own stuff. My false positive has to be zero and my mis-detection rate has to be whatever it has to be to get there. So if you can't define your loss function you can't optimize, and if you can't optimize then basically you're fishing.
Accuracy winds up coupling in real world systems not to other properties of how this thing scales. Of course it does that at some level, but the thing that you have to really care about is some notion of implementability. If I can't put this stuff into practice, it doesn't matter how good it is on paper. So I think one way to illustrate this is by looking at both the Netflix prize, which I think is now about 10 years old, is a million dollar prize to try to get people to improve their recommendation engine by 10%, and the team that won, won basically with a few decimal places better in the metric that Netflix cared about over the other teams. What I've done here is I've taken both the Netflix prize and all the public leaderboards from that and all the public leaderboards from all the Kaggle prizes that have ever been done, which are running Netflix-like prizes, and you can split this up between very valuable prizes and not very valuable prizes. And I've normalized it by whatever the winning metric is, so if it's a number they're trying to minimize, I've basically reversed the normalization, so all of this stuff should be largely on the same scale, because again, the way that Netflix was evaluated and the way that Kaggle is evaluated is with typically a single number. And what you wind up seeing is that there are lots and lots of teams for these expensive prizes that get within some normalized metric of the final answer, and they do really really well. For the ones that aren't very valuable, you wind up seeing fewer teams actually participate and fewer high quality teams participate.
So which one do you wind up implementing? If you get within Epsilon of the best answer you could possibly get, maybe somebody who came in second place has essentially a linear classifier and you could write this in 12 lines of code, and maybe the one that wound up winning has some incredibly convoluted classifier that would take a long time to do. So when you think about the trade-off between accuracy and putting things into production, you have to be mindful of this, and Netflix themselves saw this in spades. Those of you that know the history of what happened after the Netflix prize was given out, is I think they figured that it would take about a 100 man years to put the winning algorithm into production. And I encourage you to read this blog post by Xavier and Justin on just what they learned out of that. They looked at all the different pieces of that code, that was one of the requirements of that competition, and they realized there are a couple cool things they could take from that, it just didn't justify putting it in production. Now you could say this was a complete failure in some sense, to have launched the data science competition world, not to have had a bigger impact on data science more broadly, but you could also say, well, a million bucks was a pretty small price to pay for Netflix to not have to throw tens of person years, data science years, at trying out a bunch of stuff. They just got everybody else to try it out for them. So it's not clear that this was obviously a failure.
So putting algorithms into production is absolutely critical. In my company at wise.io, we wound up realizing that the kind of things that we wanted to do for our learning algorithms we weren't able to do at the time on existing public code bases and existing implementations of algorithms. And what you wind up realizing is that while these algorithms work really well on paper, until you start putting them into computers and start dealing with issues of RAM, you wind up realizing there's a lot of inefficiencies out there. So this is a very big picture view of the core ML stack within the company. This of course is orchestrated in a much larger environment with containerized data science workflows, etc., that interconnect with the rest of the world. But we needed fast, memory efficient, scalable, composable data science that we could actually do on a laptop and then move it into the cloud very easily. And really the end goal of the company from a data science perspective and engineering more broadly is to have our implementation folks manage that entire setup. So it's not the data scientists or the engineers that are working on and setting up new clients, where we basically wind up seeing the difference between customers from the back end is the difference between config files.
So what our API was meant to do, from the very low level, was reduce the friction of common tasks in data science, like subsetting, cross validation, computing ROC curves, etc. And we really focused a lot on computational efficiency, so most of the work that we've done is at the C++ level, in the form of something we call Wise data set, which I'll spend a little time on, and Wise transfer, and then I'll also talk about this other component called Wise wind tunnel, both of which we're hoping to get out into the open at some point. Some of these things are pretty new, especially Wise wind tunnel, so we're going to work on it a bit, but I've already started talking to many of you in the audience about it. So let me just talk very briefly about Wise data sets and how they're laid out. Again, we're trying to be able to do most of this work very efficiently in C++. The idea here is that once I've laid out the data very well and intelligently in memory, then I can build fast algorithms against them, again in C++. And really I think one of the highlights of what we've done is we take a given data set, say in CSV, and we pre-parse it into what we call variable groups. So all the things that are ints go into one variable group, the things that are strings go into other variable groups, and you can keep track of those at a metadata level, so that you wind up having mappings between, and actually mappings between the strings and the ints for instance, so that when you're doing comparisons you're doing them with ints and not with strings. And the nice thing about it, of course, is that this is an abstraction layer that the Python folks don't have to really think about, it just gets exposed to them, and I'll show you what that looks like. The cool thing also is that, because we've abstracted the notion of what these different data types are, we can simultaneously store dense versions of our columns and sparse versions of our columns.
The way this surfaces back up into the Python land is using something we call Wise transfer. See if I can get that to work, so you can see how that actually looks. And the goal here is to move as little data as possible from C++ land back into Python land. We found that we had to do something like that, and so we also had to play around with transferring data across from C++ into Python, and we wound up realizing that all the various implementations of this, like protobufs, etc., really didn't work. There were enough limitations on
those that we had to build our own transfer protocol, so we can basically transfer arbitrary data, can arbitrarily serialize C++ stuff back up the stack if we need to. And so we've tried to make this as easy for the data scientists to be able to use without having to think about all the stuff that's happening under the hood. So you see here we basically instantiated a data set, and the important thing that isn't clear from what you just saw is that train and test, which were part of basically a stratified sample, aren't copies of the data, they're just copies of pointers into C++ land. So we're effectively just creating views on the data, and it allows us to operate again very efficiently in C++.
So where does this live? It's funny, because in the 80s everybody wanted to be a DJ, I feel like now everybody wants their own data set abstraction. We did that as well. This is somewhat opinionated, backed up by our own internal tests against these various different implementations. Starting from the far left to the far right, it's the sorts of things that data scientists care about down to the sorts of things that implementation engineers, C++ folks, care about. So slicing induces copy is a really important one. If you've got a big part of the data and you want to build, let's say, a train test for a holdout set just by slicing your original data, if you have to make a copy, if that data is very large, that becomes problematic. Immutable columns, things like queries, transfer between Python and the lower levels, that's actually very important as well, that's something that Spark data frames is doing quite poorly right now. C++ SDK, it means that I can write against these data set objects at the low level. Is it distributed? Is there a memory efficiency with the way that you've laid data out in memory? Have you optimized the notion of what categories are of those data sets? And again, can you use sparse and dense at the same time? So this isn't to say that what we've built internally is a panacea, but for us, building very fast machine learning algorithms that are memory efficient without giving up accuracy and having to make trade-offs is, I think, really really important.
Okay, so let me move on to a somewhat larger view of this system, and that's about enforcing weak contracts between the lower levels of the system and the higher level ones. And in some sense this is really how we monitor deployment. So here's a very generic example that those of you that are working with ML systems in production probably know about pretty well. So I build a data science workflow on a test set, I like the offline testing accuracy, so I'm like, great, I'm going to deploy it, and I'm going to start monitoring the results in practice. And of course, because this is how I've set up the problem, online, when we actually start looking at the results, it's worse than we expected from our offline test. So what do we do? Well, you can bang your head and say, well, I overfit on the model, I'll go back to the drawing board, I'll do some more data science, and that's probably fine. I can just retrain because more data has better answers. You've probably heard that a lot of times, that's Peter Norvig's thing. Or maybe it got worse because there's this thing called concept drift, that the underlying data distribution changed and the model that approximates the predictions you wanted to make should indeed get worse. So if you've built up a data science workflow that, let's say, hardcodes which features you wind up using and you're not doing automatic feature selection or automatic hyperparameter optimization, you can wind up basically hurting yourself quite a lot. If it turns out that upon retraining, where you figure out those other parameters, you still get a pretty bad answer in production, you can go back and change that data science workflow, maybe you realize that new features that you hadn't thought about become important. Or maybe, just maybe, that's okay, because your prediction influenced the outcome, right? That was the goal in the end of what these ML systems are.
If I'm making predictions, as we heard earlier today in the context of Stripe and in the context of payments from Mike, if you're basically making predictions about whether this transaction is going to be fraudulent, and you don't let what you think are going to be fraudulent transactions through, the next time you go and retrain, unless you're being very very careful, you've basically influenced the world. And so there's something about the way the data looks today that's different than the way the data looked yesterday, and so when you try to glom all that stuff together you get a really strange view of the world. So there's a lot of people obviously thinking hard about these problems, and real companies that are putting real ML systems in production are thinking about this probably as the forefront of the things that they have to care about to get good answers. There's also a very good talk by Chris yesterday on A/A testing and A/B testing that's relevant to some of this discussion. But there's more, of course. There's obviously the testing that you normally do, because all of you are awesome, across your entire software stack: you've got unit tests, regression tests, you're looking at integration tests, etc. But then there's also what I'll call a semi-new concept, at least in the Python world. We heard yesterday from Trey about a couple of cool tools that are helping along these lines. Basically, is the data changing, and are my predictions different than they should be given that changing data? Is the contract that I set up with the larger software infrastructure that I've set up affected by that changing data? And then there's another component of this, I'm still kind of trying to get to the words myself, of model deployment testing. I need to know when things are too different than they were before, I need to alert a human, and then I need to use automated tools to try to isolate the cause of that change. Was it data that changed, and that's what the ETL stuff gets to, or was it the code that changed? Because again, you're not deploying a machine learning system in production and then walking away and looking at it every year, you're constantly deploying new things into the system, so you need to make sure that you're not actually introducing new bugs and new errors that other parts of the stack didn't know about.
So we built this thing in pure Python called Wise wind tunnel, and the idea here is that if you start from a known good implementation of a known algorithm and then you start monkeying with it and start adding bells and whistles and things, in some ways you can basically start understanding whether, with the data that you're throwing through, you're getting the answers out that you expected, given the fact that I've made changes. So instead of hard coding, the AUC should be this value or the accuracy should be this value, say within some constraint, instead the way that we allow ourselves to do it at the decorator level is to say the AUC shouldn't change by more than a few sigma, where we've defined what a few sigma is from the previous run using different random seeds given the same data set. That has to be true. But it's not just testing on things like accuracy, we're also testing on things like memory usage and total time it takes to train, total time it takes to load data into memory. And so when you start doing that you get a probabilistic inference about how well you're doing relative to what you had before. Now you still have to have something at the very beginning that you wind up trusting, but to the extent that we're thinking about this as almost like a git project, everything that we do when we do a new deployment becomes a new git hash and allows me to go back and reproduce the code that got us those exact answers along with the same seeds, so we can go back and make sure that we're not messing anything up. Unexpected surges in memory cause crashes, for instance, and you can protect against that, or at least test against that, and that can be as critical as protecting against loss of accuracy.
So when we think about these weak contracts between the different layers here, one way to think about it is that the abstractions of how you think about the implementation of your algorithm, in the context of the world that you have heads-down focus on, those abstractions wind up leaking into the other parts of your system. So one thing that Leon Bottou basically said at ICML along those lines, he came up with the very clear example I think that illustrates this leakiness very well. You've got a smart programmer which makes an inventive use of a trained object detector, recognizer. The object detector recognizes data that it hasn't seen before and it produces an answer, and then this smart programmer basically gets an answer that's complete garbage. And there's nothing in the way that we think about, say, an algorithm in deep learning that protects us naturally against this, we have to build larger systems around this. And why is that? It's not just because the theory is really bad for this algorithm or that algorithm. The way that the Google paper put it is machine learning packages are inherently a tool that mixes data sources together, and because they're doing that you create entanglement, and that entanglement is very hard to take out. We have not software to blame for this, or theory to blame for this, we have data to blame for this. We like to think, when we're building out our algorithms, of this platonic form, this crystalline sphere of data, but real data is not platonic, it's plutonic. Sorry, I had to do it. It's ugly, it's a dusty snowball with mountains and warts and geysers, and it's got NaNs and missing quotes and all that stuff. It is incredibly ugly, but at the same time it's incredibly rich.
If we wind up looking at, say, one document or a series of documents, you wind up realizing that the different pieces of that data require different sorts of attention in the featurization of it. So you've got natural language processing, which could produce very sparse output, you've got computer vision, which may be a nested thing, you've got metadata, which can produce dense output, time series, so there's a concept of streaming, and of course if you're pulling over third-party data about this, many times that turns out to be missing and noisy. So the richness is wonderful, and what you wind up realizing is that to do this right, to do this featurization correctly, you need to have very deep domain knowledge. And so doing machine learning without that knowledge is very painful, and in fact may even be too hard. And when you think about domain knowledge and you think about your requirements of having that, in some sense what you're really asking is, are you asking the right question? How many of you know a botanist? Wow, that's amazing, are we near a big botany school or something? I wasn't expecting that at all, I was expecting crickets. So the reason why I'm asking is because there's this famous machine learning data set, it's the hello world of machine learning, called the iris data set, where you have to predict the different types of irises. Who's asked a botanist, of those that know a botanist, do you even give a crap about that? I've got this amazing classifier, it's got 100% accuracy and I can take an image, and they'll say, no, I don't care about that, right? Who's asked themselves or maybe somebody else about the provenance of this data? How come there are no errors on the length of the petals? Were all these data points taken of the same area with the same type of soil at exactly the same time after they were planted? No, or we don't know, right? We need to start asking ourselves about questions that are actually important. Not to say that this isn't an interesting data set as a hello world, but this doesn't actually have any real value, and because it doesn't have any real value it's hard to define what the real loss function is over that data.
One of the things that we found very hard in our academic work was the recognition, which should be clear to all of you, as me being essentially a trained physicist, that I'm not a machine learning expert, I'm not a statistician by training. And so for me to be able to answer these questions in a production environment that mattered, I needed to find people that were really good at the things that I wasn't, in the computational sciences specifically. I could ask the domain questions, but then getting those people to essentially build these large-scale machine learning systems so that we could answer really important scientific questions was hard. What I started realizing after we started building some projects around this in astronomy is that when you look at time series data in the sciences, there's a lot of commonality, from neuroscience to seismology. And the way that we started thinking about building out a project in astronomy, we started realizing some of the same tools that we were going to need to do that were going to be broadly applicable to those in other sciences. And so what we did is we got a grant from the National Science Foundation to basically build a platform for domain scientists to very quickly go through and get essentially featurization off of time series data, to be able to build out these scientific projects on time series data and answer questions that they want to ask themselves without having to spin up a whole team of domain experts in machine learning and statistics. So I won't go into the details of this, this project is on GitHub and we're certainly very happy to have people contribute to that. One of the things that we're really excited about is the way in which we do continuous integration, basically leveraging Docker, and we've been very thankful and fortunate to have Microsoft Azure devote some time on their cluster for us to be able to work.
What was the output of this? We wanted to produce basically some of the scientific papers that we put out several years ago with the same raw data sets with essentially just a couple clicks of a button, and our MVP, which we're now arriving at, is our ability to produce results that are comparable to the ones that we had before. So one of the things to recognize is that making predictions, let's say, about the classes of variable stars is one thing, but you really want to use that as a launching-off point to something else of some sort of deep interest. So based on all of the work that we did in the past where we classified tens of thousands, if not hundreds of thousands, of variable stars, we were able to find some really interesting needles in a haystack that were essentially impossible to find by any other means. And I won't go into the details of all these things, but the point of this is that we basically had an end goal in mind, we had something that we really cared about, which is pushing the envelope of knowledge, and in doing so we had to do machine learning. I didn't do this because machine learning is fun. I mean, it is fun, but it is really hard.
And one of the things that we all have to ask ourselves is, you see this data set, you say, I've got this pretty awesome hammer, I'm just going to start hitting it because it looks like a cool nail. There is a real danger in starting off and trying to solve problems with machine learning systems before you actually think about the problem deeply and can you solve it in other ways. In the company at wise.io, we're basically selling products into the CRM world with a non-technical buyer and a non-technical user, and we're basically focused on routing, say, support tickets. When you send an email to a company, there's a lot of people that have to look at it, and optimizing and automating that is the domain problem that we decided we're going to solve. What you wind up realizing whenever you're solving some sort of large machine learning problem, even if you're doing it well, is that you're still going to make mistakes. And the way that you make mistakes and don't shoot yourself in the foot is you build what's called fault tolerant machine learning. Gmail is famous for doing this really well: I've categorized this message as spam, if it's not spam, that's okay, tell me why, and next time I go back I'll basically build a slightly better spam filter for you. Microsoft Kinect uses random forest, and if I get the pixel wrong of, is this the left thigh or the right thigh, the next frame you get it right, it's not a big deal. At wise.io, this is a view of what an agent sees within Zendesk as they're basically going through the tickets. We're giving them recommendations about how to respond, and if they don't like those recommendations, they can search and figure it out. And it seems very simple, it's abstracted at the UI level, but you have to think about your audience, right? You have to think about, as you build these and deploy these new ways of doing machine learning in production, what's the impact?
I think it was about a week ago, maybe two weeks ago, that the Gmail group said we're using deep learning now and look how great the ROC curves are getting for all of our people using our spam filter, and it was pretty impressive. And then a couple days later Linus Torvalds posted this, he said, something you did recently has been an unmitigated disaster. Of the roughly thousand spam threads I've gone through so far, now 228 threads were incorrectly marked as spam. Google pissed off Linus, he didn't like his ROC curve. I mean, let's face it, Google is the best machine learning company in the world, and they sometimes struggle at what is one of their core end-user facing products in machine learning. And it isn't just Google, Netflix also makes this mistake sometimes. This was on Reddit a couple days ago. Is this fault tolerant ML? Yeah, I mean, this guy isn't going to cancel his subscription, Linus isn't going to stop using Gmail. This is very very hard stuff. And one of the things that Leon said in ICML was that machine learning disrupts software engineering. So what should be the machine learning engineering process? The my-scaling-curve-is-better-than-your-scaling-curve is not the answer, or my version of this algorithm has a slightly better accuracy than this. You have to have, and this is the scary part of it all, you have to have a 50,000 foot view of the entire system as you're building out and working on these various different components. It's not an intractable problem, it's a really hard problem, and I guarantee you it's not going to be solved by one type of person.
And really the last thought here, the parting thought, is that when it comes to ML systems, it takes a village. Or another way to say this is, data science as a team sport. You have to have interdisciplinary teams. And this should echo, for those that were there at the keynote this morning, some of the things that you heard there, of the cry for that. We often talk about in academia the need to build these pi-shaped people who are really good at domain questions and really good computer scientists and PhDs and everything. And my view of this world is that there isn't this notion of a pi-shaped person. It's dangerous from an educational perspective to be thinking about how we create pi-shaped people. We need to create people that are very deep in something and maybe somewhat deep in some other parts, and we need to get a bunch of gamma people together, gamma-shaped people, to solve some real world problems. So, unless you haven't been paying attention, I think it's reasonable to say that just because you've got a good software package that implements a whole bunch of machine learning algorithms, it doesn't mean that that's a functioning ML system. There are many other considerations when you build these ML systems. It better be because it's the last resort, because you have to. What I like to say is, if it's not worth doing, it's not worth doing well. Machine learning systems require optimizations across components, so we better understand the true loss function, not just in our restricted domain but at the largest level. And hopefully this isn't a completely lost problem, and the novel testing frameworks can strengthen those abstractions within components and the contracts between them. And regardless of what you do, the last line of defense is some notion of a fault tolerant machine learning. So with that, I'll say thank you. [Applause]