As Wise.io co-founder/CTO: what it takes to deploy machine learning in production enterprise settings, and how industrial ML practice differs from academic ML.
Month per FirstMark slide upload (May 2015); one listing says Nov 2015.
“Astronomy in some sense had a big data problem 120 years ago. When we had this big data problem, what we did is we hired people.” – Joshua Bloom
“When you're deep in the weeds in a data science organization that isn't highly connected to the product… you can lose sight of the fact that what you build and what you put in production has to be useful.” – Joshua Bloom
“Netflix and Google, some of the best machine learning companies in the world, and these are their core products, and they still make mistakes. Yet these are not fatal mistakes. They've built fault tolerance into the machine learning.” – Joshua Bloom
“We don't really see ourselves as a big data AI company… we're solving interesting problems that don't show up in the academic literature, more around featurization than it is around the actual model building itself.” – Joshua Bloom
Hello everyone. My name is Josh Bloom. I'm co-founder and CTO of wise.io. I'm also a professor at UC Berkeley. Tonight I'd like to talk about machine learning in production. And when I talk about machine learning in production, what I'm actually trying to talk about is the data products and the interactions that all of you are applying to data that's unique to your own business and transactions that are unique to your own business. But many of you are also thinking about building outside of the core of your business into some of the aspects that actually are very similar across multiple industries. So the key question that I want to pose tonight is: where is the AI build-buy decision boundary? And many of you as data scientists are going to wind up having influence on that decision. So I hope I can give you some thoughts that help you frame that question for yourself and your organizations.
My own point of departure is as an astronomer, and astronomy in some sense had a big data problem 120 years ago. When we had this big data problem, what we did is we hired people. So we had what were pejoratively called computers. These are mostly women who are looking at data, and they were making high-level decisions at some times, when many times they were basically just pulling data out of images. There were too many photographic plates that were coming from the southern hemisphere, so the director of the Harvard College Observatory had to invent grid computing or parallel computing. And what's amazing is that if you look at this slide it seems so quaint, but this is actually how a lot of science and actually a lot of industry is still done today. When we look at data and when we touch data, we tend to think of needing people in the real-time loop. And one of the things that's so amazing is that this sort of repetitive knowledge work is still rampant throughout organizations.
After we wound up building essentially a machine-learning-based workflow that helped us to do astronomy better on some more modern problems, we started to turn our attention to other aspects where repetitive knowledge work was being applied in an industry. The place where we wound up landing is in customer success. So for those that don't know about wise, we basically build an artificial intelligence layer on top of SaaS-based data sources. And so we touch the conversations that you are having with your clients, and we try to make them more efficient and better. And we see customer support as a pretty interesting launching-off point to other value propositions within organizations. So rather than ask people to build new workflows around AI, instead what we do is we add essentially an augmentative or automated layer on top of the workflows that people are already using. I won't go into the details, and I'm happy to take questions on what the company actually does, but I just wanted to give you a presentation of our stack.
And this is the first time in this talk we start thinking about the build-buy decision. In this case not build-buy around AI, but build-buy around engineering practices. So it would be very easy of course, if you've got a good group of people, to stand up your own Kafka cluster or manage some multi-zone Postgres server. But instead, why not just have Amazon do it, or your other favorite compute cloud? So we made, as a young startup, the decision that if there was a managed service around what we needed to do to get something into production — and we're at the millions of predictions level a month over dozens of customers now — we were just going to buy it, or in this case we're going to wind up leasing it. So obviously I won't go into the details of where our stack is and all the things that we've tried to build, but suffice it to say we try to build a composable set of services that wound up talking to each other, and we try to make the most vanilla plain architecture as possible, so that we could focus on our core differentiators, which was building very fast and very memory-efficient AI and solving some real problems.
I also won't have time to go into the details of what our stack looks like, but what we wanted to be able to do is let data scientists in our own organization build, almost at a config-file level, the differences between different clients that we have, and put those into production without the need to talk to engineers. So we try to abstract away all that other stuff that you saw on the previous slide and allow a data scientist to essentially work in the kind of environment that he or she wants, typically in Python, and then be able to put it directly into production after testing it on their own laptop. And so that's what we've been engineering towards, and that's what we've been able to accomplish. And in part it's getting into some of the nitty-gritty of what the core IP of the company is.
When we're making these decisions — when I hope you're making these decisions — now in the context of machine learning, there is this question of what it is that you're optimizing for. And obviously if you're a machine learning expert and you come from academia, you might really be thinking about how your scaling curve is better than somebody else's scaling curve on some large amount of data on some toy dataset. And that's fine, right? So getting better scaling is important, getting better accuracy is important. But oftentimes when we're building these systems, we often will wind up neglecting other parts of the stack, and more importantly we'll start neglecting who our actual customers are of the types of products that we wind up building. There's this multi-axis optimization that many people don't think about. So deep inside you work on the algorithm, but you have to put that into hardware, right? And so is this algorithm that you've just built actually memory efficient? Are you trading off accuracy, memory usage and speed, and are you actually explicitly making those trade-offs in your head as you're starting to build these systems into production? And then of course there's a time to implement, which is often highly neglected when organizations are starting on a new project.
How long is it going to wind up taking you to take this cool algorithm that just got published in ICML and put it into production? How many person-years? How many person-years after that is it going to take to actually run the system? And then of course, is this actually valuable to customers? And that's something that we obviously always have to have in mind. But typically when you're deep in the weeds in a data science organization that isn't highly connected to the product, that can be a real challenge, and you can lose sight of the fact that what you build and what you put in production has to be useful. So I'll talk about some of these different trade-offs.
There's several hundred people in the room, so I'm probably pissing off most of you right now. In this trade-off between accuracy and interpretability, you'll hear more from some of the other speakers about their thoughts on this, but oftentimes in organizations the data scientist is trying to optimize towards higher accuracy, which makes perfect sense, but they may be biasing themselves towards algorithms that are harder to understand why you got that answer. There's a great example from a friend of mine who gave a talk from basically a real estate analytics company, saying that he showed us all the different accuracies for all these different algorithms, and he had everything from a linear algorithm all the way to random forests and even some deep learning. And he said, which one do you think we wanted to put in production? And it was very obvious that as you go far to the right there was a better and better accuracy on this one problem he presented. And in the end he said, we just chose the linear one. And the reason why is because their clients needed to be able to explain it to the CEO, so that CEO could go to the board and explain, well, there's this A times B times X plus C, and there you go, that's why we chose what we chose. And so explainability or interpretability turned out to be a very important optimization that very few data science teams will be cognizant of unless their investor is really thinking about it.
There's obviously this other optimization metric of the trade-off between accuracy and implementability. And so here I've taken old Kaggle prizes, including the Netflix Prize as well, and normalized by essentially the winning metric. So if there was a low number I normalized it, if it's a high number I normalize it, so the winning team was all the way on the right on this graph. And then what you see on the y-axis is the percentage of teams that did within some normalized distance of the final answer. And you wind up seeing, for low-value Kaggle problems, you wind up having not a lot of teams doing really well, and for high-value ones a lot of teams come out of the woodwork and do quite well with those things. And so when you wind up winning one of these competitions, you typically win in the fourth decimal place. But there's an obvious question of, what about the second team, and the team before that — was their algorithm or their approach easier to put into production when you actually want to start doing this at scale? And that is the question that you have to ask yourself. Accuracy is not the only thing that's most important.
And what I love is this quote from Xavier Amatriain, who talked about what the Netflix Prize meant to Netflix. And essentially what they said is — for those that don't remember, it's now a 10-year-old prize, where they gave out a million dollars for getting an improvement in the recommendation engine from Netflix — they basically say, we couldn't put it in production. They paid a million-dollar bounty, and they wound up looking at the code, and there were hundreds of separate models that were then boosted together, and they said there's no way we can do it. They took a couple of interesting pieces of that, they put it in production, but accuracy — the team that won in the fourth decimal place — wasn't the most important thing for them. It was about implementability and usability. I think some of those lessons of data science are echoed in a very scary way in the new paper by Google, who titled this “Machine Learning: The High-Interest Credit Card of Technical Debt.” And they talked about machine learning in production as a very different type of beast than typical engineering efforts, because of the lack of composability and because of the abstraction boundaries that wind up bleeding over from say algorithm into software into hardware, etc.
And by the way, if you are a data scientist, this should be in your canon, right? This is like your required reading. If you're a CIO or a CTO, I think you have to read this as you're making your build-buy decision. If you're an investor, this is the kind of thing that you want to be asking companies that are coming to pitch you, where they're talking about how they've got this next great machine learning algorithm, next great machine learning platform — asking these questions, because these are the things that people are going to want to know as they start putting it in production. And so there's this quote up here where they basically tell all the academics of the world, yeah, by the way, algorithms are important, but ninety-five percent of all machine learning code in production is actually just glue code, connecting all of these different pieces together. Again, happy to talk about this in the question-and-answer session. There's another wonderful talk by Léon Bottou, who is now at Facebook, in ICML, talking about some of these concepts as well.
So again, I'm going to piss off a lot of people who don't see their favorite platform up here. When you are in a data science team and you are making that build-buy decision, are you trying to optimize for something that's very quick and dirty to get an answer easily but may be very hard to put in production? Or are you doing something that needs to be on-prem because perhaps the data is very private or very secure? Do you need to do something at a very fast speed with low latency, or are you willing to do something as a service? And there's nuances in all of this, right? Are you willing to give up interpretability to get a very good answer, in which case you might go farther to the left? But again there's multiple axes here. And the other ones that I think are critical when making those decisions about which platforms to use and implement within your own organizations is: what does that cost in time to actually put this into production, and then once you put it in production, what is that cost to actually maintain it?
So when we think about putting machine learning in production, we're often thinking about giving our end users the ability to go everything from what they're currently doing now, which is manual processes or manual business rules, to some sort of augmentation — we're giving the end users of our products the ability to weigh in on our decisions — to a full automation, where we're basically taking data off the table and removing conversations that agents and others actually have to see just to solve those problems. So on the left-hand side is just a schematic of what you might see as the output of the confidence of a prediction from a machine learning project on this piece of data — purposely being abstract here — and then as a function of basically the frequency over say a day's worth of data. So when you first start out you may not be very confident in your answers, and so once you've built this into the organization, into the workflow, most of the time they're still going to just keep on manually going through these normal processes, but some of the times they might like to get your suggestions and act upon those suggestions. And over time, if you build the appropriate feedback loops into your systems, the system itself will wind up learning from those processes and get better and better, so you can actually start automating those processes.
So really on the right-hand side what you see is perhaps that risk and cost, because this is now another thing that you have to optimize over. If I make a mistake, what's the cost to the business, right? If I send a “here's how you do your password reset” back to somebody after they said “hey, I like your product,” it's not a terrible thing that you've done, right? But if you say “here's your password reset” and somebody said “I'm going to be suing you for X, Y and Z” and you've missed that interaction, that's a serious problem. So you have to understand the cost of being wrong in really both directions. So we like to think of what we do, and many others, as building fault-tolerant machine learning. So when you make a mistake you have the ability to give feedback into the system. So Google does this obviously with Gmail — oh, this was in a personal message, this was spam — allows you to move that, and then behind the scenes there are models that are getting rebuilt specifically for you and your feedback. And what we do in wise is we give essentially the agents the ability to take our suggestions for how to answer a support ticket, and if they don't, then that becomes feedback for us, and if they do, that also becomes feedback.
I'll just end with a couple of slides showing you how hard machine learning in production actually is, lest you think that this is something that you can wrap your heads around. You might have remembered Gmail getting very excited about the fact they're using deep learning now to get better spam filters, and they talked about their accuracy rates. And then a couple of days later Linus Torvalds wound up putting this on his Google+, saying “something you recently did has been an unmitigated disaster. Of the roughly thousand spam threads I've gone through so far, 228 were incorrectly marked as spam.” So Google pissed off Linus Torvalds because their machine learning wasn't perfect. Netflix does the same thing, right? So here's this showed up on Twitter a couple of weeks ago. Netflix and Google, some of the best machine learning companies in the world, and these are their core products, and they still make mistakes. Yet these are not fatal mistakes. They've built fault tolerance into the machine learning. So as you think about building your own machine learning products, please give some thought to that build-buy decision of whether this is your core competency or whether this is something that you perhaps should bring in-house. Okay, thank you. Thank you.
[Q&A with Matt Turck]
TURCK: This was awesome. So many very interesting things you said. So you were talking about the ability to explain to the ultimate buyer in an organization, the CIO, what actually is going on, as being a very important aspect of this. And so I understand this is not a completely black box, you can provide some feedback to the system, but are you finding that it's one of the big issues — that living inside the technology, but the reality of selling to the enterprise, that the issue is that people need to understand what the hell's going on behind the scenes?
BLOOM: Yes and no. I think the reality is that there are multiple players in that enterprise software buy decision. And fundamentally there has to be an ROI discussion. If there isn't that discussion up front, then you're going to wind up having a big challenge later on the road, because even if they can understand how it works, if they don't like the numbers it doesn't make sense. But once you get over that hurdle, then there's the data scientists themselves, who are typically part of, in our case, the types of decisions that are being made of whether they're going to actually purchase this and bring this in-house, and they're really the influencers on this decision. And for them, typically they wind up asking — not to denigrate all data scientists — but what algorithm you're using, doesn't it scale, why is this not this one that I just read about or I just wrote about? And the answer you say to them is, look, we would love your help on understanding the data and the exact problems that you're having, but we just want to provide — the proof is in the pudding, right? This is a great answer, and your boss's boss needs to understand why we're getting these answers and how much the accuracy, or what other ROI metrics they care about, are actually improving over time.
TURCK: I'm going to open up to people in a second, but you very kindly didn't turn this into a product pitch, which I appreciate, but can you tell us actually a bit more about what you guys do — what is the experience of a customer service agent when using the product?
BLOOM: Yeah, so the interesting thing is we have a bunch of different users throughout the organization. At the base level it's the agent, and for them it's really where the augmentation comes in. We're recommending how they should respond to an incoming message. Some of our clients have hundreds or even thousands of macros of how to respond to somebody's inquiry, and instead of having them go through all of those or memorize those, we're basically giving them the top three suggestions and allowing them to search deeply into all the other macros when they don't get that. We're also doing triage, so as a ticket winds up coming in, we're routing it to the correct queue. And so because you asked, I'll say a couple of numbers that we find pretty exciting. Typical time will be, especially when the triage team is in another country in another time zone for a US-based company, it could be six to ten hours before a ticket that is your email where you say “I like your product” or “I don't like your product” even gets looked at by a human. And when they do, the first thing that happens is they wind up routing it to another team, and that team's job is then to respond, because this is actually a fairly straightforward process. And we're baked into systems like Zendesk and Salesforce, it allows us to route those almost instantly, so within a minute or two you're taking an eight-hour process to a few-minute process. And so even if we do nothing else after that farther downstream, the amount of time savings is crucial, let alone the labor savings.
I mean, I think what's really exciting is that obviously there's this buzzword around customer success now, and there's this understanding that customer success is the core of what a company has to do, obviously after they get a customer. Customer satisfaction is only one of the metrics that really drives whether you're doing well in customer success, and we think all of that really starts with support. The other thing that's quite interesting is that these companies are growing very rapidly, and we've been told multiple times by people running support desks worldwide that if they had an infinite budget they wouldn't be able to hire fast enough for how much their support needs are growing, to keep customer satisfaction at a fixed level. And so what's really interesting, I think, is that you could have a software company that's figured out how to do all the scalings correctly, so you have one person and you get a hundred customers, you add another person and you get two hundred customers on your own team — support is still that last place where you wind up having to scale the total number of people in your support system and support operations by the number of incoming tickets. And since tickets scale with revenue, companies that are growing quick start feeling this pain very much. And that's where we wind up coming in.
TURCK: One or two questions. Okay, that's this one here.
BLOOM: So the question is, are we going to try to solve the problem or create the best response? That's really where that automation-augmentation layer lives. At first we're just going to basically do augmentation, so we're giving help to the end-user agents. And I didn't answer part of your other question — their bosses can wind up understanding overall macro usage and things like that, and understand their efficiencies and how those are changing. But over time the system becomes so confident on some fraction of the answers — five to ten to twenty percent — it can basically just answer them without any humans on our client side actually looking at it, and those tickets get solved and people are satisfied. So that, we hope, shifts over time. But what we start off with is that proposition to the agents, so that they can feel like we're essentially their assistant.
TURCK: Quick one more.
BLOOM: So the question is about low-hanging fruit. I think we actually are solving a pretty hard problem, in the sense that we have to build data science workflows against data we haven't seen yet. And the only way that we were able to do that, I think with some success, is by bounding the problem from just that general statement to: it's going to be around natural language conversations that people are having with their clients. And there's a small vocabulary, as it turns out, of the ways that individuals are interacting with companies. And so what we wound up doing is essentially build models off of those specific interactions. So for us actually I think it's a pretty hard problem, and by bounding it to natural language for multiple companies in a real-time environment — not just in English but in other languages — we see a whole bunch of really interesting problems. And one of the things I'll just add is that we don't really see ourselves as a big data AI company. Most of the problems we're solving are at the hundreds, or sort of maybe thousands, of gigabyte levels at biggest for the types of models we're building. And so for us, we're solving interesting problems that don't show up in the academic literature, more around featurization than it is around the actual model building itself.
TURCK: Great, well thank you very much. We're going to keep moving.