An 84-minute interview on pioneering ML for robotic-telescope imagery at Berkeley, the founding and evolution of Wise.io, its data-science pipeline, and open challenges in machine learning.
“The classic example I go back to is Galileo, who said, hey, there's this new thing that's been invented to look at the horizon for ships coming towards us — what if I just took it and pointed it to the stars? What could I do with that?” – Joshua Bloom
“In some sense, the way I view this now is that algorithms, and the accuracy that they can produce, and the ability to optimize them around a loss function, is really only table stakes for the utility of these in a real environment.” – Joshua Bloom
“We even got to the point where we were finding interesting objects in the sky without any humans in the loop… So by the time people woke up in the morning, we not only had the discovery, we not only had the initial inference, we then also had real follow-up — scientific follow-up.” – Joshua Bloom
“Your brain is a 30-watt supercomputer, unrivaled — at least for now — by anything else that's out there, and anything else that's out there is likely going to take megawatts or hundreds of megawatts to get anywhere close to that computational capability.” – Joshua Bloom
“Doing machine learning for machine learning's sake really doesn't make sense. It's probably the last thing you want to do if somebody hands you data — you do it because you have to do it.” – Joshua Bloom
Sam: Hello everyone, and welcome to another episode of TWiML Talk, the podcast where I interview interesting people doing interesting things in machine learning and artificial intelligence. I'm your host, Sam Charrington. I think you all are really going to get a kick out of this show. My guest this time is Joshua Bloom, professor of astronomy at the University of California, Berkeley, and co-founder and chief technology officer of machine learning startup Wise.io. I was in California last week, and Josh graciously agreed to host me in his company's office for this interview. We had a wonderful discussion, and as you might have guessed if you happen to have noticed the length of this episode, we covered quite a lot of ground. But I promise you that you'll find this 84-minute interview to be jam-packed with great information, ideas, and war stories.
In this show you'll learn how Josh and his research group at Berkeley pioneered the use of machine learning for the analysis of images from robotic infrared telescopes. We talk extensively about the challenges they faced in doing this and some of the results they achieved. We also discussed the founding of his company Wise.io, which uses machine learning to help customers deliver better customer support — but that wasn't where the company started, and you'll hear why and how they evolved to serve that market. We talk about his company's technology stack and data science pipeline in fair detail and discuss some of the key technology decisions they've made in building their product. We also discuss some interesting open research challenges in machine learning and AI. Of course, I'll be linking to Josh and the various things we mention on the show in the show notes, which you'll be able to find at twimlai.com/talk/5. That's twimlai.com, talk number five. And now on to the interview.
Sam: Hey everyone, I am here at the Wise.io offices with the CTO, Joshua Bloom, and we've got a great conversation lined up for you. We'll start with Josh — why don't you introduce yourself to the audience here?
Josh: Great. This is Josh, and I am the CTO and one of the co-founders of Wise.io. I'm also a professor at UC Berkeley in the astronomy department. One of the important things that we'll touch on today is how does somebody go from astronomy and teaching to building an AI application company. A big part of the origin story of the company is, of course, my history, but I think it also has some interesting lessons for how we think about AI in production systems and why having diverse backgrounds is pretty important these days.
Sam: That's a lot of good stuff to talk about there. Why don't we start by learning a little bit more about you and your background and how you got to where you are?
Josh: I was trained as a physicist and an astronomer. I went to Harvard as an undergrad and caught a bit of the research bug over the summers working in Los Alamos, then went to Cambridge, England to do a master's, and back to Caltech, where I did my PhD, all in the context of astronomy, and then back to Harvard, where I was a postdoc — all the while working on what we could broadly term time-domain astrophysics: understanding the variable sky and why things do what they do, explosively, cataclysmically, or otherwise. And while there's a deep interest in understanding the origins of those events and how they're connected to other things that we study in the universe, I got more and more interested over time in the informatics of actually just doing the science — the statistics on variable sources and the presence of noise. Then as I became a faculty member at Berkeley, I started looking ahead to really a series of new surveys, particularly imaging surveys of large swaths of the night sky, and one of the great interests for myself and many others was in finding new events — essentially new explosions or new variable, eruptive stars that hadn't been known about before — and doing that as quickly as possible. Now, the traditional way in which that was done, and in some cases is still done today, is that as you acquire more data, you linearly hire more grad students. That scales with the total number of images that you're getting and need to sift through, and as I was becoming involved in some of those projects, I ended up realizing that as time went on, that really wouldn't scale anymore, and we needed to find effectively a replacement for domain experts who otherwise would have been looking at and opining on data, using alternate techniques. About seven, eight, nine years ago I stumbled upon machine learning as a real interesting potential avenue, and at the time machine learning really hadn't been applied to anything in astronomy in the variable sky, in a time-domain context. There had been a number of studies in using machine learning to do special types of inference on the static sky and understanding demographics of stars and galaxies and their distribution in space. So we really felt like there wasn't a lot of precedent in us applying some of these capabilities to this data, but we wound up realizing it was sort of an imperative. One of the things that people who know astronomers would probably say about them is that they tend to like to use tools that help them, and seek those tools out — be those new types of detectors (CCDs, for instance, were something that astronomers adopted almost as soon as they were invented) and obviously statistical techniques and computational techniques. Astronomers are willing to try things out to solve their problem. The classic example I go back to is Galileo, who said, hey, there's this new thing that's been invented to look at the horizon for ships coming towards us — what if I just took it and pointed it to the stars? What could I do with that? And so our use of the telescope was essentially a co-option of a technology that had been built for other purposes. That kind of proceeds apace throughout the history of astronomy, and so the idea of bringing essentially a fairly new technique into the fold is not at all unusual.
Sam: Do you remember how you stumbled across machine learning?
Josh: Part of it was just asking the question: if I've got a bunch of data and I need to decide, is this part of the sky interesting or not, is this event new or not, what type of event could this be — very quickly you wind up realizing this is a classification problem of some sort. And talking to people at UC Berkeley in the department, as I was starting to introduce some of these interesting challenges, it became very clear that machine learning, and particularly supervised learning, would be a fertile ground for us to start exploring. But one of the challenges that I saw is that even though we're in a very rich and fertile environment at UC Berkeley, and there's a lot of crosstalk between departments and individuals within departments, it was very hard to get even the language on both sides up and running, where both sides understood — the methodological folks who deeply understood what machine learning was and what it could be used for, and then people like myself from the physical side even learning to ask the right questions. So thankfully we wound up getting a group from the stats department and those folks from computer science together with me and my postdoc, and we were able to get a proposal together that the National Science Foundation funded, which allowed us to start building out new ways of doing inference on astronomy data. And that turned out to be a very fruitful place for us — for me in particular — to learn about the landscape of what other techniques were out there that we hadn't been taught in school.
Sam: Can you tell us a little bit about, from that initial discovery, what the research arc looked like? What were some of the first things you started exploring, and how did that evolve over time?
Josh: I really just started looking at toy amounts of data that we already had in the can, and we could start applying these different techniques and looking for tools that would be useful for us. Really the best thing out there at the time that we started was something called Weka, which was — and still is — a collection of machine learning algorithms that one can apply in a GUI, graphical way, all written in Java. Really, that was our original playground and benchmark, and we used that as a launching-off point to start understanding what are these different modeling techniques that are being exposed here, what is a support vector machine, what does it mean when people say random forest — and we used that as a way to educate myself and those in our group. And we started seeing some interesting results — we started seeing accuracies that were better than what you could get from random. Then as we poked farther and farther, we wound up seeing how far can we take these algorithms, how well does one of them work relative to the others to get the kinds of answers that we want, how do we build in a loss function — which turns out to be very important to get good answers, because in the case of what we did, when we're discovering something in the sky, it's not easy to articulate that loss function. And by that I mean: what is the cost of missing an interesting place in the sky? It means that you don't get to do new science. Versus what's the cost of saying everything in the sky is interesting, which means you burn all of your follow-up resources. And then we started to think about context-aware classification — now not just in the context of really just resources, but now time constraints: making an inference about something that could be of interest may be more important than waiting to get another couple of data points and saying something with even more confidence. So understanding how to calibrate confidences and probabilities, doing this in the presence of missing data and irregularly sampled data in time — all of these also wound up showing to us that there were parts of the machine learning sphere, at least in the academic world, that were not often exposed to the kinds of data that we were exposed to. And so noisy data, for instance — no one ever, when they talk about the iris dataset, says the petal of this is red plus or minus purple. So even just having uncertainties in your features, let alone your labels, became an interesting challenge, and we wound up realizing perhaps there were some new techniques that we needed to start innovating on to even do the kinds of inference we wanted to do with our data.
Sam: You mentioned the loss function and needing to wrap your arms around what that means. Can you succinctly describe how you grappled with that? How did you approach it, and what did you end up coming up with for the types of data that you were looking at?
Josh: One of the things we wound up realizing is that one person's loss function is not the same as another person's loss function, and so to get traction on your answers, one needs to at least be clear about what it is that you're optimizing for, and at least give people the ability to imbue their own loss functions. If, for instance, you're producing a catalog of different types of variable stars on the sky, we have a specific notion of what it means to get something wrong about, say, a very minority class versus a majority class. I wouldn't say that we solved that problem by any stretch, but at least we were trying to be clear about what our assumptions were of the loss function and articulate what it is that we're optimizing for. When people are doing AI or machine learning in a production environment, there is always going to be an optimization of some sort, and the typical one people will go to, without knowing exactly what the business value is, or scientific value is, of the answer, is to go for some notion of inaccuracy. And then when you get a level deeper in that, you say, well, what I really want to do is minimize false positives at a false negative rate of 0.1 — and that is an implicit statement of what your loss function is, and you hope that by defining it that way and by optimizing on it that way, you're actually getting very close to an optimization of the result of what you're emitting out of your modeling.
Sam: And so you're primarily looking at image-oriented data over time. Are there other fields where you've seen them adopt the same types of approaches to what you were working with?
Josh: Well, one of the nice things is you can work at the sensor-level data, which is effectively photoelectrons in a CCD, and counting those up as a function of position in x and y, and then trying to map that back to the sky — so that's what you might call noisy image sensor data. We worked at that level, but then we also worked at a metadata level, which was: now let's use traditional astronomy techniques to extract the brightness of a star as a function of time. And so we got ourselves out of the image plane and into the time domain, and then we're basically working with effectively tabular data. And again, there are lots of different models and feature engineering approaches that one can take to all of that. I wouldn't say that there was a common thread in our work across a bunch of these different sub-questions, other than to say that over time we wound up realizing that there were only really a couple of different machine learning models that did as well or better than everything else. And so even though, for instance, support vector machines are very popular because they have some great theoretical, provable properties, they tend to be kind of unwieldy, and for dealing with the kinds of data we were working with — which is heterogeneous, noisy, dirty, sparse, missing, and multiclass, where you needed to also get probabilities out that you could then calibrate — models like support vector machines really fall short for practical purposes. And so we wound up recognizing in our group — and I think that was validated in a conference that we ran at UC Berkeley on essentially streaming inference with machine learning; it was a week-long conference that involved folks from Netflix, folks from Google, and then domain experts, everything from biomedical to physics — a number of people would stand up, give their talk, and say, yeah, we wound up realizing that decision forests pretty much always won. Now, this was in 2012, before the resurgence of deep learning. I bet if we ran this conference again, half of the talks would be about how that's a better algorithm, as it were. But it was pretty eye-opening, and it was one of the things that we took to heart when we wound up starting the company: a recognition that to succeed, to produce value, very generally, the algorithm itself is not necessarily the key. In some sense, the way I view this now is that algorithms, and the accuracy that they can produce, and the ability to optimize them around a loss function, is really only table stakes for the utility of these in a real environment. So yes, you need to use a model that's very, very accurate and potentially can be retrained and get slightly better, but as most data scientists or most people that work in machine learning workflows will say, almost all of that work is in feature engineering — and if you're a deep learning person, you'll say almost all that work is in figuring out what the shape of the network should be and iterating over that.
Sam: On that note, before we jump into what you're doing at the company today, what were some of the results you saw out of your research on the astronomy side?
Josh: We looked at a couple of different realms. One was looking at large catalogs of variable stars and coming up with probabilistic classifications of what type of variable stars they were, what was the physics that drove them. And we did that in a bootstrap way, starting with effectively a few hundred known classes and a few hundred or few thousand known labels, and then extrapolated that to tens of thousands, hundreds of thousands of variable stars and produced probabilistic catalogs. One of the things I became adamant about as we were doing that was that producing a catalog where you say, hey, this object in the sky is of this type with this probability, is effectively useless unless it's then used for some new kind of science. And one of the things that I became — I won't say frustrated with, but I noticed often, is that people started using — not just in astronomy but in many other fields — machine learning as an end unto itself, saying, I'm going to apply machine learning to this data and I'm going to get a result. Until that result itself is novel, or until that becomes a stepping stone to another result which becomes novel, it's sort of an empty exercise. And so what we wound up saying is: what can we do with this probabilistic catalog that couldn't have been done with any other means? So one of the things we did is we looked for very strange types of stars that had certain properties and then followed those up with big telescopes and actually wrote science papers with those.
So we used that as a launching-off point in a real-time environment, where we actually were looking at images as they were streaming off of telescopes in Southern California, off of Palomar Mountain. Every 60 seconds or so it would get transferred up to Lawrence Berkeley National Lab, and we'd apply our machine learning to that to find new interesting objects in the sky and then populate databases of, for tonight, here are the interesting objects. And then we also had another machine learning code which would go into those databases and periodically make statements about what types of objects those might be, what they could be. And we wound up having, I think, of order 100, maybe 200, papers that came out of that — refereed papers — which, again, the machine learning part of that was really the stepping stone to discovery. The other parts of the machine learning were the stepping stones to initial inference, and obviously in the end you needed people to actually write the paper.
Sam: But it goes back to the grad students, right?
Josh: What's that?
Sam: It all goes back to grad students.
Josh: Exactly. But I really thought about removing people from the real-time inference loop and getting as far up the inference stack as we could. We even got to the point where we were finding interesting objects in the sky without any humans in the loop, identifying that not only is it a new object, but it's something we probably should be following up, and we were issuing alerts to robotic telescopes to go follow those up. So by the time people woke up in the morning, we not only had the discovery, we not only had the initial inference, we then also had real follow-up — scientific follow-up. One of the, I think, great achievements of the work that we and others did in one of our collaborations was to build this production system that had real consumers on the other side of it, and when it was broken or was wrong or didn't take feedback properly, we'd get nasty emails from our collaborators saying, your thing didn't work for an hour, you kind of screwed my science while I was at the telescope. So having an end user really keeps you heavily focused on making sure the things that it needs to do, it does right and robustly. But because we were discovering things even faster than a whole army of grad students would have been able to pore over all of these images, we were able to find, for instance, the nearest Type Ia supernova that had been found in 25 years, and get a whole bunch of people looking at that part of the sky and taking lots of data that led to a bunch of papers in Nature and Science.
Sam: Wow.
Josh: Not because that object wouldn't have been found by even amateurs — because it got so bright you could have seen it with binoculars eventually — but because the interesting science happened hours after the event blew up. And so it wound up also driving home for me the need for not only something that's working and is robust, et cetera, but where it's able to make statements quickly and do it in a way that's reliable.
Sam: Interesting. I'm sure that has led you to some interesting perspectives on the relationship between this technology and society and jobs and stuff like that. I'm hearing parallels to — a lot of people projecting that as AI is deployed, shifts in the job market will take place that put a lot of people out of work. But I'm also hearing in your example the counterargument you often hear, that really what it does is empower people and allow people to do different things. I don't necessarily want to go deep into the society stuff at this point, but —
Josh: Yeah, it's certainly a valid concern. What we do in our company at Wise.io is help customer support agents become more efficient at their work: by suggesting answers of how they can respond to an incoming inquiry, by automatically triaging incoming inquiries or tickets, emails, et cetera — that is, getting them to the right person or the right group who's going to be the best at answering that question — and then in some cases we will automatically respond to incoming inquiries. So when you write into an e-commerce site and say, my package didn't arrive, there's a growing chance that us or somebody else may be answering what looks like a bespoke question of yours with what looks like a bespoke answer — in the end it's just a templatized response that we ourselves are using. For us to be able to do that — obviously we can talk more deeply about how that works from an AI perspective — we have to get very confident in what our answers are. But what does this mean, on one side, to your question about labor displacement? Companies don't need to hire as many support agents — so where would they have gone? The other side of that is that the agents that they do have become better and more tuned at working on some of the harder parts of what their own products are about, what their customer complaints are about, in a way they wouldn't have been able to because they would have been distracted by the mundane. So if you say, how do I reset my password, and there's a person or sets of people that have to look at that email and decide how to respond, that's time that those people are not spending on really complex problems where empathy is required as well.
So we think of our product and what we do as a way of freeing people to work on the things that people are uniquely suited at, that machines really aren't going to be that good at until somebody solves the Turing test. Chatbots are not going to be able to understand people in the subtle ways that they need to, but we can take a lot of easy stuff off the table — we have. And so there was certainly a concern as we were starting to roll this out that we were part of this labor displacement movement, but we heard time and time again from our customers that their support agents became more and more happy the more involved we were. There was an entire team in Asia who had been tasked with basically reading an incoming inquiry or ticket and then deciding who else should be reading this to solve the problem, and because our triage capability came into play, one of our clients effectively deprecated a 20-person team off of triage, because we're effectively automatically triaging now. And we were worried — what's going to happen to these people? They have families to feed. And we got a whole bunch of really great quotes from them saying that because they had been reassigned to actually work these support tickets rather than push them along, they were much more happy in their job.
Sam: That's fantastic. So we jumped right into what you're doing at Wise.io, but the transition is a fascinating one as well. How do you get from astrophysics to a software company doing CRM stuff? And I know there was an intermediate step there as well.
Josh: Going back to the original part of the conversation, we had recognized in the team that I'd built and the people I'd worked with that, A, we had some great technical orthogonalities — some were good at software engineering, some good at ML, some at UI, et cetera — and, B, that what we had learned to do — recognizing the importance of putting AI into production and having real end users give real feedback in potentially a real-time loop — was something we at the time didn't see anyone else doing. We knew obviously that the Googles and the LinkedIns and the Netflixes of the world had this kind of baked into their overall data flow, but we certainly weren't seeing companies helping other companies do it. And one of my now co-founders had, more or less while he was between jobs, figured out how to make one of the algorithms that we liked a lot — these decision forests — scale very, very well, at least on a single machine, in a multi-core environment. And so we realized that we might have some interesting firepower. And given that there seemed at the time to be so much emphasis on massive-scale machine learning — it was certainly the pre-Spark era, but very much in the Hadoop heyday — it looked like most of the interesting ML that was starting to come out, and some of the other companies that were coming out, were really focused around helping the — I won't say exascale, but certainly petascale-level, Google-scale amount of data companies bring machine learning into their workflows. So we thought about skating to a place — using the analogy that's heavily overused — to the part of the ice where the puck was going to be, which was helping smaller companies and midsize-scale companies bring machine learning into production environments. And that was the impetus for starting the company. What the domain was going to be, we didn't know. Who the customer was going to be and who the buyer was going to be, we didn't know. We were, I'd say, blissfully ignorant about all the business challenges that we would wind up encountering over the next couple of years in bringing this to market.
And when we emerged out of our first accelerator — it was the Alchemist Accelerator — I gave a talk at our demo day where I said we're going to be GitHub for data scientists, and produce some interesting UIs of interactions to help data scientists like ourselves more easily build models that they could then push into a production environment. We wound up seeing over the next couple of months the challenges of selling products like this into data science teams. First, data science teams were few and far between, and the ones that existed were either too sophisticated, believing they could build it all themselves, or not sophisticated enough to get a large enough budget to pay for the things that we wanted to provide them from a tooling perspective. All the while, we were building out our underlying platform to be able to do exactly that: to be able to build templated machine learning models against certain types of data for certain types of use cases, and then, even though you built it at a laptop level, push it into the cloud and have, in Amazon or other compute frameworks, the scalability to be able to serve large numbers of customers around those same use cases — where what's emerged for us is that the difference between a customer is not new code, it's just a config file, if they're using that same use case.
So, all of that to say that we evolved — you could call it a pivot if you'd like, but I think of it as a series of pivots — to a place where we wound up seeing in customer support a lot of data, a lot of manual work, and some really nice CRM systems with open APIs and a fairly fixed schema. So the Salesforces and Zendesks and ServiceNows of the world really are the data lake and the transactional layer for doing customer support and related activities. And we thought if we could build now an intelligent system on top of that and do all the things that I mentioned — empowering these agents to become more efficient at their job and making the whole support desk more efficient at serving customers — we would solve a bunch of pain points. And as we wound up going into the market and started leading with products that could be more or less installed by a non-technical user and could be used by a non-technical user, we wound up getting a lot of feedback that indeed we were solving some pain points. There's obviously the efficiency question of needing less headcount, but there are also some really interesting customers of ours who were growing very quickly, and one of them said to us that if the CEO had given him an infinite budget, he wouldn't be able to hire good customer support agents quick enough. So helping them capture all the internal knowledge was something that, it turns out, machine learning actually does quite well at.
Sam: What were some of the biggest challenges in going from a product direction focused on generalized tools and platforms to one focused on a very specific application area?
Josh: Interestingly, it was all the things that we hadn't thought about, which was product management, and how do you get structured feedback from customers, what does it mean to build an MVP, roll that out, iterate on it, et cetera. A lot of Lean Startup 101 stuff was something that we hadn't really been thinking of when we started the company, and certainly didn't have, frankly, a lot of expertise in. And then as we started scaling, it was a recognition that there are large parts of a machine learning pipeline that don't naturally scale. So figuring out ways to containerize the parts that need special attention from PhD-level data science, and abstract that away from other parts of our engineering group that don't need to know about what's happening deeply, but need to be able to ask predictions of some other part of the stack RESTfully, in a services-oriented way — we just wound up realizing that what worked for us from a scaling perspective, a compute-scaling perspective, also wound up being what we needed to do from an organizational and HR perspective. When we hire front-end engineers and middleware engineers who are great at writing scripts against databases and managing Redis queues, et cetera, those folks don't need to know about machine learning. They need to know that there is a contract between their part of the stack and somewhere deeper in the stack: that if I ask you for a prediction for this client, for this model, I'm expecting to get it back in this format on this timescale, and if I don't, then our contract's broken. But likewise, I'm going to hand to that deeper part of the stack that's going to be providing those predictions effectively some data, in JSON or otherwise, that will have a fixed schema, so that the group that built that machine learning pipeline knows that this column is going to be of type datetime, this column is going to be an int, and it's going to join using these four indices on some other data. So once we wound up realizing that we could lock down the schema for a given use case, it meant that we could write data science pipelines against data we hadn't seen before. You need to see it once to make sure it's all working, and make sure it cross-validates in an offline sense and has the kinds of accuracy properties that you want, but then it means that we could start spinning up new customers where they get the base template that operates on their data, and when we need to make changes, those can happen from a more or less technical person rather than somebody with a PhD in statistics. So there were a bunch of challenges around that, and as we started solving those, it just sort of fell out that our stack really mimics what our organization looks like.
Sam: Okay. Can you talk a little bit about the data that you typically see in a customer environment? I'm imagining just loads of trouble tickets, but I imagine as well that there's ancillary data, supporting data as well. And you mentioned that there's lots of it — can you talk about the size you typically see, those kinds of things?
Josh: Our typical customer is doing of order five to 20,000 tickets a month, and we need to be working with companies that are achieving that level of volume — A, because the price points are reasonably high, and so it's typically the companies that have those large volumes that are willing to pay for what we do, and, B, because the machine learning models are built specifically for and on their data, and we don't use a common model, for instance, across our customers, so we need a lot of training data for a given customer. Now, again, this is not petascale amounts of data — we're talking tens to hundreds of gigabytes at the per-month level, per customer. The data is indeed a lot of human-to-human interaction, from emails, web forms, even chat. And there's also a lot of metadata: what is the value of this customer, what products are they using, how often have they been emailing — so there's a time-series component to this as well. And we've had to build these pipelines that are generic enough that we can then apply them to other use cases, but specific enough that they give good enough accuracies that wind up rivaling what humans can do. And so oftentimes our goal is to get to — we don't call it accuracy, we call it matching capability, because oftentimes when a human's labeling something and saying it belongs to this bucket, or this person should answer it, or it should have been answered with this template, they oftentimes can be wrong.
Sam: I think we've all had that experience.
Josh: We want to get ourselves to that kind of level of quality, let's say. So from a featurization perspective, we're doing lots of natural language processing, getting it to the point of rectangularization, a grouping in semantic space of outgoing tickets — that is, how agents are responding — that don't look like templates that are sanctioned by the company, which means that they're coming up with their own templated responses and potentially even sharing those with other agents. So we have a dashboard, for instance, that we show our customers — the ones running the support desk — of potentially new templates that they can use, because obviously if there's a new issue, for instance, with a product, then agents who are on the ground have to figure out a way to answer it, and if it's a recurring problem, within a couple of emails they will end up essentially having the right answer that they've already pre-formulated. So that's an unsupervised problem.
Sam: And do you see in that last example a future place for generative types of AI approaches, or is that more — do you think when you hear that, is that like the technology chasing the problem kind of thing?
Josh: Yeah, it's a good question. We've shied away from the generative component, and in fact we make that a big part of our sales pitch, to say: you, our potential client, really know the voice that you want to speak in and speak with your customers, and who is it for us to come in and say we're going to autogenerate — at the character level, CNNs or something — a bespoke answer, the way that Google Inbox does? Well, if it gives you five different words — sure, sounds good, I'll see you then, or how about Friday — those are fine. But because we're really focused on not just getting results into the hands of agents — where they can actually see, in a UI sense, within the dashboards they normally see, what our predictions are, and consume it in a way that they like to — we also want to take a lot of these tickets off the table in an automatic sense. The only way our customers get comfortable with that is if we're showing to them in an offline way: here's our accuracies for these types of templates. So every now and then somebody says, I'm very unhappy with what you've done, and we're going to send, thanks for your feedback, when it should have gone a different path — but we're only going to do that 1% of the time, at this level of false positives. And once we can do that, then our customers essentially can turn on a specific macro for us to auto-respond without any agents in the loop. To do something like that to potentially irate customers is pretty challenging, so — we certainly won't rule it out, but we certainly don't think about ourselves as producing generative answers in a bespoke way. We're just more or less turning all of our problems into multiclass classification problems of: which of the hundreds or potentially thousands of canned responses is the right one to answer with?
Sam: Okay, and just so I understand the comment that you made a second ago, in terms of sending out a given response a small percentage of the time — are you describing an error type of situation, or are you describing a feature, like an exploration type of feature?
Josh: Good point. There's an explore-exploit component to what we do, in a multi-armed bandit sense, that's typically not exposed or a knob that's tunable by our customers — so that will happen, and some of that will happen naturally. In the case of auto-response, we hold back 10% of the ones where we know what the answer is, or we believe we know with a certain threshold of confidence, and then compare after the fact with an agent who wound up now having to see it, because we didn't auto-respond, implicitly. No — the one I was pointing out is ones where we are essentially wrong. And that gets back to the question of the loss function, of what does it mean to be wrong. If one of the canned responses is, I'm so sorry for your loss, I will refund your entire vacation in the amount of $10,000 — the cost of being wrong on that is very, very high. But if somebody is mad and says, my vacation got ruined because of something you did, I want my 10K back, and we say, thanks for your feedback — being wrong on that side is not nearly as bad as being wrong on the other side of that. And so we give and empower our customers to make the decision about, let's do the easy stuff where the cost of being wrong is not a big deal. And that's for the automatic response, but for the recommended types of responses, if our first canned response is, here's your money back, and an agent looks at that and says, no, that's crazy, the right answer is farther down the list, they'll select that, and that becomes the feedback — our models wind up getting better as they wind up learning over time.
Sam: What are some of the most interesting challenges that you've run into in putting together this kind of hybrid ML-plus-human solution? One of the things that pops to my mind is just user experience, user interface — are there challenges there that are interesting, or what surprised you the most in trying to field these types of solutions?
Josh: Certainly. Because we're getting into the space and the face of agents who do this all the time, when we first started releasing our products, we didn't have a good training program for them. And so when they would see what we thought was an intuitive set of responses in the form of widgets that would show up on their desktop, they didn't know how to consume it, and they didn't know how to use it as effectively as we thought they should. There's all the mundane stuff around UIs, like responsiveness, and somebody saying, well, it doesn't look like your product's working because now there are no responses, and we'd say, well, that's because you've already responded — you're bringing up an old ticket that already has a whole conversation, and we're only getting involved, at least for now, in the first part of the conversation: what's the first response you should do. So then we weren't showing the results — and so how can we modify our widgets so that the agents understand we're not showing it for a purpose, it's not that our system is broken?
And then realizing also that many agents wanted parts of our UI — and UX more generally — that don't have anything to do with ML. So they wanted keyboard shortcuts, because we thought everyone would just click on stuff, but high-velocity support desks want to just use the keyboard. So we had to build that in for a set of customers, because essentially it was like the mouse had a disease on it — they didn't want to touch it. Getting feedback from the UI itself back into our system, making sure that we're getting the right metrics back, making sure that the KPIs that we're measuring are aligned with the KPIs that our customers wanted. I think one of the hardest things for us — and it frankly continues to be a challenge — is really just thinking about how fault-tolerant ML needs to happen. And again, going back to Google Inbox, for those of you that have used it: it makes a couple of suggestions about how you could respond to an email, and if you don't want to use those, you don't use it. So I would call that a great fault-tolerant ML experience. And the same thing in a spam filter within your mail system — it'll say, we think this is spam; if it's not, move it over, and then later on we'll figure out how not to call these things spam anymore that are like that. That sort of fault tolerance, where you're also getting feedback either implicitly or explicitly, is just something we've had to build up over time. But I think more broadly, that kind of approach needs to be built into any AI system in a production environment. Unless the AI outputs that you're building are going to be consumed entirely by machines, you need to have some level of understanding of who it is that's going to be consuming it, what are their concerns, and how can they give you feedback, so that your models will end up getting better over time.
Sam: Can you talk a little bit about the algorithms that you're employing and the toolchain, the pipelines? What does all that look like?
Josh: We stay out of what we call internally the algorithms arms race — we're not really selling the platform to other data scientists, so it gives us the freedom to focus on parts of the pipeline that we find most important. All of our algorithmic learning parts and then prediction parts are built in C++ and then surfaced back out into Python, which is where the data science team winds up working. We have our own notion of what a pipeline needs to be, and the data science team works entirely within the confines of what that pipeline ought to be. Which is: some sort of pre-filtering — so, for instance, if a ticket is from a voicemail, don't predict on it or don't use it for a build; so get rid of those that have, in this column, this value. Then there's the data transformation parts of that, and the joining across multiple data sets if that's needed, and then the featurization, which will often use open source tools in the Python ecosystem — pandas is one we use very regularly. And then once we wind up realizing that we've created a bottleneck — which typically will happen not so much in time but in RAM usage — we'll wind up rewriting other people's algorithms or code so that we create a RAM-efficient pipeline. And then once the featurization happens, the learning winds up happening in the C++ layer, and we've built a whole bunch of hyperparameter optimizations and feature selection capabilities, and then post-processing capabilities to get calibrated probabilities out of a multiclass problem. So we have a bunch of pieces that we've been building up that are not in the open world — it's something that we've decided not to open source for now — that allow us to work efficiently. So we think of it as high-velocity data science in building out a template for the first time.
But then, because the models have to rebuild every single day for every single customer on cloud infrastructure, which is not super cheap, we needed to make the cost of doing that as small as possible. And what we wound up realizing is that open source tools that many people use, like the scikit-learns and the TensorFlows of the world, even Spark ML, were vastly more costly to run — even if you could do it in the same amount of time, which we think we're much, much faster than most of those tools — because of the RAM requirements needed on multiple machines, or even a large single machine in Amazon, the cost of building a model just was X% higher — X being in the thousands. So having a RAM-efficient, speed-efficient, and — obviously, again, getting back to the original conversation about table stakes — highly accurate set of algorithms, which produce the kinds of answers we want, that we could then get into and modify if we needed to, was where we wound up settling as where we needed to spend our R&D engineering time.
Sam: Now, one of the areas that many of the machine learning platform companies have focused on is trying to close this gap between data science and production, and in essence eliminate the, hey, I've got this model that kind of works, throw it over the wall to developers and have it implemented. And it sounds like you guys have maybe embraced that, and you're using that as a way to build out the models in C++, presumably for performance. Are there ways that you've then compensated for that in terms of automation tooling, or do you just accept that, or even — we just have the best people on both sides of that fence that can deal with the existence of the gap? How do you maintain a level of efficiency and innovation in terms of the development pipeline — not the machine learning pipeline — so that it all works for you?
Josh: There's definitely this separation of concerns, which again is both an organizational one and also a computational one, to the level where we often talk about what we call the organizational API of who within this stack is the customer of whom. And so, for instance, the people who are the core ML and algorithms folks in the company are working in C++ and surfacing the great results back into a Python layer. Their customer is the data science team. The customer of the data science team is the people working on our architecture, who have to maintain this scalable, robust infrastructure, and their customers are the people working in the middleware, and their customers are the ones who are in the UI. And so each of them has a set of contracts of what it is that each part of that stack is looking for and how, in fact, they're supposed to engage with each other. And that's become very, very helpful for us, because what you find is that when you put somebody in a box, they figure out a way to innovate very highly within that box. So if there is a very strong contract of what data is expected to come in and what data is expected to come out, and everything in between there is really up to you to decide how to do well and efficiently — that's where, for instance, our data science team and implementation team will wind up working in building out a new template. They can work at their laptop — or glorified laptop — level on a toy data set, get some confidence that the pipeline is working, offline accuracies look good, and the whole thing is going to work. And once they're comfortable with that, they literally are just pushing a new version of a Docker image into our registry, which then is farther upstream from anything they ever have to think about from a production sense. Once a new build winds up getting kicked off for that customer, for that type of template, the new image will just get pulled and it will just get built with the config file for that customer. And so the data science team can wind up working within their confines, and of course we have a whole testing suite to make sure that if they build something new, they're not going to break something downstream from them, to gain confidence in that. And then they're literally just pushing the results of what they're doing on a semi-weekly basis into the Docker registry; that becomes the latest template for, let's say, triage, and then all the customers in production are automatically migrated to that. So having the data science team be able to push stuff into production without having to be on the ops side of things, nor have to think about the architecture, has really freed us up in great ways, I think, to innovate. And likewise, when they need a new bell and/or whistle from the core algorithm folks — because they say this part of our entire build chain is really inefficient — they can ask the people working on that to improve it, and they go through their own testing suite. And I think we're at 300,000 regression tests in our core ML, where we're also testing against every open source algorithm on customer data to make sure that we're staying as efficient or more on all these different axes before we cut a release. Then the data science team can just pull essentially a Python egg from our registry and use that in their system. So having those separations has been great. Obviously, if you're abstracting everybody from what the end use cases are, there can be a huge danger, but it's the job of people like myself to make sure that everyone is focused and innovating towards the right set of goals.
Sam: Great. I'm glad that Docker came up. You guys publish and maintain a set of Docker images for data science tools — I've come across that. My impression is that in general, Docker adoption within the data science, machine learning community is not particularly high. Is that yours as well?
Josh: We certainly haven't heard of many other companies using it in the ways that we are, but it seems like such a natural way to literally containerize and abstract the work of one part of an organization from the other — so long as that container will respond with a slash-build, predict endpoint, feedback endpoint, et cetera, in the way that everyone expects it to. I think that's a wonderful way to do abstraction. And then obviously it also helps you wind up achieving scale, because for us, scale is not, can we serve a billion of our customers with the same app? It's instead, well, we've got a new customer — we just spin up more containers to do the builds and the predicts for that customer, and if we need more compute capability, that's elastically scaling for us for free on top of Amazon. So I think of it as a very natural way to separate concerns from a stack perspective, and also a very natural way to do what is, for a company that's serving lots and lots of customers, a very embarrassingly parallel type of compute.
Sam: Interesting. I got into a conversation on Twitter or Reddit or someplace where someone was griping about just the dependency hell with Python and pandas, and trying to come to terms with managing different versions of different tooling versions and things like that. And I suggested — I might have even pointed to your Docker repository — and the response was, no, I want to make this simple, not more complex. And obviously you find it to be simpler. Can you give folks that aren't familiar with Docker and containers your 30-second Docker-for-data-science pitch, and where they can learn more about it?
Josh: Docker is a way of explicitly specifying not only what your, let's say, Python requirements are, which you can do with a simple file, but also what the entire OS shall be for running whatever scripts you're going to need. And once you build that, and you gain confidence that that image is doing what it ought to, you can essentially very rapidly turn a container on that is the almost instant instantiation of that entire OS, plus that script and all of the dependencies built inside of that. And you can hand somebody a link to the Docker Hub registry — or, if you maintain your own private registry, an explicit URI to that explicit version of that explicit image — and more or less guarantee that when they run that, with whatever data is contained inside of that, or whatever will be pulled over, so long as it's the same, you'll get the same answer out. So I tend to think, from a data science workflow — and then, getting back to just doing science more generally — Docker is a very nice framework for reproducibility. And so the idea is that now I don't have to share a machine with you, or an Amazon machine image with you — I'm just handing you effectively a Dockerfile that says, if you run this, you're going to wind up getting the same answer that I got. But again — doing the types of work that we do at Wise, and in some cases what we do on the science side of things, the final result is not what comes out of the Docker image or container. It's not, okay, here's a report of what my ROC curve is going to be, my false-positive-versus-false-negative curve, and then let me write a paper about that. It needs to be, for us at least in a production environment: now I've produced a prediction that now needs to get consumed by something that's farther downstream. So Docker is quite nice in that sense as well, because you can also now connect Docker containers explicitly, using something like a Docker Compose — there's many other tools out there as well — so that containers talk to other containers, and you allow each container to have, again, its own separated concern from the other ones, but still pull the results and push results to the other ones around it. In addition, some containers can just contain data, and you can build databases around that data. So now it allows you to build up a very lightweight version of what might be your entire stack, and do this in a way that's programmable. So we found that to be incredibly useful for testing purposes.
Sam: So is your GitHub the place that someone can go to learn more about what you're doing there, or —
Josh: Yeah, we've got a public Docker registry — you can go to the Docker registry and search for wiseio — or you can go to GitHub, wiseio, and see our other public projects that we've pushed out. So there's one around Docker and data science, which in that case — because we're not releasing any of our internal tools we've built — we're basically building up a container with open source tools that we find are really useful for doing lots of different types of data science. The other major project which we have up there in GitHub that's open is something we called ParaText, which started as just sort of a weekend hack from one of our engineers, Damian Eads, who wanted to see what it would be like to read data from disk in parallel, just to see what kind of speedups you could wind up getting. And it turns out pretty much every open source tool out there doesn't read in parallel, and the ones that do are explicitly parallelized, like over multiple machines — but if you just made use of the multi-core environment, how well would you do? And we wound up getting 100,000x speedups over some of the other tools that are out there, and importantly, also using vastly less memory. ParaText is not yet in our production environment, but we thought it would be a good example of showing off the philosophies that we try to adhere to within the company: of creating efficiencies that isn't just the one thing around accuracy, but around how fast can you read data, and how big is your model on disk — all these other aspects of what it means to do machine learning that have nothing to do with the algorithm. Once you're happy and you've reached some level of plateau with the algorithm accuracy, all that you're left to do is optimize all these other pieces of that pipeline. And so a lot of our engineering over the last year in particular has moved away from just optimizing accuracy to things like creating interpretability around the models that we build, making the model smaller on disk, making the other parts of the featurization pipeline be more RAM-efficient. And once you start playing whack-a-mole with, let's just say, RAM usage, you wind up finding really interesting parts of your entire pipeline that very few people wind up talking about — again, reading data, which should be the easiest part of your entire toolchain, is vastly inefficient, and whacking that mole, it turns out, you save a whole bunch of Amazon cost, because now you need a smaller-RAM machine.
Sam: That's great. You mentioned interpretability — have you spent a lot of time working on that, and what were the drivers for that?
Josh: We have spent a lot of time on that. It's one of what we think of as our trade secrets, around — getting back to the question of UI and UX for end users — we were asked often, at least in the early days, well, why are you getting the answer that you're getting? And you can't say, well, it's a thousand-dimensional feature space and there's covariance between all of these, and the model importances over the entire thing say that this is the most important feature — I don't know why we said for this one what the answer is. But that answer is what's called in the financial services world reason codes, and turns out to be really important — some places it's actually regulatorily required that you tell somebody why you got the answer that you got, even if it's a machine learning black box. And so some of our early R&D effort was around how to make, at the instance level — so an individual prediction level — how do we make these models interpretable, by saying these are the important features and these are what's driving this specific prediction. As an example, if you're working on customer churn and you want to predict somebody going to churn 90 days from now — it's a use case that we've also used on our platform, but not something we go to market with necessarily — two customers can have an identical probability of churning, but one of them may be churning because they haven't really used your product and they haven't done the training videos, and the other one may be churning because there's a high probability they're going to go bankrupt. In the first case, that's something you can do something about, and in the second case, you're kind of — and so even though they're identical in what their predictions are, and the probabilities of those predictions coming to pass, one is actionable and one isn't. And so it's not just people gaining a warm fuzzy about why did you get these predictions, and does it jive with my feeling about why that could be okay — which is critically important. It also then starts tying into next best action, because, I think again, an important part of machine learning in production is to drive value. If the value isn't the prediction in and of itself, then the prediction in and of itself is really just there to drive the next thing that happens. And so next best action is heavily coupled with the importances around which features are driving the prediction.
Sam: Okay. You mentioned value, and that's a great transition to one of the things that I really wanted to dig into with you, and that is this blog post that you wrote about cost-optimized AI, that I've incidentally mentioned on the podcast a couple of times. Do we have time to go into that?
Josh: Of course.
Sam: So I guess the first — it's actually come up several times in our conversation already, this notion of cost and value, but was there a specific thing that prompted you to, I really got to write this down now? What drove that?
Josh: That was a bit of an intellectual journey. I was wondering, to be really frank, why the hell are all these companies building these neuromorphic chips and all these specialized hardware to do deep learning — because I think much of the world's data and much of the world's value in data is tied up in — I'll use the word, quote-unquote, small data or medium data, not massive-scale, Google-scale data, Facebook-scale data. I was wondering why all these people are starting to build these very specialized pieces of hardware when deep learning — I think magnanimously one could say, or charitably — is incredibly good at a large number of inference problems, but not very good at a large, probably even larger, space of inference problems. That may be changing over time as people start applying it to these new realms, but the place where deep learning winds up shining is in really large amounts of data, because effectively what you're doing is turning millions or even billions of knobs to optimize a model, and to do that credibly without overfitting, one needs lots and lots of data.
So I wound up asking this question of myself: why are people doing this, and why isn't what we already have out there, even just in GPU land, good enough? And if you look at a plot, which I have in my blog post, of the efficiency — gigaflops per watt, which is something of, if I put this amount of energy in, which has this amount of cost, how many computations can I get out — that efficiency has been growing over time, but it's nowhere near what some of these other chips or these specialized pieces of hardware can do for these specific types of calculations. And those themselves are nowhere near what the human brain can do, which is of order, if I remember right, about 10 to the 5 gigaflops per watt. So your brain is a 30-watt supercomputer, unrivaled — at least for now — by anything else that's out there, and anything else that's out there is likely going to take megawatts or hundreds of megawatts to get anywhere close to that computational capability.
Sam: Incidentally, I don't know if you've come across it, but there's a parallel to using DNA for storage, and the storage density per unit energy is incredible in DNA.
Josh: Yeah, something like a drop in a teaspoon or something — it can take all the world's data. It's incredible. So getting back to this — that's an obvious one. And I started thinking about it when AlphaGo had its big set of results, the national or international championships, and you wind up looking at the computational capability that it took to win those competitions — it's just huge: thousands and thousands of computers, thousands and thousands of GPUs. The amount of power required there was several orders of magnitude larger than what was going on in the champion's head that it was playing against.
So I was thinking about that vast gulf, and wound up realizing that the companies that are pushing towards these specialized pieces of hardware — it's because they realize that for a given amount of time and a given amount of data, because these algorithms are all basically saturating on near-perfect answers, the only thing left to do is to get more energy-efficient machine learning for building. And the step after energy efficiency, when it comes down to it, is really cost efficiency. And so my takeaway on that part was that people are building these chips because that's sort of the last frontier of squeaking out and eking out the last amount of dollars coming out of the system for the number of dollars going into the system. And then taking a step back from that, I wound up realizing — or at least realizing for myself; it's probably obvious to most out there — that because machine learning is optimization, your good optimizer will find the optimal answer by definition, and if you're not writing down the function that you're trying to optimize — to get a minimum of or maximum of — in terms that actually matter, then you're creating, by definition, a suboptimal answer or system. And now that system doesn't just involve, is my algorithm more optimal at getting an accuracy better than yours, but now translating the accuracy into — well, let's go back to our loss function: what's the cost of being wrong in saying this thing is A and this thing is B? Translating that to a business term is something that's critical, and almost everybody knows that that's important.
But then you wind up realizing, well, if I'm going to build a model, what if it takes me 12 days to build one of these models, to get an accuracy which is only epsilon better than one that takes me 10 seconds? And what if I can build a model that may take 12 days, and the accuracy is much higher than one that took me less time, but the labor costs are very different? So I had to spend more data science time building one versus the other. And what about the opportunity costs of those data scientists not working on another problem in your business that may be more important? And when you wind up couching the problem that way, you get out of, again, just focusing on accuracy in the algorithm, to what is my cost of doing the entire pipeline. And now the entire pipeline isn't just running a machine learning model in production for this specific use case, but how does that couple to all the other things you're doing in your business? Are you hiring a data science team to do this, and then paying pensions? Or are you going to do a third party to do this and just write a check one time? And then, what are the societal benefits of all this? It becomes unwieldy at some point if you're actually being very honest about what's the cost of doing this. But at least I just wanted people to start thinking about, as we've started thinking about within our company, that accuracy is the table stakes. Let's assume that you all have your good algorithm that's going to do well — is it going to have strong scaling properties, so that if you needed to get the model built in X amount of time, you could just have N number of machines that get you X-divided-by-N amount of time on the clock, because maybe you need that model built very quickly, very often? And then questions around pipeline and RAM usage and AWS costs — in the end, as a small startup, when you start getting down to the brass tacks of what's our revenue and what's our cost of goods sold, what's our COGS — the COGS component is really, what does it cost to build a model and predict? And until we were able to boil down the fact that the cost per prediction for one of our customers is X, and we're going to be making X times some number — everything else is sort of moot. If you're losing, every time you make a prediction, effectively hand over fist, then you've got something wrong — that's unsustainable. So I started thinking about it as we were going through the exercise of, what's our cost of doing business? And the cost of doing business is running an AI system in a cloud with real customers. The labor part we can get, but all the other pieces — in the end, there's an Amazon bill, and because we put it all inside of Amazon, and we know how much money we're taking in, we can see how those two things relate to each other.
Sam: So you started with the question and ran through the thought exercise — what's next there? Whether it's you or someone else that does it, do you see this evolving, or co-evolving with someone else thinking about analytical frameworks for thinking about this, or tools — whether that's a spreadsheet, or — it almost lends itself to a machine learning algorithm trying to figure out how to deploy resources to do the machine learning.
Josh: Yeah, it's a great question. One of the nice things about a blog post is you emit it out to the world and you hope somebody runs with it. It's been helpful in focusing, for me, my own thoughts, and as we drive those sorts of efficiencies in our company — and then, again, more broadly, in doing science, doing astrophysics: choosing the right tools, choosing the right skill sets and people, choosing the right problems to work on or not work on. Those are very obvious outcomes from me having thought about it and framed it that way. One of the challenges — and I think people may wind up being able to pick pieces of this up and work with it — is coming up with articulations of, essentially, what is the conversion term between that item in the entire optimization equation and dollars. So one I'll throw out there that I don't know the answer to is: what's the dollar value of interpretability? And once somebody starts getting some handle on that, then optimization takes its wonderful toll or approach, or at least shall lead to a great outcome, which is: once you can really put a dollar cost to all these different pieces, then I think you can do a real honest-to-goodness optimization. So I know what the dollar cost is, for instance, of needing a RAM machine of this size versus that size on Amazon — okay, great. But what's the real dollar cost of — and can I know — how much time it's going to take for a data science team to build up this template from scratch and then push that into production? And how many people do I really need on that? Is it good to have one data scientist or multiple ones? And so all those things I think wind up becoming really interesting over time, once people potentially even wind up agreeing upon what's in bounds for this equation and what's not. Obviously out of scope is, what's the probability that my machine learning algorithm is going to start World War III — probably not worth talking about. But something smaller than that, smaller in scope at the company level, is probably worth starting to get some clear understanding around.
Sam: So now we've maybe come back full circle to graduate students. Sounds like there are a lot of interesting research questions in here for a PhD student or something.
Josh: Yeah, I think for those in computer science thinking about systems optimization who are also interested in machine learning, this is hopefully some fertile ground to start thinking. The other statement, which hopefully is clear from what we've been talking about, is that doing machine learning for machine learning's sake really doesn't make sense. It's probably the last thing you want to do if somebody hands you data — you do it because you have to do it. It's painful, and to run it in a production environment, given all the crazy bugaboos that many, many people have talked about — there's a great paper from folks at Google — D. Sculley is the first author — called Machine Learning: The High-Interest Credit Card of Technical Debt.
Sam: That came up on my last interview as well.
Josh: Yes, I'm not surprised. It's an important paper. It's got, I think, no equations in it, but it's a whole bunch of important lessons about how machine learning systems tend to be very different than typical engineering systems. So there's a lot in there to get right, a lot of bugaboos there that people who haven't done this before tend to get wrong. But what you wind up realizing is that once you realize machine learning, or broader AI, is the right set of tools to apply to the problem that you have, what you'll often wind up finding, I think — at the graduate student level, in terms of graduate student projects that could be worked on — is that it's still very much early days for the types of algorithms, pipelines, et cetera, in dealing with real-world data. I've often said to my colleagues on campus that real data is not doing sentiment analysis on Twitter — and yet many, many, many papers saying my scaling algorithm is better than your scaling algorithm will wind up using that as a toy data set. The real world is not toy data sets. Yes, we need to have benchmark data to have a lingua franca of who's doing better on these different axes, but when you wind up getting exposed to real questions, you wind up realizing that all the stuff that people know out there in the academic world, that people write about and do Kaggle blog posts about, are not what you really need if you're being truly honest about what needs to get optimized.
Sam: That's great. So how does one manage being CTO at a high-growth startup and being an astrophysics professor? It's becoming increasingly common to see folks, particularly in the machine learning community, have professorial posts and do academic work, or do work in these research labs and things like that, but — yeah.
Josh: I've been on what's called an industry leave for a number of years, and so it's allowed me to have also that separation of concern. So not getting paid by the university, not having health care, has made it easier for me to spend all my time as need be on the company, while still maintaining the kinds of links that I think are important. As I started thinking about coming back into the university setting — obviously there are a number of things I've picked up in management, ideas and capabilities, and then also thinking about how to evaluate new technologies: when is it appropriate to bring this into your toolkit, or when is it appropriate to wait? Those become really practical uses that I can take with me. But then also, again, recognizing, as I was saying before, that there's a whole interesting set of problems out there that are not being addressed by pure academic R&D research means that I can also start looking for those white spaces to actually do some pure academic research around those. I'm particularly interested in questions around interpretability and how you put metrics on interpretability, and that's something that I think I benefit from — having come from, having felt the pain of, customers asking about that — that I at least have a fresh lens on that. It doesn't mean I'll solve any of those problems, but at least I'll have a direction of potential interest. So it's certainly a challenge, but I think despite the challenges, the benefits to both myself, the company, and the university and my students at the university far outweigh all the gray hairs that I wind up getting. I'm teaching a data science class — essentially a Python-ecosystem data science class — right now. It's aimed at graduate students, and the things that I've seen in the business world have really helped me hone that class, and I'm directly giving back to the students from those learnings.
Sam: And is that a MOOC, or is that available only to —
Josh: It is not a MOOC. Other incarnations of that class that I've done in the past are probably online somewhere in the iTunes sphere or elsewhere. All the material can also be found on GitHub, and then we'll hopefully post some of the lectures online as well.
Sam: Okay, great. So if folks want to learn more about the company or get in touch with you, what are the best ways for them to find you guys?
Josh: Easiest is drop me an email, and you can find that by Googling around. So I'll add that as a little bit of a bar: if you really want to find me, you'll have to do a little bit of work. You can tweet at me — profjsb is my Twitter handle — and we can do a direct message. Maybe that's probably the best way to get at me.
Sam: Right, great. Well, I really appreciate you taking the time. It's great to finally meet you in person, and I really enjoyed the conversation. I think folks will enjoy it as well and get a lot out of it.
Josh: Great. Well, thanks so much. Thanks for your interest.
Sam: Great, thanks. All right everyone, that's it for today's interview. Thanks so much for listening. I haven't asked you all to do this in a while, but if you enjoyed this episode of the show, please, please, please do these two things. First, share it with your friends on Twitter, Facebook, good old email, or however you share cool things with your friends. Second, reach out and let me know how you like the show, who you'd like to hear on it, and how I can make it better for you. You can reach me on Twitter at twimlai and at samcharrington, and you can email me directly from the contact page on the twimlai.com site. Thank you so much for your support, and catch you next time.