Filmed at a16z's 2014 Academic Roundtable: how machine learning became a discovery engine for astronomy, sifting real transients from roughly a thousand bogus candidates apiece and catching the nearby Type Ia SN 2011fe about 11 hours after explosion, and why running ML in production led to Wise.io and to products with machine intelligence baked in.
Recorded after Bloom's talk at the roundtable, Practicable Machine Intelligence in Science & Industry, whose slides are on that page.
“Machine learning has this wonderful generic advantage of being able to approximate human intuition and cognition, and obviously go beyond that.” – Joshua Bloom
“The gauntlet that I lay down in front of anyone is that you're not doing machine learning because it's cool and it's fun and you can learn something about the data from the past; you're trying to really use it as a tool for the future, as new data winds up coming in.” – Joshua Bloom
“Because the software exists that can actually sift through and look at that data as if it's essentially virtualized graduate students with a huge amount of domain knowledge, and do this at scale, it allows you to take more and more data.” – Joshua Bloom
“If your dad is a Viagra salesman who travels to Nigeria, who just won the lottery, you probably want to see that email from him, right? But most people don't, because those are all signals of potential spam.” – Joshua Bloom
“In the future people are just going to be buying products where machine learning and machine intelligence are baked in. It's just how that product winds up working.” – Joshua Bloom
My name is Josh Bloom, and I'm a professor of astronomy at UC Berkeley. I'm also co-founder and CTO of a machine learning company called Wise.io.
From the domain-specific view of machine learning, we basically see it as a tool, just like you might see computation or a Hadoop cluster as a tool for you to deal with data and do inference on data. Machine learning is already starting to creep into the physical sciences. We are using it in astronomy; it's being used in physics and biology, a number of different places where there's so much data that we just can't have people looking at all of it. And so in some sense, that's the break point: if you can't hire enough grad students, or you can't get enough people in the crowd to opine on data, you have to start looking towards other techniques. And machine learning has this wonderful generic advantage of being able to approximate human intuition and cognition, and obviously go beyond that.
So when we were dealing with more and more images coming off of telescopes, my colleagues’ approach was, “Well, why don't you just hire more grad students to look at the data?” And I wound up stumbling upon machine learning because I was being opportunistic; I obviously had heard about it before. But I wound up recognizing that when you start putting machine learning into production (the nuances of how you do that, of keeping a robust system that's up and running where you've got real clients on the back end; these are your colleagues saying, “Hey, the thing went down last night, and I wasn't able to point my telescope anywhere, because I didn't know where to look”), you wind up realizing that the kind of machine learning that's done in the academic world, where you write a paper and you wind up saying, “Well, my scaling curve is nicer than your scaling curve,” doesn't really apply anymore. I mean, you might use some of the basic concepts, and there might be some breakthroughs that allow you to do things at scale in a better way than you would have been able to by other means. But when you wind up realizing that there are people on the back end of whatever your machine is saying who rely upon it, you get to a whole other level.
So we're using machine learning in my academic research really as a way to do discovery on images. If we take a new image of the sky, typically we're looking for new things, and the way you look for new things is you subtract off what was there before. That subtraction process is an inherently noisy and really dirty process. And so in the end, if you just make very simple cuts on the data, you wind up getting a whole bunch of candidate new things: about a thousand bad candidates, or bogus candidates, for every one real one. And so when you show one of these postage stamps to a human, after about three or four examples they just know what's real and what's not. So something in your brain is actually looking at this data, looking at these images, and doing some high-level inference, where it's basically learned off of previous experience.
So what we do is we train machines to look at known answers: they look at known bogus examples and known real examples, and then it becomes the real-time classifier that's saying, “Yes, that's real, that's real, that's real.” That then becomes the gateway to another set of machine learning algorithms that are saying: given that this object in the sky is real, and given that we saw it last night and the night before, but we didn't see it the night before, and given that the galaxy nearby is red, and given that it's in this part of the sky, et cetera, what type of new thing is it? Is it a supernova? Is it a star that's burping? Is it a black hole that's eating a star? And so you wind up doing another set of inference runs on that data.
Now you're getting into some interesting places where machine learning hasn't spent a lot of time, which is time-series data. And what we wound up realizing in our own domain-specific research is that there weren't a lot of tools for us from the machine learning world to be able to apply to this. And so we actually wound up hiring some postdocs in computer science and statistics to start innovating machine learning algorithms. So it was this very interesting confluence where we were doing novel inference (that, by the way, if you do it well, means that you can discover things that people couldn't have done in the past, because it's a massive needle-in-the-haystack problem) and really pushing the envelope of different types of time-domain science. But at the same time, we were bringing datasets, and challenges with those datasets, back into the hardcore stats and ML world, and finding some beautiful places where PhDs could be written, and actually were, in my group, in statistics, because we needed to know how you take learning from one project, for instance, and transfer it over to another one in the context of noise and the different ways in which you sample the sky.
So what you wind up finding is that connection of novelty, where domain science is using machine learning as a toolkit, yet the kinds of domain-science questions that we have, and the data that we bring to bear, wind up actually becoming novel sandboxes for basic methodological research. The great example is Galileo, which I noted in my talk, where he takes this telescope, invented for military purposes to look for ships coming over the horizon, and he basically points it upwards, finds the moons around Jupiter, and really changes our whole notion of where we are in the universe with just a couple of observations with this new tool. We're using advanced statistics; we're using new computational engines all the time. And in fact, one of the great examples in astronomy, not so much from a software or algorithmic perspective, is the invention of CCDs, which are now part of our smartphones, right, and part of essentially everything we do for imaging. Astronomers had been using photographic plates, which are less sensitive than CCDs are, and we were essentially digitizing them into something that we could actually measure very precisely in zeros and ones. Doing that directly with charge-coupled devices was one of the main use cases that drove CCD development at Bell Labs, some 40 years ago. So astronomers have also been pulling on industry in various different places for a long time. And now we're using machine learning as, in some sense, just yet another tool for us to be able to do the novel science that we're trying to do.
One of the great indications of putting machine learning into production, for me, from a scientific perspective, is that we were able to do things that we wouldn't have been able to do by any other means. In some sense, that's the greatest imperative, and the gauntlet that I lay down in front of anyone is that you're not doing machine learning because it's cool and it's fun and you can learn something about the data from the past; you're trying to really use it as a tool for the future, as new data winds up coming in.
So one of the great things is our machine learning algorithm and framework wound up finding a new supernova that was in a very nearby galaxy. And because it was found about 11 hours after explosion, which was days earlier than had ever been found for that type of supernova before, it allowed us and a number of other groups around the world to use the world's largest telescopes, at all different wavelengths, pointing at that one place in the sky, gathering lots and lots of data. And that led to a number of Nature papers and a lot of insights into the nature of that explosion. And these types of supernovae, for instance, are the ones that tell us about the expansion of the universe. And it's amazing that we had one essentially in our own backyard that we were able to pick up and get so much data on so quickly, because we're now able to understand something that we didn't understand before with these types of events.
So the reason why it was found, in some sense, is because of the hardware. The hardware was capable of basically looking at a large part of the sky and taking data essentially every minute, and it happened to be that the telescope was looking at essentially the right place at the right time. In some sense, the existence of toolkits like we built for machine learning for this telescope project was the enabling factor that allowed the hardware to be built, in a sense that if you were taking data and no one was looking at it, throwing it into the can, and five years later somebody was going through the archives and realized you had a really important supernova in it, that would be great, and we'd be excited about it, but we wouldn't have been able to do the science that we did. So the two work hand in hand: because the software exists that can actually sift through and look at that data as if it's essentially virtualized graduate students with a huge amount of domain knowledge, and do this at scale, it allows you to take more and more data.
So it seems a little bit strange to think about an astronomer starting a company that's addressing more pedestrian or more earthly problems like churn, like the ways that companies interact with their clients, and trying to improve and make those interactions more efficient. But when you end up thinking about it, we wound up learning in astronomy how to deal with real-world data. I mean, of course it's not real-world; it's outer-worldly data, but it's taken with conventional sensors. It's got conventional needs of being able to serve insights on a regular basis. And rather than create some sort of IP that we wound up transferring over, what we wound up doing is creating the know-how of what it means to take not some super-sexy algorithm and apply it to a toy amount of data, but to take that and the toy amount of data, wrap it up in a platform that we wound up building, and then serve the results out in ways that are consumable, and in particular ones that create what we all call actionable insight: this idea that I'm telling you this is a place in the sky to look, do something about that; or I'm telling you that this customer is going to churn three days from now with this level of probability, you might want to do something about that.
Well, the point is, and this is the conceit of all data-driven approaches, that you don't need to have a theory about why something's going to happen. The idea is that you've got enough data, both in terms of the number of examples and also the dimensionality of the data, that you can train a very good grad student, or you can train a very good algorithm, to learn from the past. As new examples wind up coming in, they're not exactly the same as the previous examples. But if I know that a client isn't logging into my software product for one week, is three days late or more on making a monthly payment to me, and the only person that's used it is the CTO and not the people it was actually meant for, that becomes a very huge signal, so that when another client does something like that, you can wind up basically surfacing that. And so the actionable insight that we wound up learning how to do with astronomy data, in real time, on noisy streaming data, is exactly the same know-how that we end up applying to more of these conventional problems.
I mean, one of the things that's important to recognize is that machine learning, or machine intelligence, is already part of our lives. We all know about Amazon recommendation engines. Spam has gotten a whole lot better as we wind up building up these adaptive filters that are using a large amount of machine learning and, importantly, are personalized to you. So the story that I wind up telling is that if your dad is a Viagra salesman who travels to Nigeria, who just won the lottery, you probably want to see that email from him, right? But most people don't, because those are all signals of potential spam. And so what you wind up getting is, if you can do machine learning at scale, you're not just creating a single model for everybody. It allows you to create many models for cohorts of people, or the manifest destiny is everybody has their own machine learning models built upon their own past behavior, perhaps leveraging some of the insights that you wind up getting from the whole system.
But it's already part of our lives, right? And what you wind up noticing is that the why of why an answer is told to you is something that Google's already doing, for instance, with the new way of thinking about mail, where they say, “Well, we think this is a travel update,” or “We think this is a social update,” and that allows you to go in and say, “Well, actually, it's not.” So it gives the feedback loop, which is very important for learning. And every time you give that feedback loop and a new piece of data, that becomes a new moment for the machine to learn. So it's there. It's obviously mining over credit card interactions, et cetera.
But the view of our company, and the thing that I'm very excited about, is that there are so many different places where machine learning can make the interactions that companies have with their clients more efficient and more meaningful, and those places are not yet being well served. If you think about where analytics is, there's the descriptive analytics world, where at best you're rolling up into beautiful pie charts for people to look at and get an aggregate sense of what's happening. That can be real-time, but it tends to be something that's more in a reporting sense. And then beyond that is maybe some level of intelligence, where you wind up building some very basic models into the system and do some level of forecasting, and so you get some notion of prediction out of that. But the destiny of where all this is going is that you don't have to come to the table ex ante with a notion of what the physical model is that describes this behavior. The data itself winds up becoming the thing that you create your generative model out of, and that's deeply what machine learning is all about: you don't have to come to the table with that set of biases.
Being able to do that at scale, being able to do that in a production environment, is very hard. And what we believe, and I think it's being very well borne out by the difficulties that many companies are having in hiring their own data science teams, is this idea that in the future people are just going to be buying products where machine learning and machine intelligence are baked in. It's just how that product winds up working. It is an assistant; it is the intellectual appendage to the people that are using that product on a daily basis.