Talk on the automated, ML-driven software stack of modern time-domain astrophysics at the Simons Institute's inaugural public symposium; reportedly the earliest video on the Simons Institute YouTube channel.
Verify YouTube ID at build time.
“It's easy to think about astronomers as portrait artists, but we like to think of ourselves as celestial cinematographers, because for us time is a very important axis in understanding the physics of what's happening.” – Joshua Bloom
“It's of course preposterous to think about scaling up, say, 10 to the 6 Jodie Fosters to listen for what it is that we think we may hear in the sky… observing and listening to the universe doesn't guarantee discovery, and discovery of course doesn't guarantee insight.” – Joshua Bloom
“We have to learn as astronomers to say goodbye to black-and-white catalogs, because these catalogs, if we're getting lots and lots of data, have to be made in this probabilistic way.” – Joshua Bloom
“Even though we're using machine learning to really automate the whole astrophysics stack and remove people from the loop, can machines be taught to ask the questions that we haven't, or we can't? Will machine intelligence ever replace the eureka moments by people?” – Joshua Bloom
Maybe a little bit darker — oh good, that's good. Those of you that knew Jim Gray know that this was an endearing quote; he meant this in the best of all possible terms. Astronomy's data is big, there's no privacy concerns per se, and unlike satellite imaging or with medical data, if you make an incorrect statement about it because you're trying out some new algorithm, no one dies and we don't go to war unnecessarily. Jim was a computer scientist at Microsoft and an alumni at Berkeley, and he saw astronomy as a sandbox to develop new formalisms of fundamental computer science research. In some sense we're trying to reach back the other way: astronomers are starting to play around with some of the new concepts in computer science and view all of that as our own sandbox, and we're trying to do something novel with it.
I want to thank the organizers and thank all of the other speakers — it's been an absolutely amazing couple of days. In some sense, as a way to end this workshop, I wanted to try to make this talk fundamentally about a conversation between computer scientists and astronomers, or physical scientists in general. Our stack really ends with insights about the universe, but it starts with the things that you all do, and I hope, as you'll see throughout the talk and especially towards the end, that there's something astronomy is bringing to the kinds of thoughts that you're working on every day.
One of the fairly well-known astronomers is this guy named Galileo. He was pretty opportunistic, and in some sense he's, I think, a pretty good example of the way that astronomers look at toolkits that they themselves didn't necessarily invent. He knew about this thing called the telescope, which most people have been pointing out at the horizon looking for enemy ships. He said, what if I just do that and look up? And the rest in some senses is history. But repurposing telescopes that were invented for military purposes to point towards that horizon is in some sense what astronomers have been doing ever since then. And obviously with those new eyes came some amazing insights. In some ways he was the guy that really put the nail in the coffin of the Ptolemaic order of the universe — the planets were supposed to be fixed spherical orbs, and they weren't supposed to have moons — and that picture that he made about 400 years ago of this discovery of the moons around Jupiter was obviously an amazing landmark. I also wanted to show this because this is really our first glimpse of so-called time domain astronomy. Time is moving in the axis going down and angular position on the sky in the axis on the right, and synoptic observations — taking observations repeatedly over time — plays a very big role in the work that I do. I also wanted to show this to you because it emphasizes the crucial role that humans have in data collection, or at least had in data collection — and not just data collection but data analysis and inference. You could think of this piece of paper as not just a log book but a database. What's so amazing is that this one man with his toy and intellect really changed the world.
What happens — and it's not a lot of data — what happens when astronomers get more data? Well, we just add more people to the loop. Harvard College Observatory in the late 1800s was dealing with a vast inflow of photographic data that was coming from remote sensors, in this case photographic plates from the southern observatories, and there was a set of questions in the time domain around finding and understanding variable stars. But there weren't the same types of computers we have today. So these computers — these people who are poring over actual data — were in some sense a prototype of crowdsourcing, or, you could think of it another way, a prototype of grid and parallel computing. It's also sort of a personnel innovation. In some sense it also sets the groundwork for modern day data mining. Viewed from a modern lens, this is obviously fairly pejorative, to call these people — and they were all women at the time — computers, in a modern way.
However, when we look at it, we can also see great things in the fact that they were close to data, and they made some very important contributions, as you'll hear about later. At the time they were really interested in understanding the fundamentals of stars: what are their masses, what are their radii, what are their temperatures? And in making measurements of binary stars in particular, it was thought that one could, in concert with other types of data, make some fundamental measurements of stars of a whole different type. So this is fundamentally what they were looking for — binary stars. We'll come back to that later on.
Today, of course, we have graduate students and post-docs who are actually using the types of computers that we all know very well, and the concept behind what we try to do in our group, which is called the Berkeley Center for Time Domain Informatics, is figure out ways to take people out of the real-time loop. As you'll see throughout the talk, there's a great need, as we get more and more data, to be able to automate the sort of tasks that people way back when — and even just a few years ago — were doing in a manual sense, even ones that seem fairly cognitive. These are ones that we're actually trying to automate away.
We're concerned with the dynamic universe, this notion that everything in the universe changes if you have enough sensitivity, and we can see those changes unfold before us. It's easy to think about astronomers as portrait artists, but we like to think of ourselves as celestial cinematographers, because for us time is a very important axis in understanding the physics of what's happening. How fast or by how much tells us a great deal about the underlying physical processes.
With that, I wanted to show you a very important — probably the most important — three-frame movie you've ever seen. I'm sure you've seen longer-length movies before. Why is this important? Well, first of all, this is a snapshot of the same place in the sky taken about 45 minutes apart between these images, taken on a fairly small old telescope, about 60, 70 years old. And why is this so unbelievably important? First of all, you can see the kinds of raw data that astronomers are dealing with. There's some noise in here, there's this effect called fringing from the detectors, and there's an object moving here. Can anyone find that object? Yes, you're all pointing right there — very good, there it is. This is the discovery movie of this thing we now call a dwarf planet, called Eris. It turned out to be larger than Pluto, and this happened 10 years ago. For those of you that have children that are upset about Pluto being demoted from its planet status, this is the movie that essentially did it. This was a massive data mining effort, and it was done not entirely in manual mode, but obviously had a huge impact upon our understanding of our own place in the solar system.
Here's another two-frame movie, actually just superimposed on top of each other, from the Hubble Space Telescope. There's a dot here, a dot here. A very, very bright star, through various different techniques, was occulted out, so we could see, essentially behind the glare of that star, this very, very faint object, which moved from here to there in two years. It turns out from there to there is a Keplerian orbit, and this is now believed to be the first image of an extrasolar planet by direct imaging. And this is just happening essentially in our own lifetimes. Watching things move in space is obviously having a transformative effect upon our understanding of the universe.
The thing that I spend most of my time on is how things change in brightness or change in color. Everything, as I said, if you have enough sensitivity, appears to change. Stars, even though they might seem fairly quiescent — if you have the ability to collect enough photons quickly enough with enough signal to noise, you will actually see it burble around. That's not what I work on either. What I spend a lot of my time on is working on more dramatic things, where I'm not looking for changes in one part in a million to the light; I'm looking at something that wasn't there and now it's there, and then it's going to go away. Typically this means that something catastrophic happened: stars die and they blow up. This is the realm of supernovae, gamma-ray bursts, and potentially even new phenomena. One of the important themes that you'll see throughout this talk is the notion that just because we can make measurements of these changes doesn't mean we actually understand anything about it. In particular, even if we've made an interesting discovery of something new that's happened in the sky, the greatest insights in astronomy happen when we're able to do follow-up imaging or follow-up spectroscopy or look back at archives.
The important thing about all of this follow-up after discovery, which is in some sense where the real fun comes along, is that it's expensive. There are real resources involved in all of this: there's people, there's time, there's precious telescope time, and it just goes on and on and on, fundamentally, which I think makes this whole problem quite interesting. We could probably turn all of these limited resources into a notion of a dollar term, and what we learned is that when we start couching the problems that we have in astronomy in raw terms like that, the metrics that we wind up applying to decisions actually become even easier to follow.
It's not just the objects in the sky that we know about that are interesting. And I should point out — this is one of the terms that I'll introduce here — this is what we call light curves. This is basically time on the x-axis here, and this is brightness in crazy astronomer terms, where this is faint and this is bright. And there are a bunch of different behaviors of brightness as a function of time of these very important types of exploding stars. These are the two most common ones: Type Ia supernovae are the exploding stars that are used to measure the very precise expansion rate of the universe and have a great importance for cosmology, and Type II-Ps are more ordinary stars blowing up. These are the things that we know about and we find all over the place. Although they're incredibly rare on the sky, people have gotten pretty good at finding them. And then there's things that have been hypothesized to exist — the so-called pair-production supernovae, which is brighter, and all these other kinds of exotic events. It doesn't matter what these terms actually mean; the point is that I'm showing you things that we know about and things that we think we know about. In a Donald Rumsfeldian view of the universe, there are the known knowns, and we can go after those pretty easily. There are the known unknowns — maybe you might say that the theoretical work might guide us in the types of pictures and the time scales that we'd be applying our observations to, to find these things. But then of course, by definition, I can't plot for you the unknown unknowns.
One of the major difficulties in doing needle-in-the-haystack work — discovery — in astronomy is making sure that you keep your eyes open for all these different classes of possibilities, and making sure that if you're going after one of these events — which, by the way, would be a major result if we could actually find an electromagnetic signature of two neutron stars merging — we have to make sure we don't completely exhaust our resources and throw everything we have at every single possible event. So we need to be very careful.
It's of course preposterous to think about scaling up, say, 10 to the 6 Jodie Fosters to listen for what it is that we think we may hear in the sky. Yes, we may be able to hear subtleties that might be hard to program a machine to uncover by itself, but observing and listening to the universe doesn't guarantee discovery, and discovery of course doesn't guarantee insight. This is especially true today as we just get more and more and more data. For optical astronomers, those of us working in the visible regime, this is really the golden era of time domain astronomy — and Big Data time domain astronomy. This is a depiction of the Large Synoptic Survey Telescope, which is now on track to come on sky in about 2020. This will be surveying the sky every couple of days, essentially imaging every available part of the sky to unprecedented depths, and the amount of data and information we're getting just about the changing parts of the sky is truly mind-blowing. For us, we're going to get light curves of almost a billion sources, which will be updated every three days. We're expecting to discover about a million supernovae a year, and it goes on and on — pick your favorite time domain object, and LSST is going to find these things in spades. Twenty terabytes a night is more than most astronomers at most wavelengths are used to collecting, although there's some other wavelengths where that's actually not that much.
But in the context of scaling up those computers that you saw at the very beginning, if you think about that as actually happening today, with saying, well, I've got more data, I'll just hire more graduate students — this is obviously not going to work anymore. This is easily a petascale problem, and this is happening at a number of different wavelengths: LOFAR and SKA is an even higher data throughput problem. What's exciting, I think, is that we know this is coming, which is good, so we can start preparing for it. But there are a number of precursor surveys that we're starting to cut our teeth on, and this is for me my real sandbox, saying, well, we can try out new ideas of how to do this sort of automated astrophysics so that we're ready for LSST, the Large Synoptic Survey Telescope.
So for us the Big Data challenge, and this question that we ask, is: how do we do discovery, follow-up — which remember I said has got resource implications — and inference when the data rates and the requisite time scales preclude human involvement? It's an interesting question, because going back in retrospect and looking at a database and saying, oh wow, there was something really interesting a year ago, I sure wish we had used all of our telescope resources to do some real great science with it — that obviously doesn't cut it anymore.
What's exciting about the time domain is it adds this extra element of urgency: I need to take action on the things that are happening in the sky right now for me to maximize the return. I think I've mentioned this a couple of times already, where we're trying to automate the scientific workflow, and in a cheeky way I'm plotting the barrier to entry as a function of intelligence required to do this. So there's the observing stack, there's the finding stack, the discovery stack, the classification, follow-up, scientific inference — writing papers, getting awards, standing at podiums, et cetera, that's not on the plot. And some of these things we're starting to really get a good understanding of how to do well under the types of conditions that we all operate in. I'll talk more about all that later.
Observing is getting to be fairly routine: robotization, queue scheduling — it's a constraint satisfaction problem. Finding — that is, actually identifying those objects in the sky that potentially could be new; maybe they're just noise, who knows — but really getting down massive amounts of data into, say, medium amounts of data. And then discovery, and this is where we're going to draw from computer vision and image recognition. And I make a distinction, by the way, between discovery — where we might say, aha, this is something of interest, I don't know what it is yet, but I recognize that this is something that I may be interested in, I want to do more work on — I make that distinction between discovery and classification for the following, or at least drawing on the following story from history.
Again it goes back to Galileo. In all of his observations of Jupiter, it turns out that Neptune, at the time when he was making these observations, was pretty close to Jupiter in the sky, and Galileo made note of it in his database — in his notebook — and he said, here's a fixed star. Galileo is not credited with the discovery of Neptune, even though we can go back and with essentially no doubt believe that he actually recorded it. He found Neptune; he's not the discoverer of it. It took another 220 years or so for the actual discoverer to find Neptune and realize that it was something interesting. I think that's remarkable — he would have been a very famous person if he had actually discovered Neptune. Once you've said, aha — I've not only found it, I've not only said this is something that I care about — now you want to ask, well, what is it? What is the source of this change on the sky? What's the physical origin of this? And this is where we're starting to apply some machine learning techniques. And then you want to do follow-up, et cetera, et cetera.
Let me just talk a little bit about automating observing. One of the great leaps, I say, for astronomy — and optical astronomy in particular — was when we went from photographic plates to digital imaging in the form of charge-coupled devices, CCDs, because for the same size aperture telescope we could collect essentially light at 50 times higher quantum efficiency, which means that we were able to build up a much better signal-to-noise image much quicker. It also has some great properties, like they're linear, and so the data that you get out of it tends to be more accurate. But after we went from photographic plates to digital imaging, we also had to take humans out of the loop of operating telescopes, and so we've gone from human operations to robotic telescopes. My colleague Alex Filippenko has been operating the so-called KAIT telescope on Mount Hamilton, not so far from here, for almost two decades now, and that's been incredibly important, because getting these people sitting on top of a mountain night after night taking data is a lot harder than just getting a robot pre-programmed to do things.
This is something that I did early in my career: I actually roboticized an old telescope that had been mothballed, and then I decided to go beyond just the robotization of it, so that I could put in a whole list of objects that I cared about, where I actually asked the telescope to figure out what it thought was most interesting to observe at any moment, given all the constraints that were put in by all the observers that had gotten time on the telescope. So beyond robotization, we started creating what I guess we could deem intelligent data collection agents. This is me standing in front of my telescope. I highly recommend people getting pictures of themselves in a smug pose in front of stuff they built, and if you haven't done that already, that's okay — for those of you that code a lot, you can just print out code and have somebody take a picture, so you can show it in talks like this.
What it does is autonomously schedule itself based on complex prioritizations — again, that's a constraint satisfaction problem that we worked on for a while. It does detailed weather sensing, so it senses when it's in danger and will try to shield itself and close itself up when there's a problem. And it will react to new things in the sky without any people in the loop — again, remember, we're trying to take people out of this loop, because the idea here is then we can be more efficient with the way that we do science. And so if there's an event, say from a satellite, it will be able to slew if it believes that it's important to look at that object in that place on the sky, before anybody even wakes up. It tweets where it is on the sky, so that other people can follow this up and actually try to draft off the locations of where the telescope is. And under the hood it's some interesting stuff that we worked on: daemonizing all the different subsystems and making this into a formal state machine, so that we could wrap our heads around all the different problems that wind up cropping up when you're doing a software-slash-hardware problem.
The results have been pretty great — we've been very excited about it. This is a movie up here going from one minute to 30 minutes after one of these gamma-ray bursts exploded in the sky, essentially without warning. A minute after an event was recorded by a NASA satellite — and it more or less beamed down a text message to the ground — this telescope had slewed over and started taking data. And what you see is a negative image here, where it's very bright at the very beginning, and you see after about 30 minutes it's completely lost into the noise. It wasn't just a burst of gamma rays; it was also a burst of visible and infrared light, and you can see the light curve here just fading from very bright — your eye can see down to about here. This was incredibly bright, and then it was tens of thousands of times fainter than what your eye can see within just a half an hour. If we had had a grad student in the loop, then maybe they would have had to have their cell phone go off, and they rub their eyes and they fumble for their computer and they forget their password and they log in, and all of a sudden we're way down here and we get to miss all the juicy science in there.
We're not the only people who are trying to push the envelope a bit on robotic telescopes. This is now becoming a more and more interconnected system: instead of well-connected graduate students talking to each other, we're trying to figure out ways to have telescopes talk to each other and actually federate themselves to actually do observing campaigns without any people in the loop. And so what we see here is an initial network of these telescopes which are starting to talk to each other, and we obviously have different roles for the notions of publishers and subscribers. What's pretty neat is that this topology of this network has been growing organically, and I think it probably looks a lot like some of the early days of the internet. And for those that are interested, I created essentially the language that telescopes use to not actually just talk about where they're going to look in the sky but describe different events in the sky, and this has been tremendously exciting, and there's been some actual great results that are coming out of just this network.
I want to keep going, though, and talk about what happens after we've now scheduled the telescope to look at something and we've now gotten the data — we then automate this reduction process — but now how do we wind up doing discovery? I've been involved in one of these precursor programs to the Large Synoptic Survey Telescope called the Palomar Transient Factory, and what you have here is the kinds of things that, for the first several weeks of this project, which started in 2009, we were having ourselves — and not just graduate students and postdocs but also faculty — looking at images. These are two little places on the sky. What you see here is a new image — it's a little postage stamp around a new place in the sky, and all that nebulosity, by the way, is from a nearby galaxy — and then here's this new image right here of a different place in the sky. And what we have is a deeper reference image of the sky, built up over a number of nights before this night, and we more or less just do a subtraction to look for something that's new.
Nature is not usually so kind as to draw green arrows and cross hatches around where there's actually a new object, but I wanted to point your attention to this object right up here. It got this name, 11kly — that's not all that important. The important thing to note here is that this is a pretty easy find for an automated algorithm that's just passing over these images: it's just noise, and then there's something that's obviously not noise. But then down here you see something which is a little bit fainter — which, by the way, to your eye it's pretty hard to see. This is a new supernova that happened on the outskirts of a galaxy which is quite far away — halfway to the edge of the observable universe. And what you see are these red boxes around here, where the subtractions were actually imperfect. It's a very hard problem to do a subtraction in the presence of noise, because there's subtleties in how the atmosphere winds up distorting the image; the telescope is not exactly in the same state today as it was in the past. So this is a really hard problem. We get about a thousand images a night from this project, and from that we're able to extract — including all these red boxes — all the possible candidates: about 1.5 million of those objects a night. Of those, about a thousand of them are actually bona fide new objects in the sky, about 300 of them are variable stars, and about 10 of them are new transients — new things that weren't there and are now essentially there. These are new things that are exploding somewhere in the universe, and we want to now follow these things up.
So we've been working on trying to automate this process of discovery, and one of the things you wind up really realizing right away is that discovery is imperfect. Whether something is new or not, especially when I'm getting close to the noise floor — you don't really know if it's actually a new object. You might learn later on, after it's gotten brighter or fainter than it actually was, but if I want to take action on that right now, this is an important thing to understand. So we started more or less taking those images and trying to do even further dimensionality reduction, more or less creating of order about 50 different so-called features that really describe each one of the candidates: fits to the Gaussian, what's the signal to noise, what's the distance to the nearest object, et cetera, et cetera. And what you can see in some of these pictures right here, in the two different colors, are examples from our training data of real candidates — so these are real astrophysical candidates — and bogus ones. I didn't point out before, but going from 1.5 million down to about a thousand is a pretty massive needle-in-the-haystack problem, so there's a huge imbalance in these two classes of good versus bad or, as we call it, real versus bogus. And so we have all of these different features, and it's really just a two-class problem: we're asking, is this candidate real or not? And so we've couched this in the way that people who do machine learning can understand, and we threw as many algorithms at this as we could, and we wound up finding that some of them did much better than others. This is the so-called ROC curve — receiver operating characteristic curve — of false positive and missed detection rate. It's an important place to be able to understand where you want to be on this curve. If I really didn't mind more or less missing all the interesting objects in the sky, and I was really afraid of going after and using resources on all my false positives, I might put myself right over here, where I have a very low false positive rate — that is, calling things that are actually bogus real — but I would miss a whole lot of interesting stuff. If I put myself here, I'd miss half of the interesting stuff, so that's a problem. If I went all the way down here, and I didn't want to miss anything, well then I would really start blowing up my false positive rate. So you have to decide, if you're now trying to build a framework around this in terms of follow-up, where do you want to be on this curve.
What we wind up doing is bootstrapping the survey, so that we started off with just a couple of training examples, and then as the survey went on we wound up getting real ground truth about what actually was real and what wasn't. And we wound up finding, as you might expect, that as we added more data to the training set, we're able to get better and better classifiers. Here's where we were in the original paper that I wrote in 2011, and just by adding a larger training set we went all the way down here. For me this brings back one of the well-known statements from Peter Norvig, who said that more data beats clever algorithms, but better data beats more data. I have a little bit to say about that on the next slide, but it's very clear that more data helped us tremendously in getting a better classifier, even just using the same algorithms. But what's interesting is that what we wound up doing is we started flipping the training data. So we took a real object and we said that we actually now believe that it was bogus; we took a bogus object and we flipped it around and said that it was real. And we kept re-running things as a function of how much we messed around with the ground truth, and what we wound up finding is that only a few algorithms did very, very well in the presence of what's called label noise. We got up to about seven or ten percent label noise — that is, being wrong in the training set — and still produced a very, very good classifier. So that was pretty interesting for us.
Now, those that were paying attention to that thousand number going down to 10 — it turns out that most of the real astrophysical objects in the sky are actually asteroids. Asteroids are very interesting to a large number of people, but they're not interesting to me at all, so we needed to figure out a way to get rid of all of those so we weren't wasting our resources on following up known objects. So we built essentially a parallelized version of a resource that was being used and is still out there called the Minor Planet Checker. We just did this in a couple of months with one graduate student; it's now being used worldwide. But for us this was really the last step to be able to say, aha, we have actually made a discovery — and moreover, because we have training data, we can go back and figure out what the probability that we're wrong about that discovery actually is. So we can ascribe probabilities to that discovery.
This has been a great project — and in fact it's already ended, and now there's a new incarnation just starting up called the intermediate Palomar Transient Factory — but one of the things that we were very excited about was illustrated in this three-panel image here. On August 23rd in 2011 there was a nice image of this galaxy — there was, again, no green arrow that nature was telling us to look there. On the 24th of August there was this little new object that popped up, and it didn't move, so it wasn't an asteroid. And then on the 25th it was getting brighter and brighter and brighter. This turned out to be a discovery of a supernova which, after we did a lot of analysis, happened just 11 hours after the explosion. It's the first time we've been within 24 hours — or maybe even three or four days — after the explosion of what turned out to be an ordinary Type Ia supernova. It's the nearest Type Ia supernova in three decades, and it was discovered by our machine learning framework that essentially promoted it to the top of the stack for people to actually do more inspection on and follow up.
And because we got on so early, and because we were able to throw as many resources at it as we could and we recognized the importance of it, we were able to make some pretty interesting and useful discoveries. I'll highlight two of them here. This is a bit of a busy plot, but I want you to just look at this green area here and note that these are all different regions of exclusion in effective temperature and average density of the actual object that blew up. There's a lot of physics that goes into making this plot, but fundamentally all these different lines rely upon a single observation of a non-detection — what turned out to be four hours after the explosion — that an amateur astronomer had essentially taken in Spain. The non-detection of that turned out to be very, very useful for limiting the actual radius and the properties of the thing that would eventually wind up blowing up. And what we were able to do is rule out all types of normal stars and only allow, in this space right here, very dense stars — compact objects, either a white dwarf or a neutron star. And people have been talking about white dwarfs as the objects that blow up for Type Ia supernovae — and as you know, Type Ia supernovae are incredibly important for understanding of the universe as great probes of the acceleration of the universe — and yet we haven't really known for sure what it is that's blowing up. Astronomers are pretty great at this: they'll say, I don't really understand my probe over there, but it does its purpose, so that's good enough for me. Here we're actually able to rule out all other reasonable possibilities for the first time.
Type Ia supernovae are also thought to be part of a binary system, where the larger star is transferring mass onto the thing that winds up blowing up, causing it to explode after it reaches an important limit, and there's lots of different possibilities of the configuration of that binary system — what is the size and what is the age of the donor star, et cetera. And we were also able to exclude, in this yellow region up here, many of the well-described theories about what makes Type Ia supernovae, because there was pre-existing imaging of this place on the sky with the Hubble Space Telescope, and a non-detection of objects there — in the place where a supernova was going to explode just six years later — turned out to be incredibly important for ruling out all but just a few models. I'm trying to point this out to show that by putting machine learning into practice, we were able to do this massive needle-and-haystack problem incredibly quickly, and we're able to do this very important science that informs really all aspects of astrophysics, just as quickly as one could hope for.
But I still haven't told you what happens after we do discovery. After discovery you want to ask that question — what is it? And here again we're bringing in some machine learning concepts, where we're using machine learning as a surrogate to ask that question: what is the nature and the origin of the variability that I see? And in this sense this is what I would call classification. The problem is, in the context of variable stars, it is a zoo — it is a variable zoo of variable stars. And if I gave you an object which is changing in the sky and asked you to tell me, even if you're an expert, what it is on the sky, you might have a hard time putting it into one of these broad classes, and you might have a very hard time, until you got more data, putting it into one of these subclasses. If you've got great data, you might think that you can just do some template matching. This is four examples of variable objects on the sky that are varying on a variety of different time scales, from three different large classes of variable stars. There's the so-called eclipsing systems — these are binary stars; there's pulsating stars that are changing their radius as a function of time; and cataclysmic variables — these are things that are undergoing catastrophic mass loss or some type of eruptive explosion. Here again are some light curves. This is a pretty nice-looking so-called W UMa class star, and it's changing on time scales of hours. Here we've got a so-called Beta Lyrae, which is changing on time scales of order days. Here's again a Type Ia supernova light curve, and here's a long-period so-called Mira variable.
So if you had great data and you had great examples from all of these different classes of variable stars, the easiest thing would just be do template matching. The problem is, even though we have these broad classes, this classification system is really kind of an admixture of phenomenology and physical thought and physical belief. Some of these classes are called X-ray because they emit X-rays in addition to optical light, and some of them are called eclipsing because we actually have a physical belief that one star is moving in front of another. But we don't typically have perfect templates of all the things that are changing in the sky, and even Type Ia supernovae, which are incredibly regular in their light curves — again, which allows us to make these great measurements to large distances and measure the expansion of the universe — they're all different in their own right. And when you get down to the subtlety of it, each Type Ia supernova and every other type of star is actually different in its own way. It's sort of like Tolstoy and Anna Karenina, I guess — every unhappy family is unhappy in their own way. So there's really a lot of interesting problems that start coming up when you're looking at the actual type of data that astronomers get. We don't get to have those perfect light curves, and one of the things we have is noisy and irregularly sampled data. When we actually extract information out of the data that we've now reduced from the raw data down to the nice images, sometimes we could get the answer vastly wrong and underestimate the uncertainties in our flux measurements — might call that spurious data. And if we're thinking about follow-up, the telltale signature of what it is that we might want to be actually observing might not have happened yet, so you also have to have some predictive modeling of what might be happening in the sky.
So the machine learning approach to all of this is to take all of the data that we can possibly get our hands on and coerce it into a large-dimensional feature space, and then with training data hope that we can build a classifier — now not just a zero-one classifier but a multi-class classifier — that would actually allow us to do this well. And so we do this with about a hundred different features on a very large data set of variable stars. All the things in blue are time-domain-related features — again, doing this in the presence of noise and irregularly sampled data, so some of this stuff is non-trivial — but then also context features, this notion of where did this thing go off in the sky.
What's interesting is, if you came up to me afterwards and you said, Josh, I saw this interesting object in the sky last night, it got brighter and it went away, I might ask you first, well, was it in the ecliptic plane — where a lot of the asteroids actually live — and you'd say no. And I'd say, well, was it in the galactic plane, because then maybe it's associated with a star, and you could say no. And then I'd say, well, but was it near a galaxy? You'd say, oh yeah. And was it at the center of the galaxy? And you would say, no, it's on the outskirts of the galaxy. And I'd say, was the galaxy red or was it blue? And you'd say red. I'd say, oh, that's a Type Ia supernova — and I'd be right 99% of the time. So without telling me anything about the time domain, just the context of where this thing occurs in the sky — if we're able to go after all the different databases of that place in the sky and extract all this information, some of it rich, some of it in places of the sky we've never even looked at, and build a classifier on that, we can do great things.
And so we're able to build a pretty good classifier over — I think this is about 25 classes of variable stars — where this is the true class and this is the predicted class, and this is a whole cross-validated error matrix. And what you see is that you want to have all your power along the diagonal here, meaning that your classifier is always right — but unfortunately it's not. What's really interesting for us, though, is that when you draw boxes around these larger classes and don't get into all the different subclasses, what's pretty interesting is that a lot of the off-diagonal power still lives within these larger classes. And what's pretty neat about this, I think, is that even though we didn't tell this machine learning classifier anything about the physics or anything about the actual connection of all these different stars with respect to each other, there's something about the fact that they all kind of look the same as why the classifier maybe got some of this wrong. So the classifier in some ways is discovering some of the physical behavior and some of the physical connections between these larger classes. And we're getting down to about 15 percent classification error, which is pretty good, but when we actually start using a structured classifier, where we actually use the taxonomy of this classification tree as it exists, as people created it, we get down to five percent gross misclassification rate, which is pretty good.
One of the things that has been touched upon throughout the last three days, which I found pretty interesting, is this notion of how we can learn on, say, one survey. And I should say that when we take data with one survey, we're doing it with a certain set of filters, we're doing it with a certain cadence — so we go back to the same place in the sky at some level of frequency — we're doing it with a certain telescope, et cetera, et cetera. Every survey is different. So now if I handed you that classifier and I gave you all of my codes, and I said, go ahead, now you've got your new survey, go ahead and classify — you would do a very terrible job with my classifier. And that's because the way that you observe the sky actually winds up changing inherently the underlying feature distribution.
This is illustrated pretty well in James Long's thesis — he's finishing up right now; he's been doing a PhD in the group — and what you have here is a depiction of a three-class problem where you have feature one and feature two, and you can build a perfect classifier — a decision tree of, if you're on this side here, then you belong to the yellow triangles, et cetera, et cetera. But if I take that exact same classifier from this survey called Hipparcos, and I apply it to data — now where we actually know the answers for all these three different classes — you see I've produced a very lousy classifier. And so this question of how you use decision boundaries and classifiers from one survey and transfer that knowledge onto another survey that has different characteristics is a very interesting question. And what's pretty neat is, of course, if you're observing with a different telescope in a different part of the sky, you're going to have different reasons for doing so; you're generally going to be probing a different population of stars. And so there may be whole places in this large-dimensional feature space that are not even populated by one survey and are incredibly well populated by another survey. That's illustrated here with a plot of the period of known periodic stars and the amplitude of how much they're changing as a function of time over the course of their period. These are very large amplitudes — changing by an order of magnitude — all the way down to very, very subtle changes. And what you see here are data from two different surveys. One is a survey called ASAS — this is where we had all the ground truth. And then you have this other survey that we'd like to know the answers in — sorry, this is where we didn't have the ground truth; we have all the ground truth from the training data — and you see that down here we have no examples. We have no idea what classes of objects those things are.
And so what we've been doing is building up a framework for doing active learning, to figure out how we can imbue experts into this problem to allow us to identify and manually label the testing set of data, so that when we use the finite amount of expert resources, we'd be able to, in future iterations, do a better job on the classifier that I took from survey A and then applied to survey B. I won't go into the formalism here of how we did this, but I just wanted to show that, with a few iterations — if we have our off-the-shelf error rate on this essentially transfer learning problem — very quickly, with just a few iterations of active learning, we're able to get to a very, very nice classification error that's essentially acceptable for this problem.
I'm sure all of you know this, but astronomers have a very hard time with the slide I'm about to show. The classification statements that we make on this noisy data is surprisingly fuzzy. I can't tell you it definitely belongs to that class; I can only give you probabilities that that object belongs to that class. And what you really hope for is that I do a great job in calibrating those probabilities. I can get something that comes out of my classifier and say there's a 20% chance of this type of supernova, but what you really want — in the same way that weather people will say there's a 50% chance of rain — you actually want it to rain 50% of the time. If I'm wrong in either direction, you might get angry with them for different reasons. So the catalogs of transients and variable stars that we're building up are necessarily probabilistic, and this is incredibly subversive to astronomers, because they like to go to a paper of known class of this type of star and say, ah, I'll observe that. But those were all populated by people and experts over the last 100 years, and a lot of those things actually turn out to be wrong — that's another sociological question. But we have to learn as astronomers to say goodbye to black-and-white catalogs, because these catalogs, if we're getting lots and lots of data, have to be made in this probabilistic way.
And so we tried to make it easy for astronomers to digest this. How am I doing on time? Yeah, okay — okay, good. What we did is we built a website that allows people to probe through the probability distribution for any one of these stars and search around for the different types of objects that they care about — this is using the Google Fusion Tables under the hood, if people are interested. So I can go to this type of RR Lyrae, I can actually click on one that I might be interested in, and there's the probability vector of the top 10 most important ones, there's the actual light curves — and you see how crappy that data is. And then we even made it social, just in case Marissa Mayer wants to potentially buy us — maybe Yahoo will buy this thing, I don't know. But we've gotten a lot of traction with this, in the sense that astronomers who care about these different subclasses are actually pretty interested, and they're learning how to actually use these catalogs. We've been trying to go beyond this and now actually do our own science. Again, just because you make a catalog doesn't mean that it's actually all that useful. You're doing this not just for the exercise of, cool, I can do machine learning and make probabilistic catalogs; you're doing it because you actually want to do novel science. So building a catalog so that you can do novel science, I think, is only justified after the fact if you actually do that. And what you really want to do is figure out what kind of science do you want to do. Do you want to do demographic science, where you want the purity of your sample to be very high — and there you might really tamp down and put yourself on a part of the ROC curve where you have very few false positives — or do you want to work in novelty discovery, to go after the rarities, where maybe only one out of 100 objects that you actually spend telescope resources on turn out to be actually all that interesting?
And so we were able to do this with one of my other graduate students, Adam Miller, where we wind up discovering some very, very bright stars that were essentially misclassified for the last five or six decades, where the probabilistic classifier said it was of this type — so-called R Coronae Borealis stars or DY Per stars — and with just a couple of spectra we're able to confirm these very, very weird objects. I don't think I'll go into the details of how well we do relative to just blind searches and blind cuts.
I want to just end with a couple of statements not about looking at catalogs of data that have already been taken, but about doing this classification on the fly. I already talked about how we do discovery on the fly; now we actually want to say, is this a supernova, is this a variable star? And one of the great places we got some training data from was this group called the Galaxy Zoo, where we presented to them — citizen scientists — a bunch of images just like the ones that I showed you at the beginning of the talk, and asked them, is this a supernova or not? And this is not a supernova, and that is a supernova. And the citizen scientists got very good, and they wound up marking up a few hundred of these images a night and built up a huge training set for us. We then of course went back and we looked at those places on the sky and applied our resources of follow-up, and we got some ground truth. And what's pretty amazing is that we were able to build a classifier using their training data, and the classifier outperformed the people at any single false positive rate. So our ROC curve — you actually want it down here as much as you can — was actually below the citizen scientists themselves. So that was pretty exciting.
And we're now putting this into practice, where we're not just doing discovery on the fly; now we're actually able to make classification statements on the fly about what type of object it is. And we don't have a lot of data — when we just have one new data point, we have to more or less rely on context, and so we have a simplified version of the taxonomy. What's pretty cool, I think, is that we now have the robot that's doing this classification log into the system that everyone in this group actually uses, and it actually will log in now and say, I think this is a transient, and I think this is a supernova. And so then you see a whole bunch of people who logged in and said, oh okay, it's at this distance, it's this type of supernova, after they got more and more data. But the robot now is responsible for more discoveries and more classifications than the people are in this project. What's exciting about this is this actually turned out to be a fairly interesting supernova — it was a paper that appeared in Nature a few months ago, because we were able to again get great data and maximize our resource usage.
I've been thinking a lot about machine learning, because we've been drawing from all the great literature on machine learning to do the kinds of science that I'm interested in doing. But I've been thinking about how that may be a bit of a misnomer. That notion of learning is a difficult one. In a cheeky way, I would say if you went to the Library of Congress and you learned everything there was to know in the universe, and you had some amazing insight, and then you got hit by a bus, it wouldn't be all that useful that you learned all that. So for me, trying to find another term that has meaning, I've come to this term of machine intelligence — of putting machine learning into practice.
I think it's easy on the theoretical side, and maybe with a computer science lens, as you're thinking about a new machine learning algorithm, to think about classifying irises with your new great algorithm. But the data that we actually get in the real world is noisy and it's subversive and it's fuzzy, and we don't really have a great answer, and the color of a petal of an iris is not purple plus or minus red. The kinds of data we actually get in the real world has noise, and sometimes it's missing, and it's dirty. And so putting the results of machine learning into practice — I haven't seen much of that in academia. We see it obviously a lot in industry, where they've been able to recognize the importance of doing great machine learning that works in practice, because if they got it wrong, there are real resource implications. Does a botanist really care whether we've got a great classifier of different types of irises? I don't know — has anyone actually asked the botanist, do you care about that? What's important is that we need to figure out ways to get machine learning working for the types of science that are collecting lots and lots of data, asking these very complex questions on that data. And I think it's only really just beginning.
I wanted to come back at the end here to this picture that I showed you at the beginning — it's very, very dark — of the computers in the room at Harvard, and not point out the light curve but actually point out this person in the foreground here. Her name is Henrietta Swan Leavitt, and she was the discoverer of the period-luminosity relation of Cepheid stars. And that discovery that she made in this room, around the same time when this picture was taken, was because she decided she wasn't all that interested in binary stars, and she started thinking about pulsating stars. The work that she did became the basis for Edwin Hubble's first measurement of the expansion of the universe. It's the genesis of modern cosmology.
And so what I end with is a question that I don't have the answer to: even though we're using machine learning to really automate the whole astrophysics stack and remove people from the loop, can machines be taught to ask the questions that we haven't, or we can't? Will machine intelligence ever replace the eureka moments by people? And will the entire astrophysics stack — not just from the data collection, but to the initial classification, but to the real breakthroughs — will that ever be able to be truly automated?
I'll end with just a couple of concluding statements. The modern astrophysics stack is obviously drawing very deeply from contemporary computer science theory. It's great because these frameworks exist; it's not so great because it means it's a massive educational challenge now. It means I have to teach my students not just about the mundane things, like exploding stars, but now I need to teach them about toolkits that perhaps I don't even know about. But it's pretty clear, in the context of the data deluge for astronomers, that there is a huge demand for the abstraction of the traditional roles of people from the ordinary scientific process that we've been used to. And yet despite all that, I think it's pretty clear that there remains a very important role for people, and I don't want to give that up. So with that, I'll end the talk and say thank you.
Well, thanks for a beautiful talk. Let's have a few questions.
Slide? Yes — good, okay, I can do that. Yep. So the question is, what are people actually doing with this sort of emergent network? I think they actually are looking at asteroids, and using these networks to follow up interesting near-Earth asteroids that could be pretty dangerous — that is actually happening. What's great, though, is that if you have this network, they don't all have to be doing the same thing at the same time. Maybe they spend 10 minutes of their night doing this; maybe one telescope is more interested in doing it than another telescope. Again, because this is all about resource maximization, we've tried to think about how you actually have telescopes from different groups that have different needs sharing data with each other, when perhaps they've never even collaborated with each other. So we've invented a currency for these telescopes called Starbucks — hopefully we won't get sued — but that again makes it very easy, once there is that currency and people believe in that currency, to actually do the best with the resources that you have.
The classification scheme you showed us — has it been frozen for ages, or it started developing? And do you expect during your lifetime to add another branch?
Yeah, so the question was about the classification scheme, that rat's nest of taxonomy. It has to change, because first of all we're making new discoveries, and we usually try to slot a new object into one of the existing classes, and eventually it becomes pretty clear that it just doesn't belong there anymore, and it will wind up branching out. There are new classes of variable stars found every, say, couple of years, and usually it happens now with the very subtle changes of light curves, because the really explosive events have been announcing themselves for 300 years and people have been following them up and trying to understand them. So yes, I think that classification taxonomy will change, and one of the things that I think might be interesting is to take machine learning concepts and figure out actual distances between all of these different objects, and see whether there's any phylogenetic tree that winds up emerging out of it, or concepts of species.
You folks are trying very hard to use computer science, but don't you also want to recruit from computer science?
This is an excellent question, and I'm glad to be able to address that a bit. The question was about whether we actually want to recruit from computer science and bring people in to essentially help us out. That would be great, but I think what's more exciting about this conversation that astronomers are having with computer scientists — and I hope you all see this — is that we're presenting to you data sets and types of questions that perhaps there is not yet a well-established theory to answer and to address. And so I see this much more as a collaborative kind of communication, rather than one where we're just drawing on people in a more rote way. I think what we've done in the Center for Time Domain Informatics, with other faculty in the computer science department and in stats, is produce for them a set of really important science questions that we would love to be able to answer, if only they could develop the theory and the formal mechanisms for us to do that with enough grounding that we believe we're on to something.
Okay, yeah — so the question is, if you're gaining confidence that you've got this awesome pipeline that doesn't require graduate students anymore in the real-time loop, how do you gain confidence that you're not missing something really important, something you hadn't thought about? I don't know the answer to that. One could try to throw dummy events into the data stream and hope that you actually wind up catching those. But then again, if you're throwing a dummy event in, it's still within the realm of the known unknowns. If something is truly different than anything we've ever thought about, it's going to be pretty hard to guarantee that you're never going to miss things like that. And that in some sense maybe is where the frontier gets pushed to: the next generation of people doing machine learning and automating workflows have to figure out ways to uncover the unknown unknowns in a more systematic way. It's a good question.