Boot-camp lecture on large-scale statistical inference for time-domain astronomy, during the Simons Institute's Theoretical Foundations of Big Data Analysis program.
“Make no mistake, computation is not our goal. Instead, the novel computation and algorithmic techniques are enabling what we do, which is fundamentally to conduct physical science.” – Joshua Bloom
“We'll find more supernovae, which are basically big explosions from dying stars in various different ways; we'll get more of those types of supernovae in the first two weeks or so once the survey turns on than mankind has found throughout all of history.” – Joshua Bloom
“We haven't quite got to the point of having machines write the papers for us, but that's maybe one of the ambitions.” – Joshua Bloom
“Very few people actually talk to botanists and ask whether they care about classifying irises; if they go out in the fields, is that the thing that they really want to know?” – Joshua Bloom
[Music] So I love this quote from Jim Gray. He meant this in, I think, the nicest way you could possibly take it, in the sense that when people get to work with astronomy data, and in particular people working on large-scale algorithms and sort of novel computation, they get to do so in a sandbox that is a lot easier to work with and work in than some of the other types of data that are large and streaming and noisy and dirty, like astronomy data is. So he saw this as this wonderful playground for Microsoft, and more generally for people like yourselves, to be asking questions of scientific importance,
but not so much on the physical side, more on the computational, algorithmic side. And of course if you make mistakes with astronomy data, you don't start wars, you don't blow up your company, you don't have privacy leaks, etc. So there is something very special about that. And of course from the physical science side we think our pursuits are reasonably orthogonal to commercial ones; I won't try to claim more consequential. So I'm very excited to be here and want to thank the organizers for the invitation to speak. So while our data is maybe what you could consider a fertile ground for interdisciplinary work between
the methodologists and the physical scientists, make no mistake, computation is not our goal. Instead, the novel computation and algorithmic techniques are enabling what we do, which is fundamentally to conduct physical science. That is, we're working with novel computation and algorithmic techniques because we have to, to solve a physical science question. For those that don't know, astronomers pride themselves in actually using new tools, and in the context of this talk we'll talk about algorithmics and computation as some of the tools that we get to play with, and we really pride ourselves in using those to achieve our end. So this is a picture now essentially 120 years old, from
the Harvard College Observatory, and you notice a lot of people looking down and meticulously taking notes and essentially looking at data. There was a big data problem back then, 120 years ago. The Harvard College Observatory had just opened up a new observatory in the Southern Hemisphere, and they were getting more images of the Southern Hemisphere than they knew what to do with. And they had a specific science problem in mind: they were trying to basically look at binary stars. You can see what's called a light curve in the back here, which is basically the change of brightness of a star as a function of time, and these are actually binary stars
that they were interested in looking at, because if you measure the periods of these oscillations you can actually wind up inferring, with some other data, what the fundamental properties of those stars are, like the mass, the temperature, etc. So that was of great interest to people and still remains of great interest. But in some sense you could view this as a prototype of grid computing, or crowdsourcing. And we used to call these people computers, in a pejorative sense, because they were almost all women at the time. But in a positive light, what we wind up learning is that many of the people
that were in this room were making fundamental contributions to astronomy, in particular Henrietta Swan Leavitt, who's pictured here. In a modern sense we have our own data deluge to deal with, and the 800-pound gorilla in the room for us is the so-called Large Synoptic Survey Telescope, which has really just kicked off construction essentially this month, and is a sort of billion-dollar-level project funded in part by your taxpayer dollars, that will be taking essentially more data on the sky at visible wavelengths than has ever been taken before. And just to give you a sense of the sort of raw numbers that we care about, the light curves that you saw
before on the back of that room, we'll be getting something of order a billion of those updated every 3 days. And so obviously getting people to look at that data doesn't really make any sense. We'll find more supernovae, which are basically big explosions from dying stars in various different ways; we'll get more of those types of supernovae in the first two weeks or so once the survey turns on than mankind has found throughout all of history, and it goes on and on and on. It's of order 20 terabytes a night of streaming image data. There's a satellite which recently launched called Gaia, which is in some senses a pared-
down version of the Large Synoptic Survey Telescope, getting sort of hundreds or so updates on the light curves of essentially a billion stars in the sky over the course of several years. And one of the things that you'll see throughout the talk today is that in advance of these surveys, a number of us are trying to cut our teeth on real data with sort of precursor surveys that have largely the same ambitions but perhaps pared down in the size and the velocity of the data. It's not just visible wavelengths that we're interested in; basically across the electromagnetic spectrum there are endeavors already ongoing that have very specific scientific ends.
In particular, one that I wanted to highlight here is one that's headed by my colleague Aaron Parsons here at Berkeley, and the goal here is to detect the signatures of the emergence of the first stars and galaxies after the Big Bang. And these are enshrouded in essentially lots of hydrogen, and what happens is you wind up sort of burning away, that is ionizing the hydrogen, at these very large distances, and the signature of that would wind up being bubbles that we might be able to detect on the sky. But it's many, many orders of magnitude below the foreground emission and just general noise, and so this is an incredibly hard measurement. And just to
give you a sense of the data rate that comes out of these radio interferometers that's already being achieved essentially today, it's something like 210 terabits a second. And over the course of an observing season, several months, after it's been correlated — that is, basically you take all the interferometric data and multiply it by itself and do a little bit of math after that — you wind up getting something like 200 terabytes. And this is a pretty significant endeavor for astronomers, in particular because these telescopes are in very remote sites. So one of the innovations here, while the algorithmics are actually pretty straightforward, the actual work that had to get done
was to build very specialized hardware, going from custom FPGA boards down to GPU and then ultimately doing the final analysis in clusters. Astronomers really do consider ourselves quite lucky in being able to play around in this large data inference space, and so depending upon the problem that we have and the tools that are available to us, we get to put on different hats. And so there are certainly cases where we need to be incredibly theory-driven — we've got very deep physical reasons why something needs to happen, and so we might apply physical theory to the data that we get — and in other cases we don't have much theory at all and we need to be
much more data-driven. In some cases a Bayesian analysis makes sense, sometimes a frequentist analysis makes more sense, and we have those tools available to us, and really it is just deciding — and one of the hardest parts is in making that decision about which toolkits we apply when asking questions of the data. One of the things that I've been interested in over the last several years is understanding this taxonomy of so-called variable stars. I already presented really two of them to you: one, I said, supernovae, exploding massive stars; another are these so-called eclipsing binary stars. But the taxonomy of these stars that we know about that
go bump in the night, that is, what we think has separate physical explanations, there are about 150 different classes and it's growing every day. Some of these classes have essentially one example in them, others have thousands or tens of thousands of known examples. And so what I wanted to do, and the question that I wanted to ask, is, given existing data sources — essentially synoptic images of large parts of the sky taken repeatedly in similar wavelengths — could we actually infer just from the change of the light curve itself, the change of the brightness as a function of time, could we infer which class a star would wind up belonging to? Now you might imagine we just take
all of the stars that we know that are the prototypes of each of these different classes and just apply templates to them. But that's actually very difficult, because almost all of these classes are sort of fuzzy in some sense, and people have made judgments about whether something belongs in one class or another, so there is no actual real prototype that everything has to look exactly like. The other important thing of course is that our data is noisy, irregularly sampled, and so that adds complications to the analysis. And then of course we also have spurious data; that is, we're pulling essentially values of the brightness as a function of time out of
raw imaging data, and sometimes we make mistakes, we basically make mis-estimations in how much noise we think is involved in each one of these observations. So these are actually wrong data, or incorrect data, or incorrectly inferred uncertainties. And then if we're going to try to actually extract more information out, it could be that the interesting things of what's going to happen in this event — and this is actually a theoretical curve from a gravitational lensing event — it could be that those actually haven't happened yet, and so we have to have some sort of notion of where things are going so that we can start marshalling resources to be able to do even more science. So
taking that sort of heterogeneous set of light curves and turning it into something that we can actually do inference on is something that I've been doing for a while, and we've taken a machine learning approach to this, and basically we try to imbue some of our domain knowledge into what distinguishes different sources from each other. And we build basically featurization codes that take this heterogeneous data and turn it into a homogeneous, if not sparse, large dimensional space — a rectangularization of the data. But the location of where something is on the sky — and so we have of order a hundred different features that we apply, really just looking at a whole bunch of different things you might ask
of time-variable data. And that's gotten us pretty far along in being able to make strong classification statements about stars that we've never seen before, without any people actually looking at the data. So on a data source of about 50,000 stars where we had some notions of labels, we're able to get sort of a gross misclassification rate across three different classes of only about 5%, which hadn't been achieved previously. And making use of the taxonomy to build a loss function was one of the things that we worked on for a number of years, and that actually improved the classifier. Another thing that improved the classifier, as you can imagine, because
in some sense this was a semi-supervised expert expedition, is to use active learning, where we would build a classifier on those stars, about a thousand of them, that had really strong labels, and then the classifier would ask questions of experts and said, if you told me the label of this other star, that would improve the classifier dramatically, or at least the most. And we'd go through several rounds of active learning where we're essentially getting experts to essentially buy more labels, which are expensive to do, and we improve the classifier a whole lot from that. So really what we wanted to be able to do in the end is
to take a subset of the really well-studied and well-known classes of variable stars, down from 150 to about 25 classes, and take essentially a light curve that looks like this and then spit out probabilities that it belongs to some class of variable stars — in this case 94% probability it belongs to a so-called RR Lyrae class. And we tuned down the total number of features using modern feature selection techniques, and we got a fairly good classifier, 15% error. And what you'd like to see of course is that you have basically all your power in this confusion matrix on the diagonal, and you can see that there is a bit of confusion
across some of these different classes, but you notice in some cases we only have sort of one or two examples in our training data, so we're not too worried about those. So the output of this endeavor was to take a survey that had been around for about 10 years and produce a probabilistic catalog of all these different stars. And we made a website that, we tried to help people basically go through and be able to search through this taxonomy and find the objects that were highly likely to be part of that class. So here you can see we're parsing through the taxonomy, and then we bring up basically a list
here of all the different classes of stars that are part of that, and then we're going into some of the subclasses of the RR Lyrae, and each individual star has its own page made for it: here's the raw data, here's the folded data, here's the probability vector of what we get. And in here we have a little social thing; we're hoping maybe this gets bought by Facebook. Okay, good, you're still awake. But what's kind of interesting about this, of course, is we're making these statements, but they're fuzzy statements, and this is quite unusual for astronomers. Astronomers like the idea of being able to go out and do science on a catalog of stars that are part of
Class A or Class B, but doing science with probabilistic catalogs is not something that people are generally used to in the variable star field. So we really have sort of two different ways that you can imagine doing your science with this catalog. And to emphasize again, we're not doing this machine learning endeavor because it's fun — it is indeed fun — but we're doing it in service of novel science. And so what are the different kinds of things one can do with this? One is to do sort of demographic surveys where you don't have a lot of capability of following up any of these individual objects, and what you're willing to do there
is trade high purity at the cost of lower efficiency. And so if I want to use a subset of stars that I know are very good for probing the 3D structure of the Milky Way, I want to make sure that that survey that I have, and that bucket of stars that I have, really only includes those types of stars, and even if it means losing a large amount of them out of my catalog, I'm really happy to have a highly pure sample. On the very opposite side of that spectrum is what you call novelty discovery, the notion that I don't mind dedicating lots and lots of
follow-up resources if it means I can find something that's quite novel that's actually really interesting. And so I'm willing to follow up with many telescopes and burn lots and lots of time and people resources and dollars to be able to find that sort of needle in the haystack, and for that I'm willing to have a very high efficiency sample at the cost of low purity. And so that's sort of what we did; we said, well, now that we've built this survey and this catalog, what is it that we can do with this to actually do something novel that wouldn't be able to be done by other means? And we found, in the novelty discovery
part of the spectrum, some very strange stars. And there had only been about 10 known at the time; we found basically about seven more of these really weird stars that change over courses of decades in their brightness. They basically will fade by factors of 100 to a thousand, and people still don't even know what causes this fading. But the fact that we're able to find it in a survey and data that had been open and public for the last decade, I think, is a testament to the fact that if you're doing these fuzzy catalogs you can get a lot of traction. One of the other things that I'm interested in is
not just variable stars, things that are changing in the sky, in our own Galaxy generally, but things that are blowing up. And there we have another sort of interesting challenge in that we're sometimes looking for things that we know about — these are so-called type Ia and type IIP supernovae, these are pretty normal explosions in the universe — but then there are other things that have been theorized, really bright events that take a really long time to evolve, and really short events, and faint events, that will evolve over the course of just a couple of days. And in the case of these events we really benefit from being able to recognize
these events more or less in real time, so that as new data streams in, if we can make inferences about the fact that something like this is happening, we can train our telescopes and optimize our resources to do the follow-up. It also means that if we're wrong about things, and we basically only figure out that there was an interesting object happening over here and we figure that out a year later, well, that's basically useless to us. As scientists we want to be able to optimize our scientific follow-up now. The other thing that I can't put on here, by definition, are sort of the unknown unknowns. We want to be able to build classifiers, streaming classifiers, that are going
to be able to identify objects essentially in real time that we've never even envisioned before. And having people listening or looking at data clearly doesn't make sense in this sort of streaming environment, especially as the volumes of the data are increasing. So a big part of what I've been doing is really trying to automate that whole inference stack, from the strategies of how we observe in the sky, to how we actually schedule our telescopes, how we do the observations and the preliminary analysis, how we do the finding and the discovery, and all the way up to actually getting robotic telescopes to follow up on discoveries made by other robotic telescopes
before any people actually wake up and know that something's going on. We haven't quite got to the point of having machines write the papers for us, but that's maybe one of the ambitions. So I'll just talk for the next couple of minutes about the discovery engine that we built, which we put into a real-time system, basically asking the question, are these objects on the sky real, or are they bogus artifacts of having observed the sky with a detector that has noise properties associated with it, and our inability to actually very cleanly subtract out new images of the sky from reference images of the sky? And what we did is, again,
featurize these images and then used what amounted to a number of different classifiers, and figured out that for our purpose here random forest was quite good at producing a classifier that bested any of the human labels that we had even made. We had noisy labels and we were able to figure out that the humans made mistakes with that. One of the new endeavors that we're working on along these lines is to be able to use deep learning, so we don't have to imbue our classifiers with domain knowledge to build the features. And some of the preliminary work on that, of being able to use essentially these different image postage
stamps, has already been ongoing, and we've been doing that up at LBL and in conjunction with some people here in the EECS Department. But we were able to get this classifier using random forest into basically a survey that was ongoing over the last four years, and found a number of interesting events, as you might hope. One of the ones we're most proud of is having identified a supernova in a nearby Galaxy. And one of the important things — and you can see basically there's an arrow, which doesn't appear in the sky, by the way, you actually have to draw that afterwards, to tell you where to look — a supernova that wasn't
there, and then it appeared, it got brighter and brighter, and it was the earliest supernova after explosion ever observed; basically about 11 to 12 hours after explosion we think we got our first glimpse of it. The important point here is that this object was in a very famous Galaxy that's observed by amateurs every night, and probably about 3 or 4 days later, had we not found it with our machine learning framework, it would have been found by amateurs, but it would have been too late. We got a lot of great data in just the first day after this event that would have never been able to be obtained by
any other means, if we had observed it and detected it 4 days later or so. And so it shows you, I hope, the sort of onus on getting your inference not just robust but getting it very quickly. Just in the interest of time I'll skip over one of the results that came from that, but just show that we have in the survey basically a robotic inference machine called the PTF robot, which goes in, looks at every new image that's been streaming off this telescope, and not only does discovery but then starts to say, well, I think this is a supernova, and it actually logs in before any
people are awake. And then people over the next couple of days wind up saying, oh yeah, this actually looks like a supernova, and then people say, actually this looks like a weird supernova. And this thing then turned into a Nature paper that came out last year, where people wound up realizing that there had been sort of a pre-explosion in that part of the sky, which was quite interesting for understanding the progenitors of what makes those types of events. And then last month another sort of early discovery of a supernova by our team came out in Nature, where we were able to make observations very, very early on, which
helped us make some inferences about the nature of the thing that wound up exploding. So this is maybe a bit too pedantic here, but what I'll end with is a statement about using novel computation techniques, or machine learning techniques, or algorithmics, for the sake of physical science. Those of you that are in the machine learning world know the iris data set, where you're trying to classify three different types of flowers based on four different sets of observations of the length of the petals, etc. It is a really interesting sort of toy data set for people to test new algorithms, and
if you're building a new one you want to make sure that it would scale to a larger data set. But very few people actually talk to botanists and ask whether they care about classifying irises; if they go out in the fields, is that the thing that they really want to know? So a big emphasis when I talk to groups that are not astronomers is to point out that when you're working with physical data or social science data, there has to be a very specific question that obviously taxes your new algorithmic techniques but ones that you hope are of great interest to the actual physical
scientist or a social scientist. What we've done here is not just build sort of real-time streaming frameworks to be able to do inference; we did it so that we could basically understand the nature of exploding stars, etc. And so that's an important thing to keep in mind, is how you wind up working with people and how you wind up finding problems that are of interest on the physical and social science side as well. That is really one of the main thrusts, if you haven't heard about it already in this conference, of a new institute that's just been started here at Berkeley called the Berkeley Institute for Data Science. And the idea here is
to be kind of an incubator for this type of activity, where people working in algorithmics and computation can come together with people on the physical and social science side and find sort of novel problems together that tax both sides of that house. So I'll leave you with some parting thoughts. I hope it's clear that the astronomy data deluge demands an abstraction of the traditional role of people actually looking at data in the scientific process, and what's emerged of course is that the only way to deal with these data streams is with the types of techniques that many of you are working on. And I'm happy to talk with people offline
about what we're working on. BIDS will hopefully be one of these places where more work like this winds up emerging. So I'll end with that. Thank you.