CITRIS Research Exchange seminar on ML pipelines for discovery and inference on streaming astronomical time-series data (variable stars, transients, synoptic surveys).
Exact date within 2013 unconfirmed.
“My own field is in astrophysics, and understanding in particular transient phenomena, that is, understanding objects which change and go bump in the night.” – Joshua Bloom
“Basically Kepler is able to see incredibly small changes. It'd be the equivalent of flying over New York City and looking at the Chrysler Building and noticing that one light was out on somebody's desk through looking at all the windows.” – Joshua Bloom
“The greatest insights in astronomy… come only when we do great follow-up, but follow-up is incredibly expensive.” – Joshua Bloom
“The manifest destiny of where astronomy is heading is this full abstraction of the discovery steps and even the scientific inference steps, allowing us to sit in our armchair during the daytime and make strong statements, and perhaps transformative statements, about how the universe works.” – Joshua Bloom
Great. Well, thanks very much for having me. Lauren ended his talk very nicely hinting at, well, not even hinting, essentially saying that some of the problems that he's encountering in one domain are similar to the challenges that he's finding in other domains in genomics. And I think this really speaks to a commonality of challenges that we all have on the application side of big computation, in trying to make the best out of a deluge of data and come to important scientific inferences that push our own fields. My own field is in astrophysics, and understanding in particular transient phenomena, that is, understanding objects which change and go bump in the night. Some of these are explosive events. Some of the changes are actually much more subtle. But what we're seeing, like Lauren, is that there are tools that are out there and some algorithms that actually need to be refined and perhaps in some cases be created, to make use of the data and to maximize the science that comes from that data.
What's interesting, I think, in the context of astrophysics, and this adds an important wrinkle from a computational perspective, is that when we're talking about time-domain data, this is watching things change as a function of time in the sky, there are a number of events, a whole series of classes of events, that need immediate attention. So it's not just going back a decade later and looking at a data stream and saying, oh, there was something really cool there, I wish I had known about that at the time. It's actually identifying what's worth spending resources on before you get swamped and another new event comes up. So it's a maximization problem in the context of massive amounts of data. As Masoud mentioned, this work is being done in the context of what we're calling the Center for Time Domain Informatics, which is housed in Evans Hall here on the Berkeley campus. We've been funded by the National Science Foundation for some of this work, and some of our computation time has come from Yahoo and Google. The important people here — well, everyone's important in their own right, but the important people here are the fact that we have a number of professors from the statistics and computer science departments collaborating with us. Again, we have a bunch of work happening not just on the astrophysics but also on the algorithms that will drive the frameworks for us to maximize our understanding.
So it's perhaps appropriate to start off, for those that aren't astronomers, with a traditional view of astronomy. This is Vermeer's painting, a famous painting called The Astronomer. It hangs in the Louvre, painted 1668, and this shows a fairly quaint view of an astronomer looking at a globe of the heavens and some maps of the heavens in this book here, and making sense of it all. This is not so much different from a Greek philosopher just thinking about things and making grand conclusions about how the universe works. A more modern view is one that's actually not so different than what was going on in 1949, where you had an astronomer, in this case Edwin Hubble, smoking his pipe, looking through a very large telescope, in this case just the finder, and making grand inferences about the universe. Obviously he would do this with the lights off, although legend is that he never stopped smoking a pipe. This is not so different from the way that astronomers work today. You know, replace the human eye with a charge-coupled device, replace this control here, a hand paddle, with some robotics control, and you more or less have a prescription for how we take data today in astronomy.
What's important is that Hubble discovered with small telescopes, what we now consider small, some of the grand insights that have driven 20th-century and now 21st-century astrophysics, in particular the expansion of the universe. And he did that by discovering a time-domain phenomenon called Cepheid variables in a nearby galaxy, recognizing that the Andromeda galaxy is not actually within our own, but is actually a dissimilar object that's quite far away. And he found a number of other Cepheid variables that allowed him to infer a relationship between distance and velocity moving away from us, and hence the expansion of the universe. So time-domain astronomy has been critical in a number of different ways, and Edwin Hubble is a good epitome of an early time-domain astronomer. What we're talking about here, when we think about how we're going to deal with this data deluge, is in taking the traditional way of doing science, of observing, finding objects on, say, images, making discovery on those objects, conducting follow-up, and then getting to a scientific set of inferences. It's how do we deal with this in the context of this data deluge? And from the beginning there's been humans in the loop in every one of these various steps.
I should say, by the way, there's a finer distinction between finding and discovery. One, you might consider taking metadata out of an image and sticking it into a database; that might be called the finding process. But the discovery that what you're looking at is something interesting or something new takes a bit of a leap. There's a famous example of Galileo who, while studying the moons around Jupiter, actually observed Uranus but didn't realize that he was looking at another planet, so he's not credited with the discovery, although he actually found it and it's in his journals. So there is a big leap that has to happen from finding to discovery. So let me talk to you about the dynamic universe, some of the astrophysics that we're interested in, and then we'll talk about some of our approaches to this problem. The first thing to recognize is that every single star in the sky changes its brightness, in some cases very dramatically and in most cases actually quite subtly. These are what are called light curves. Six of them from the new Kepler telescope, which is flying in space, has a very precision photometer that's capable of measuring the brightness of bright stars as a function of time. And you see a scope here of about a week. So this is something of order a month, and you see these very clear undulations in the brightness of this star.
That might not be too surprising until you start looking at the scale here. These undulations are of a size of sort of 12 parts per million. Basically Kepler is able to see incredibly small changes. It'd be the equivalent of flying over New York City and looking at the Chrysler Building and noticing that one light was out on somebody's desk through looking at all the windows. It is an incredibly small change in brightness. And what they've noticed is that there is no such thing as a quiescent star. Every star is variable, and the origins for this variability are quite varied and diverse. Stars pulsate, they rotate, that is, they spin. Some stars move in front of other stars, or planets move in front of other stars, and you get what are called eclipsing systems. There are some episodic systems where you get unsteady mass flow and you can get some outflow, that could lead to a brightening event, and there are explosive systems, basically when stars wind up blowing up. There are hundreds of classes of variable sources and events, and making use of a generic data stream means trying to put things into their respective buckets, and that's obviously a tremendous challenge.
The subtleties are one thing, but the truly exciting events, at least for me, are the ones that are quite rare and also herald the end state of massive stars, that is, supernovae. And here's a picture, essentially a before picture, looking at some of the Magellanic Clouds, satellites of the Milky Way, and you see the after picture, and I'll leave it to your eye to figure out where the new star is. It's right there in the middle. It's a really bright thing. This was found in Chile in February of 1987. It's the famous supernova 1987A. It was found by the human eye, and somebody just happened to go outside, probably for a smoke on a pipe, and happened to look up at the heavens and notice that his favorite part of the sky was a little bit different than it had been the night before. One of the most famous discoveries in astronomy in the last 100 years, and it was done by somebody who just happened to be outside at the right time. There's not just supernovae of types that we know. It doesn't matter that you know these various different nomenclatures here, but Type 1a supernovae are the things that tell us about the expansion of the universe. Type 2p supernovae are the most common types of supernovae in the universe. And then there's a whole bunch of classes of sources that could go bump in the night, that only theorists have dreamed of, that we've yet to see.
So we have sort of the known knowns, and we have some known unknowns — sorry for those that aren't fans of Donald Rumsfeld — and of course there's the unknown unknowns that we haven't even thought of yet and been able to put on this plot. This is another light curve, days since explosion. And this is just a measurement of brightness. Some things could be incredibly bright, some could be very faint. Some take time scales of only a day to evolve. And to truly extract the great science out of these sorts of events — which, by the way, neutron-star neutron-star mergers are the objects that are thought to produce most of the gravitational radiation that we should see with the next detectors that are coming online in something like 2015. This would be the first detection of gravitational wave events, and these are probably the objects that are going to do them. They will create an electromagnetic signature that traditional astronomers might be able to see if we can react quick enough, if we can actually find these things in the sky. So there are multiple thrusts in understanding and exploiting transients, and the introspection one is a clear one. We want to understand what gives rise to the changes that we see. What are these events? There's the exploitation of them, which is a statement to say that we don't really care about the physics of an individual event, we only know that they're great for studying something else — in-situ laboratories, or perhaps lighthouses or beacons to other places in the universe.
A classic example of this might be pulsars, which are rotating neutron stars putting out pulsed radio light. We don't really understand the physics of pulsars, but we know that they're tremendously good clocks. So you can use those tremendously good clocks in understanding, and maybe even finding, primordial gravity-wave radiation. They've also been very good for understanding gravity waves in general, in the context of strong gravity. Nobel Prizes have been given for the study of gravitational radiation in the context of pulsars. So exploitation is one of the categories. And then this exploration, the idea that we want to search the time domain and find new things that we haven't even conceived of yet. Okay, so let's start getting a little bit closer to some actual data. But before I get there, I want to make clear that when you make discovery on a massive data stream, that is an incredibly difficult event. Extracting metadata out of images is one thing, but saying that of the million things I just looked at, these five might actually be fairly interesting to follow up on, is quite important. But then understanding which of those five you want to spend your precious telescope resources on is an important one, and discovery is really only the start.
The greatest insights in astronomy — and this is where the real-time needs for classification and understanding come in — come only when we do great follow-up, but follow-up is incredibly expensive. We have people involved in that. This is using large telescopes to, say, follow up discovery of a number of different transients on a small telescope. People, telescope time, resources, and ultimately it's about money. So we can couch some of these classification needs in the context of some need of maximization of resources. Talking about data scales over the next several years, in the context of surveys that I'm interested in, mostly optical waveband surveys. There's a couple different ways to capture this, given that every pixel of a CCD on the sky captures about the same amount of data. When you think about this metric called étendue, which is essentially a statement about how powerful the survey is, you can translate that roughly into total sizes of data volumes, and you find out that we're living in an era now where traditional surveys are going to be producing orders of petabytes of data. But we're getting to a point, say in 2019, with the Large Synoptic Survey Telescope, where we're looking at something which is approaching 200 petabytes of raw imaging data, and making sense of all that in real time is obviously a significant challenge.
Just to give you a sense of scale, don't be scared with this silhouette man standing over here. This is the scale of the Large Synoptic Survey Telescope. 3.2 gigapixels of imaging, getting data more or less every 15 seconds. So it's a tremendous data rate. So this is really the 800-pound elephant, or gorilla, or whatever, in the room. And we're looking at having to get scientific inference out of 800 million sources every 3 days, and that amounts to something like a million supernovae a year, and something like 20 terabytes a night of data. Gaia, which is an interesting space-based mission which is going to precede LSST by a number of years, will be looking at something like a billion stars, but only sort of 70 times over 5 years, and we'll also find an appreciable number of supernovae. Now the good thing is we have the ability to now cut our teeth with real data from real-time data streams. So what our group has been doing is looking at the Palomar Transient Factory data, which is a survey using the same telescope that you saw Edwin Hubble looking through, but now completely roboticized, and now not with a bunch of Edwin Hubbles’ eyes sticking at the end of the focus, but instead essentially paving the focal plane with silicon.
And you see that we get a tremendous number, and it's actually quite a wide field of view. Here's the full moon for scale. We get a quite large number of candidate objects a night. This is more or less just detections of these individual objects. But only about a hundred, or say a thousand, of them are bona fide, actually varying sources. And of those maybe 300 of them are variable stars, and maybe about 10 of those are new transients. And our group is most interested scientifically in getting at these 10 transients. So we have a huge compression ratio, factor of ten-to-the-five, essentially, in going from things which we're not at all interested in, in fact are artifacts of the way that the data were taken and processed, down to the things that we really want to spend our resources on. So what we've introduced, because obviously we can't get a million graduate-student eyes on this data in real time, we've introduced the ability to have expert training on a small subset of the data, looking at these subtractions to find these new objects. And we present to the experts essentially an image of the sky as it is in quiescence, the new image, and a subtraction of the new minus the reference. And we ask them, is this real or bogus? Essentially, is this a discovery or not? And this, to a trained expert, no, this is probably no. And to a trained expert who looks at this, they say, oh yeah, that's a real supernova. And in fact, that is actually a real supernova.
So it allows us to get a tremendously good rejection, because we can now do some machine learning on the input, the labeling, that comes from these experts, on data on a massive scale. And just in the first month of data last year, you can see that most of the stuff that we were looking at was tagged as being essentially bogus, and a good fraction of it, although much smaller, is tagged as being something which is potentially very real. So that's allowed us to take humans out of the discovery loop within the Palomar Transient Factory. And by requiring a number of epochs of detections for a true discovery, we're able to plot on the sky where all our different transient discoveries are. And it's a very large number. You know, the Palomar Transient Factory found more supernovae last year than any other project, even those dedicated to supernovae. And that was just one small facet of the entire endeavor of that project. So we're already making great progress there. But what's interesting is not to say, well, we've got something of interest there, but we now want to know what that is. Because it may be that we don't even have enough resources to follow up every one of these objects. And that's actually true; it's quite daunting to believe that.
And it should be pretty easy. It should be that here's the taxonomy of all these variable and explosive sources I was telling you about. It should be that we just get a lot of data. This is again a light curve, flux as a function of time, or brightness as a function of time. And you see there's some time scales of different types of events. There's a Mira variable, which has a very large amplitude of variability over very long time scales. Supernovae, which is a huge amplitude of variability and then it winds up declining over also really long time scales, and then other types of sources all belonging to different classes of variable sources. And we are trying to make sense of that and try to understand that from the PTF data stream. The problem is, and this is what's kind of interesting, is that it's not like we're sampling the data in regular time and we can just get a period out of that data just doing an FFT. We are getting the data in an irregular way. The data have noise on them, and we have to try to understand what this event is. In this case it's a microlensing event, but here are all these different data points in time. So trying to get a sense of what a source is in the context of noise is a tremendous challenge for us. And perhaps even worse is that in many cases the metadata that we extract out of those images are actually spurious, that is, incorrect, or we haven't correctly characterized the systematic error, and that's a significant problem for us. And then if we're trying to actually do some follow-up, maybe the telltale signature of the event hasn't even happened yet, but we want to send out some alerts to help us get going on it.
So we've identified machine learning as a reasonable approach to dealing with this data deluge. And here we create something called features, which are a homogenization of the data from this noisy and irregularly sampled data set into one that's much more regularly gridded, in this feature space. And we have a number of different metrics. These ones in blue are related to the time domain. And then context, that is, where this object is, near a galaxy for instance, where it is in space, is also quite useful. And what's pretty remarkable is, just taking the immediately available data from the Palomar Transient Factory and trying to identify transients from that, that is, just any object which is truly essentially explosive and not just a variable star doing its thing, is that what we're finding is we can do very well in a cross-validated machine-learned way. We're getting to something like 95% efficiency at 98% purity. That is, we would lose 5% of the true transients if we were willing to take follow-up data and only have 2% of the follow-up data not be of true transients but of variable stars. So this is, I think, an impressive step for us, that we're now able to go not just from extraction of the data out of the images to discovery, but now we can make some of these rudimentary statements about what those objects actually are.
But we want to go even further. We now want to follow as a function of time how these events are changing, and try to get an even stronger statement without the need for even more data. So what we've been doing within the group — and this is a paper by Joey Richards, who's a postdoc in our group — is look at existing well-labeled data sets from, well, one space and one ground base, at Hipparcos and OGLE. These are ones that have very nice labels on them, and we can try to learn off of those labels. And what's clear is that in feature space — this is an example of two different features over all the different classes of objects we're concerning ourselves with, in amplitude of the variability and in some sort of frequency sense, if there is enough data to be able to get a dominant frequency — you can see that the very long frequencies, or, well, sorry, the long frequencies, as in long periods, are out here. This is the Mira variables. And then we have a whole bunch of short-period objects over here. And then you can see that there's an anti-correlation, for instance, of this feature with this feature. So if we learn off of this, we can then, when we get a new instance which may have a value in this feature here and another value in this feature here, we can pretty rapidly throw out whole classes of sources here and another whole set of classes here, and we can hone in on the ones that are most interesting.
And what was shown in Richards’ paper is that we can do quite well in a cross-validated sense, in showing now the confusion matrix of how well we do essentially classifying a source relative to itself. And what you'd want to see is a total amount of power of one along this diagonal. And what you could see is that there's some spillover of some classes into other classes. What's remarkable about this is that when you highlight what types of physical sources these classes are, you wind up seeing that a large amount of that spillover happens within some of these larger subclasses. And so, in some sense, what the machine is doing is figuring out that there's an intimate relationship between some of these classes here, because it has a very hard time distinguishing them. Their distance in some abstract space is very small. I'm running short of time, so I'll just say we're trying to make use now of the hierarchy and of this taxonomy so that we get even better classifications. And what we find is that if we take our variable-star classes and we just separate them into three very broad classes, our misclassification rate is starting to go down pretty dramatically, to the point where we're only misclassifying about 5% of the variables.
The idea now is that we can start abstracting this large amount of data to a human who might start asking questions about a given source or a given position on the sky, and present to them a machine-learned set of statements about what it thinks that object is, before resources are used, and in some cases abused, to follow up sources that really don't deserve it. It would allow the human operator, who's at the very end of this long tail of scientific inference, to be able to make great use of this. And what's quite remarkable is that the LSST Corporation has now put out an iPhone app where they're able to push out events using a protocol that I invented called VOEvents, that allows an amateur astronomer to essentially get real-time alerts of these things. Now these alerts are currently vetted by humans who go through and say, yeah, I think it's that. But the next step, obviously, from the Palomar Transient Factory and other new surveys that are just now coming online, is that we basically have the machines making not just that discovery step, and not just that initial statement step about whether it's a variable or whether it's a transient, but going deeper into the classification and saying this is a supernova that may be important for this type of science. So it's enabling some great citizen-science follow-up.
So I think we can come back now to this original picture from Vermeer, and it doesn't look so quaint anymore. It kind of looks fairly modern to me, in the sense that we replace this globe here with a screen that's visualizing the results of an SQL query that was asked of a machine that's probably not even local to this guy's office. And what you wind up seeing is that the manifest destiny of where astronomy is heading is this full abstraction of the discovery steps and even the scientific inference steps, allowing us to sit in our armchair during the daytime and make strong statements, and perhaps transformative statements, about how the universe works. So I'll end with that, and just remind you that discovery engines in the context of astronomy are already swamping our available resources. We literally cannot follow up every object that we find. So knowing what to follow up is key, and this autonomous classification is going to play, and is becoming, an important tool for us, and is really abstracting the traditional role of astronomer in the entire scientific process. What's also very clear is that we have this rich interface between machine learning, computer science, statistics and astronomy. And there is so much commonality with the other data-driven fields, as you already heard today and you'll hear more of now. I'll end with that. Thank you so much. [Applause]
[Q&A] We have time for one quick question before I get to the next speaker.
So you mentioned the fact of getting the people into the loop, but again, how do you actually trust when actually they represent something in a sense, because you don't know what type of rule each of these astronomers are using. So how do you actually make sure the quality of the results?
Well, so what you can benefit from is having multiple experts all labeling the same data, and then you can take some aggregate sense of what the true label actually is, and then see how individuals were, in a distance sense, from getting that label correct. So here you just say that the answer is what a large number of experts say it is. And if that's the case, then you can learn how to weight people according to their expertise across all of this different labeling process. So we have a very rudimentary algorithm for how we go about that in the context of discovery on the two-dimensional images. But we're now thinking about how you get experts weighing in on individual sources which, when identified through some techniques to tell us that it might be an important source for us to understand, because it lives in a feature space that isn't already well labeled by other sources around it. So getting expert learning as part of the process is critical. The important thing here, though, is that all of that happens in a relaxed sense, that we haven't yet identified a place where we think there's tremendous use for experts in the real-time loop. And in fact, that's kind of what we're trying to get rid of, is the need for that. So getting experts in the learning process means that we can get them out of the application process. Thank you so much.