Data-driven inference at scale: Bayesian RR Lyrae period-luminosity fits giving ~1% distances, Gaussian-process image registration for astrometry, machine-learned variable-star catalogs, predicting stellar spectra from light curves, and real-time transient classification in the Palomar Transient Factory.
“These are the tools of the trade at some level: hardware, software, and peopleware, or carbonware.” – Joshua Bloom
“If you give me an RR Lyrae and several observations of it, and I can fit its period, I can tell you its distance to something less than 1%, which rivals anything you can wind up doing with Hipparcos or Gaia.” – Joshua Bloom
“You ultimately don't want to do machine learning just for machine learning's sake; you want to do it because you're doing it in service of science.” – Joshua Bloom
“So this is a little bit of a party trick, in a sense that you give me a light curve and I'll tell you what its spectrum is.” – Joshua Bloom
So this title is a bit tautological. So any inference is always data-driven; if not, obviously, then our inferences are based purely on our prior beliefs. I think it's true. It's worth at least trying to visualize the space that I get to work in as an observer analyzing data. And this isn't to say, in this multi-dimensional space that I've shown here, that these are independent axes by any means. But what I wanted to do in this talk is highlight some of the places where we spend our time, particularly with computational challenges. So I'll have a couple of discussions about some of the analyses that we've done in this space over here, and then getting into more of the machine learning side, up into the top right.
Of course, even heavily nonparametric, almost fully data-driven analyses require some sort of insight into the result, and getting black-box results in astronomy is often very difficult to swallow. And even, as we'll see, when we do machine learning in a fully frequentist and data-driven way, we always bring some level of theory to the table, in particular upstream from where the actual machine learning happens. So these are the tools of the trade at some level: hardware, software, and peopleware, or carbonware. And implicit in this, I guess, is one of the soft challenges that we have in this field (this was very nicely discussed by Mike earlier today) of getting people that aren't ordinarily trained to be able to do these sorts of analyses, but understand very deeply the astronomical questions that we're asking. How do we get them doing novel work, and how do we get them working on large-scale computational analysis in a way that we can all wind up trusting?
So let me start off with a very simple problem, and this is near-field cosmology, if you'd like. This is basically doing distance ladder work using pulsational variables. This is the equation that describes what we see: apparent magnitude, there's a distance modulus in units of magnitudes, and what the absolute magnitude is at some fiducial period of one of these pulsational variables, and then essentially just the slope as a function of period. Hey, so I'm doing your talk while doing mine. So this was discovered, obviously, by Henrietta Swan Leavitt about a hundred years ago, and has now been understood very well at the physical level.
So we come to this problem where we're trying to basically fit period–luminosity relations for RR Lyrae stars, and it's not a whole large number that we have in this data set, but we've done it over something like 13 different bandpasses. And this is something that you would think would be pretty straightforward, except what we're looking at setting up is a 307-dimensional space where we actually want to know what all these different parameters are. We want to simultaneously figure out what the distance to every single star is, and we have priors on these stars, and some of these priors are different: some have parallax and some don't. And we want to basically throw this into a large-scale framework to be able to get these answers. And this is a deterministic MCMC model. It took about six days for a single chain to run on a reasonable-size computer. We can use parallelism, of course, to test convergence, but there's still a lot of work to be done, as I understand it, critically, for being able to do fully parallel MCMC without having to make a large amount of approximations.
But the results are pretty nice. In fact, if you look at the reddest bands, where there's very little dust and the photometry is great (these are from the WISE satellite), it actually looks almost too good. And indeed, when you go to these very long wavelengths, you wind up seeing that the pulsation period and luminosity relation is dominated by very, very simple physics, and so it actually winds up making a lot of sense. So this is the result of a simultaneous fit using those 134 RR Lyrae, but with data from many different bandpasses. And it's nice in and of itself because (it's hard to probably see) we can actually then determine what the intrinsic scatter is on these relations as well. And you see, in some of these bandpasses it's at the 1% level, which means if you give me an RR Lyrae and several observations of it, and I can fit its period, I can tell you its distance to something less than 1%, which rivals anything you can wind up doing with Hipparcos or Gaia.
So getting 1% distance uncertainty within the Galaxy is pretty novel. And what's pretty remarkable to me, not being directly in this field, is just how much of a frequentist approach people in this field have been taking for decades, and now when you apply a large-scale Bayesian analysis, you end up getting some pretty interesting results. What's also nice about this is I can do 3D tomography of dust through the entire Galaxy. And what you see here is a validation of what you might naively expect: the RR Lyrae that are well known and bright enough to be detected in these different bandpasses, the ones that are closer to the plane, are actually nearby, because of just our ability to detect these things, and they are more heavily extinguished. So now what we've done is applied for time to basically be able to get something like several thousand RR Lyrae over the whole sky, and then several thousand Cepheids and several tens of thousands of Miras. That starts to get into a non-trivial computational regime for us to be able to do these sorts of analyses in a reasonable amount of time.
So jumping over to another interesting near-field endeavor that I'm interested in, which is doing astrometry on nearby objects: measuring parallax, proper motion, etc. Then you get into some of the more juicy things like binary parameters, and maybe even discover astrometric microlensing and find black holes that are isolated, etc. And so you'd like to be able to see the motion of a star on the sky as a function of time and just fit all this stuff. But as it turns out, before you even get to this point, you have to do something even more basic, which is image registration. And again, image registration is the thing that all of us do if we're looking at images at all and need to figure out where things are on the sky relative to each other. It turns out this also has been done with a very frequentist eye, but we've taken it upon ourselves to try to create an analysis package that allows us to essentially map the location of any object from one frame to any other frame.
And that's a simple regression over multiple dimensions. Of course, you can have priors on what the different scales are on different images and what their offsets are. Excuse me. But then the extra thing that we've done is add the ability to do Gaussian processes to figure out local distortion fields. And what you wind up realizing (it's obvious in retrospect) is that if you have a group of stars up here and you have a sparse field down over here, and I want to know the location of one of those stars with respect to another star in the area where you have a large stellar density, that should be known better because you've got a lot more tie objects around there. Whereas all the traditional techniques that have been applied to astrometry that I've seen don't have a notion of what the local distortions are and being able to propagate the errors well.
So we have a package; those that are interested might want to pull this over on GitHub. And the nice thing about doing this in a fully Bayesian way, of course, is that we can get covariant uncertainties in celestial coordinates, which is also novel if you've been working in the SExtractor regime for a long time. Mapping observed points can incorporate variance through the image. Like I said, we can have non-trivial distortion fields that are completely non-parametric, and then we can think of astrometry as updating: as we wind up mapping one image to another image, we wind up getting a probabilistic feeling of what the various parameters are that do that mapping, and then I can use that as a prior when I do the mapping for another field. So for 71 observations of a high-proper-motion white dwarf, you can see the results of the motion in time, so it's blue to red over the course of a couple of years. And this took an hour or something like this, and this is embarrassingly parallel because I can do every other mapping as I'd like.
So let me jump into machine learning, which is the other part of the space, where we basically want to take something like this light curve right here and get some notion of what the probability is that it belongs to a certain class, an RR Lyrae class, for instance. So one of the applications that we took upon ourselves to work on was to look at a catalog of 50,000 stars from a southern time-series survey, and be able, with something like 810 training samples over 25 different classes of variable stars, to produce a fairly good error matrix on guessing how well we would do on the rest of those stars to figure out what those classes are. This is a very large dimensional space, and it is highly nonparametric in the sense that we're just taking light curves and just generating features off of those and letting the machine decide what the important features are.
But one of the nice things is that we were able to produce a catalog, which you can check out (it's called bigmacc.info), of these 50,000 stars and what their different classes are, and put them into some sort of hierarchy where you can then go through and browse. And individuals who are interested in RR Lyrae can wind up pulling over stars that are high probability of being RR Lyrae, and then drill down into these individual stars of interest, and then see the probability vector of what this source actually is, look at the original light curve, look at how it's folded, see what these different probabilities are, and even have a little social component, so you could tell your friends.
One of the things that's important to realize when you're doing machine learning over these large amounts of data is that it's pretty much impossible to check all of these things by eye to make sure that you're actually right. But you ultimately don't want to do machine learning just for machine learning's sake; you want to do it because you're doing it in service of science. So there's two different regimes where you can work: where you're doing demographics, where you don't have the ability to do lots of follow-up, so you want to basically take the results of the catalog itself; or, if you want to find some rarities, you might do lots of follow-up and be very comfortable with an impure sample, but with a high efficiency of having found those rarities. So one of the last papers from my student who graduated recently was to take the results of that probabilistic classification catalog and find so-called R Cor Bor and DY Per stars, essentially doubling the number of these stars that are known, with a catalog that's been around for the last ten years. We were able to find these things with very, very high efficiency.
The other thing that we've been doing, and I just have a couple more slides remaining here, is taking the results of a different time-domain survey and asking the question: can I predict the spectroscopic properties of a variable star based on its temporal properties and colors? And the answer is surprisingly yes, with a very high degree of accuracy, scary degree of accuracy. Again, an 80-dimensional regression problem, where we can predict temperature, gravity and metallicity with increasingly better scatter as we just basically don't use colors only, or actually you wind up using light-curve features. So this is a little bit of a party trick, in a sense that you give me a light curve and I'll tell you what its spectrum is. So it turns surveys like LSST into spectroscopic surveys, but it's also very useful in that it will help us, I believe, find anomalies and decide what are interesting things to follow up. Let me skip over this bit right here. Obviously many of you are aware of the different surveys coming online.
But one of the things that I've shown so far is essentially offline analysis, where if I need more computation, I can just wait, or I can just go out and buy it. But in time-domain analyses, where you're actually trying to maximize your follow-up resources to do the best science possible, there is a cost for being wrong, in a sense. There's a cost for missing things that are important, and there's also a cost for going after things that are actually boring. So especially when we get into a regime where we need to do follow-up not just every week, but we're going to try to do something robotically every night, now we really need to figure out how to do inference in a very well-calibrated way. And so some of the things that I'm involved in with Peter with the Palomar Transient Factory has been trying to really automate that whole inference stack in the places where there had been a whole lot of people who had to make decisions in a real-time loop.
Getting a little bit short on time, so what I will say is that we were able to get a fairly good classifier of just this question of discovery, and show that there are different algorithms, obviously on the same input data, that wind up doing better than others. One of the challenges with any sort of machine learning approach that we always worry about is if we're getting very highly accurate answers but we can't actually understand why those are accurate, it makes it very, very difficult to swallow. So this is that black box that I was talking about. And so oftentimes what you wind up seeing is people like to work in this regime here, where they get highly accurate machine-learned answers just so they can explain what the important factors are in the data, and give up on really accurate answers, where they tend to be more black-boxy.
But all of this stuff is important to do in real time, and one of the, again, important takeaways, I'd say, from the talk is that this is happening in real time. This isn't just happening on paper; we've actually put this into practice. We have the ability now for our machine-learned inferences to go in and actually interact with the same sorts of code bases that people are interacting with when following up. And here's an example where the machine-learned robot is going in and making the discovery: this is a transient, I think it's a supernova. And then this turns into a Nature paper a couple weeks later. We still need people to write the paper, and probably need some people to referee and understand that this is an interesting supernova, but we're getting to the point now in understanding not just “this is a discovery and I think it's a supernova,” but understanding how different it is from others.
So what I will say is that this has been a massive challenge for us, again echoing what Mike said earlier, to get the ability to not only work on these algorithms and improve the algorithms themselves that give us the answers we want, but then put it into a real-world production environment. And we wind up seeing this as a problem throughout the time-series domain, not just in astronomy but in seismology and in neurosciences, is that you've got great domain scientists who have this amazing data, know the questions that they want to ask of that data, but really don't have the toolkits or the people around them to be able to build up these datasets.
So we have a grant that recently came in from the NSF Big Data initiative that's going to allow us to essentially build up an open-source toolkit to allow astronomers with time-domain data to do essentially machine learning at scale, where they'll be educated in how to use this and be able to produce, hopefully, some great and novel results from it. So I'll put my parting thoughts up there and maybe take a couple questions.