How time-domain astronomers use ML on streaming, noisy, distorted images of the sky — the search for Planet 9, discovery of new variable sources, data-driven emulators — and building physics-informed learning architectures.
“If everybody has access to the same data, everybody has access to the same compute, everybody largely has access to the same set of algorithms, who's going to win? It's those that can get access to that and can tell a better story.” – Joshua Bloom
“If you have something in your catalog, it doesn't mean you've discovered it.” – Joshua Bloom
“If you find an object that you're extremely excited about, chances are 99% of astronomers in the world think that's boring.” – Joshua Bloom
“Without telling this neural network architecture what physics it is that you care about, the bottleneck layer is learning something about the dimensionality of the problem that it's being given.” – Joshua Bloom
Thanks, everyone, for coming. I obviously see some familiar faces and some new ones as well. Before I kick off into the meat of the talk, I wanted to say what I think a lot of you know: it's such an exciting time for data science here on campus, and BIDS and the leadership of BIDS have been such an important driving force in getting what is now a large organizational structure on campus built, and really getting the momentum going on it. So I'm excited to see where the future takes us, and I think BIDS has got a lot of credit for getting us to where we are now, and I've been very happy to be part of that.
Today's talk is on astrophysical machine learning, and in particular some of the things that I've been thinking about, mostly in the time domain. But I wanted to couch this conversation in a larger sense of the democratizing trends in the physical sciences, because astronomy is very much at the heart of this. In some sense it's these trends that I think are driving the way that I think about our future discipline, and think about the types of people that I want to work with and the types of students that I want to train.
So the first one is pretty obvious. This is data, and data is becoming increasingly easier to obtain. The cost to not only get it but then also move it and store it is dropping precipitously, and has been for a while. So in some sense it's that that is driving this very, very large push in the scientific community towards open data, and more freely shared data amongst large collaborations. So that's becoming easier. From the compute perspective, obviously, as you know, decreased costs in compute and increased specialty in the compute architectures that we can get access to readily and spin up programmatically make that whole infrastructure for doing inference that much easier. It's becoming more and more accessible. And while it's not zero, and as you know getting access to large computational resources is not trivial, in some sense it's getting easier and easier.
And the last, on this technology side and algorithmic side: algorithmic innovation continues to happen, and it's happening not just on paper but in open-source tooling. All of that is, as you know, very exciting for people that are in domains like astronomy or physics or life sciences, because as an algorithm winds up getting published from our friends on the methodological side, it only takes a few days, if it's going to be any good, for it to show up on GitHub and implemented in our favorite frameworks. And so what used to be all these different barriers to entry have all been removed.
And so one of the interesting questions, if you buy my premise of this democratizing set of trends, is: in this world, who wins scientifically? I think the answer is really, in some sense, twofold. It's those with the deep domain expertise. It's she who knows how to ask the right questions, and is able to, obviously, identify where all the data is to be able to answer that question. But that's also where the generalized data science skills need to come in. So it's she who can answer those questions better and faster than others. If everybody has access to the same data, everybody has access to the same compute, everybody largely has access to the same set of algorithms, who's going to win? It's those that can get access to that and can tell a better story. And I think fundamentally important to all of this is to work with people with a diverse set of interests and backgrounds, who can bring to bear all these different skill sets that need to happen.
And so in that sense, this is my way of really teeing up the group that I've been very fortunate to work with since coming back from industry and now working in the academic setting again: a number of these students and postdocs and staff researchers, some of them are here. But as you wind up drilling into their different backgrounds, you end up realizing they have a different set of skills. And Manik Varma, who's a visiting Miller professor, is also somebody that we started thinking about collaborating with, and he comes from the industry background, from Microsoft Research. So I would just implore all of you, as you think about building out teams doing domain-specific inference, to think about bringing people with a diverse set of skills to the table.
So this really leads to the agenda for the day. What are the impetuses that we see, and what are the things that are driving us for our work in the time domain? Then I'll get into some of the nitty-gritty about how we do discovery and how we do inferencing, and talk a little bit about some of our forays into the neural network world applied to time-domain data. And then I'll talk a bit about some of the Bayesian modeling work that we've been doing, and try to end with some of the open challenges for ourselves, and hopefully for those in fields adjacent to ours who get excited about the things that we're doing and may want to join forces.
First, I should say machine learning is becoming a very pervasive tool throughout astronomy, not just in the time domain, and I'm excited about a number of things that are happening in the static sky. In fact, one of the really interesting ones that came out a couple of years ago now was the ability to do super-resolution imaging. You take an image like the one that you see on the left, and you want to basically get that from the image that you see in the center. You want to basically recover that using some sophisticated network architecture, and that's what's done on the image on the right. Super-resolution imaging has a lot of utility in the static sky, and then there's a whole bunch of people on campus: Gyro and François are working on some really interesting problems in the static sky. But what I wanted to focus on here is the time domain, and try to help you understand the different ways in which we see the different problems evolving.
So first is discovery, and doing discovery in images is in some sense our bread and butter. We have lots of data coming off of telescopes. The traditional path that you would use to actually find new objects in the sky, or new variable stars, for instance, would be to have grad students look at the data, or, if you're lucky, you have an army of undergrads looking at the data, because they're slightly less expensive and maybe more talented. But in the end, what we wind up realizing, as we were getting more and more data coming off of telescopes, is that machine learning had to be part of that real-time workflow if we were going to make any traction and find things that were actually useful for following up.
And then after you do discovery, there's a question about inferencing. So here's what we call a light curve, in the middle. This is time on the x-axis, flux on the y-axis, and you can see what astronomers face, right? We have data that's somewhat noisy; we have data that's taken at different times, on different timescales. And it's interesting: obviously there's a lot of information in the frequency domain, not just the time domain. You'd like to apply some Fourier transform to this, but we don't have regularly sampled data. In fact, when I talked about this about ten years ago in the stats department, people were asking, well, how come you just don't take the data in a regularly sampled way? And I said, because the Sun comes up and the Earth moves around the Sun. And you wind up realizing, as you start talking about domain-specific data, let alone domain-specific questions, that a lot of the tools that have been built up in other methodologies make some assumptions about how that data lives that aren't always true. So this is kind of interesting, anyway.
Presented with some data like this, ratty, heterogeneous, noisy data, the question is: is it worth spending telescope resources to keep going and get more data? Because oftentimes it's on these discovery images that you just get started; it's using other facilities that you actually wind up getting some of the interesting science out. And then the third category, which also ties back into some of the static-sky stuff, is surrogate modeling. The idea is that you've got very computationally expensive models. In this case, you might have models for supernovae, and each one of those different lines might actually be incredibly expensive to generate, supercomputer-level expense. And now, presented with new data, you want to be able to very quickly hone in on the different parameters that gave rise to that data. By having a good surrogate model using machine learning, which is a data-driven model as opposed to just an entirely physics-based one, one can imagine getting to parameter estimation in a much faster and more robust way than you would if you had to run lots and lots of coarse gridded models.
By the way, I should say, as I bring things up, I'm trying to not use that much jargon, but if any questions arise or I pique your interest in something, please feel free to stop me. What I wanted to say here is that, as we think about the different places throughout the whole workflow, in the way in which astronomers acquire data and do inference, there's a whole bunch of different steps here. So number one, you can imagine the planning and the acquisition of the data, and these are strategies for actually how you tile the sky. There's a lot of politics in this, but there's also a lot of algorithmic and technical work that needs to go into doing this, and then obviously, from a software engineering perspective, questions about how you move data around and store it.
Then there's the categorization and the cataloging of that data. So as you wind up obtaining the data, turning that efficiently and robustly and reproducibly into a catalog of, let's say, flux measurements of how bright everything is on the sky as a function of time, and doing associations of individual objects with other ones: that sounds pretty easy, because you just say, well, that object is in the same place on the sky as that other object. But objects move, they change colors, and interesting things happen on small spatial scales, so this is a nontrivial task.
But what I've been most focused on over the last decade or so is discovery. This is a question of saying, well, you've got something in your catalog (a large database is probably the way to think about it more broadly): is this an object of interest? Is this something new? Is this something that I should spend some time on? And this sounds kind of obvious, because if you've got something in your catalog, well, then you've obviously discovered it. And the counterargument that I have to that comes from Galileo. This is a little snapshot from his notebook. I think this is the second night that he wound up observing what are now called the Galilean moons around Jupiter. And what you notice is he made this little mention of a fixed star on the far left, and he basically didn't say that was all that interesting. It turns out that when he was observing in 1612, Neptune was almost co-aligned on the sky with Jupiter. And while he had it in his database, it's very clear to say that Galileo didn't actually discover Neptune. And as many of you know, if he had discovered Neptune, which took another hundred years or so, he would have been very famous.
So that means that if you have something in your catalog, it doesn't mean you've discovered it. And then, because you've discovered it, it doesn't mean that you've prioritized it to actually do something interesting with it, and you certainly don't know what it is and how it fits into the potpourri of the types of objects that you've studied before. And this is where the inferencing and the hypothesis testing wind up coming in. So what is this source? Have we seen something like this before? If we're correct about the classification of this object, what will it look like the next time I look at it? These are all interesting questions that you can pose, obviously, the way I'm posing it, but you can also pose it in a more technical way and get some potentially quantitative answers out.
And then, what do I do with this? How do I prioritize this object over another object? These are real-world problems that we have every night. There are precious resources in the world: telescope glass is incredibly expensive to obtain, and if you find an object that you're extremely excited about, chances are 99% of astronomers in the world think that's boring. This is how we all work together: we don't get in each other's way, except for the fact that we're competing for these resources. So figuring out clever ways to do fast inference on a little amount of data, so that you can convince yourself and others that it's worth pursuing, is a problem. And then, obviously, all this feeds back into how you acquire more data.
So what I'll do now is just give you a sense of how we've gone about those different parts of that time-domain inference workflow, and start off not with the cataloging and the telescope planning (that's certainly of interest to me) but in the discovery space. And this turns out to be a pretty hard problem. What you see on the top set of panels are bogus detections. This is essentially me taking an image of the sky multiple times in that same part of the sky, co-adding those images, maybe doing some special median adding, so that I wind up getting what I think is a static version of that sky, and then a new image that comes in tonight, overlaying the two of those and subtracting them. It turns out that that process, while very effective in finding, or at least surfacing, potential objects of interest, winds up giving you a lot of spurious types of data.
So what you see up there are all these subtractions that look like new point sources but actually are spurious for different reasons. You've got bad misalignments of images; you've got cosmic rays that have come in and hit the CCD, and it wasn't there in the median image. All the ones on the bottom are actually real detections. So this is basically the differential flux, the fact that this part of the sky has a little bit more flux than it did yesterday or over the last year or so. It turns out that these bogus detections, which are easy to just find with some sort of “give me a five-sigma cut over this difference image,” outweigh, for at least this project that we worked on, the real ones by a factor of about a thousand to one. Maybe we're getting down to about two hundred to one for the state of the art, for some of the newer surveys and some of the new ways of doing image subtraction.
But what we realized we had to do, as new projects were coming online, in particular the Palomar Transient Factory, which started around 2008 or so, is, instead of having people looking at these different images and then with their eye making some decisions, we trained a machine learning algorithm to rip through these different postage stamps as quickly as possible. That gives us parallelizability. It gives us the transparency, because we understand what our algorithm is doing and what it's using. It's deterministic in a way that people are not, and it's obviously versionable. Was there a question that came up?
[inaudible audience question]
Well, so we've couched this as a supervised problem, where we actually know the answers. So we know which are the bogus postage stamps in a small corpus, and we know the real ones, because while it may be faint today, if it's a supernova, it's going to get brighter and brighter over time. So we know, oh wait, the day before, when there was a little something there, it was probably the supernova getting brighter. So we have a way, and we did for a long time, of getting ground truth on a small subset of the data.
So bogosity, which I contribute to you as a new term, has a physical reason for why something is bogus, but the real objects also have a physical reason: that's actually something in the sky changing, there's a star blowing up or a variable star changing its brightness. We take a very data-driven approach to both of those, in a sense saying, if I'm given a corpus of postage stamps that are real and some that are bogus, can I train a classifier that rips through these postage stamps at scale and gives me a score, essentially from 0 to 1, bogus being close to zero, real being close to one? And then it's up to me to decide what I want to do with things that are in the gray.
[inaudible audience question]
Yeah, it's an interesting question. I will try to come back to that, this interplay between physically driven modeling and this fully data-driven modeling. It's an interesting question that comes up more starkly in some of the other problems that I'll talk about. Thanks for that, though.
We were able to put this onto paper, not just saying, “My false positive, false negative ROC curve is nicer than your false positive or false negative ROC curve,” offline. We felt very fundamentally that we had to put this into, using an interesting term, production. This had to be working in a real-time workflow. And this project and the successors to those projects wound up actually relying upon these machine learning inference frameworks for them to succeed. And one of the big results that came out of this was the detection of the nearest Type Ia supernova in a generation, about 25 years. And this was ML-assisted, in the sense that we wound up running our codes on the data as it was coming through and wound up surfacing the most interesting objects, the ones that looked most real, and then correlating that with nearby galaxies on the sky. For me and my colleague Peter Nugent, that wound up revealing this new, interesting object.
Why that's interesting is not, in some sense, that we did the discovery; it's that we did the discovery very quickly. This happened about 11 to 12 hours after what we think was the actual explosion time, which at the time was about two to three days earlier than had ever been found for any Type Ia supernova. This turned out to be so bright in the end, a couple of weeks later, that you could have seen it with binoculars in a dark sky from the Northern Hemisphere, so it would have been discovered for sure. But because we were able to get on so quickly, and recognize that this was an interesting and potentially unique source, we were able to trigger not just the world's telescopes but satellites around the world to look at this.
This is somewhat of a wonky slide for the non-astronomers in the room, but we were able to put some interesting limits on what the progenitor is, the object that actually exploded, and we did this because we were able to get on so quickly. It's those green curves, the regions of exclusion to the left, that are the most interesting. Basically, what we did is we ruled out a whole class of interesting progenitor objects, so-called main-sequence companions to the thing that blew up, and basically left behind the only thing that was possible: it was a compact object.
And while we've known, or believed, for a long time that white dwarfs are the sources of Type Ia supernovae, this is in some sense fairly direct and constraining evidence that the object has to be something like that. A neutron star is still allowed, but we don't think that that's possible. So it's because we were able to identify something quickly that we were able to get on other telescopes quickly, and we were able to do inference that we wouldn't have been able to do had we been waiting around for an army of people to sift through all of that data and find it eventually.
This sort of discovery, this notion of real-bogus, which in some sense we pioneered on the Palomar Transient Factory, has now become a bit of a cottage industry. There are certainly new techniques that are being applied to imaging data, and the rate of false positives and false negatives on different surveys is getting better and better over time. So this is becoming really what people are using whenever they're taking images of the sky. This is now becoming a central part of the whole discovery chain.
What I started working on fairly recently, with a student at Berkeley named Mike Medford, was looking for the so-called Planet 9. So this is an object that lies outside of the orbit of Pluto. Just a few years ago, people looking at the orbits of a number of different solar system objects wound up realizing that there may actually be a perturbing large-mass system, of order 10 Earth masses, at a fairly large distance, 300 to 400 times the distance that the Earth is from the Sun. This would be that much further away, so way beyond the orbit of Pluto. And the hunt's been on since that inference that there is maybe something in that part of the sky. The problem is that any one telescope that has observed a very large part of the sky to date may have observed it by accident, but likely they didn't get deep enough. They didn't have enough signal-to-noise on a given image to be able to see it.
So what Mike has done, as part of his thesis, is use a very clever technique of taking lots and lots of images from the survey that I just mentioned, called PTF, and stacking images by year, and then looking for subtle changes of motions of objects that happen to line up with hypothesized orbits. This is a very computationally expensive problem. And if you just go through and do the real-bogus thing, he's generating of order a trillion bogus detections. These are just five-sigma fluctuations that aren't real. And because he's able to simulate what a Planet 9 might actually look like in our data, we can actually get a training set of bad detections and real detections. So going through the same process we had before, we were able to build essentially a real-bogus detector.
The problem is, if you have a lot of possible bogus detections, and you're looking at a needle in a haystack of maybe not one in a million, or one in tens of millions, it turns out that there's not enough data space at NERSC, which is where this is being run, to save all of the candidates and then deal with it later on. So what we had to do is not just develop an algorithm that could find potential candidates and throw out all the ones that were bogus. We had to do this where it worked at scale, so we could make use of the infrastructure that we have available to us at NERSC. But we also had to do it where it happened very fast, because we didn't want the supercomputers, essentially, that were sitting around waiting to do the next computation to have to wait for the machine learning algorithm to give you a score before you made a decision whether we're going to save that object or not.
So we had a new, interesting constraint. We had to build a machine learning classifier that was very good, which was not a new constraint, but it had to work at scale, meaning we had to basically stick it in a bunch of Docker containers, each of which would serve thousands of requests of “Given this postage stamp, should I save it or not?” And it had to answer that question within a few milliseconds. And so that's what we did. It's been deployed, and the first pass through all of the data is now coming to an end. You'll have to stay tuned for what the answer is, but needless to say, we were able to pull the false positive rate down to a very, very small number and still keep our true positive rate reasonably high. So at a few percent missed on true positives, we were able to get false positives way, way down below the 1% level, which is what we needed to do to be able to rip through the data. Yeah, there's a question?
[inaudible audience question]
Yes, we made this again into a supervised problem. It turns out it's an even harder supervised problem than what I showed you, because in the previous example, the boguses all look weird and different, and the reals all look the same. It turns out that in the other example, the reals don't all look the same, because you have to make some guesses of what the orbit is of an unseen object so that you can stack images appropriately, and you get interesting, weird bananas and other sorts of things. This is like the Anna Karenina problem: all unhappy families are unhappy in the same way. Oh, unhappy families are the bogus, and all happy families are all the same. This is a much harder problem. So to be able to do that, we had to generate a whole bunch of fake reals, ones that we thought were what it would look like if we were actually seeing Planet 9.
OK, so in some sense, that's just-in-time inferencing. It's one of the first examples I know of in astronomy where, effectively, the decision to save the data is being made by a machine learning algorithm. In the other cases, the data was already in the can, and a machine learning algorithm is helping us sift through this. This is one where we're basically saying, do I save it or not? Because if I don't do that, I bring down NERSC, which no one would be happy with.
But just-in-time inferencing doesn't happen necessarily just at the millisecond timescale if you're now interacting with humans, and those are the ones that are going on telescopes to make the next set of decisions. We also put into play this capability of looking at a new detection and new discovery and doing some rudimentary inferencing over this hierarchy that you see up there on the top right, about what is this object likely. So is this a supernova, or is this some sort of activity of a black hole at the center of a galaxy? Is this a variable star? Is this an asteroid? Etc., etc. And what we did is build the system so that as a new object was saved, as it was discovered by the ML algorithm, another one would pick up and make some inference and write this, using an API, into, basically, the websites that people were using to discuss these different objects.
So it might be hard to see over there, but at the top, as this object is saved on August 25th, 2010, right away the robot that we built winds up saying it's a transient of some sort, not a variable star, and I think this is of type supernova. And that's the thing that then — you see the conversation from other astronomers saying, “Hey, this looks actually pretty interesting, we should do more with this.” And this actually led to a Nature paper, which was the pre-detection of some activity before a supernova itself. So obviously we're not getting to the point where the ML algorithm's writing Nature papers (that would be a great place to be) yet, but it's still being part of the assistive workflow in what it is that time-domain astronomers actually do.
What we wind up realizing is that this codebase that we used was proprietary and had been built with some of the older web technologies. Stéfan and I and others in the group have built something called SkyPortal, which is an open-source version of the thing that I showed you, that allows others to basically use it and collate and aggregate their own data and not have to use somebody else's proprietary system. And this is a project that's ongoing; happy to answer questions about that later on. That is not ML per se, but what we've thought very hard about is making API interfaces to this system really the first-class citizen, so that if you've got some ML system sitting off and you get some webhook notification that there's a new object, that ML system could then apply on the data and write it back in, so that it could basically live alongside of people talking about those same objects.
Just-in-time inferencing isn't just for following an object up that night. We also looked back retrospectively at a number of large surveys of variable stars, and the typical variable star that we get out of this survey of 50,000 variable stars is something like that at the top left: a pretty ratty light curve. And again, what we want to know is what is its classification. And this turned out to be a fairly successful project, the output of which was basically a probabilistic catalog over 26 different classes. It's actually a fairly hard problem, because we only had 810 known labels over those 26 classes, so it was a very small label problem, and that was an interesting paper.
But what I often say is, if you're doing machine learning and making probabilistic inference, what is the actual purpose of that? And so where we took that catalog was where I got very interested. We can now query that catalog and ask interesting questions: give me an object that is unlike any other object I've seen, so give me anomalies; or give me an object that looks like one that I only have a few of in the training set, because those are interesting and rare. And so we did that within my own group. For instance, we found and tripled the number of these interesting DY Per stars, for the astronomers in the room. Another outcropping of that was we were able to query our catalog looking for so-called detached eclipsing binaries that allowed us, with spectroscopic follow-up, to get mass and radii of these different stars, which is actually quite hard to get by other means. So the point is, doing ML inference on a large amount of data, ultimately, as domain scientists, we want to have an end in mind. It's fun to do it, although ML is hard, but the question is, what are you going to get out of it if you do it well and you do it right?
Another thing that we got fairly excited about is the idea that we could look at the variability of how stars actually change on the sky and, given a little bit of color information, we can infer something fundamental about those stars, that is, something that you would only get from very expensive spectroscopy. We were able to get temperature or surface gravity and metallicity. It's a very strange idea, the idea that you could look at — in some sense, it might be like listening to an opera singer in the other room, and by hearing them sing, you know what their gender is, you know what clothes they're wearing, right, you know what their hair color is. That is interesting. It seems a little bit like a party trick, but given that it is very expensive to get spectroscopy of objects, the fact that we could look at variability and infer something about their fundamental properties is very exciting.
I mentioned before that we had this project to basically collate all of this data and put it in one place using this thing called SkyPortal, but before that we also built an inference framework that used a lot of the tools and toolkits we had built up to do machine learning on time-series data. We called this project Cesium, and it allowed us to apply it not just to astronomy data, but we were able to get some state-of-the-art results in some other fields, basically just by taking a bunch of time-series data, again in a supervised sense, and applying it to that data, and then, in an offline way, seeing that we're getting good answers. So anyone that's interested in contributing to machine learning frameworks, this is one that we're pretty excited about.
OK, so with the time that I have left, I wanted to get into some of the nitty-gritty of the types of things that we're now thinking about and working on within my group, and a lot of that comes down to what is the utility of using neural nets for some of the same problems we've looked at, and also applying it to new problems. So one of the things that my student Keming is working on — and he's here in the room — is taking raw images of the sky like you see on the left (this is taken with the Hubble Space Telescope) and turning it into the one on the right. And it's hard to see a little bit of contrast of some of the spiral arms, but suffice it to say, what you have on the left has got a lot of noise, and in particular what it has is a bunch of streaks that come, because you're in space, from charged particles which are hitting the detector. You want to be able to remove those a priori, and you want to be able to figure out what would have been the flux of the sky or the galaxy or the star that you're looking at had I not had a streak that went through that.
Now of course, if you found the objects and you did some sort of bicubic interpolation, that might be pretty good, but it turns out that we're finding that you actually may be able to do a little bit better. That is because there are lots of differences in the sky; galaxies of the same size and of the same type all basically look alike. So if you train a neural net to basically learn what it means to be a galaxy, and you remove some little pieces of it, it can figure out what it should have looked like underneath that.
And so for those in the ML space, you'll probably know right away the types of architectures you might want to use. This is using the classic U-Net architecture, which is a way of finding objects, but then feeding that U-Net object, which is a U-Net detector of the cosmic rays, into another network, which is all trained simultaneously to be able to figure out what things look like under those pixels. And while we haven't published this yet, and we're still working on it, and actually in particular looking at how well this does relative to other algorithms that exist in the world right now that are being used, things are looking very promising. We're able to basically get very nice reconstructed images that at least visually look very much like the ones that we know to be the ground truth.
We also started looking at how we can use neural nets to do classification on time-series data, and this is something that I did with Stéfan and Fernando a while back. What you do here is take that noisy light curve, you compress it down to a few numbers — and that's this B here, that's called the bottleneck layer — and then you uncompress it, so you basically blow it back up to its original shape, and then you compare what you started with and what you ended with. And this is what's called an autoencoder. It's an architecture that's widely used in the imaging space and not so often used in the time-domain space, and one of the challenges here is that the different types of layers that are used in time domain are not easily amenable to the irregularly sampled way in which we wind up acquiring the data.
So it turns out that this bottleneck layer is a very powerful way of generating features that we can then use to build classifiers, and the reconstructions of the light curves turn out to be quite good. For now, you just take a look at the panels on the left-hand side: what you have at the top is the original data as it was obtained, and then you have the phase-folded data on the bottom, and it's the red data that is the reconstructed light curve using this so-called autoencoder. And we have another example on the right-hand side here, where the bottleneck layer's got 64 numbers.
What's really interesting, and what's really become a bit of a point of departure for me and my thinking about the utility of neural nets — not just to help us do better classifications, but something deeper, potentially within physics — is the fact that when we use this architecture on objects, or fake generated objects where we actually knew the answers (so these are basically sinusoids), and we said, can it wind up reconstructing these sinusoids, it turns out that the values in the bottleneck layer wound up correlating with things like period, like amplitude, like phase. And so what you wind up realizing is that without telling this neural network architecture what physics it is that you care about, the bottleneck layer is learning something about the dimensionality of the problem that it's being given. And we don't quite know where to go with this yet, but there's a burgeoning activity within my group to start thinking about how we might use machine learning itself to actually learn something about the underlying physics, or at least the dimensionality of what drives the physics of the objects that we're looking at. Anyway, so it turns out that this technique wound up beating essentially all the other classifiers, including our own using some of the more traditional featurization techniques, so that got us very excited.
But one of the things that we're very happy about is what we think is the generalizability of this network architecture to other types of irregularly sampled data. So in particular, this natively handles irregular sampling. We can actually use the uncertainties in the measurements themselves as part of the loss function. And I just skipped over one here, sorry. And what's also nice is, because we understand the uncertainties in our measurements, we can bootstrap resample our data and get a much, much broader sample of the original input data for training. And fundamentally, what's exciting about this from my perspective is we have an unsupervised learner. We have a way of learning features in time-series data that — I haven't told it what the classification is. I'm able to basically throw a lot of data at it, and the more data I throw at it, even if I don't know what it is that I'm looking at, in principle this autoencoder is learning something about the physics of what it means to be in this domain. It creates some sort of semantic understanding of that, which allows us — because astronomers live in this world where we don't have a lot of labeled data; we wish we did — to actually learn from the data that we have in hand.
For those that know a little bit about this, it's also interesting to see that there is a strong possibility that some of these layers could basically be frozen and you could retrain the network on different surveys, and the transfer learning from one survey to the next appears to scale linearly with the amount of training data we have. So that's all very exciting. Where we're going with this work — Sara Jamal and Mike in my group have been pushing on this quite a lot — is now not just taking this unsupervised part, which is part of the original work, but now adding, for the ones that we have classifications of, a component to the loss function that is supervised. So this is basically a semi-supervised problem. I think this is a bit of a kitchen-sink approach: we're taking all the data we have, including the metadata, and we're throwing it in, so that when we wind up wanting to get classifications, we can either do it in the bottleneck layer or we can do it in the classification layer. So that's very exciting; we'll have to stay tuned to see where this goes.
Another thing that we're also interested in is looking at that bottleneck layer, given that kitchen-sink approach, and asking the question: if we throw in a whole bunch of known classes and learn, in a semi-supervised way or even an unsupervised way, all of those classes, and then we throw in a new class of objects, is there a way to detect this? So can we find anomalies? Can we find new classes of variable stars in our data, not by saying, “I know the classes of all the stars, now tell me what this class is,” but say, “Here's all the data that we know of; find me the things in that data set that don't look like other things”? And that's a hard problem that we're pushing on, and we're using not just the work in the bottleneck layer but also in the reconstruction errors. So there you have an example at the top where we threw in known classes. So this is a toy example where we throw in sinusoids, and then we actually ask it to reproduce some weird-looking light curves, and it does its job. So both in the bottleneck layer and in the loss layer, it looks like things are doing reasonably well.
And lastly, on the neural net side, we're also interested in simulation, and not just simulating the vast potpourri of weird variable stars that we know of. We're also interested in saying, given a known variable star that I throw into a learned system, what are the physical parameters that I can get out of that that are of interest? And so here we're using something called generative adversarial networks, where we're basically throwing in — and we're training simultaneously — a generator of essentially fake time-series data, and we're asking another network to basically say, is this real or not?
So every now and then we throw in some real data and we throw in some fake data, and you train these simultaneously, you reach some Nash equilibrium, and then you wind up getting something that's really good at generating fake data and something that's really good at discriminating on that data. What we're interested in doing is taking the random latent space of the generator and being able to marginalize out the different physical parameters that we're interested in, so that would allow us then to generate new objects where we say, give me an object that has a temperature of this, a metallicity of this, an age of this. And again, this is also looking promising, but maybe you'll have me back and I'll tell you some of the final results as we wind up publishing this out.
I think in the interest of time I will skip over some of this Bayesian inference work for now, so I'll just skip through that. But I wanted to just hone in on some of the activities that we're interested in and the real deep questions that are arising through the work. So one thing is that I already mentioned our work on anomalies. This last bullet point, I think, is really interesting: if you build some sort of GAN that is capable of generating fake data, and you build a discriminator that's good at this — whether you've been given something that it's seen before, or something like something that's seen before, and something that's not — if I now give it real data, can this discriminator actually tell me what looks different? So using a discriminator in a GAN is actually quite interesting, and we think this may be one of the more robust ways to do this at scale as more data is coming off of telescopes.
I also said we have a small label corpus problem that persists today. Being able to transfer-learn from one survey to the next is of interest to us, but also co-training over multiple surveys, each with their own multiple ideas about what one object looks like, is also really interesting for us. And there again, I think the unsupervised and supervised approaches that we've been working on may actually be really useful. I won't go into great detail about what we can do here, but that first one is really the project that my postdoc Jorge is working on, which is this ability to use GANs for physical parameter estimation.
And another bigger question that I am certainly not trying to bite off myself right now, but I think is an interesting one for some of the people in this room, is about the overall systemic optimization of how you do federated follow-up across multiple different heterogeneous telescope resources, given these automated inferences that you're able to do using some of the frameworks that I've just laid out. And perhaps also interesting for people in this group is the role of people in this. I've been talking about it largely as a “here's a supervised problem with data that other people have labeled; let me now go ahead and build a classifier and run it.” There's obviously a chance to build feedback into these systems, where you build a classifier, you run it, people say, “No, it's not what you said it is,” and then actually taking that feedback, getting better labeled data. But of course people aren't always right, so how do you do this when there's some noise in the labels themselves is also pretty interesting.
So as I wrap up, I just wanted to invite all of you to get involved in our work, if that's of interest. There's really a couple of different ways to do that. One is I hold group meetings on Mondays here on campus; please email me if you'd like to participate in that, if you think you may have some problems or some methodologies that might be applicable. And then another one is that we're running a machine learning and astrophysics and physics seminar on a roughly monthly basis. Our next speaker is Ruth Angus from AMNH in New York, and I would encourage you to get on our mailing list for that; the URL is at the bottom right here. I don't know if that's her title or not, but it's something she's written about recently, so she may be talking about this, but she may be talking about something else.
And another thing that I hope many of you get involved in is a set of workshops that we're having in this very room next month. One is what we're calling physics and machine learning, and so this is doing some of the work that I was starting to hint at, of using physics to help us train better models, and maybe using trained models to help us learn something about physics that we didn't already know about. And then the second day is doing inference in the so-called multi-messenger astrophysics world, where we're looking at simultaneously gravitational wave data, neutrino data, and electromagnetic data, and trying to do better than the sum of the parts. And there's a great set of speakers coming to both of those. I'm very thankful to BIDS for their support; we're really looking forward to it. So with that, I will stop, and I'll be happy to take some questions. Thank you.