ML across astronomy workflows: the Planet 9 search, new variable-source classes, semi-supervised VAEs generating realistic time series to optimize observing schedules, and deep-learning cosmic-ray detection and correction.
Originally scheduled Apr 17, 2020 in person (cancelled); delivered virtually.
“There's obviously a lot of interest in optimizing ad clicks, and understanding Twitter sentiment perhaps, but understanding how the universe works I think is pretty interesting and pretty fundamental.” – Joshua Bloom
“It's a little bit like Anna Karenina: all real objects all look the same. All bogus objects are all different in their own way.” – Joshua Bloom
“When it comes to machine learning, it turns out we have very few examples or exemplars of the objects that we care about. So we have a big data problem, but we also have a small label problem.” – Joshua Bloom
“With machine learning, you always get an answer. And that is a wonderful thing, but it is also an extremely dangerous thing. Always getting an answer means if you've put something into production, you're getting a result that hasn't been vetted by people.” – Joshua Bloom
Admit all.
Pavlos, would you like me to stop sharing my screen just for a bit while we're waiting for people to come in, or just keep this?
Up to you, Josh. If you like, I usually start without it and then I share. But whatever you like. I think we can start. Natasha, can we start the course?
I'm letting people in. Whenever you guys want to go.
I'll start with this slide. I'm all set when you are.
All right. Natasha, recording?
Yes, it is.
Recording, okay. Welcome. Good afternoon, good morning, and good evening to everyone. My name is Pavlos Protopapas. I am the scientific program director for the Institute for Applied Computational Science. You're here for the seminar. We have about six, seven seminars every semester. So today, it's my special pleasure to introduce Josh. I've been working with him for a while. Josh started from Harvard College, then he went to the other side of the pool, Cambridge. He did his master's there. PhD at Caltech, and now he's a professor and chair of the Astronomy Department at Berkeley. He's also a senior fellow at the Division of Data Science Berkeley. And if I'm not mistaken, he was instrumental in the creation of that division. He was one of the first people to start data science at Berkeley. His research is largely time domain astronomy, transient events, or telescope insight automation, and AI. He's also in the cusp between astronomy and machine learning. He teaches radiated process, high-energy astrophysics and at the graduate level Python for data science. Also, he has been awarded the Data-Driven Discovery Prize from the Gordon and Betty Moore Foundation, and the Pierce Prize for the American Astronomical Society. Before that, at one point, he was the CTO of Wise.io, which was sold in 2016 to GE. And today, he's gonna give us a talk about physics informed machine learning in astronomy. Let's welcome Josh. Josh, the floor is yours now.
Great. Thank you very much, Pavlos. I really appreciate the introduction and the invitation to be here. I'm excited to talk to you today about some of the work that we and many others have been doing in machine learning in astrophysics. I thought I'd start off with this quote for a little bit of levity. Recognizing that Jim Gray, for those that don't know, who is a prototype of a data scientist, many years ago wound up realizing at Microsoft that by using data from astronomers, they could test algorithms. They could test ways to scale compute. And Jim loved working with astronomers because unlike with lots of other big data, when you make a mistake and data leaks, or you actually make an inference that's wrong, credit cards aren't leaked and people don't get actually hurt. So working with astronomers was seen, and still is seen by many computer scientists and statisticians, as a kind of safe sandbox environment to play.
But I also wanted to bring up this quote because there's something interesting about how astronomers look to computer science and statistics for the development of our own work. And while we're fairly famous for taking work that's been done in other places, not so much on the algorithmic side in the early days, think of Galileo, pretty famous astronomer, hearing about this new discovery called the telescope, which was meant to point over the horizon. And instead he said, what if I do this and point upwards? And the rest is history on that front. But astronomers are pretty well known for co-opting both hardware and algorithms and approaches computationally to our own work. So we love working with computer scientists because there's a lot of fantastic work that's coming out, especially in the machine learning world, that we can try to bring in and adopt for our own purposes.
And last, what I'll say is I think computer scientists and statisticians, with all due respect to the work and the data sets they do, I think they need astronomers. And I think they need domain scientists more broadly to ask interesting and hard questions about the physical world. There's obviously a lot of interest in optimizing ad clicks, and understanding Twitter sentiment perhaps, but understanding how the universe works I think is pretty interesting and pretty fundamental. So we're hoping that we provide an interesting set of data and questions that help push the algorithmic side as well. So that brings us to this talk, which is really, hopefully, an introduction for many into the kinds of ways in which ML is being used in astronomy today, but also somewhat forward-looking into the sorts of things that we're trying to do to push the envelope, not just on the domain-driven questions that we have in astronomy, but what we're seeing is that there are special ways in which astronomers can contribute fundamentally, I believe, to the machine learning literature itself, in trying to change the ways in which we actually do learning by informing the learning process with the physics that we know of the objects that it is that we study.
So I'll be talking a lot about the work from my group and my students and postdocs, and mention the work of many others. But I just wanted to pause for a second and say that some of the most important work in this front is happening right there with Pavlos's group. And he's really been an inspiration for a lot of the things that we've been thinking about as we move forward. I also wanted to say before I moved on how wonderful it is to be here with all of you today virtually. It is a shame that I couldn't go back to my old stomping grounds and see many of my favorite colleagues. But that'll happen at some point soon, I hope. Just as a brief primer on the place where I've come from to try to get to the kind of research that we're doing now. You already heard some of this in Pavlos's introduction of me. But really maybe starting from the right-hand side, I've been focused most of my career on studying transient events in the universe. And I'll talk a little bit about the types of objects that we're interested in, and obviously talk about how the work that we're doing, and many others, can help gain insight into the phenomenon itself. Which ultimately, from a domain science perspective, is the thing that we're trying to get to.
I've also been doing a lot of teaching. Started Python bootcamps about a decade ago on campus. And that's now grown and blossomed, and is now part of a much larger effort of data science on campus, which is now a new major and now a new minor. And as you've heard, is also a large administrative division on campus that sort of matrix into many of the other things that we do at Berkeley. And then I also have somewhat of, for an astronomer at least, a special view on machine learning in production, having founded and been the CTO of a machine learning applications company originally focused on natural language processing when we were an independent company, and then moved on over time as we wound up being acquired by GE into automated insight on machines. So the agenda for today's talk is to give you an introduction into the centrality of machine learning in the way in which we do discovery and inference today. Talk a bit about how self- and semi-supervision is becoming more important, at least in the work we're doing in astronomical time series.
And then also talk about how some of the self- and semi-supervision is being used in, or we have a paper, at least, hopefully other people will start using it soon, for actually reduction of data. So looking at raw data and improving it, and doing it in a fundamental way using neural nets. And then I want to spend some time where we talk about exploiting the physical symmetries that we know, baking these symmetries directly into the learning process. And how we can, because we are physically constrained in the sorts of objects that we study, when we actually do machine learning on them and want to do inference, how we can use variational autoencoders with physical constraints to produce realistic light curves, that is brightness versus time of the objects that we care about. And then at the end, if there is time, I want to touch on some new work that we've just put out on likelihood-free inference. I should say that I think I'm monitoring the chat room on my iPad over here. So if there are any questions that arise, please put that into the chat. If I do miss it, I'll ask Pavlos or Natasha to just speak up if there's something that I've glossed over that somebody is interested in. Hopefully, we'll also have some time at the end to chat. So this is the big overview of the time domain of the kinds of objects that we and many people are interested in.
What you see on this plot is time on the x-axis versus some astronomer units of brightness on the y-axis. And what you see overlaid here is a couple of different light curves. As I said, brightness as a function of time, of different types of objects of interest in astronomy. First, I'll point out the type Ia supernovae in the blue curve. Type Ia supernovae are incredibly important for understanding the end states of some types of mass of stars, but also more importantly, for being probes of cosmology. And the Nobel Prize in Physics, in the context of cosmology, was given for using type Ia supernovae to discover the accelerating expansion of the universe. So finding lots of type Ia supernovae in a data stream is actually pretty important. Type II supernovae are the most common types of supernovae in the universe. And those are the most common way in which large stars wind up dying and exploding. If you want to understand the creation of elements in the universe and star formation, understanding how type IIp supernovae work is also important.
So those are the objects that we know about. And those are the objects that we know we're trying to find when we look through our large datasets, but then there are other types of objects that we have theorized about. This NS-NS merger, which stands for neutron star-neutron star merger, is a very quick and faint event, at least it was predicted to be. This is a slide that's about 10 years old now. And that was thought to be the kind of signature we might expect following a gravitational wave event where these two very compact objects, very massive objects, smash into each other and produce not just gravitational waves, but this light that we could potentially see. It turns out that in 2017, one of these events was actually seen. And it looked a lot like the theorized light curve that you see here. So this is something that we were pretty sure we would see. We didn't know the exact details. And then there are these other objects that are up here, neutron stars colliding with red super giants, or these mythical pair production supernovae. These are also theorized to exist. And we're also trying to find these things in our data streams.
But the reason why this is a Rumsfeldian challenge is because we're not just looking for the known knowns and the known unknowns. By definition, I can't put the unknown unknowns up here. And so our big goal, if you think about it from a time domain perspective, as we're trying to find and study the sky and find interesting objects in the sky, isn't just to optimize on the things that we know about, but also to try to find, in the presence of really pernicious noise, a whole bunch of interesting new objects out there. So this is our big challenge. If we had the world's capability of following up every discovery that we make in one of our imaging surveys, then this wouldn't be all that hard because everything that looks interesting in the sky, we would just throw a big telescope at, and get spectra and try to understand them in more detail. But we don't have that. And there's a very strong competition, for those that don't know how a lot of astronomy works observationally, for these very precious resources to observe different parts of the sky. So even though I might want to look at every object that looks interesting to me, there are plenty of other people that are trying to do other types of science with other types of objects. And so this is indeed a grand challenge.
And what looms very large for us in the astronomy community is the Vera Rubin Observatory. And this was formerly called the Large Synoptic Survey Telescope or LSST. This is gonna be producing something like 20 terabytes a night of raw imaging, tracking 18 billion objects over the course of its survey lifetime. And it's going to require something like 150 teraflops just to produce the first data release in a few years’ time. The final catalog, after about 10 years, is gonna be 15 petabytes of just essentially metadata. So this is going to be something of a tremendous interest for us, but this is the kind of data stream that we're trying to figure out how we can make sense of it and how we can optimize our scientific return. So this gets me to the first part of the talk in the context of machine learning, which is to say that we already are using machine learning to help us power some astrophysical discovery. And we're able to do this at scale. So there are parts of the inference chain, as it were, with new data streams, where we feel fairly confident we've learned how to extract and mine interesting sources out of that.
And this gets back to an original problem that I and my group worked on, which was to do something which seems pretty simple. That is to find new sources in the sky. And the state-of-the-art way to do that was and still is taking a so-called reference image of the sky. So a deep median stack of the same part of the sky that was done previously. And for a new image, align those two and subtract them off. And when you've done a good job in the subtraction, you see what it looks like at the very bottom here, these reals. These are new sources, basically where there's a little extra brightness above where there was previously. But it turns out, doing the subtraction is not all that easy. And this leads to a lot of artifacts, these bogus detections. And it's a little bit like Anna Karenina: all real objects all look the same. All bogus objects are all different in their own way. This is actually one of the major bugaboos in us being able to find and then study these objects. Because it turns out that the number of bogus in some of the surveys 10 years ago, the number has come down a bit since then, as subtraction techniques have gotten better, is about 1,000 to one for the number of reals. So we're really doing a needle in the haystack problem here.
And so what we did is developed one of the first machine learning, not just algorithms, but computational infrastructures that ran on real-world data that learned from both bogus and real, and essentially scored every single object that came off of these new telescopes producing these images. And that allowed us to be fast, obviously, compared to people in doing inference, parallelizeable, transparent. And interestingly and importantly, for science, deterministic and versionable. Unlike when you ask people to look at data, you always will get the same answer out of these machine learning algorithms. So that helped us quite a lot. And this may seem obvious in retrospect, but when we started this work, I was reminded of this amazing picture from Harvard College Observatory over 120 years ago where astronomers, and particularly at Harvard, had a very big data problem. They were getting more images coming off of telescopes from the Southern Survey than they knew how to deal with. And so they hired a number of people, mostly women, to look at this data and try to opine on it. And the person that you see in the back, just for historical reference, Henrietta Swan Leavitt, who is considered one of the major figures in modern cosmology, discovering an important relationship between the period of pulsation of some types of stars and their overall brightness. There's also a very nice play by Lauren Gunderson called “Silent Sky”, which dramatizes the life of Henrietta Swan Leavitt. So for those that haven't seen it, I really recommend it.
But this looks old, and it is, but it's actually pretty much how most people, until very recently, were dealing with large data problems in astronomy for discovery, is essentially hire more graduate students. So bringing the machine learning component into this problem was a fairly dramatic shift away from needing experts. Because as we know, domain experts don't scale. And one of the highlights I'd say of some of the early work that we did in this space was to automatically mark up the most interesting objects in the sky. And this led to what was then the earliest discovery of an exploding type Ia supernova in one of the most nearby galaxies in the last three decades. And this ML real-bogus discovery was important not so much because we found this object. Because in the end, this object is so nearby and got so bright, you could have seen it with binoculars if you knew where to look. And it would have certainly been discovered by amateur astronomers. But we found it early. And because we were able to find it early, we were able to get the world's telescope resources pointed at it and trained at this position. And we were able to do some interesting science that we wouldn't have been able to do had we waited longer.
This is a bit of a busy slide for non-astronomers. But what I wanted to say is that all of the regions that are colored were ruled out by some of the data that we were able to obtain very, very early. And in particular, what we're trying to understand with type Ia supernova, even though we use them for cosmology and people have won Nobel Prizes using them as probes, we still don't know all the details of why they explode, and whether there's one star involved or two stars involved. And it's almost certainly two. But one question is, is one of them actually compact and so-called degenerate? And because we were able to rule out the green region, as you see here, we were able to rule out all but the most compact objects that we know about locally. And so this became very strong evidence that one of the objects which blew up was a compact object. Not a huge surprise, but it became a new line of evidence that we didn't have before. And again, we could do this because of the ML assistance.
Many of you who have been working in ML for a long time know that ML, not just on paper, but in production, is really hard to do. And if it's really hard to do, you really have to seek other reasons and other ways to do it if you're actually gonna put this into production. And oftentimes, what you see in academic settings, especially in the domain sciences, is that we try to apply a new machine learning algorithm to some existing data, write a paper about it, and say, here's my ROC curve, let's move on. The prize is really the sorts of plots that you see here. We're able to do domain science in a new and novel way, faster and better because we've applied machine learning in production. Now this whole idea of doing a real-bogus to discovery and classification is now really a cottage industry. And all significant surveys are now building their own real-bogus detectors, not just using random forest algorithms like we did way back in the day. Which seems silly to hear just because that was only 10 years ago. But now a lot of people are using neural nets and putting those into production as well. So that's fantastic. We are using this. It's become an important part of the way in which we wind up obtaining data and using data.
But one of the things we're trying to do now is to push the envelope a bit more still, on this kind of real-bogus idea, but recognizing that we need to think beyond the score, beyond the accuracy, beyond the how good is this algorithm, to something a little bit more expansive that I'll explain in just another slide. The place where we're having to push the envelope is in our search for Planet Nine. So there is a purported theorized massive object beyond the orbit of Pluto, gravitationally bound to the sun. And one of the interesting ideas is that we've already imaged this object somewhere in our archives, but what we need to do is, because this object is too faint to see in one image, we need to actually shift and add these images slightly to follow the unknown orbit of this planet. And if we do that right, then this object may wind up getting popped out of the shift and added data. So there's a student here at Berkeley named Mike Medford, working with his advisor, Peter Nugent, basically doing the shift and add. Now, as you can imagine, the space over which you have to search isn't just one direction of shifting and adding. It's all possible allowed orbital phases of this unseen object.
And so what happens is, after we do a shift and add, we wind up finding candidate objects that are a little bit brighter than the typical noise. And we have to very quickly score them to decide, do we want to even save this object? And we have to score it because it turns out we don't have enough data available to us at the nearest supercomputing center at LBL to actually save all the data that matches some sort of simple criterion above some noise threshold, which is fairly amazing in its own right. And it's because we're producing tens, or hundreds of billions of potential candidates as we run through this old historical data. And so what we had to do is build
a machine learning algorithm that could be extremely capable of deciding, is this real or bogus, based off of simulated data of what an unseen Planet Nine might look like. And then we had to distribute this over a very large supercomputing cluster. And because we didn't want this opining on do I save the data or not to be the bottleneck in the work we're doing, we had to make these predictions in 10 milliseconds. So what we're finding is that to do this right, we needed to shrink our models down to a very small size. And we needed to build an extremely lightweight infrastructure around those models that could serve a prediction at scale very, very quickly. And so what we're starting to see now in astronomy isn't just a push to get better answers, but to get better answers with other types of constraints like smaller models that can opine on data very, very rapidly.
I want to turn my attention now to self- and semi-supervision. I'm seeing some of the questions that are coming up in the chat. And I think, at least my quick view of the questions that I've seen so far are ones where I think I can address some of that through the rest of the talk. So I will do another pause when I get to the next section. And then we'll see if there are any questions based on everything that's happened up until then. So the traditional approach to classification, and this is a visualization of 50,000 variable stars all over the whole sky. The galactic plane is in the horizontal direction. And you can see, obviously, there are more variable stars around the galactic plane. And then as you go farther up, there are fewer and fewer. The typical light curve of one of these observations looks like what you see here. And unless you're very good at doing Fourier transforms in your head on raw data, you might not be able to discover that this actually has a known period of about one day. And this is an object called an RR Lyrae star.
So what we did, also about 10 years ago, was start building something that could look at raw data as it came off of telescopes or historical archive data, and build a bunch of features from this heterogeneous data that is noisy and actually sometimes has spurious detections in it. And we featurize that data in a very traditional, domain-specific way, taking all the things that we could think about from building Fast Fourier Transforms or Lomb-Scargle Periodograms, what you need to use on this sort of irregularly sampled data. And then throwing that into a random forest, and then doing some probability calibration post-processing, we were able to get some pretty interesting results, to be able to classify over something like 20 different classes of variable stars in a held-out way, in a semi-rigorous way, to know that we were able to get errors of order 10% over this very large catalog of 50,000 stars. The challenge, for those that have been working in machine learning for a while, with this approach is that we have to hand-code features. We have to actually say, okay, I can put my domain knowledge hat on, and I can write some code and I can take out the relevant features in this noisy data. That also means that at predict time, when we have to run through all of these features, we have to write and run a code that essentially scales with the total number of features. We also had a very small number of labels: with 50,000 objects in this survey, we only had about 850 known labels over 20-ish classes of stars. So this is a very small label problem.
And that's actually an interesting thing. When you talk to astronomers who talk about machine learning, oftentimes, the first thing we talk about is how much data we have. I'm guilty of that myself at the beginning part of the talk. But when it comes to machine learning, it turns out we have very few examples or exemplars of the objects that we care about. So we have a big data problem, but we also have a small label problem. And it turns out that after you build one of these models and try to apply it to a different dataset, it doesn't really work that well. So what we did is build a semi-supervised autoencoder using a recurrent neural net architecture that allowed us to do basically automatic generation of features. And for those that haven't worked with recurrent neural nets or autoencoders, the simple way to think about it is the picture that you have here. You have this raw data and you build a neural net, which learns to encode that down to just a few numbers. And that's depicted here with this B. That's what's called the bottleneck layer. So think of this as a compression. And then you uncompress the data with a decoder. And the goal here is simply just to reproduce the original data you had. And so if this is a very simple, let's say, sinusoid, we probably only need a bottleneck layer of size three because we've got to capture the amplitude. I'm getting a little bit of feedback. I don't know if somebody is off mute. We need to get the amplitude. We need to get the phase and we need to get the period. And so if you have a more complex-looking source, then you need a larger bottleneck layer to capture all the relevant data.
So the idea here is that we can use this bottleneck layer as the creation of features for us. And I won't go into all the details of what the actual architecture looked like, but what I wanted to point out, first of all, is that it wound up getting essentially best-in-class classification accuracies on several different datasets. But what I wanted to point out is we had to modify existing autoencoder infrastructure to handle the irregular sampling of the data. And also, because we understand something about our noise properties very well in astronomy, we're able to make use of that to change the traditional loss function, which is usually some mean squared error, to a weighted mean squared error. And that allowed us to not over-index in the learning process on poorly measured data. That's one of the things that we had to do, where we had to monkey around with the actual architecture itself. But the other thing that we realized during the course of this work was that even though we had a small number of labels, what we're looking at here is a self-supervised feature learning process, where even if we don't know what the answer is, that is, the classification of a given object, we could throw something that doesn't have a label into the system and actually have the system learn from that to reproduce essentially that data. And the goal there would be that something could get encoded in this bottleneck layer, that even though we didn't have that label, we'd still be able to get a better and better set of features out of that.
The extension of that work is, in some sense, not just taking what you see at the bottom here with a depiction of what I just showed you from the previous slide with the encoder in blue, and the decoder, also in blue, on the right-hand side of the bottleneck layer, but also using both self-supervised lines of thought and lines of learning, and supervised lines of thought where we actually know for some of these cases what the classifications are. So this is what I've been calling the semi-supervised kitchen sink, where we're throwing everything that we know at this problem to get better results. And a survey of not just this approach that you see here, but also different types of specific architectures is something that we put out on the archive and is now in press. And I've given you the link there. I did this with a former postdoc of mine, Sarah Jamal. I want to turn my attention now to what I'll call denoising autoencoders, which is a very similar idea as what you saw before where you wind up taking data and scrunching it down to a smaller number of bits, and then blowing it back up with an encoder. And that's in an application to raw images.
What you see on the left-hand side is a raw image from the Hubble Space Telescope, which looks kind of busy. And it turns out that most of the detections that you see there are from cosmic rays hitting the detector. These are charged protons, or usually electrons or muons, which are hitting the detector and leaving streaks in the data. What we want to see is the thing on the right-hand side. And this is a very nice image of a cluster of galaxies where it's cosmic ray free. So the way that we did this, I did this with a student, Keming Zhang, in my group, was to use a modified U-Net architecture which takes these little postage stamps that include the cosmic rays, and tries to predict the mask of where those cosmic rays are. That's, in some sense, task number one. But then task number two is to take that mask in the original data, and then in-paint with a different network over the data with cosmic rays and get a clean image after that. And as you know, if I give you an image and I give you a mask, you can pretty easily interpolate over that. And the question is, can we actually learn to do better interpretation?
One of the things that was amazing in this process of learning was that we wound up, in the initial layers of the model, discovering, because these are just convolutions in this U-Net architecture, we wound up discovering that we actually had the network find the current state-of-the-art way in which we find cosmic rays. There's another code called LACosmic, which does a Laplace transform on the data to find sharp edges. The network actually learned a Laplace transform in addition to many of the other convolutions that you can see depicted here. So while we didn't direct it and say Laplace transforms are important for edge discovery, it actually wound up finding it, given the training data we gave it. And we wound up getting some really nice results. What you see at the bottom is some zoom-ins to some especially pernicious places on the sky with large streaks of cosmic rays. And you see the discovery of the mask and the inpainting over that. So it looks visually very clean and very good. And then by all the metrics that we knew how to go through, we were getting not just better answers, but we're getting faster answers than the traditional methods that are used in this data reduction.
So these are, what good talk in machine learning wouldn't have a couple of ROC curves on this, this is two of them, false positives versus true positives. And our results are the solid lines versus the previous results from this LACosmic that I mentioned before. And we do better in some types of fields than other fields, but we always wind up doing better than the previous results. And then on the inpainting side, we're basically doing better than median masking and biharmonic interpolation, and actually able to do it much faster. So here's a place where machine learning isn't really gonna be used directly for inference or discovery, but we're hoping to see how machine learning might be part of the data reduction analysis itself, even upstream from some of these high-level discovery and inference techniques. Okay. So let me just pause for a second. Somebody asked, “Was the U-Net part supervised? Did you provide the mask of where the cosmic rays were?” Yes, it was, that was a supervised problem. We were able to determine
the mask by actually doing a median smoothing over many images of the same part of the sky. So we knew what the real view of that part of the scene should be. And so that was a supervised problem where we went through and said, we know the answers there. Similarly, on the inpainting job, we also knew the answers. And so we're able to say, hey, inpainting network, please learn what it is to paint over a cosmic ray when you're in the presence of a galaxy. So it seemed to work pretty well. It looks like I'll move on and I'll take some other questions perhaps at the end of the talk.
What I wanted to transition now into, in the time that I have remaining, is a discussion about physics informed ML. Pretty much all the things that I've been talking about so far is where we're changing some of the learning architectures, asking very domain-specific questions of these learning architectures to get better answers than what we had before. But now we want to be able to use our knowledge of physics to try to learn better and faster. And one of the points of departure for this comes from a very nice paper called “Why Does Deep and Cheap Learning Work so Well?", with the recognition that these researchers had from MIT, that while there is an extremely large space of possible images, if you have a thousand by thousand image and it's grayscale, you have something like 256 to the millions possible states of that image. We're able to learn on images of cats and dogs or galaxies and stars in a very small amount of time. And that's because there's something about natural scenes that are already naturally regularized by the physics of what it is that produces them.
So if that's the case, can we actually use our knowledge of physics to impose constraints on the architecture? And this has been done and started being done over the last couple of years. First in computer vision with a recognition that if I have an image and I rotate it, it's still the same image. Or if I blow it up, or if I add a little bit of shear, it's still the same. So why not have neural architectures that look at an image and have the same output, regardless of whether it's rotated or zoomed in? In high-energy physics, people are starting to build QCD-aware neural nets. As you're looking at particle events from these higher-energy colliders, you want it to be knowledgeable of things like conservation of energy and momentum. And people are doing this in quantum chemistry. What you see here is a depiction of, essentially, they're trying to predict the forces on a molecule. Well, the molecule is the same in the left side picture and the right side picture. You want to build networks that are invariant to these rotations, translations, and permutations. And so this is a very important part of what we could potentially do.
Well, where does physics come in and where do these symmetries come into the work that I've been talking about? Well, first of all, in asking this question, how we can actually embed in our architectures and maybe even the data itself some of the things that we already know about the taxonomy of the objects we're looking at, conservation laws, and symmetries. In the objects that I've been thinking about and working on is a recognition that with periodic variable stars, the previous approaches to looking at time series data were not very aware of the fact that in phase, essentially we end up wrapping around and getting the same observations as you go around in phase. And when I say periodic variable stars, I mean just like the RR Lyrae that I mentioned before. You have a star which is changing as a function of time, but after you go through two pi radians of phase, you get back to where you were at zero pi. And that source generally winds up repeating itself fairly regularly.
So what we did is recognized that instead of zero padding out in these recurrent neural nets, as we wind up doing these so-called dilation layers, we would basically wrap the data around to the other side. And using that we were able to get what we call invariant temporal convolutional neural nets, or ITCNs. And this invariance is a very simple change to existing architectures. But what it does is it gives you very realistic results on essentially doing convolutions not in a one dimensional direction, but in the polar coordinate direction. And so I won't go into the details of what you're seeing here. But essentially because we're doing this padding where we wrap around, we're basically teaching this network that the outputs should be invariant to wherever I wind up starting a phase. And this paper appeared in a NeurIPS workshop last year. And we're working to get this into a journal now. But needless to say, for all different types of invariance that we wound up adding to the well-known types of networks like ResNet and temporal networks, ITCNs, we're able to get better answers from that just by making this small change.
And then on non-astronomy data, it's actually kind of hard to find a good benchmark dataset on periodic time series. We did what all ML people seem to do, which is coerce the famous handwritten digit MNIST dataset into a periodic time series. And we did the following. We took the original data, and the three and the eight you see there, with a random seed essentially permuted every pixel somewhere in this 28 by 28 image, unraveled that. So we got back something that looks like a light curve that's going up and down. And then started this at a random phase. And that was our way of producing this, what we call periodic permuted MNIST. And that also produced very, very good results over the baselines that we studied. So I want to talk now a little bit about how we can start using our physical constraints on learning to be able to get better answers for the types of things that we're trying to do.
And what are the challenges that we had? What we realized is that we don't have an issue of physical models for a lot of the types of events that we look at. To produce a type Ia supernova event requires a massive supercomputer. And even then, we don't know all of the physics to produce realistic light curves. RR Lyrae, we can fit with a template and there are some physical models that give us something that looks like an RR Lyrae. But what we'd like to do is to create a non-parametric, non-linear set of models that can capture the range of physically plausible conditions for all the sources that we may be interested in. And what we built was a variational autoencoder, with my postdoc Jorge Martinez-Palomera and Ellie Abrahams, who is a student of mine, where we built an autoencoder that takes raw data, real light curves with known labels. So these are ones with known classification, and then physical parameters that we know of these objects, and then compress all the data that we have of this object into a latent space and allow us to dial around not just in random directions in the latent space, for those that are used to building variational autoencoders, but also to orthogonalize the latent space relative to physical parameters. And I'll talk a little bit about what that means in a second.
But the output could be that we get these generated light curves that are realistic. The architecture is also fairly simple in the sense that we have a generic encoder on the left-hand side and a generic decoder on the right-hand side. Except that now, we're injecting after the latent space some of the important physical parameters that we want to have as part of our output for our reconstructed light curves. And one of the interesting things is, to do this well, we built the traditional reconstruction of those light curves from a variational autoencoder. That's the first term that you see on the left. But we also had to build a so-called KL-divergence term that was regularized by a hyper-parameter beta. And also because we were trying to simulate survey data, we created another KL-divergence term to make sure that the output light curves were realistic in what their uncertainties were. And by tuning all of this up, we were able to do some pretty exciting things. We were able to produce RR Lyrae light curves realistically. And now we can produce as many as we want. Not just randomly where we generically sample the entire RR Lyrae phase space, but where we can dial up and down physical parameters of these sources and get realistic light curves out. So for instance, as we dial up the temperature on these stars, we wind up seeing they become a little bit more (indistinct) as we go from left to right.
So it's this kind of thing that we think may be very useful in creating these emulators of real physical sources. As people try to build survey cadence optimization, where now they can select from a basket of variable stars, where instead of having to run a big computation to get the result out, they can just run one of these small variational autoencoder models where they can sample from a realistic distribution of temperatures and masses and metal densities. Now, this generative modeling is not a new idea. This has been done in a lot of different places in the context of cosmology. What you see at the top is realistic simulations from supercomputers of the distribution of dark matter over a large swath of the sky. And then you see at the bottom generated samples that have been learned off of the samples that you see up top. And visually, these look very similar. But then also on the top left, you wind up seeing that the distributions also wind up looking pretty similar to the more realistic simulations. People have also started generating images of galaxies, which is extremely helpful in mocking up surveys so that you can test your sensitivity to different types of parameters.
There's something else that's interesting as well in the context of generative modeling. And now this is solving the so-called inverse problem. So instead of starting from a set of parameters and trying to create a realistic version of whatever it is you're looking at, on the left-hand side, you may want to create a gravitational wave signature. On the right-hand side, this is an image from a very nice work that studies the use of likelihood-free inference in physics. Where there, you're looking at a particle physics experiment. What you're trying to do now is, instead of going in the forward direction, where you start with parameters and you get realistic realizations out, what you'd like to do is what a lot of us do in the physical domain, where we have data. And we want to understand the physical parameters that generated that data. And in a Bayesian sense, what we're trying to do is, given data, we're—
We lost him.
I think we lost Josh.
Yeah, he got bumped out. Let me go back in and see.
Let's wait a little bit then.
I realized that my laptop died. And so now I'm switching over to my iPad. So why don't we take this occasion for me to answer some questions that you may all have while my laptop reboots, and I can finish the last couple of slides of my talk? Pavlos, maybe I can ask you to moderate this?
Yeah, there is a question by Jordan. That's, “How would your network handle any variable signal whose parameters are changing over time? For example, stellar position whose period exhibits a long-term oscillation?”
Right. That's a great question. So semi-periodic variables are one of the hardest things for us to do in this invariant network. And so what we're trying to figure out is how we might be able to extend, or at least augment, these types of networks with a more traditional recurrent neural net. The other thing that one can do, and I think Pavlos has spent some work on this, is build sliding windows where you have the same network, but over time, it winds up seeing different parts of a semi-periodic variable. And that actually can get very, very good results without having to use this strictly rotationally invariant network.
Also, Josh, Michelle is asking, “What do you think are some of the biggest challenges to the widespread adaptation of ML in astronomy?”
That's a great question. And I'm gonna try to answer that while I'm multitasking because my laptop is now back up and I can try to rejoin the Zoom. So, one of the hard parts obviously is just training. A lot of us have not gone through the formal training on the stats side and the computer science side to be able to understand deeply the tools that we're using. That's maybe okay in some contexts because we don't always have to understand the inner workings of how a computation works for us to make good use of it. So there's definitely gonna be places where astronomers can just benefit from off-the-shelf algorithms. But I think to truly innovate and to truly push the envelope of these algorithms, we're going to need to be able to have a training curriculum that starts maybe even before college, that gets people that are gonna be going into domains to be able to be functional and conversational and understand how these approaches work. That's one. And I think the other one that's almost certainly worth mentioning is that some types of learning that we do require very large computational resources. And that is already something that not everybody has access to. Now, what's good about this is that not a lot of the problems, at least the ones that I've been interested in recently, have massive computational constraints. But in ones that do, having access to big iron is going to be fairly important over time. I think I'm about maybe 30 seconds away from getting into my talk again. I have only a couple more slides left. Any other questions that I can answer before we get into it?
A couple of them more, but the sub-question, or the previous one, is any argument against widespread adaptation of machine learning besides the big iron, maybe inference is a problem?
Yeah. So that's a very good question. And I have a long diatribe on that. In part, I touched a little bit on that already, which is with machine learning, you always get an answer. And that is a wonderful thing, but it is also an extremely dangerous thing. Always getting an answer means if you've put something into production, you're getting a result that hasn't been vetted by people. And machine learning algorithms don't know whether they've made a mistake. They're just putting results out. And in the context of normal imaging tasks, if I learn on an image, is this a cat or a dog, and then I give that model a CAT scan, it's gonna tell me it's a cat or a dog. It has no idea that it hasn't seen this data before. So I think that is an extremely dangerous problem. And again, because we're not all trained in being able to diagnose and understand deeply what's happening in some of these, what seem like black-box models, we wind up having, I think, a hard time learning when we're making mistakes on that front. So that I think is the most dangerous thing, is that we could be saying things about the universe that are unvetted and unchecked by our own intuition. Putting machine learning in production that's just allowed to run and actually not just do discovery, but potentially trigger other telescopes to take data there, can be extremely dangerous unless you put a lot of guardrails on it. Okay. I'm joining the meeting in progress. Of course, Zoom is asking me to redownload another version of Zoom because why not? Any other questions?
Yes. There's one more by James. James says, “In searches for negative signal in images due to eclipses or others, are there either significant differences in techniques being applied or significant difference in the sorts of false detections on the data?” I think it's a great question.
Yeah, that is a great question. What we've done, honestly, is because a lot of the interest in what I've been involved in—it's now not letting me in. I have to go and reregister. Sorry.
[Pavlos] Sorry, Josh.
That's okay.
If it were up to me, I'd immediately permit you.
Okay, now, I think I can be let back in.
[Pavlos] You should be back.
Okay. And now it's very bizarre for me to see myself from this direction here. So let me share my screen again. And I think this might come back up. So yeah, so what we've done, as you know, just out of practicality, is mostly learn on positive images. I believe that there are some groups that are now learning on negative images. So if you have a reference image and you subtract, and a source got fainter, then if you invert that image, it will look positive. Indeed, the types of artifacts look different in those negative images. So people, I believe, are starting to do that. We didn't do that in the original real-bogus that we implemented about 10 years ago.
Now you're muted, Josh, on this machine. You're gonna be unmuted on the other ASU.
Okay. Can you see this?
Yeah, you're good.
So I already talked about the likelihood-free inference. What I wanted to say is that we're starting to do this and apply likelihood-free inference to interesting hard problems in astronomy. What you see here is from a paper that we submitted about an hour and a half ago to the NeurIPS physical science workshop. And what
you see on the left-hand side is a posterior of a bunch of different parameters that describe so-called binary microlensing events. So it's when you have a star and a planet, or two stars, moving in front of a background star. Because mass bends light, you can get gravitational magnification. And what you see zoomed in here on the bottom right-hand side is a depiction of the light curve from this event. And what's interesting, and makes this problem really hard, is that this is a really ugly landscape of a posterior space. While some parameters are very well-behaved and it's pretty clear you're in a global minimum, there are other parameters, like the second or third one down that you see there, where you have two different islands. And it's unclear whether you're in island one or island two. And just because you found a local minimum doesn't mean that you found a global minimum. And so what we're able to do is learn on lots of microlensing events, run that through the entire neural net architecture that we built, and produce now inference on the parameters that produced these light curves. And we're doing this in the context of a new satellite, which is gonna be launched, called the Roman Observatory. It used to be called WFIRST.
I think just in the interest of time, I'll skip over this bit. But just end with an overall depiction of the places where we're trying to bake physical constants and constraints into our entire learning process. So in some sense, the old-school way of doing that was featurization. You take your domain knowledge on raw data. You featurize that into a set of features that you think are gonna be informative for, let's say, a classification task. Then there's also the symmetry preserving layers. And I talked about in this talk a way in which we're using symmetry padding for periodic variable stars. This is something that Pavlos and his group had also worked on, in the context of solving differential equations, is building symmetries and conservation laws into the different layers. Then there's also the bottleneck layers and imposing sparsity. This is in some sense where we, as people working in the neuro world, get to impose an Occam's razor understanding of how the physical world works by scrunching down bottleneck layers and forcing large amounts of data to be coerced and compressed into a small amount of data before you blow it up again. This is a place for us to impose this sense of sparsity. Then there's the loss function curation where we can actually enforce physically meaningful results at the instance level. So if we're getting results out that look good, but actually violate some conservation law, we can heavily penalize that from a loss function perspective. And then something that I didn't have time to go into today, is as you wind up building these networks, enforcing distributional loss, so that you're getting realistic ensembles of how stars are over the whole sky, for instance.
So with that, I'll end. Apologies for the logistical snafu. And just to summarize here, that machine learning is already very central to astrophysical discovery and inference at scale. And what we're now starting to realize, obviously, is that it's not just the accuracy or the score. We're starting to optimize on things like deployability, versionability, understanding of what these models are doing, size of the models, and speed of inference. These are all things that are important in other realms outside of physics and astrophysics, but are becoming more and more central. If you get a good score on data that's coming in, but it's too computationally expensive, or you can't fit it in RAM on the little machine that you have, it's no good. We're also starting to recognize, because astronomers live yes, in a big data world, but also in a small label problem world, that self- and semi-supervision approaches I think are becoming very key to us to be able to get good answers. And I think the most emergent area of research for all of us is in the acceleration of learning with potentially less data on physical systems when we can figure out ways to imbue our knowledge of physics and symmetries and conservation into the learning process itself. And as I touched on, I think there is a growing symbiosis between first-principles simulations that require potentially supercomputing amount of effort to get a realistic set of observations out, and generative modeling, so-called surrogate modeling, and likelihood-free inference. So with that, I'm happy to stop and take your questions.
Thank you, Josh. (clapping) Great talk. I'll be moderating. I answered some of the questions, but a couple of them that I think it's better you answer. There is one from John Wu. John is asking, “Since the variational autoencoders have intractable likelihoods, does that make it difficult to identify out of distribution examples? Are they good alternatives to autoencoder models?”
[Josh] That's a great question. We found autoencoder models were the most straightforward to train. We certainly started off in the GAN world. And the nice thing about generative adversarial networks, of course, is that you get for free an out-of-sample discovery engine in the form of the discriminator. The discriminator, when you give it new data, can basically say, “Yeah, this doesn't look like data that I've seen before.” So there's actually some great utility in GANs, not so much for data generation, but for looking at data as it comes through and potentially discovering out-of-sample events. The whole problem in general with concept drift that you wind up seeing in industry, the idea that you built a model on how the world was, and now, if it's an NLP model, people are starting to use different words today than they were using yesterday. Your model is constantly getting out of date, and that concept drift can be a big problem. Luckily, physics isn't changing all that much that we can measure. And so when we build models on existing datasets, as long as the new data is being taken in a similar way with a similar set of instruments, we have a reasonable amount of belief that while we may get some noise that comes in that we've never seen before, for the most part, if we've done a good job in modeling the data broadly, we don't have that notion of concept drift in the same way that you have in other types of industries.
Thank you. There was one more. Someone said, I missed it, what do you mean by likelihood-free estimation? That was Shane Lake asking that question. Can you clarify what likelihood—
[Josh] Yeah, so a way of thinking about likelihood-free inference is it's very much in a Bayesian context. If you're a Bayesian, you come to all problems with a prior understanding of what the parameters are that are gonna generate your data. And then you look at your data. And the data winds up informing the results. That information is what's called the likelihood. And the multiplication of your prior times your likelihood gives us the posterior, which is the plots that I showed you for the microlensing example. Those are the parameters and the uncertainty in the parameters and the covariance between those parameters that we'd like to understand. What the neural nets that do likelihood-free inference allow us to do is skirt the need to actually build a likelihood on the data, which is generally computationally expensive if you're doing it in traditional techniques where you actually have to do lots of simulations on the fly. And instead, go directly from data to posterior. Now, in some sense, there is a likelihood that is actually generated under the hood. But likelihood-free inference is sort of a shorthand for saying directly from data to posterior.
Thank you, Josh. There's one more question. I'm not gonna reveal the name of the person.
[Josh] Is it you, Pavlos?
Yes. Your autoencoder with a kitchen sink, right? Did you use the whole data set to train the autoencoder, or you preselected variable objects, or you use everything? And I ask you because if you use everything, you have some kind of covariance shift between the unsupervised part and the supervised part.
[Josh] Yeah, so in that paper, we basically threw in everything that we had classifications for. And you're absolutely right. There is an unspoken bias in this that's possible where if you don't have classifications for an object, there may be a good reason why. The underlying assumption in this kitchen sink approach is that there is a random reason why you don't have a classification for something. But for instance, a good counter example would be if you have classifications for all the bright objects, but not good classifications for all the faint objects. Then clearly, you're not gonna learn much about what it means to be some of these fainter types of objects. And this network is not protected at all from just over-indexing on the brighter objects, and giving you more likelihood on the classification of these other bright types of objects. So it's an interesting question. I'd be curious to see how you think we should and could protect against that.
All right. Thank you. Good. Josh, excellent. Really enjoyed myself, of course. And we should thank Josh again. And I think, Natasha, the video is gonna become available, right, for everyone?
Yes, it'll be posted on our webpage and YouTube channel on Monday.
[Pavlos] Yep. Okay. Thank you.
[Josh] All right. Thank you, everybody.
[George] Thanks, Josh.
[Josh] Bye.