Practical generative neural models in astronomy: semi-supervised VAEs mapping physical parameters to latent space for telescope scheduling, cosmic-ray artifact detection and inpainting (deepCR), and autoencoder-based semantic indexing for compressed sensing.
“The second reason why it's interesting to work with computer scientists is because they're pretty clueless about important problems to work on, and they need our help in understanding what are the important questions to be asked, rather than optimizing Twitter sentiment. They should be helping us understand how the universe works.” – Joshua Bloom
“Up until about a decade ago the state of the art was to hire more grad students to look at these images and go no no no no, yes yes yes.” – Joshua Bloom
“Imbuing some notion of the physical constraints that we'd like to have into our networks is a way of learning more quickly, which is another way of saying it allows us to learn with far less data.” – Joshua Bloom
“I don't hand you this 1.2 terabyte catalog, I hand you a 12 megabyte file, and with that 12 megabyte file you could produce whatever catalogs you want. How much compression can we have that would actually preserve scientific inquiry?” – Joshua Bloom
Hi everybody, thanks for having me here. See if I can do this. I thought I'd put up a quote from a computer scientist, Jim Gray, who recognized decades ago that astronomers are really fun to work with if you're a computer scientist or mathematician. That's because our data is worthless, and if you make a mistake nobody dies. There's no PII that gets leaked out and causes billions of dollars of damage. He loved working with astronomers and we loved working with him. There's sort of two reasons why it's great to work with computer scientists as an astronomer. Number one, there's incredible tools that are being created all the time, and in the grand tradition of Galileo who said, oh, there's this thing called the telescope that's meant to point on the horizon to look for enemy ships, what if I just did this, pointing new algorithms and new approaches at our data with the kinds of inference problems that we have has been tremendously fruitful for us for centuries.
So we've been watching the machine learning world develop and we started taking a lot of those techniques and bringing it and making it our own. But then also the second reason why it's interesting to work with computer scientists is because they're pretty clueless about important problems to work on, and they need our help in understanding what are the important questions to be asked, rather than optimizing Twitter sentiment. They should be helping us understand how the universe works.
So that's really the point of departure for this talk, and what I'll do in the talk is help motivate our use of machine learning in more of a bread-and-butter way, traditional supervised learning, and then get into the kinds of uses of generative modeling that we're starting to see emerge essentially in real time. So it's a very exciting time for us. But to kick off, what I thought I would do, because obviously there's not a lot of astronomers in the room, I'm going to try to give you a little bit of a feel for the kinds of things that we're interested in. In my world this is one of the most interesting plots to look at because it captures both the things that we know about, like type 1a supernovae. This is what's called a light curve, so as a function of time we have these weird units which is essentially, think of it as brightness, so brighter up top and fainter down below.
Type 1a supernovae are the kinds of objects that we need to find en masse to be able to do cosmology, to measure the accelerating expansion of the universe to better and better accuracy. These are the sorts of objects we need. We want to understand how stars explode and how the elements in the universe are created, we need to understand other types of supernovae like type 2p supernovae, and we see lots of these two types of events. But there's other theorized events that, when this plot was made, merging neutron stars, which gives rise to a very bright and loud gravitational wave chirp, those hadn't been found yet, and the EM counterparts, the electromagnetic counterparts, also hadn't been found. It turns out that the one event that we've seen actually follows pretty well the theoretical predictions. But then there's these other types of interesting objects, neutron stars merging with large stars and pair production supernovae. These are theorized to exist, and so as we get data coming off of telescopes what we want to be able to do is find these things in the presence of the mundane. We also want to find some of these anomalies, these needles in a haystack.
Of course those are the kind of known unknowns, and then we can't, by definition, using Rumsfeld's words, write down and plot the unknown unknowns. The things that we don't know that exist out there that would be potentially transformative for understanding of the universe are not on this plot by definition. So the thing you would like to do if you could, for a given observatory or even the class of observatories over the whole world, is write down some optimization metric for how you do discovery, how you do follow-up. Because it turns out discovery is actually fairly inexpensive these days. It's the actual, now that I found this thing, what do I do with it. You need bigger and bigger telescopes, you need more specialized instrumentation to follow it up. This is incredibly hard to do, and in fact even if you could write it down on paper there's the socio-political reasons why you couldn't actually implement this, because there's lots of competing voices about what we should be using with our precious resources.
What looms very large for us is the advent of what's called the Large Synoptic Survey Telescope, which the American taxpayers have paid for already, and will be going for about ten years. It's a billion-plus-dollar facility surveying the entire night sky basically every three nights, capturing something like eight billion objects over the course of its 10-year history, and getting thousands of observations per object. So it's not just covering essentially the whole available sky from the southern hemisphere, but also capturing the time histories of all these objects that are changing. Now most objects of the sky are not changing in interesting ways, at least for most of us, but there are these new events like the supernovae, like these very rare things like emerging neutron stars, that we'd love to be able to find and follow up and do that effectively in real time. So the size of the data are fairly large, and if you think about this again from a needle in a haystack perspective, I'm sifting through all this data just to make the initial discoveries, and that in and of itself is a challenge.
Okay, so for the agenda, I'll talk to you a bit about how we're using machine learning today in a very bread-and-butter way, and then get into some of the versions of self-supervised and semi-supervised learning, show you a couple things that we've been doing in our group around astronomical time series inference, and then dealing with images, doing inpainting during the process of data collection and data analysis. Then I'll talk about physical constraints being built into variational autoencoders and the like and the kinds of things that we can do with it and we have been able to do with it. And last I'll end with some speculative thoughts around generative catalogs and some notion of creating compressed sensing so that we can actually acquire more data from remote sites and be able to
make really good use of that.
Okay, so to start off, bread-and-butter astronomy and ML. As you can imagine, for that needle in a haystack problem, what you'd like to do is find things that are changing in the sky. So what you do is you make essentially a median stack of a whole bunch of images to get the ground truth of what the static sky should look like, and as a new image comes in you subtract your new image from your old one. The problem is the atmosphere is turbulent and your optics are not perfect, so doing that subtraction in the presence of Poisson noise and all the other pernicious noise properties of your detector is essentially an impossible task to do perfectly. So you have to do it imperfectly, and what you wind up getting is a whole bunch of bad subtractions. These are some postage stamps of those, and the number of these in classic surveys that are ongoing right now vastly out-swamp the number of new objects. So the new object is essentially something that is brighter than the static sky, or it could actually be something that got fainter than the static sky if it's an object that's changing over long timescales.
So we have to be able to sort through this very quickly. Up until about a decade ago the state of the art was to hire more grad students to look at these images and go no no no no, yes yes yes. As you can imagine this is a great place for ML, not just for the parallelizability of it, but for the fact that it's deterministic inversion of it all. So this is stuff that I worked on about a decade ago, got it infused into a project called the Palomar Transient Factory, which is in some sense a very good precursor to the Large Synoptic Survey Telescope. And it worked, it helped us sift through the sky very quickly, and perhaps the greatest result of that is represented in this image here where in 2011 a new supernova was discovered. So we call it ML-assisted, by the work that we had done allowing us to hone in on the topmost interesting candidates as quickly as possible.
This turned out to be a type 1a supernova, which as I said before is incredibly important for cosmology, and it was the nearest type 1a supernova in about three decades. So in the digital era this was the nearest supernova, that allowed us to study it in great detail. The important thing from the perspective of where did ML actually help in this world wasn't that it helped us find this quickly, it's that it helped us find it quickly and that we were able to do something with it. This got so bright that if you had had binoculars and you were looking in the right place you would have actually seen the photons with your own eyes, which is actually amazing, that generally hasn't happened since Kepler's time since we had a supernova that you could see as bright as that. So it would have been found by amateurs, but it would have been found by amateurs days later. But the fact that we were able to find it within 11 hours after explosion, which at the time was unprecedented, allowed us to throw the world's telescopes at that position in the sky, get some interesting limits on the change of brightness as a function of time.
This is a very busy plot that captures a lot of physics that we're interested in. These objects essentially allowed us to rule out all the regions that are colored here, which allowed us to rule out the things that exploded to be not traditional types of stars, which are right here, but they had to be what are called compact objects. So a neutron star or a white dwarf were the only things that were left over that were allowed. Had we only detected this days later we wouldn't have been able to make this plot and get the first direct constraints on the nature of what it is that exploded. So that's one example of the kinds of things that we have been doing for a while and are working at scale fairly regularly across multiple different projects. Now it has really become a cottage industry.
There's other things that people are trying to do which I think of mostly as a supervised learning problem. There we have these simulations, for instance, of the distribution of dark matter in the universe as a function of different cosmological initial parameters. So these are things that cosmologists think about and we're trying to measure these parameters very very carefully. Given this few set of half-dozen or so initial condition parameters in the universe, you can simulate forward and model in a supercomputer and get these distributions on the sky, essentially in 3D space, of how matter should be distributed some time after the Big Bang. So the idea is, given an observation of this, could you go and do the inverse problem, and given that, could you actually infer what those parameters are. And so with enough simulations you can imagine using a 3D convolutional neural net idea, one could then try to back that out. So this is the kinds of things that people are trying to do.
Another interesting problem that has recently come out is the idea of looking at what are called gravitational lenses. So these are individual galaxies that have been lensed by a foreground galaxy which has been subtracted off here, and when you get this lensing effect that galaxy, through gravitational effects, winds up basically bending the light around the galaxy and you wind up seeing multiple images of that object. The objective here, given lots of observations of gravitational lenses, is to figure out the size of the radius on the sky, because that then gives you access to other interesting cosmological parameters like Hubble's constant. And so in this project here they simulated a whole bunch of gravitational lenses and then asked the question, could I make a measurement of that angular distance on the sky and recover back the thing that was in the simulation, and they said yes. And then all these different plots up here with the boxes and different colors correspond to these images right here, where they actually were able to infer what these Einstein radii are. Now compared to very complex modeling, this generally is a very hands-on, very labor-intensive, large Monte Carlo problem that you have to work on, because you don't know the properties of the galaxies that's being lensed and you don't know the property of the thing that's doing the lensing.
So this is where the state of the field is at in the context of supervised learning. But one of the things that we started thinking about in our group is how we start making use of semi-supervised and even self-supervised learning for the kinds of problems that we like to work on. So in some sense the pain point that's being captured in this slide here is the reason why we felt like we had to try something new. Typically what we have are observations, again this is a light curve, not of an explosive event but of a variable star. This is called an RR Lyrae star, and it actually
has a period. Unless you're good at doing periodograms with your eye you probably don't know this period is about a half day. And what we traditionally have to do over the whole sky of variable stars, and this is a catalog of 50,000 variable stars that we looked at, we want to infer the classes over dozens of potential classes of variables and transients, and this is essentially all the data that we have. So because we started this project about 10, 15 years ago and started working on it before the resurgence of neural nets and deep learning, we were doing the traditional learning with random forests, and there we had to build features on every single object. So we have a light curve, sometimes we have some metadata, and we want to infer what those objects are, again pretty bread-and-butter supervision.
The problem with that of course is that it requires lots of hand-coding of the feature engineering, which can be labor-intensive and actually can be pretty expensive at predict time, and of course the predict time will then wind up scaling roughly with the number of features that you have. And the other big problem is that we only have a small number of labels. So while we can produce lots and lots of features of these 50,000 objects, it turns out for this study that we did we only had about 800 labels across 26 different classes. It was a very very small label problem. And so we wanted to know how we could actually learn features in a way even when we didn't have the labels. So this is where our self-supervised approach came in, where we wound up using an autoencoder. And you've already seen autoencoders in a couple different contexts today already, but the general idea of course is that you have an encoding of your light curve, you have some sort of bottleneck layer, you have a decoder, you try to get back your light curve. And then what we did, because we're still kind of in the random forest camp, is use these features and augment them with some metadata about each object and throw that in a random forest to see how well we could do.
So here, instead of hand-coding hundreds of features, we were able to let the network essentially learn, not just from the sources that had classes and good labels but from the entire catalog, and it turned out to work pretty well. So here are some original light curves of these sources unfolded, and then when you fold them you wind up seeing the reconstruction in red compared to the blue. When you have very fast periods, in this case here on scales of days instead of scales much longer than that, it turns out when you don't fold, the network spent most of its capacity trying to learn how to do periodograms. So we just basically said let's give you the folded, phased light curves, and as you can imagine it actually did pretty well compared to some of the other results. So across three different data sets this autoencoder approach basically bested all the other feature-based approaches that had been done, across not twenty-six classes because these were smaller data sets than our original one, but across a reasonable number of classes. So we're pretty pleased with that.
But this is one of these interesting places where the computer science work on recurrent neural nets and LSTMs wound up conflicting a little bit with the way in which our data is distributed. In particular when we take data on the sky we're not taking data in regular intervals, we're taking it in irregular intervals, sometimes perniciously, where you wind up having these bad window functions. So we needed a network that could natively handle the distribution of data. So we had time and flux measurements, and then we also have uncertainty in our measurements. There's a Poisson error when we wind up making these measurements, so we needed something that could naturally account for, and actually help what we had, highly uncertain light curves. Those would count less for the building up of these bottleneck features. So we built a stacked RNN.
Was there a questioner? Yeah. No, this is not microlensing data. So MACHO stands for one of these microlensing surveys. There's other data that they had that wasn't just on microlensing events. They observed large swaths of the sky for decades and produced a whole bunch of different classes of different variable stars, so we just focused on periodic variable stars.
Was there another question? Yeah. Yes, we have uncertainties. That's the cool thing about astronomers, is we believe we know what our measurement uncertainty is, because again it's just Poisson statistics. As it turns out though, we're often wrong. So if you were doing this right from a hierarchical Bayes model, if you're doing some forward model, you'd also want to have some nuisance parameter of what's the likelihood that your uncertainty is way off at any one given measurement. So people do have to take that into account. We didn't use that, for instance, in our study. But where uncertainty comes in is basically in a modified MSE. So now the loss function of the reconstruction is basically just using our measurement uncertainties in a per-epoch way, and so if we had a data point that was very uncertain it counted less in the loss, which is just an obvious place to stick it in.
Yeah, no, this is, we're just doing an autoencoder here, so this is just, we're just doing it like an MSE or an L2 loss, but now normalized by the uncertainties. So if the uncertainty is large, then it winds up, one over the uncertainties, it winds up knocking that data point out from contributing a lot to that loss. And then the other place is we had to use the notion of, here are flux measurements, we had to use the notion of the time offsets. So this allows us to naturally use RNN type of architectures where we have these interesting two properties of how astronomy data is taken. And the nice thing about the uncertainties actually is that this also allowed us to do a bootstrap resampling. It's a very natural way of doing data augmentation. So if I have a given light curve and I have uncertainties at every measurement, as long as you assume some sort of iid between all those measurements, I'm free to basically bootstrap resample every measurement that I made, and now instead of one object where I get one pass through this network, now I can potentially have hundreds. And that actually wound up helping us a great deal.
So as you can imagine the thing we were excited about here is being able to leverage a large corpus of unlabeled data for us to be able to build features across all these different classes. And the interesting thing that we hadn't expected is that when you actually use this network on a different survey, where you take data in a different way with a different distribution of classes of objects you're interested in, the transfer learning actually worked pretty well. So that's been out in the literature for a little bit. The thing that we're starting to work on
is what I call this kitchen sink approach, which is essentially taking this blue train here where you have what we did before, but now, given that we are actually interested ultimately in the classification across all these different sources, is to use the classification and have some admixture of the loss across that. But instead of doing this all where you're using this just to build some features on the time series, now these features are actually going to be learned so that if you know the class you should be able to get that class back very well. So this is the semi-supervised approach that we're working on, it's starting to show a lot of promise. We're also thinking about co-training across multiple different surveys, and not just a single band-pass flux measurement of a given object but multiple band passes, so multiple colors. So we're basically throwing everything we know at this problem across a lot of different surveys, and stay tuned for the results on that.
Another thing. Yes, good. Yes, so the question is about source metadata. So I just showed you data on the objects, how they varied in time, but of course there's a lot of observations and information that we have about where that object is in the sky, what's the nearest object to it, what are its colors, does it have radio emission, is there x-ray emission associated with it. And this is one of the challenges, is that every object on the sky has a different catalog that you can look up from the past, and some have a lot of information and some have very little information. So the heterogeneity of our data is actually one of the most confounding, difficult problems here. What I also didn't mention is that the light curves that we're using are variable length, so it's kind of hard to think about how you might do that in a batch sense. So using masked autoencoders is possible, but it's actually pretty painful.
The other thing of course that one can imagine doing is looking at the encoding that you get. And here's just for a toy problem that we're working on, two different types of light curves, what is the distribution of those encodings. And here's across all of these classes. I don't know why this looks like a bird, but it does to me. And this is just UMAP, and then all of the ones that weren't part of the training set wind up getting a very nice separation in this space. And this is where we wind up imposing an L2 norm, so you wind up putting, over all your dimensions, you basically put something on a hypersphere, which actually helps for the separability. And then of course you also can look at reconstruction loss, and here's where we get good reconstruction. And here what we wound up putting in weird objects, these anomalous things that weren't part of the sinusoidal systems that we started off with, we get very poor reconstruction, so we have large loss. So we're trying to build anomaly detectors as well, which of course gives us access to interesting new sources should they be out there.
Another thing we're using autoencoders for is cleaning up data. And so here we have an image from the Hubble Space Telescope of a galaxy cluster. Now it's an inverted image, so the darker it is the brighter the source is, and all this scruff up here, all these little things, are what are called cosmic rays. So these are charged particles that are actually hitting the detector in space, and you want to be able to get rid of those and know what was underneath. These spurious charges, anything that was underneath that is actually completely wiped out because this basically saturates the detector, it gets close to saturating the detector. So what you'd like to do is learn an autoencoder where you have ground truth, so that you can not only detect these objects but you can actually inpaint where those objects were, and there you have to learn what it means to be an astronomical object that you image. So here we're basically just using a modified version of a U-Net. So for those that have seen this before, you basically have a bunch of skip connections, you again have some notion of a bottleneck. And what we want to do is take our original image, and we did this in small postage stamps, and you want to predict out that mask, that's task number one. And then task number two is, given that mask and given the image, do an inpainting so that you wind up getting an observation of what you would have seen had there not been any cosmic rays.
So we just published this fairly recently, and one of the things that we were excited to see is that in the top layers of our convolution, we got excited about this one right here, which if you're not used to looking at these things is actually a Laplace kernel. And it turns out that Laplace edge detectors run over our images is the current state of the art, a way in which people actually find cosmic rays. So the network actually learned Laplace, which is not so surprising, but it also learned interesting things like find symmetries on the left and the right and the top and the bottom, etc. So that was very exciting for us. Visually it turns out it works extremely well. But then of course the thing that you need to do is show ROC curves, and so compared to that state-of-the-art Laplacian edge detector in different types of fields, we wind up having essentially better ROC curves for all of that, so for our false positives and false negatives. The thing that we're also very concerned about, and we actually spent a lot of time thinking about,
is model complexity and its impact on the speed at predict time, because we're talking about taking images that have already been acquired and running it through this network. The longer it takes the slower your pipeline is, and it turns out the Laplace edge detector takes a very very long time to run even if it's been parallelized on a CPU. So we actually were thoughtful I think about the size of our network, because if we had a very big network and it did much better than the current state of the art but it took 10 times longer to run, no one would use it. So this is actually starting to creep up in a lot of the problems that we're thinking about, is not just the quality from a ROC curve perspective, other sorts of impact like RAM usage, like your ability to run even on a GPU. And then from an inpainting perspective, compared to median masking and biharmonic interpolation, we did quite well and we're much faster than the other processes.
Okay, so let me turn to something that's adjacent to that work of AEs and semi- and self-supervision, to physics-informed ML. And this is an interesting paper if you haven't read it already, asking the question, why does deep and cheap learning work so well. There it's a very nice quote making the statement that if you've got an image of grayscale of 256 and it's a thousand by thousand image of a cat or a dog, the size of the available space is, whatever, it's 256 to the million, and yet these networks which have far less capacity than that seem to do extremely well, making the case that the reason why the networks that we're using are working well is because they're very quickly honing in on a much smaller space of what's physically plausible and physically relevant. So as many of you have started thinking about, and some people in this room will be speaking about, the idea of now not just having the network learn this from scratch but actually imbuing some notion of the physical constraints that we'd like to have into our networks is a way of learning more quickly, which is another way of saying it allows us to learn with far less data, and make sure that when we make predictions out of these networks they're actually physically plausible and physically relevant.
So Tess is going to be talking about these rotation, translation, and permutation equivariance types of neural nets that, regardless of what your orientation of your molecule is, you still wind up getting out the same force vectors just rotated appropriately. But this is now happening across a number of different fields, and one of the things that I think in general what we're trying to do is find embeddings and network architectures that don't just conform to the known physics but, if we have a hierarchy for instance of classes, can we actually build that in as well. We want to make sure that our networks are physically informed and are basically constrained by the physics that we have.
So this is a new project that we've started off on recently. I'll tell you first about the application of that and then show you what we're doing and how we're thinking about that. So here, going back to the Large Synoptic Survey Telescope, what we're trying to learn before the survey gets going, next year or a year from now, is how we should observe the sky. And you think, okay, well you just go like this, but then somebody goes no, because then you're not going to find the interesting solar system objects. Over all these different science objectives there's a lot of competing needs on the cadence of how you actually observe the sky and what filter when you come back to the same part of the sky. And this for instance is one of the outputs of a large simulation, which isn't just saying where do we observe on the sky, let's just make measurements of that, but then forward folding through photons from an ab initio cosmological simulation that includes galaxies to figure out what our detection threshold is so that we can then turn that into figures of merit for how well you can detect dark matter and dark energy. So this is a very complex part of this project. This is a terrible cadence if you're
interested in periodic variables, because you've just built in all these aliasing window functions that really make it very hard to find the appropriate periods. But one of the challenges is that there's some objects that are easy to simulate that you can throw into this, but there are a lot of types of objects like the ones I'm interested in that are very hard to simulate, because there aren't these kind of ab initio models where I can just say here's all the physics, now make me a whole bunch of interesting objects of type RR Lyrae or of type Mira or of type whatever. So what people will generally do is take observations of these known classes, try to do some sort of Gaussian processes and then do interpolation so that they can do these simulations. Well, this is very difficult and winds up leading to an interesting set of biases where we don't actually have the full range of the specific class that we're interested in that we could then throw into our simulation.
So what we set out to do is try to find a data-driven, nonlinear, nonparametric model over all the different classes of variable stars that we're interested in that would allow us to walk around in some space, i.e. some latent space, and produce a whole bunch of variable stars where we wind up now not just producing things that look like the variable stars that we're interested in but are ones that are actually constrained by the physics of how we know variable stars to work. And the overall sketch of that is that we have all these different light curves, we wind up having our latent space where we have nice separability in that latent space between the different classes, different colors or different types of objects, and then walking around not just in the abstract latent space but actually now infusing into the latent space some of the physical parameters that we're interested in looking at. In this case this is called T effective, so it's the effective temperature of that object. We want to be able to produce realistic generated light curves that are constrained by that.
So this is a complicated diagram. It's essentially a VAE that all of you have been seeing before, except what we're doing is we're taking our labels of our objects that we care about and want to simulate and the physical parameters of those objects, and we're injecting them after our recurrent neural net space and then basically making sure that the latent space knows about those. But then afterwards, after we wind up sampling from the latent space, we inject it back in so that we wind up having our input back in so that we can construct our light curves and be constrained by the temperature and the size and other types of properties of the objects that we care about. Like I said, we're currently, because we have sort of a swap-in notion, now using these things called temporal convolutional neural nets, which is something that's based off of WaveNet. It has some nice properties, and they act like convolutions and they don't have some of the challenges that you get with LSTMs and GRUs in their computational problems.
Yeah, what's that? Oh yeah, these are the citations. So the citation for WaveNet is van den Oord, that's the original one. Oh no, we're putting it out soon, yeah, we're still working on it. But TCNs, it's a very nice paper that compares that to all the other sequential learners. And so what we're doing is obviously, in the loss, having a reconstruction of our likelihood the same way that we had before with the previous study that I showed you, but we're also actually, in some sort of curriculum learning sense, stepping up the amount of requirement that we have, that first of all we learn our reconstruction really well, and then we start stepping, as we're learning, through this value of beta that winds up adding more power to the KL divergence to basically regularize the latent space very well. And because we're trying to make simulations out of our data, we also want to simulate what our uncertainties in our data would be in our light curves, so we're also building in a KL divergence for the predicted distribution of uncertainties.
Yeah, when I have the uncertainties, again, so these, in some sense you can think about this as another part of the data. So we're trying to reconstruct the light curve itself, but here we're also trying to reconstruct the distribution of uncertainties in the flux measurements, and again we have those because those all come with the catalogs that we're using. So just a little movie of that. So if we don't include the physical parameters we can still walk around in that latent space for a different label that we're interested in producing, but when we have that we can actually see the effects of what temperature does on the objects that we're interested in. And for those of us that have been working with these types of objects, it's kind of remarkable, when you move temperature up and down, that you wind up seeing the effects that you might nominally expect. So it's actually learning some of the properties of the distribution of the data very well.
Yeah, so another part of the project that we're doing but it's less far along than what I'm showing you here, is, so the inspiration for this was what's called a transparent latent space GAN, where you may have seen pictures of it, where you want to walk around in the space of what you're generating where you're kind of linearized on things that matter. So in the case that this TL-GAN was using, he was trying to do face reconstruction and face construction, and so there it'd be dialing up and down the amount of hair of somebody. And so rather than walking in this abstract latent space, essentially what you do is you linearize the latent space, you make a prediction of what are my six values of my latent space and what direction do I need to head in that latent space that is parallel to the physical parameters. Here we're actually just injecting those physical parameters, but your intuition is exactly right, if you walk around in this and you move Z0 down by a little bit and Z1 up by a little bit, you're actually heading in a parallel space to the physical parameter. So what
we're doing, because we have these for a subset of our data, we're also trying to see what if we just kept this latent space like this but instead of dialing in these parameters, what if we just purely dialed in the temperature direction or the mass direction or something else. Yes, correct.
And are you disentangling, are you doing something to encourage disentanglement amongst the abstract? No, we're not, well, so yes, the encouragement of the disentanglement comes from this part right here. You wind up actually getting a nice disentanglement over time. So it's interesting, when you don't have this KL divergence and you're just trying to do reconstruction likelihood, all the classes of all the objects wind up just being jumbled all over each other. But here we're actually encouraging disentanglement of the classes, and you're using the reparameterization trick anyway as well, and so you wind up getting this to look like a Gaussian.
Let me just go on for now, just in the interest of time. How am I doing on time? Say that again. 20 minutes to lunch, okay good, yes, well I'll be done. So that's our foray into the generative modeling space. Many other people are doing this as well. In the context of cosmology here, this is again producing what are called weak lensing maps. So this is in some sense a 2D projection of the dark matter distribution on the sky. Here's the original data from simulation, and it's basically indistinguishable, at least visually, from what these GANs wind up producing. But more importantly, when you actually look at the important properties of these, like a two-point correlation function for instance, they wind up falling along very well, not just producing visual similarity but also these similarities in this distributional space. What's interesting is that I don't believe that this study actually imposed regularization that the distributions of the images should wind up having these properties, it just came out naturally from the GAN.
Related to that of course is surrogate modeling, where you have extremely expensive computational projects. So here's an example of just a two-dimensional model of a supernova, where you have a couple of input parameters like the mass and the distribution of mass in the star before it blows up, and other sorts of things like its rotation, magnetic fields, etc. You have just a few numbers that you start off with, and then you need a massive amount of supercompute to be able to produce these pretty pictures here. But of course this is not what we observe from an object which is giga-light-years away. What we observe is these really noisy measurements of the total intensity as a function of time, which is the integrated light that comes out of this. And this one simulation for instance was 360,000 CPU hours at NERSC just to produce this one object, which produces one light curve for one set of input parameters. What we'd like to be able to do is make measurements like this with these black points and figure out what are the correct input parameters to the explosion. And there, surrogate modeling has been very helpful. As far as I know that's mostly only been done with Gaussian processes, where they have a family of different explosions and then they have a way of doing interpolation using Gaussian processes in multiple dimensions to be able to back out what the parameters of interest are.
This is also starting to be used as well in gravitational wave measurements and simulation. What you see on the right-hand side is what's called a chirp
signal as a function of time in milliseconds before you have a black hole merge with another black hole. The problem is the breakthrough in making the actual exact calculations for this only happened about 10 years ago. To do the full general relativistic simulation in 3D that's numerically correct and stable, you need all these different input parameters. These are dimensionless spin parameters, this is the ratio of the masses of the two black holes, and then some orientation parameters. You need all of these to be prescribed exactly and then you wind up getting these different waveforms. What people are doing in this study is taking, across these simulations of what these waveforms would look like in these different parameters, they've got these 1500 or so simulations of waveforms, they're creating a surrogate model that allows you, if you had a measurement of this chirp signal, to be able to back out what those parameters are, and they're doing pretty well. This is this study and this is a previous one compared to the full numerical relativity supercomputing simulation. Each one of these data points is like more compute time than probably most of you have ever used. This is a massive endeavor that's been ongoing for decades to be able to generate this coarse grid, so the idea is to be able to build something that is more finely grated, and that's where the surrogate modeling was coming in.
And of course, related to that are inverse problems and likelihood-free problems where you want to do some sort of inference. And in this case, and you'll hear more about this from Ben at the end of this week, you've got simulations of a particle shower from a collider and you throw that in, including the original parameters, and you use this as a way to get confidence intervals on the parameters that you're interested in.
Last, in the time that I have left, I just wanted to give you some other speculative ideas, because we're talking about generative modeling here, on how we could actually make interesting use of generative modeling in astronomy beyond the sorts of problems that I've given you. So here's actual data, this isn't simulated data, from a satellite which is operating right now called Gaia. It's making measurements on about three billion stars in the sky, and the amount of data that's coming down from the satellite is about 200 terabytes of raw data. The final compressed catalog is 1.2 terabytes of data, and there's a lot of interest in this. By the way, this is the projection of the sky if you haven't seen it before. Usually you don't get to see the Milky Way really nicely. This is the center of our galaxy. See these Magellanic Clouds, very nice image. And this is actually an image that's now been generated not from the original 2D pixels but from the catalog itself. And you can see the distribution of colors and some interesting properties.
One of the things that people like to do with this isn't just study individual objects but be able to find large structures in our galaxies that hadn't been known about. When you look at other galaxies you wind up seeing that they've actually been eating other smaller galaxies. This is called galactic cannibalism, because astronomers like to make up fun words, and it's actually true. This is the way in which galaxies form, is they basically assemble themselves from smaller little building blocks over time. And we know that the Magellanic Clouds for instance are sort of waiting to be eaten by us eventually. These are little dwarf galaxies that are in the tidal influence of our Milky Way. But there are other galaxies that had been around that actually got totally ripped apart and are now part of our galaxy, we've eaten them. We've known about these streams for a long time, these are called tidal streams. It turns out though that there are a whole bunch of other streams that we hadn't known about until Gaia started producing its maps of the sky, because it's making not just a map in 2D space, it's making a map of the locations of these objects using parallax. And this is an amazing study where they're basically finding all these essentially old galaxies that have been eaten up by the Milky Way.
So the idea is, what if you could take this catalog and you lose information about an individual object but you could compress the catalog down to a small amount, that you could still do these aggregate studies to find interesting things about large-scale structure. And the idea is again some sort of autoencoder where now I'm actually just trying to autoencode the catalog itself. And so I have my normal latent space regularization, but for my numerical columns I do some sort of weighted MSE, for my categorical columns you do some cross-entropy loss. But then what you want to make sure is, when you wind up having your batches, the outputs from this, this is the raw catalog output, so this would be like location on the sky, proper motion, velocity on the sky, etc. Some admixture of some of these winds up basically giving us some interesting insights into the distributions of the physical parameters of these objects. This is a proxy for temperature on this axis and this is a proxy for brightness here. You notice that the universe doesn't populate this space uniformly. This is what's called the main sequence, this is where most stars lie, these are where white dwarfs lie, and these are objects that have fallen off the main sequence. What you'd like to do is ensure that you actually have some regularization for all the different physical plots that you'd like to make that come out of that catalog. And so that's that extra little piece that I think is going to be tremendously useful. And then
it gets us to an interesting place. We could use this for anomaly detection, so if we run all of our objects through our mock catalog creator we could actually probably find some data quality issues where an object is not where it's supposed to be in that space and its loss, its reconstruction, is pretty bad. We could produce thousands of simulations of the Milky Way that are all consistent with the observations that we have, which would be really nice for us to be able to compare to theoretical statements about what the Milky Way distribution properties should look like. And the other interesting thing maybe is that if you were able to compress a catalog down, you can also do some option where the people who make this catalog, who want to do the science, could release the aggregate results to the world and then they could work on the individual objects themselves, but it would allow people to basically build up their own catalogs where you're not actually leaking out the individual information about any one individual object. So one of the things that we're kind of starting to get interested in is, what amount of compression could you have, essentially I don't hand you this 1.2 terabyte catalog, I hand you a 12 megabyte file, and with that 12 megabyte file you could produce whatever catalogs you want. How much compression can we have that would actually preserve scientific inquiry?
So in some sense the way that I think we're all starting to think about where physics comes into the learning process is the fact that we have the ability to infuse physics across that whole process. We already do this in the old-school way when we do featurization, when we take our raw data and we featurize it down to a couple of numbers, we're using our physical knowledge and our domain knowledge to be able to compress that data. I mentioned to you, and you'll hear more about this throughout this week, these symmetry-preserving layers that preserve conservation of energy, that preserve rotation, etc. I talked a lot about the bottlenecks and making sure that the models are sparse enough that you're not actually having too much capacity in your system, that's a way of saying we want to have Occam's razor be in effect when we produce our models. We can have loss functions that enforce physically meaningful results at the instance level, so I talked about that as well. And then what I just started introducing is distributional losses where we make sure that when we actually throw a batch of objects through we wind up getting a distribution that falls along the lines of what we know to be correct physically.
And then, last and closing up here, where I think this could be really interesting in the context of catalog compression, is the fact that we often are taking data in remote parts of the solar system. So the James Webb Space Telescope, which your taxpayer dollars have also paid for, order of magnitude more expensive than LSST, which is going to fly in a few years from now, it's sitting at the Lagrange point, at L2. The data rate from there is not very good, it's about two megabytes a second, and you can only get about 56 gigs a day down from this satellite. Yet a single instrument, and there are multiple instruments on this, at full bore can produce essentially a terabyte a day. So we're already having to make decisions about how we should observe the sky because of this bottleneck of us getting data down from these satellites. And somebody once said, these photons took a really long time to get to us, we owe it to them to actually detect them, the fact that we're not doing that and throwing out data is absolutely ridiculous. And the state of the art, by the way, in a lot of these satellites, is they wound up taking data and then they have prescribed regions on the sky that they're interested in and they only send down postage stamps of all the pixels from around that and they throw out everything else, that's their way of getting around the data rate. So, and this is very speculative, there's no reference, don't quote me on it if it's wrong but quote me on it if it's right, the idea here would be, what if you could train a denoising autoencoder on
lots of simulations of what, let's say, the James Webb Space Telescope is going to look at. There's a picture of a simulation, people do this a lot, where you have all the data available you possibly could, you build a nice beefy enough network that given the raw data you could actually not send down the raw data anymore. What if we just sent down the bottleneck layer? And of course the ground station has the model because you flew it up there in the sky. But you can also wind up periodically updating your model on the fly on the telescope and send down the diffs of the weights so that you wind up syncing up over time, when you have free bandwidth, the current state-of-the-art model. So I don't know if this is going to work, but I think it'd be really interesting, convincing NASA that I could actually fly something like this and not send down the real data would be pretty interesting.
Yeah, the question is could you install these things remotely. There's no way that they would let you do that, right? Every line of code has got vetted over hundreds of times, which means there's probably lots of bugs. And they literally generally fly not a lot of data space, and in fact JWST is flying only about 10% more data space than they can download in a day, because if they start accumulating more data and they can't get it down, there's no reason to have it. So the amount of data that's there, the amount of code and the code that's running there is all very prescribed. And at L2 you don't have a huge energy budget problem because you're reasonably close to the Sun, but as you get to even more remote sites then you really have to start worrying about clock cycles, and so all of that generally is fairly locked down before it flies. This would have to be a totally new mission, and I think the way to do it would be to do it with a CubeSat, which is only a few million bucks, and say I'm going to take data like every half second of the entire sky and send down not all of that data because I don't have enough bandwidth, but just the bottleneck.
Yeah, real time, but we have a stream which is, yeah. I mean, yes, you could of course do that. The nice benefit of having just a model that's small is that the amount of clock cycles to do a forward pass wouldn't be all that much, but I think if you were going to do this, yes, you could totally do that, you could do a hybrid approach where you're sending down all the interesting data in the old-school way. But the problem also is that we take data in a different way depending upon what we know our bandwidth is. And so what they're doing effectively is longer integration times, which opens you up to more cosmic rays and other sorts of problems. They do longer integration times because they know if they just accumulate one image instead of a hundred images, that's much easier to send down. So we're already taking data differently because of the bandwidth issue.
You're probably taking as much data as possible, then after the fact the data is taken you're deciding what to keep or not. I haven't decided whether I want to take this on because this is going to be more of a political fight, but it'll be fun to write the paper where you say this is possible, here's the kind of losses that you get. And by the way, it's not just, your loss function now would wind up being very aware of the kind of science you want to do. You basically say I want to preserve the PSF and limit down to some brightness level, so if the sky is blank there you're not basically having to reconstruct that part of the sky.
Yeah, let's say you find an anomaly. Yeah, good question. So around anomalies, I thought about that a little bit. The idea would be, as you throw the data in, whenever you have a good reconstruction loss on board you throw down the bottleneck, when you have a bad reconstruction you say that's an interesting thing that just came through, I don't know about that, I'll use that to update my weights of the model but then I'll also throw down that full image. Yeah, so I think that's the way you might do it, yes. It's a good question.
Why don't I just summarize here, and then maybe we come back to that, we have a little bit of time. But anyway, just to give you this overall sense as I close: machine learning's already fairly central to the way that a lot of astronomers are working. We're kind of using it for the bread-and-butter inference and discovery at scale, essentially replacing grad students. But we're starting to get interested in the semi-supervised and self-supervised approaches, in large part because we have so few labels, we need to do this unsupervised or self-supervised learning of labels. But where I think there's a lot of excitement, and this is really the crux of what this workshop is about, is how in this context we can actually imbue some of the physical constraints and the distributions of our catalogs into the actual loss itself as part of the regularization. And I think what we're starting to see is that growing symbiosis between the first-principle simulations that are extremely expensive and the generative and the surrogate modeling that we're actually really interested in, getting to these fast ways of getting to inference of the parameters that we care about. And last, I noted some interesting things we might be able to do with generative modeling for compressed sensing. So with that, why don't I stop, and then we can start taking some of the questions. [Applause]