How the astrophysical data deluge has swamped traditional workflows and driven AI/ML tooling: real-time telescope control, survey optimization, simulation-based inference, multimodal foundation models (AstroM3), active optics, and neural compression (AstroCompress).
“When people talk about astronomy being a big data place, it's true we have a lot of data, but we also have small numbers of labels, and we are label starved.” – Joshua Bloom
“The end goal of doing AI in our work is not to do AI in our work — it's to enable novel science.” – Joshua Bloom
“We're not just looking at simulated data to find simulated results. We're actually stumbling upon something that's actually fairly deep in nature.” – Joshua Bloom
“The best chess player is not you anymore as a human, nor is it a computer; it's you plus a computer. And I think that's where we're all hoping to go with this AI-assisted science.” – Joshua Bloom
“Ultimately I think we need to stop building foundation models for foundation model building's sake and start putting them into the hands of people that don't build foundation models.” – Joshua Bloom
All right, we're going to go ahead and get started. Hello everyone, thanks for joining. We're very excited this month to be hosting Josh Bloom for our IAIFI colloquium. Josh Bloom is the Miller Professor at UC Berkeley. He has been a pioneer in the use of and development of machine learning for time domain astrophysics. Broadly speaking, you can probably think of a time domain topic; he has probably written several papers on it. He's done work in SBI and optimization of telescope resources and is now a key player in the rapidly evolving field of multimodal foundation models. Hopefully, we'll hear a little bit about that today. He also founded a startup in AI over a decade ago, before that meant writing a wrapper around API calls to OpenAI, when it actually meant something. Today he's going to be talking a lot about the various projects that he and his group have been working on in accelerating inquiry and insight in astrophysics through AI. Great. Go ahead, Josh.
Thank you. Thanks for the introduction. All right — I've been told to mention that if you have a question, then please raise your hand and I'll run the mic over so that the folks on Zoom can hear you. Thanks for the introduction and thanks for the invitation to be here. I realized when I was getting this together that the last time I gave a colloquium at MIT was the day they announced the Nobel Prize for the discovery of the accelerating universe. And I don't know what's going to come from the rest of the day, but hopefully something amazing also will happen. I also realized after I submitted the title of this talk that I just inadvertently created a really cool acronym — which maybe is not advancing. Let's try this. There we go. Maybe that's the sound of all of us screaming out into the void or something. But I'm happy to be here. It feels a little bit like coals to Newcastle, because a lot of what's happening in the machine learning world is happening in and around this institute, in astronomy. And I couldn't be more pleased to meet all the people I did today. For those that are on the Zoom, happy also to take questions by email later on.
I'll be talking today in some sense about the things that are near and dear to my heart, and a lot of the people that you see up here have worked on various aspects of what I'll be presenting today. But I hope you'll see that we're telling a broader story about where AI fits into astronomy. I thought I'd use this slide as the point of departure to help orient everyone around the grand challenges that we have in front of us. This is a light curve of brightness as a function of time for a bunch of explosive events that are outside of our galaxy. Some of these things are incredibly important and very common: Type Ia supernovae, if you want to understand something about the evolving universe on a large scale. If you want to understand the death of most massive stars, you want to understand Type IIP supernovae. But then there's some other weirdos here. This plot was made before the famous event where a kilonova was discovered — what happens after two neutron stars wind up merging, a little blip in the sky that lasts a short amount of time. And then there's these other light curves from other weirdos that have been theoretically proposed but never seen. And then of course there's the things that we don't even know we don't know, so we can't even put them up here.
We're trying to devise systems that can be looking at lots of data to be able to discover these things, to be able to characterize them, to be able to find the things that aren't on this slide. And we're trying to do this in a way that optimizes our scarce resources for follow-up. We're going to be getting — and already have really started getting — large amounts of incoming data for being able to do discovery and initial characterization, but getting spectroscopy, getting more data from other wavelengths, is going to become increasingly hard, and there's going to be increasing competition to do this. What we'd like to do obviously is write down some single optimization metric that says if we did this right — and then we could somehow train some large network to figure out how to do all of it — then we get the best answer. But we don't have those metrics. It's not number of Nature papers or negative number of Nature papers. It's something else, and we all have different metrics, and we're all competing for the same resources. So this is a very vibrant and lively and really timely problem that we have in front of us.
And this is the telescope for many astronomers that looms very large. It's been decades in the making and is seeing first light approximately this month, maybe even this week. And so we're about to get this torrent of several terabytes a night of raw imaging data, hundreds of thousands of event discoveries that are going to be coming, and we're going to have to be sifting through. The size scale of this data obviously pales in comparison to some of the data sets that some people see in physics or in radio astronomy. But for many of us, this is the thing that we're all looking towards and have been working towards, building machinery to answer some of the questions that we have in front of us. And so there's this growing need for ML imbued into the way in which we do science, into the entire data flow. Some of the things that are obvious to the people in the room, but may not be to others, is that we have a challenging problem when we take lots of data on large parts of the sky. We have a new image here you see on the left, and we have a reference image taken from many images taken over a long period of time before this new one, and we subtract the two, and we're looking for a new supernova or some other sort of transient in there. This is a challenging problem not just because of the size scale of the data but because the subtraction process itself can wind up introducing lots of spurious artifacts. I'll get a little bit more into that later on. So are we looking at something real? Are we looking at something bogus? Once we found something, we look at its light curve and we ask questions like: who cares? What is it? Do we spend precious telescope resources to follow this thing up? And then if we're trying to do inference on the things that we are observing and have gotten a lot of data for, oftentimes the forward model — the forward physical model that we have, if we even have one — is extremely computationally expensive to implement. So if we're trying to do parameter inference, we need ways to make that faster, and surrogate models and simulation-based inference are emerging as some of the tools of the trade. I'll try to touch on all of these throughout the talk today.
This is — and I'm sorry for those that are conscious about how the oil and gas industry is screwing things up these days — a terminology from the oil and gas industry called midstream: all the things that happen in the middle of the data flow. But there's some really interesting stuff downstream, after the data has been taken, after we've done the analysis. This is where the human-computer interface comes in; this is where the AI plus scientist becomes a really interesting question. And then, as you can imagine, some of the places that I'm most interested in these days is in what I'll call upstream: before data is even taken, where are the places where AI can help us accelerate and do better? I'll start off with my introduction to machine learning, which was this world where we're getting lots more images coming off of the Palomar Transient Factory telescope circa 2008, and instead of asking undergrads to look at data and find whether something was real or not — which at the time was really state-of-the-art for grid computing, and kind of still is in some realms — we wanted to create this real-time framework that could act as that surrogate. And we get some other benefits from building these machine learning systems. They're fast, they're parallel, transparent, deterministic, versioned, in a way that even with crowdsourcing you don't get all of these sorts of things. And we're talking, at the time, the subtraction capability was producing, for every one of these real subtractions of real new objects, about a thousand bogus ones that were above some sort of detection threshold. So this was a massive needle-in-a-haystack problem that we solved at the time with random forest plus handcrafted decision rules, and we put this into production. I don't know why that's dropped off here. We put this into production, and it's led to about a thousand papers so far from the PTF and now ZTF collaboration.
One of the things I'm most proud about is that we discovered the nearest Type Ia supernova in 30 years. And because this was in the front of this galaxy and not behind it, there wasn't a lot of dust, and we're able to do some really interesting things. This was found about 11 hours after explosion, and it was assisted with this real-bogus technology. We also had in place this thing which we called Oarical, which was a hierarchical classification — not just on the discoveries but then also what is this thing — and this was happening in a real-time environment. Oarical, because I also can't spell, was supposed to be cheeky: when you're up transients creek without a paddle, you need an oar from Cal. That's where that came from, just in case people are interested.
But here's the important thing. This supernova, over the next couple days after discovery, wound up getting bright enough that amateurs could see it with telescopes, and eventually you could see it with binoculars. It was so bright and so close. This would have been discovered by lots and lots of people had we not discovered it when we did. But because we were able to do this quickly, without people in the discovery part of the loop, we were able to get on this object very quickly with the Hubble Space Telescope and with Chandra. And because of that, we were able to rule out large parts of parameter space of the possible progenitor of the thing that actually exploded, invoking some theory — but we needed observations in the first couple of days. Without going into all the detail, we were able to rule out all the places in color here, leaving behind only compact objects. It's no surprise to the people who study Type Ia supernovae that we think they're probably white dwarfs that are blowing up of some sort, but it was really nice to show that we were able to find this and infer that nothing else ordinary was able to do that explosion. And again, this is because the AI system was in production at the time.
As I hinted at, this has become a bit of a cottage industry, and now instead of random forest plus handcrafted decision rules, people are using lots of deep learning techniques. And pretty much at this point, this is part of that pipeline where we can't even imagine having people in a real-time loop. We just have so much data coming in. That second question, though, is still an interesting one: what happens after we know that there's something new in the sky?
We started asking this question a few years back, looking at existing catalogs of not explosive transients but variable stars, the benefit there being that after we make an inference, even if it's been 10 days since the last data point was taken, we could actually go and get spectra and get some assurance that we were right. The goal is to take light curves like this and infer what those objects are across a hierarchy of classifications of variable stars. And unless you're good at taking Fourier transforms in your head, you don't know that this period is about a half a day, and this is a typical so-called RR Lyrae star. The catalog that we spent a lot of time with had 50,000 variables in it — the ASAS catalog from the Southern Hemisphere — but it only had 810 labels over 26 different classes. One of the things that you should know, if you're not in this game, is that when people talk about astronomy being a big data place, it's true we have a lot of data, but we also have small numbers of labels, and we are label starved. And so it demands from us a lot of attention in thinking about how we can do some sort of representation learning over much larger corpora of data sets when we don't have a lot of these labels, when ultimately it's determining the labels for all these objects that we're really interested in.
In our first foray into the deep learning world, we started building autoencoders using RNNs — there's now better techniques — but the simple idea was to take that light curve, go through a network that creates a small bottleneck that compresses this entire light curve information into a small number of numbers, and then decompress it to try to reproduce the original data. And then what we did is use this bottleneck to do a downstream classifier — in this case, we were using random forest. This is one of the first examples of using self-supervised feature learning in astronomy that allowed us to build great representations of what was happening in this variable star world using a massive number of unlabeled data sets. And it turned out it did pretty well — achieved state-of-the-art. But we're still interested in trying to figure out ways in which we can do inference with less data and less training, because there's just not enough to train these extremely large billion-parameter models for most of the problems that we're interested in. And here is where some of the work that's happened in Jesse's group — and Tess has been involved in this as well — is trying to figure out ways, like has happened in other fields, where they've been able to introduce some sort of inductive bias in the shape of the network and the way the network's trained, to be able to make use of known symmetries in these systems, so that we could train with less data and get potentially more meaningful answers out of that. And so that's the question that we're asking: can we find ways to do embeddings? Can we find new network architectures that allow us to conform to what we already know about the systems that we're trying to infer more about?
One of the ways in which we attacked that was recognizing that with variable stars, many of them are actually periodic. And so that means that if you phase-fold them with the correct period, they keep repeating themselves over and over again. The typical thing to do in a convolution, as you're trying to get knowledge of what's happening on large distances in time or in phase, is you go to larger and larger receptive fields. Typically, though, what happens in most networks is that you wind up having to pad out beyond what you've observed with your actual data — you pad out with zeros. Zero padding is a very common way of being able to infinitely scale your receptive field. We did something really simple, which was, instead of padding, we just did what we call symmetry padding, where we took the answers from the far right side, from the farthest part of the phase, and appended it over here. So we were guaranteed that when we did convolutions on these systems — instead of doing convolutions on a sinusoid and getting this out of some part of the network, where depending upon where we started our phase we get different answers — here, regardless of where we wound up starting our phase, which is an exogenous thing to the actual object (the RR Lyrae doesn't care if we started observing it around max or around min), we wanted to make sure that we get the same answer out regardless. And so by just doing this very simple trick and introducing this invariance to this wrapping, we created convolutions in a polar context, and that allowed us to tack that on to a whole bunch of different types of networks. Essentially, over a couple of different data sets, we were able to achieve again state-of-the-art in the classification accuracy, just because we made this little simple trick where we introduced a symmetry that we knew exists. So we get these probabilistic catalogs of variable stars, and the question is: so what? Here's a great paper — referee liked it, he published it — but the end goal in this work, and I would hope in all of our work as we think about doing AI in the context of astrophysics and physics, is asking the question: how are we doing novel physics or astrophysics with the result of the AI that we've built? And so I see this really as a critical place where we inform the humans to use the precious follow-up resources that we have available to us.
With a student of mine, we wound up querying our large catalog, and we were able to triple the number of very weird so-called DY Per stars, and we got a couple of new so-called R Cor Bor stars. Some of these stars were so bright the Babylonians saw them with their own eyes, and they weren't known to be of these classes because they didn't match the exact hard cutoffs that people had made to match the canonical versions of the ones that we already knew about. Instead, we're getting these fuzzy examples where we get a ranked list of objects that could be this. We took spectra of a bunch of them, and a bunch of them turned out to be right. We're also able to find detached eclipsing binaries, which, with a large amount of spectroscopic follow-up, allows us to place those objects on the mass-radius relation and compare that directly to theory, and then on and on and on. So again, the end goal of doing AI in our work is not to do AI in our work — it's to enable novel science.
Another place, which Alex alluded to, where the community is getting very excited is in these so-called multimodal foundation models. And here's another place where we could in principle train on a large corpus of data across not just light curves but maybe spectra, maybe images, maybe metadata, maybe comments about things, and hope to get out a better answer for some other so-called downstream task. And so here is one of the simple ways to do that, where you have two different modes: here you've got words and here you've got images. This idea called CLIP — contrastive language-image pre-training — allows you to do a data-appropriate embedding and then map that to an embedding space, and then a similar sort of thing. And when we know these two objects are from the same tuple of image and word, we hope that we get a large amount of diagonal power here, which allows us to align the embedding space, and then, over a large corpus, without any other knowledge of what the classes of these things are, one would hope that you could create an embedding space that has meaning across these different modes.
And so this has been done in astronomy. Some of the people in the room here have been involved in the one in transients. There's also one that was done using galaxy images and then galaxy spectra to try to coerce this raw data into a shared embedding space, with the idea that both spectra and galaxies, and the images of those and the colors of those, are telling the same story. Same thing in the transients world: training or pre-training these embedders in simulated data space, and then taking the results of that, and then taking observed data and fine-tuning that to get a good embedding of the system, and then applying that to downstream tasks.
So this is the point of departure for our own work, where we said: what if we wanted to do more modes? We want to do not just images and spectra, but we want to do other things. And here we were able to do light curves and spectra and other sorts of metadata. And we extended the CLIP idea to a multimodal setting beyond two, where, using again data-appropriate types of embedders up here, we're able to coerce that into a shared embedding space that allowed us to build this interesting model on a fairly large corpus of data, to do interesting things, we hope. So the question is: what were we able to do?
Well, first of all, we wanted to see whether this was even worth doing at all. So we checked to make sure that if you trained a data set over one of these modes — let's pick spectra for now — how well would you do if you trained using CLIP for the pre-training and then did a downstream classification task? Here we're actually just trying to classify a bunch of variable stars. And you see we did, over the entire data set, about the same pretty much through all of this, which means this didn't help that much if this was the only data you had and you already had all of the labels that you had in hand. But where things get really interesting is where we start reducing the amount of data that we have available to us, and we train in this case with no pre-training, maybe a classifier just based on the spectra. Well, if we pre-train with all of the data and we just use the spectra with only 50% of the data, now we're starting to see the big improvements. And as we go down to smaller amounts of data, we can see the improvements get very dramatic. So this is a surrogate for the world where you have lots of data, but you wish you had a whole bunch more. You could potentially train on all of these different modes and apply it just in a single-mode context, or apply it across all of these different modes. And so this is giving us some hope that these foundation models across multiple modes are actually sharing information, learning a reasonable representation, and those can then be used for at least the classification downstream task.
And what's the first thing you do when you have an embedding space? You plot it. But you can't do it in 512 dimensions, so you use a way of bringing that down to just a few dimensions. So with a UMAP projection of those 512 dimensions into two-dimensional space, we have all the different classes of variable stars that we put into our system. And you notice, for this class called M, which are Mira variables, it looks like visually there are these two different clusters that are showing up. This is not something we trained the system to do — we didn't ask it to produce two different clusters of Miras. And in fact, during the whole training it was agnostic; it didn't even know about what the classes were for these objects. So we're only colorizing them just to guide the eye. But it turns out those two different classes are actually two different types of Mira variables, which weren't in the classification training set but showed up very obviously: these C-type (carbon-type) or O-type Mira stars really show up very obviously in that separation of cluster.
There are other minority classes that we didn't even feed into the system when we were doing the pre-training of our embedding, and then we ask where they show up once we put them in after we've built this model. So these are unseen classes. The colorization might be really hard to see: there's these certain kind of RR Lyrae stars, RRds, that are showing up in the locus of where the other RR Lyrae stars are. And so again, it's giving us some indication that it's learning something about what it means to be those types of stars, or variants thereon. You can imagine using these for outlier detections. Either finding objects that have been incorrectly classified — or they maybe have incorrect periods in their metadata or something like that — or objects that have the correct labels but look weird compared to the other ones in that class. And indeed, a bunch of these things start to show up. This is an object that's a known semi-detached binary, and yet it shows up right next to where all the RR Lyrae are, and that's obviously a problem. You probably can see a bunch of these colors up here where there's obviously some misclassifications. By the way, these classifications came from a different group who use their own machine learning classifier. It's very natural for everyone to get label problems at the order of a few percent, but this is at least surfacing a bunch of the objects that may be more likely to be mislabeled.
And then there's other types of objects, which we call in-class outliers, where the classification is correct. This is indeed an eclipsing binary, but the spectra doesn't look at all like eclipsing binary stars usually do. In fact, this one has an emission line; I think it also has X-rays associated with it. It's some sort of interacting system where there's actually an accretion disc. And so this just pops straight out of looking at this embedding space and doing something like a cosine distance of an object relative to all the other objects in the space. And we have other examples of these as well, where the classification is correct — it's an eclipsing binary system, but you see weird sorts of bumps and wiggles when it's out of eclipse, maybe due to some sort of rotation or some other type of effect which isn't usually seen.
Something that I'm excited about, when we start thinking about allowing the human to go through their own journey of exploration and asking questions of large data sets, is now being able to take a query spectrum and say: I want an object that looks like this weirdo. It's an eclipsing binary system, but I want to find other systems that look like it in spectral space. And then you can find the nearest spectra in this very large corpus and ask what are their light curves. These are different types of variable stars, but you can see just visually that these are extremely similar to each other, both in morphology, fluxes, etc. But their light curves are actually quite different. And then you could imagine cross-modality searches, where we want to find things that are the farthest-away spectrum from this object, where they have photometry that's similar to each other but the spectrum is very different from each other. And you get to play all these games essentially for free once you've built these embedding spaces. So this is an enablement, I think, of a broader set of questions that we as astronomers can start asking of this data.
I want to take a different track here. But before I do, were there any questions about the first part of this talk? Yes.
I was just curious, given that you're combining all of this different survey data, if the discoveries that you've made so far are biased towards any particular survey, or do you find that by combining them you're mitigating the bias that surveys tend to introduce into these kinds of systems?
Yeah, that's a great question. I think we're heavily biased towards the data that we actually use in this case. The goal, I believe, of foundation models broadly in astronomy now is to be able to use lots and lots of data sets — spectra from multiple different instruments, light curves from multiple different instruments with different bandpasses, different detection capabilities — so that you can become more and more immune to that. I heard today about work that people are doing here where they're trying to separate out what is the underlying physical representation going on and what is the representation of the instruments themselves, so that when you apply this to a different data set, you can have some assurances that you're going to wind up getting good answers.
The thing I didn't put up here, because I was a little uncomfortable, is that we found a really, really weird spectrum, and I got super excited about it. It essentially has very broad calcium lines, 25,000 km per second, in an eclipsing binary system. And then we queried and asked, are there other ones like that? And a whole bunch showed up — some are eclipsing systems, some are not. And what I started believing, because we're just on the cusp now of getting follow-up spectra to confirm these things, is that either we've hit a very specific moment in the phase of one of these events, or, probably more likely, we've uncovered a bad reduction pipeline in the spectra. It wasn't our reduction pipeline — somebody else's — but they were very nice to put the data out for us. So it's an interesting generic problem that we're going to run into as we use these larger models, especially on new data that we haven't seen yet: when we find anomalies, are we essentially just finding interesting clusters of bad data? And unfortunately, I think the answer is going to have to be that we're going to have to start getting follow-up data to confirm whether these are real or not. Yeah.
My question actually relates to that a little bit. Can you say a little bit about how you're quantifying uncertainty in the predictions?
Yeah, so we're not — but it is an interesting question, where for a given object, given that you can see that there's noise in the data, where should it show up in this space? Is the error bar this big, or is it actually really honed in? In this case, what we have tried is the example where you bootstrap resample the light curve and move the metadata slightly around where it is — and you can do the same thing with spectra as well, because we have the noise properties — and it doesn't move that much. Most of the noise, I think, is just because we are not training on an infinitely large corpus. But there is that difference between the model noise and the data noise. But it would be an interesting question of how little data do you need to get a good enough localization in this classification space. That might get into interesting questions as you're planning surveys: for these types of objects, I need this amount of data to be this level of confident of where it shows up. Yeah, good question.
So if you use UMAP, which is typically a tool for visualization and not outlier detection — every time you run it, it's sensitive to hyperparameters and it can create a different representation. So if you use UMAP to claim a—
Yeah, we're not using UMAP at all. We just use it so we don't have to all think in 512-dimensional space. We're doing all of that work — all the math is happening in the cosine distance space.
Okay. Right. And in the full embedding space.
Yes. Yeah. I didn't show it, but you can look at in-class distances, and they follow a distribution in the cosine of their vector angles between them. And then what we're doing is essentially fitting a Rice distribution to that and then looking at the top 10% or 5% that are just on the edge of that distribution. And so the ones that are farthest away — that's for looking for in-class weirdos, but it's a similar type of idea for looking for out-of-class objects.
This is a question more on the contrastive learning thing. I assume that you have pretty good cadence in light curves and probably spectra. Do we need to assume that these two modalities have to be equally informative to the thing, so that you can actually tell they are the same thing? Rather than, for instance — I can imagine you have very low cadence photometry, or it's very early phase, so that you don't have a lot of measurements — will contrastive learning actually miss something?
That's a great question. I don't have a great answer to that, other than saying that there is this implicit notion that the data all should be pushed into the same embedding space, and we know, in particular in variable source space, that that's not true. I guess you'd call it modal degeneracy, where you could also have the case, slightly different from what you're proposing, where you have a spectrum that is from two very different types of variable stars and they look almost like identical spectra, or you have one type of variable star and it produces a bunch of different types of spectra — and it could also be as a function of phase as well. And so the idea that they should be embedded in the same space is actually wrong. But the fact that this seems to work, even though we know this happens sometimes, is some indication that the way in which these models are being trained is somewhat immune to some of those issues. My hunch is that all you're adding by having less data is more noise in the learning process. But you're absolutely right. There is a mode — like if I took an image, for instance; let's take the image mode of these objects: they're all just going to look like a star. It's a point source. So we know that would be an example where that mode should have a very hard time getting embedded into, and sharing, the same space as the others. But it would be an interesting, I think, follow-up research problem to figure out how you could potentially learn what the relative weightings are when you actually start to figure out where a new object lives. Yeah, good. Thank you. I want to turn my attention to another type of time domain event, which — those that have worked in time domain before know — is somewhere in between an explosive event and something that's happening in our galaxy around stars, and that's microlensing. You get a typical microlensing curve, which is a nice symmetric magnification due just to the bending of light around a foreground star as this background star is passing by. And these blue images are what you would get if you were able to resolve this scene on the sky as a function of time, put in these arbitrary Einstein unit times, which is related to the mass and the distance of the object and the background star. But if you have a planet that this foreground star is hosting, then you get perturbations on those nice little, otherwise smooth curves. And so the challenge here is to be able to measure the masses of these planets and the separations from their host star by looking at a whole family of these types of events.
The problem is, doing this on an individual event in the past has been very labor-intensive and very compute-intensive. It's a very large and pernicious posterior space, and even just deciding where to hone your MCMC so you don't go off the rails is actually a bit of an art and generally requires experts in the loop. And that's fine. But as we start thinking about what we could do if we could push all the way down, as some of these space-based microlensing systems are going to get to in the future — with Roman, for instance, we expect thousands of these planetary microlensing events — we just don't have enough compute and enough experts to be able to run these inference systems on every single object that we detect. So this is a place where we have these bottlenecks in people and computation, where it calls for an automated and more efficient approach.
And this is where the simulation-based inference comes in, using a so-called neural density estimator. Again, we have our light curves. We do some sort of encoding here, and we use a bottleneck around that encoding to train, in our case, something called a masked autoregressive density estimator that takes a Gaussian in the number of dimensions that we're interested in measuring, parameter-wise, and winds up producing a posterior output. And this goes back the other direction. And so with all of these neural density estimators, the goal is to be able to train a model that can, given data, go directly to parameters. So rather than try to make fast the computation, which is a surrogate model for the forward model, we're just trying to go directly to the posterior space. This field's evolving fairly rapidly; there's lots of different approaches to it. But essentially, we're able to train over a large class of these planetary microlensing systems and get, at inference time, something like 10 to the five times faster than you get for just producing the posterior space — in part because, even though it's just GR plus whatever stars look like and following light paths, the computation for just a single set of parameters can be several seconds on a reasonable machine, and if you have to do this millions of times, that starts to add up quite a lot. One of the things we wanted to make sure of when we were producing these posteriors was that we could reproduce a well-known degeneracy. There's actually two different types of degeneracies in the system that just come directly from the gravitational lens equation. You have the so-called inner-outer degeneracy or the close-wide degeneracy, which is that you can't tell from the light curve mathematically whether the planet is here or here, or whether the planet is here or here, relative to its parent star. And so one of the things we were really excited to see when we started building up these posterior systems with these neural density estimators is that indeed, for these two different types of configurations — this is what the lensing looks like in the lensing plane; this is what it looks like as a light curve — you can see by eye there's basically no difference at all between these two very different configurations of whether the planet is close to its parent star or far from its parent star. And in fact, you might be able to see in some of these places — in this case, this is the distance of the planet to its parent star — there's actually two islands of degeneracy. There's two islands of power, sorry, and this is a degeneracy. And these inner-outer and close-wide degeneracies have been studied for many years. What's known is that you can approximately, if you know the location of one of these peaks, predict the location of the other peak just mathematically. So we're very excited to see, on simulated data from Roman, we were able to do this kind of inference very quickly and get back reasonable posteriors.
And then something strange happened. My graduate student Keming Zhang started just pulling events from the prior space, just to see what those posterior spaces would look like. And it turns out that there were some weird degeneracies which weren't supposed to happen given the set of priors. And after thinking about this for a while, he realized that it looked like there was a continuous set of degeneracies that went all the way from these inner-outer to close-wide, and there was this large gap that hadn't really been studied much theoretically. And what's also really interesting is he went back to the known 23 or so systems where you have a measurement of the location of one place of the posterior and you want to predict the location of the other one. Many of these other papers would find that they were off by a few percent and would chalk up that difference to systematics which weren't well understood in the data. But instead, what we found is that we can exactly predict the location of one posterior peak to the other one. And he came up with this ad hoc equation that allowed us to do that prediction from one to the other. And so we suggested in this paper that maybe there was this deeper symmetry in these degeneracies, and published that. And then he and our colleague Scott Gaudi went off and found that indeed it exists in the gravitational lens equation and — while people had been talking about potential unification before — hadn't really been explored nor found. So this was super exciting to us, and in the News and Views it was heralded by Mróz, who wrote about this, that while this isn't going to replace people, the fact that AI is in this place now, to accelerate our theoretical understanding of the universe, is actually pretty exciting. This came out the same week — in fact, I think the same day — that the DeepMind math-proving paper came out. So it was a big moment for us, and I think a big moment also in astronomy, where we're not just looking at simulated data to find simulated results. We're actually stumbling upon something that's actually fairly deep in nature.
So this is very exciting to us, and we've been trying since to build a capability for time domain astronomers to very quickly use nbi in their own work. And so for those that are interested, I encourage you to take a look at this repo. We were trying to — I think in industry they used to say eat your own dog food, but now you say drink your own champagne — so we were trying to use this ourselves, and I had a postdoc that was working with me in my group who said that he was working on doing a fitting of APOGEE spectra with a neural model, and we said we'll just try to use our package. My student Keming said, well, the package is so good, it's just going to work out of the box. They said it's not — and it did. And they wrote this paper in a day, because they were able to get really, really good answers, where they basically had a good simulation of real spectra, doing this advertised neural posterior estimation using the codebase. So I think we're on to something, and I think for sequential data that you have, it's probably worth trying out to see if this can also work for you.
I want to just spend the last time that I have here talking a little bit about some of these other parts beyond the bread and butter, where I think a lot of people are working, in that midstream. I'll first go downstream and then I'll go upstream, and then I'll end. And this is that recognition — I'm sure all of you have started to realize — that the best chess player is not you anymore as a human, nor is it a computer; it's you plus a computer. And I think that's where we're all hoping to go with this AI-assisted science.
I run a very large software application that allows people to upload transients and visualize them and do follow-up with telescopes, etc. But there's a lot happening even before LSST Rubin gets going — tens of interesting events per night, and oftentimes it can be hard to sift through all of that. And so what we wanted to be able to do was reduce the cognitive load on astronomers as they go through this data, which many people do. We have about 100 active daily users looking at this data and trying to decide what to do next with it. So a day or two after OpenAI came out with their APIs, I wrote into this system the ability, using this prompt, to take all of the data we have about an object and produce just a human-readable summary of that. We ship off all of the data, package it, and just shove that into a vector database. So we get LLM summarizations, which is helpful to run through for a summary of a night. But we also get distances between these objects in this abstract summary space. So now we can say, if you like this object, you may like these objects. We're creating recommendation engines to allow astronomers to click on this without reference explicitly to their classification — just looking at the data and what people are saying about it. So I'm excited about that. And then there's a lot more to say obviously about all the downstream effects, but I wanted to focus a bit on the role of AI before we even start taking science-grade data. And one of the places I'm very excited about is in optimizing, again, our precious resources, both for follow-up and even survey design. Many of you are aware, of course, of that neutron star-neutron star merger that led to the discovery of the first bona fide kilonova following that event, observed in multiple different bandpasses. That itself was a needle-in-a-haystack problem: a very large swath of the sky for where the gamma-ray burst was, and then the LIGO-Virgo localization was even better. And then a number of groups semi-simultaneously wound up finding this new event. And that was a huge triumph for the gravitational wave community, for the high energy community, and for the ground-based community as well.
This is a very hard problem. We have large-format telescopes that can look at large swaths of the sky. And the goal when a new event goes off is to very rapidly find a new object like this one somewhere over a very, very large swath of the sky, covering these large bananas that we wind up getting out of our localization error boxes from LIGO and Virgo. So we've been working on a problem where we're trying to optimize the discovery of kilonovae using the Large Synoptic Survey Telescope, so that we can collect rewards for this discovery by learning a policy that, over different filter choices — and given the fact that the sky is moving and maybe our object is setting really quickly — we can catch this event by observing it in one of the possible places in its localization error box. And what I'm excited about here — and this is a paper that's still in prep — is that we're placing the reward functions not in heuristics. It's not like Atari games, whether you get the highest score or not. It's whether we can knock out the largest part of parameter space of the physics of the objects that we care about. So you can now define a metric in this, saying maybe the largest amount of volume that we're going to wind up being able to say we didn't see an object in that volume. And so that's our reward function. So placing RL in the context of the science you want to do directly, I think, is a broadly applicable way to start thinking about our use of RL. And this is for a specific type of object, for a specific type of follow-up. But now you can imagine articulating a reward function over many different science objectives.
So LSST Rubin is going to be very helpful, as hopefully we get more gravitational wave localizations from LIGO-Virgo. But it also has to work — and it is taking first light. But one of the things you should know about LSST is that it has a lot of moving parts, and because of gravity, because of temperature fluctuations, etc., we don't always know exactly the right configuration of where everything should be, to the sub-millimeter level. There's of order 100 degrees of freedom of all the different things that can be actuated in the telescope. And what they want to do is take an image of the sky, use these so-called wavefront sensors — which look like essentially images of the primary, which we call donuts — and infer what the wavefront errors are from the last image, so that when we observe that same part of the sky, we'll get a better, higher-quality image the next time. The state-of-the-art, which is now running on the telescope, is to find the donuts, represent the wavefront as a linear combination of Zernike polynomials, and then solve the so-called transport of intensity equation that allows us to figure out then what the actual actuator changes are. The problem is that at current state — as in the last week or two — it's running too slow, and the two algorithms that are running don't meet the spec.
And so we were asked a couple of months ago if we may be able to use a neural approach to this. And this is a challenging problem: we've got overlapping donuts, and some of the donuts are cut off because of vignetting. And we were able to, on simulated data at least, through basically three different networks that we're able to pre-train with data from the pre-training data set, create something that finds the donuts and figures out how we can center on those; then does inference on those donuts, for each of them, to produce a Zernike polynomial; then, with some augmented data from the locations of where those donuts are and the bandpass we're observing, uses another aggregator to wind up giving us back the results — all of this being differentiable, so that when we get new data from the telescope itself, we'll be able to learn from that.
And to cut to the chase: the two systems that have been tested and are running now — this is the spec, with this vertical line here, below about 0.1 arcseconds — you can see that something like only 2 to 5% of the data from the current system is running within spec, let alone it's running too slow. And at least now we're getting about 50% of our predictions within spec. So we're hoping to deploy this onto the telescope in the next week or so and actually get some on-sky engineering time to make sure this works. But one of the things that we think is really critical, obviously, is that it has to be in production for it to be, I'd say, considered a success — because it works well on paper, but until we actually improve how LSST actually functions, I won't consider it one of those successes.
I'll skip this here, but you can imagine extending this to active optics, where now you're getting predictions about actuation not every 30 seconds or so, or every 10 seconds, but at the kilohertz level, where you have to now move deformable mirrors potentially with 10,000 degrees of freedom — so an action space that is much larger than the types of RL places where other groups are working. So I'll end, so that we have a couple more minutes for questions, by just saying there's a lot going on both here and elsewhere in that midstream space, where we have data and we want to do inference on that. We want to do classification. We want to decide how we're going to do follow-up. And doing inference at scale absolutely demands ML — I don't think anyone would wind up arguing with that. And here the semi-supervised and self-supervised approaches and the foundation models are helping us work with this small-label problem, and showing a lot and bearing a lot of fruit. I'm excited about the HCI, AI-guided part of that, and I hope people here can continue to push on that. But it's very clear also that there's a huge white space in the upstream, before data is even taken. Can we do better? And then that means we can't just write papers about how we could do better — we actually have to put it into production. So putting AI into production is one of those big themes: we shouldn't be doing AI unless we're actually enabling great science downstream from that. So with that, I'll say thank you, and happy to take some more questions.
Other questions for Josh?
Thanks so much. This is really great — wonderful to think about the whole life cycle of observation and making sure that you do AI in the right way across that whole spectrum. Can you talk a little bit more about putting things into production, and maybe you want to use Rubin as the example: how do we have to think about designing experiments or designing collaborations or designing those things to make that happen? So for example, you were mentioning that you wanted to have some amount of time for engineering runs. How difficult is it to advocate for that time, and then what's the return on investment from having done that? It sounds like this stuff has to happen very, very quickly, otherwise it can't actually get incorporated.
Yeah, I think you're asking that question in a way that could be answered also by industry folks, in the following sense: the way to do that is with people, and people working together within organizations who have — at the smallest level, they have their own optimization functions, which are not aligned — but then getting alignment from an organization, like the leadership of LSST, saying this is important to do. And so we set off to do this because the former director of LSST came in to our office and said, we have a problem, can you help us? But there are a lot of people now working in the engineering on LSST trying to get all the other pieces together; the last thing they want to do is change some subsystem which isn't great but kind of works and will be okay for a while. So we spent a lot of time in meetings, and partly the work that we're doing in the offline mode with simulated data is to just be able to get into a conversation saying things are looking good. I think everyone in the end should be aligned on this. It means the people on the ground in Chile are going to have to do more work, because they're going to have to do this extra thing which wasn't in the original plan. But there has to be that organizational alignment about it. So one answer to your question is organizational alignment, lots of talking, and a lot of convincing — and that happens in industry all the time.
Maybe what you're also hinting at or asking about is how do we do this from a technical perspective, and I don't think most of us in astronomy or physics, or maybe in physical science, have a lot of muscle memory around the right tooling and the right conversations we have to have around that tooling, to be able to convince all the stakeholders that this is possible and it's not going to break something, and you're not going to run a telescope into the ground or do something horrible. So how do we do model management? We're going to train a model, it's going to work, and then 10 days later we'll get slightly more data. We'll have another model — we've got to version that, and that's got to propagate all the way through to the FITS header that you see on your desk two years later. And then how do you trace all that back? So creating observability broadly around how well it's doing is one answer. And then we also have a challenging problem where some of these models need to be run on GPUs, or maybe one day, in the LHC context, on specialized hardware like ASICs and things. We've got to make sure that that's in the cards, that we even have that available from a runtime perspective to be able to do this kind of inferencing.
But it also gets into a really interesting question of: if I have model A and model B, and model A is 10% better than model B in all metrics, but model B is 1/100th the size and can run a thousand times faster, people are going to put model B in production. And so we also have to have honest conversations about the non-ROC-curves and false positives, false negatives; we have to start having conversations about cost, model size, cost to run, and all the warts of what it means to do something in a real environment. So I don't know if that answers all your questions.
That's great.
Thanks for a great talk. So maybe one high-level reiteration of some of the things you're saying is that there's kind of a difference there. There's tools that help us just ask the questions that we wanted to ask in the first place — asking that with less friction, being able to access things or see similar things — and then there's tools that help us ask questions that maybe we didn't even know we could ask. Would you say — I guess maybe there's other types of tools or maybe other types of categories — how much of the benefit in AI and science do you feel is going to come, where we're at currently, from the first types of tools versus the second types of tools? And also, how generalizable is that second type of tool? Because it seems like that's really hard, to actually make a good tool that helps you ask a better question.
Yeah, there's lots of ways to answer that — that's probably another colloquium. But I'll say the following: the first type is easier and more tractable and more palatable to do, because we have optimization metrics and we can see in principle how well we're doing. We can hold out data and we see, okay, on test data we do great. And so I think it's natural that we all went to that place — that's the soccer ball that all of us five-year-olds went to — because we know if we're doing well, and we know we're beating other approaches in time or in accuracy or something like that. And once you start getting into that nebulous, is my model better than yours at surfacing more anomalies that lead to more Nature papers — or not Nature papers — then the optimization metric gets a little more fuzzy, and then the knowability around the quality of what you've done, let alone the choices you've made architecturally to do that. Given what we've seen from the extremely large companies, you tend to need a lot of data. This is not something where you're going to build a little tiny model on the weekend, vibe coding with Cursor, and you get an amazing paper that leads to a new result. So we are going to have to put a tremendous amount of computational time and people time into this, and without a guaranteed result, it's dangerous. And so I think a lot of us are trying to dip our toes into that, and maybe we think the soccer ball will be over there in five minutes from now, but it's not clear at all what we do once we get there. I was just trying to show some examples of places where we can hope that astronomers would be able to navigate through this web portal that I was showing you a little bit faster, with a little bit more magic thrown in, so that they're not having to over-index on their own experiences. I think what we need as a community is some rigor around the observability of this. There's an entire world of human-computer interfaces, and there is rigor around how you study how people use computers and how they make good use of that. So I think there is a world where we can start, maybe at the human level, asking: can we get some rigor around the work that we do? But that's really hard to feed back into the actual models themselves. And so that's why I wanted to spend a little time showing you the visualization of this embedding space, because it's starting to hint that with the right tools and with the right types of questions, and people trained up to use these, we may be able to start getting at some novel science.
Other questions? Yeah — I don't know why this is blinking on and off for those in the room, but it's keeping us awake.
Great talk. I'm someone that works in ML for math a bit, so I wanted to come back to your symmetry result. And part of my problem is I don't know too much about lensing. What was it that humans missed in the equations? And could you maybe comment a little again just on how ML helped you figure out that the symmetries were there in the first place, and how that might generalize to other problems?
Yeah. So, first of all, as we all know, it's easier to solve a problem once you know that it has a solution. And our first paper, where we hinted that there was this new unifying degeneracy — which, again, we came up with an ad hoc equation that described the bridging of those two — that seemed like that worked really well. So that gave the impetus for my student and our collaborator to go in deep, and it's deeply embedded in the quintic equation. And this is the kind of thing where, unless you knew to look for it and you were trying to solve something, why would you even go to look? Because those two degeneracies have served the community really well when our data is not so good — it's sparse, and so it pretty much has always fit. Nature doesn't seem to want to pick out the actual configurations of these events that are separated from these two well-known degeneracy spaces. So I think part of the answer is: we definitely stumbled upon this; we were not looking for that. So part of the answer is: we got to know that there was something there, and so that's why it was worth spending the six months they did on the math to actually find that it actually was there. And maybe they would have come up with something and found a different equation, but this was the equation that actually turned out to be right. So that's one answer. The other answer maybe is more organizational: there's not a lot of theorists in astronomy, and then there's not a lot of theorists thinking about time domain, and there's not a lot of theorists thinking about microlensing, and not a lot of those theorists think about planetary microlensing. So there's of order five people that have worked in this field actively over the last 50 years. And so part of it also is there's so much to do and so much to write in that theoretical space — this just wasn't a priority. I think had it been that people realized there was a big problem 20 years ago or 30 years ago, it probably would have been uncovered early. But there are people like Jennifer Yee, who's at CfA, who — she and Andy Gould were working and starting to figure out that maybe there was something there with these degeneracies that could potentially cross over. Some of the math wasn't exactly what we wound up finding, but it was somewhat in the water. So maybe they would have found it six months later, or maybe not.
But I think, again — let's summarize the answer — it was only possible to do it because we found out it was there. So: a data-driven approach, and then you said, aha, that must be something, and then some hard work to show that's really so. That's the HCI answer as well. We just had this massive accelerant of compute. We're able to get posteriors 10 to the five times faster than with the normal way of doing it. So why not just try it and just look at them? And then my student had enough domain knowledge about this space to go, that's weird. And I remember, this was during COVID — he was stuck in China — I was like, this can't be right, you have to have made a mistake in your posterior estimator; how can you find degeneracies that no one has seen before? And so there was a lot of pounding at this — are we even on to something? But it takes the sniffing out of, maybe there is something there, we've got to keep going, that allowed us to keep going.
Thanks a lot.
Final question. If there isn't, I'll take the final question. I wanted to ask — this is maybe a more targeted form of Tessa's earlier question with respect to time domain foundation models. Obviously there's a lot of hype right now surrounding foundation models. If you had to forecast where you thought the most promising trajectory would be for longer-term use of foundation models, do you think it's in going to four modalities, five, six, seven, and expanding — variable stars, including supernovae, and throwing in AGN — and building a massive framework that is as generalizable as possible, or some happy medium in specialization for specific downstream tasks?
Well, I don't have a crystal ball, obviously. But my own hope for the community would be that we don't do more than we have to for the downstream tasks of getting better science out. And so if you just take LSST Rubin data streams and ask, what do we need to do to do great with that? My hunch is the community is already really close to being able to do great stuff with that in novel ways, just because there's going to be so much more data, and when you're sampling a large distribution, 10-sigma events happen, and that's great. So I think we're already kind of there. But as we realize, oh, we're missing a whole class of events that we should be seeing, then we have to start building maybe more modalities into the system. Again, it comes down to the implementability of this and the productionization of it. My hunch is that really, really big models are not going to serve up as much incremental improvement in the downstream science, given the cost to build those models. It doesn't mean that people shouldn't try and go down some research spikes, but ultimately I think we need to stop building foundation models for foundation model building's sake and start putting them into the hands of people that don't build foundation models. One of the great triumphs in the industry world here is that they were able to build foundation models that everyone can use, either with a front end or an API call. So if we get to the point where we have foundation models that anyone can just plug into their Jupyter notebook as a plug-in and all of a sudden things get better, I would call that the success that we're looking for.
Great. Let's thank Josh again.