Astronomy has embraced AI for data analysis, but the real revolution will integrate AI across the full scientific workflow — telescope design and real-time control upstream, hypothesis generation downstream — via simulation-based inference, self-supervised learning, foundation models, neural compression, and RL.
Given as both the A3D3 seminar and UW Physics colloquium the same day.
“We can think of astronomy as this sandbox where computer scientists and astronomers can work together to build new algorithms and to learn new things.” – Joshua Bloom
“If we think about the north star of why we're doing AI in astronomy, it's not because AI is fun or easy… it's because we're trying to do novel science. And when we stop at things like making catalogs and we don't actually get to new insights about the universe, that's where I think we fall short.” – Joshua Bloom
“I can't say that I know how to reproduce that workflow that led us to this insight, but I think it is one of the moments in astronomy meets AI where AI has really taught us something fundamentally new, or at least helped us learn something fundamentally new about how the universe works.” – Joshua Bloom
“It's not going to be replacing people with AI, it's going to be augmenting them to do their very best.” – Joshua Bloom
And he's going to tell us about the real AI revolution in astronomy that has not happened yet. So take it away, Josh.
Thanks. Hi everyone. Well, thanks very much for the introduction and thanks for the invitation to be here today. I wanted to just show a couple of faces and names of some of the early career people that have been working with me on some of the topics that you'll be seeing here today. I wanted to present this with somewhat provocative title, not to say that machine learning isn't already a part of a lot of what we do in astronomy, but there are I think fairly large white spaces that I'll try to cover in parts of this talk.
And I hope to leave you with this sense that while we have been doing a lot and a lot of progress has been made in service of science using AI, there's just a lot more to do and a lot more that we're all very excited to be doing. So thanks again for having me here. I know that there's a wide variety of interest and backgrounds on this call, so I wanted to maybe start off with a motivating quote, if you could call it that, from a I guess prototypical data scientist. Jim Gray used to work at Microsoft, formerly at Berkeley for his PhD, and he has this quote. He says I love working with astronomers since their data is worthless.
And many of us in astronomy take this in a very positive way, recognizing the idea that the data that we have, while not backing a trillion dollar industry like e-commerce or medicine, still is of great interest for us to understand. We're asking some of the most important fundamental questions about how the universe works, its origins and its fate and all the constituents in it. And we get to try that out and try to explore the universe using whatever tools we have available to us, not just from a hardware perspective but from a software and algorithms perspective. And Jim realized the importance of this in part because our data doesn't have this intrinsic value in a dollar sense. And if we make a mistake with it, we don't leak PII, we don't start wars, there aren't crashes of self-driving cars. So that's the way perhaps that computer scientists look at us and our data. But we also look back in the other direction with an understanding that we have this great ability to bring in new algorithms and new approaches that are being developed elsewhere to improve our own science. And we can think of astronomy as this sandbox where computer scientists and astronomers can work together to build new algorithms and to learn new things.
My own science is in the time domain fundamentally, and we look at these sorts of plots of time on the x-axis and brightness over a very large dynamic range on the y axis. And some of these so-called light curves represent the bread and butter of time domain astronomy. For instance, if you're interested in studying the origin and evolution of our universe, you might study type 1a supernovae, and you see their very specific light curves as they evolve over time over the course of many months. The most common explosions in the universe are type 2P supernovae, so if you want to know about the death of ordinary stars, or ones a little bit more massive than our own, you might be looking at things like type 2B supernovae and studying those en masse. But then there are other types of objects in the universe that have either been theorized or maybe only been observed once or twice, the so-called pair production supernovae. There's some indications we've seen those before but not for sure, and so we live in sort of this theoretical world where these events should happen. We don't know exactly how often they should happen, and we have some ideas about what they might look like.
Then there's these very faint, just barely visible events that were theorized when this plot was made, the result of a merger of two compact objects called neutron stars. And when that happens the very faint blip lasts for just a couple of days, and if we could see that coincident with a gravitational wave event, that would be a fantastic discovery. So there are these known knowns, there's these known unknowns, and then there's all the things that by definition we can't put on this plot, the unknown unknowns. So we have this Grand Challenge in time domain astronomy where we're trying to not just find these things that we know about and don't yet know about, but we're also trying to optimize our followup of these with other telescopes. We might find them on one telescope but then want to observe them on another telescope with different capabilities, perhaps at higher signal to noise or with different instruments. But these are scarce resources that we look at. And if we had this magic wand and could own all of the telescopes and all the instruments simultaneously, we might have a hope of writing down some optimization metric that we could solve and then decide how optimally to observe the sky. But unfortunately we don't control all the telescopes and all the instruments, and oftentimes we're competing with each other to look at the same objects or to look at different objects, and so we can't write this down. And oftentimes we're just trying to make our most informed choices we possibly can of how we can maximize our science.
And a lot of what I just said has been going on for many years if not decades, but this whole urgency in being able to optimize, to discover, and then make decisions about what we observe in the sky is becoming really more poignant as we get closer to the turn on of the Large Synoptic Survey Telescope on the Rubin Observatory. And you see some of the numbers here. We're expecting this massive torrent of not just images of the sky but these movies of the sky where we're going to be finding tens of thousands if not hundreds of thousands of events per day. And trying to decide how to follow those up in an optimal way is the sort of Grand Challenge. So we're going from hundreds of potential events of interest a night to thousands or tens of thousands, and the community is doing everything it can to try to get ready for that.
So in this interesting space where we're getting lots of data and we need to make sense of it, I break down some of the tasks that we have into questions around discovery. So for instance discovery in images, we're trying to find a new thing. You've probably seen this before in some of the previous talks. What you have in the center frame there is an old image of the sky taken by making many images in one place and then adding those together or meeting them, and then you have a new image on the left hand side that was just taken last night, and you want to find the new object that's embedded in this galaxy. Well, the state-of-the-art is to just subtract those two images, and if you're lucky you wind up finding the supernova that's sitting in the outskirts of that galaxy. That's what you see on the right hand panel. But you can also probably notice that there are some other little white dots there. These are spurious, you could call them, detections due to the imperfect nature of the subtraction of these images. So finding just new things in the sky, discovering them, is this a real or a bogus object, is actually a non-trivial exercise, an important gateway into us being able to do science.
Once we found something, we get these ratty light curves like the one that you see here, taken over days or weeks or even years, and we have to ask ourselves this question, what is this thing that's just been found or was found recently? And given its metadata that we have about how its light is changing and maybe where it is on the sky, is it worth following up, is it worth spending our precious resources to look at it with other telescopes? And then I think the last major task in the context of time domain, although this is shared across lots of different parts of astronomy, is doing model-based inference. So we have a theoretical model that we can potentially put into either a large supercomputer, or maybe we can write down the effects of a set of hyperparameters that describe this model, and we wind up getting out potential observations. So oftentimes we go the other way, where we have new observations and what we want to do is infer what are the parameters that describe this model. So what you see here are some example light curves from supernovae, and these are theoretical light curves from large scale supercomputer simulations where we put in not a large number of parameters, something like the mass of the pre-explosion object, the energy, etc., maybe a couple of other parameters, and with some random seed and throwing it through all the physics that we know, we wind up getting out these families of light curves. We want to be able to do the inverse problem of take data and go backwards and get the values of those parameters.
So this is what I call the Midstream. But one of the things that's really interesting, and when I say Midstream the way to think about it is after data has been taken it's in the can, we want to do something with it, we want to do inference, we want to understand what telescopes we might trigger autonomously or semi-autonomously to do the observations. But there's this whole sort of Downstream thing of the data's now been taken and now we want to involve the human in the real-time loop, or in the part of the loop that we're potentially best at, of coming up with hypotheses and testing those hypotheses. And I'll talk more towards the end of the talk about what I think are the opportunities in this Downstream. But as you might guess, Upstream from the Midstream is this idea that before data is even taken there are opportunities for machine learning and advanced techniques with computation and data to be able to do better, to take data in a more intelligent way, to optimize the kinds of things that we're interested in. And I'll also talk about some of those opportunities in the Upstream towards the end of the talk.
Focusing on the Midstream for now, I mentioned this idea of trying to do discovery on lots of images. This became an imperative about 15 years ago as these large scale survey operations were coming online, and Matthew Graham was heavily involved for instance in this. From the onset we wanted to be able to find these needles in a haystack, and what you see are some examples at the bottom of real detections. And you see things that if you just were counting five sigma detections above some noise floor, those are actually bogus detections. And this is sort of, you can see that the real ones already sort of look similar but the bogus ones all kind of look different from each other. We wanted to create a real-time framework that would allow us to find and discover these new objects in the sky as quickly as possible without humans in the loop, because humans were becoming the bottleneck. We literally couldn't find enough graduate students and undergrads to look at data as it was coming off of telescopes circa 2008, 2009. So we built this machine learning system that we put into production called real bogus, and that was at the time about a thousand to one needle in a haystack problem. So for every thousand bogus subtractions or artifacts there was one real one. The subtractions have gotten better and the techniques have gotten better so it's now down to maybe one in 100 or maybe one in a few hundred, but it is still something that is absolutely essential that we have.
And one of the things that we're very proud of is all the discoveries that happen across lots of different subdomains in astronomy. The one that I'm most excited about and still most proud of is that we were able to create a ranking for astronomers of the most likely new objects, crossmatching that with nearby galaxies. Our colleague Peter Nugent and company were able to identify this new very young supernova that occurred 11 hours after explosion in our first image in 2011, and that turned out to be the nearest type 1a supernova in over three decades. Now what's interesting about this is that while amateurs would have wound up discovering it a few days later with their small scopes, and actually people could see it with binoculars when it reached its peak brightness, this object would have been discovered. But what's important is that we were able, because machine learning was in production and looking at real data in real time, we were able to get on this source with our precious resources, triggering Hubble Space Telescope or the Chandra X-ray Observatory. And it's because we were able to get there so quick that we're able to do novel science. I won't go into the details of what you see on this plot here, but all the regions in color were able to be ruled out, in some of which because we were able to get on this source just within a few hours after explosion. And that allowed us to rule out a whole bunch of possible ideas of the progenitors of these type 1a supernovae. It didn't come as any surprise that the only white space that was left over in the bottom right were compact objects. But it was sort of a I think a great vindication of the idea that we're not just doing machine learning for machine learning's sake, and we're not just doing discovery on things that eventually we would wind up discovering, but because we had ML in the real-time loop we were able to do novel science. And that I think that's for many of us
the north star of why we're putting ML into production environments. Now real bogus has become a cottage industry, and our old school computer vision techniques that we were using with handcrafted features have now given way to a whole deep learning set of approaches. This is largely a solved problem, A, and B, this is also I think a reasonable place where we can't have surveys anymore, at least at the optical infrared domain on wide swaths of the sky, where we're not using some types of machine learning. It's just too much of a bottleneck to involve humans in that part of the workflow.
Now once we do discovery, lots of science. I mentioned the type 1a supernova science that we're excited about, but the other types of things you might want to do is understand these periodic variable stars, again sticking with the time domain that's nearest and dearest to my heart. And what you see here is a depiction of 50,000 variable stars projected on the sky and their locations, and one of the light curves of those taken over many years in the top left. And unless you're good at taking Fourier transforms in your head, you probably don't know that this source actually has a period of about a half day. And if you could see the period fold of this light curve and you saw the peak to peak amplitude, you would probably know that this is an RR Lyrae star, which is a very common type of variable star. What we wanted to do is take this database of light curves and ask the question, what are these things? So classify a large data set. But the interesting thing, and this is potentially lost on those that haven't been working deeply in the context of AI and astronomy, is that when we think of astronomy we think of these petabyte scale data sets, that's a large data problem for sure, but we actually don't have a lot of labels of the things that we're interested in. I think astronomy has what I would call a small label problem.
And just to give you some details here, for this one data set there were 26 different classes of sources of variables over just 810 of those sources. So we had some of these variable stars in this data set where only three or four were known to be of this minority subclass, and so trying to build a classification model on top of this data turned out to be incredibly hard, and we had to build some Active Learning techniques for us to involve human experts to quote unquote buy more labels. But we eventually wind up getting into some of the neural techniques. The previous work was using random forest and handcrafted decision rules, but somewhat early on, right as recurrent neural nets were starting to hit other parts of computer science, we brought that into our field. And we're able to use what I'm sure is very familiar to most of you, a somewhat off the shelf auto encoder, although we had to change some of the loss functions and other types of approaches to handle the irregular sampling of the time series, to build these self-supervised networks that would allow us to learn features without the handcrafting of the features themselves. And the simple idea, as I'm sure you see here, is that you take these light curves, you create an encoding network that creates a small compressive bottleneck layer of maybe 64 numbers, and then you create a decoder, and your job is to just reproduce the outcome. Now this is self-supervised in the sense that we were able not to use just the 810 of our labeled sources, but we're able to use all of the 50,000 to learn these features. And then once we had this bottleneck layer we could use traditional classifiers like random forest to be able to do better, and we achieved state-of-the-art on this data set when this paper came out.
Now one of the other things that we started asking questions in the context of this time series data was, instead of just building these generic auto encoders that didn't know anything about astronomy, could we make use of some of the symmetry that we know exists in the data that we're looking at? And the idea of imbuing and imposing physical constraints and symmetries into neural nets in the context of science is itself not a new one. It's been done in computer science, high energy physics, quantum chemistry, and beautiful depictions of these rotation and translation equivariant convolutional neural nets that operate on the intermolecular forces in these different particles. The idea is that when you rotate something, even if you go into a different direction, you should still get the same forces out if you're doing convolutions on these objects. And so nothing should change just because you rotate something around. And so we had the same thing we could exploit in variable stars, and that is the periodic nature of many of the types of objects that we're looking at. And so we did a very simple thing, which is to take a set of different types of neural architectures where, as you look forward, as you look backward towards these sort of deeper layers, oftentimes as you go farther and farther back you end up doing zero padding. We replaced the idea of zero padding with symmetry padding. So as you go farther and farther back for periodic variable stars, eventually you wrap around and you want to get the data that you had in the previous period. And so just by doing that and using the convolutional layers, we created this idea of convolutions in polar coordinates and not in Cartesian coordinates. And the result of that, without going into all of the details, was being able to do convolutions where regardless of the starting point of where we are in time, or I guess more precisely in phase, we essentially ensure that we get the same exact results out of each layer. And so just by the symmetry padding we're able to add this back into a whole bunch of different types of networks and achieve, over a couple of different types of data sets of different sizes with different numbers of classes of objects, state-of-the-art for pretty much all of the approaches. So here we're making use of our knowledge of the data, our knowledge of the physics, exploiting that, and achieving very good results from that.
Now what's kind of interesting, and it's maybe a bit provocative to say, is who cares, right? You have a probabilistic catalog that you have state-of-the-art classification on. What does it matter to astronomers that you now have a big catalog of stars that have some potentially right and potentially incorrect labels? What do you do with a probabilistic classification of variable stars, or more broadly probabilistic classification of astronomical objects? The important thing to recognize is that all of that work that we do in AI is in service of doing novel science. And so because we had these probabilistic catalogs, we were able to ask interesting questions. We had a ranking question where we said give us your most likely of the unlabeled sources, the ones that were going to be of these very rare objects, and we got something like 20 or 25 ideas. And we took spectra of all of these with our precious resources, and we were able to triple the number of very rare types of what are called dipper stars in the galaxy, because we were able to just rank order the classification probabilities. Not all of them were right, but enough that we were able to on these very bright stars add significantly to this subfield of astronomy. We're able to identify highly eccentric detached eclipsing binaries, which allowed us with lots of follow-up observations to put a number of sources on the fundamental plot of mass versus radius, which is something that others had done before, but we were able to do this very quickly and identify the objects and the candidates because of the existence of this catalog. And then a number of other different science results came out of this work. But again, if we think about the north star of why we're doing AI in astronomy, it's not because AI is fun or easy, because it's kind of not either of those, it's because we're trying to do novel science. And when we stop at things like making catalogs and we don't actually get to new insights about the universe, that's where I think we fall short.
So we have lots of data, we have some labels, we're trying to now use all of that data. And you've already had a really nice talk from Kramer who talked about this new project called Polymathic where they're trying to build up these foundation models to take in lots and lots of data, do some fine-tuning for downstream tasks. This is a question that we have and many others have, like that group, in whether taking in lots of data and then fine-tuning it for a specific task are actually going to pan out and do better than if you did direct training for those tasks. We're now getting interested in this idea of kind of a kitchen sink where we're throwing in not just time series data but also source metadata, like locations of where it is in the galaxy or what its nearest neighbor is like, and even comments about the sources that people have made about some of these objects, creating these large multimodal foundation models for us to be able to do these downstream tasks. But it really remains to be seen how well this is going to work for the types of science that we want to potentially enable with it. So this is an open question and I hope you'll be seeing more about this in the coming months and years.
Another thing that I got interested in is in the context of simulation based inference and just doing model-based inference, is could we use ML in a way that would allow us to unlock some of the bottlenecks that we saw emerging. I talked a little bit about the computational bottleneck of being able to model supernovae with supercomputers. That's maybe you've got one object and you really want to know what its parameters are. But we're also entering this world where we're going to be getting so much data on so many different types of events and objects that even if we have not super compute amounts of compute that are needed for us to produce realistic light curves from models, we may have so many of these objects that even if it takes a few minutes to do some sort of model-based inference on one object, if we have tens of thousands or more, it starts to become computationally uncomfortable if not intractable. So I started getting interested in a field that I'll try to motivate some of the science of here very quickly, in exoplanet microlensing. So the basic idea is that we have a light curve on the bottom left hand side, it's varying in time, it's a magnification that happens when a mass moves in front of some background light source. If that mass also has a planet next to it, that is it is a solar system, you can get these sort of very fast blips that you see happening a little bit after time T equals zero, because the mass of the planet itself sort of perturbs the lensing that you wind up getting from general relativity. The light would be bending around this single mass, it's also bending around in complicated ways, a little bit like the bottom of your pool, around these solar systems that are the kinds of things that we're interested in. And so what you'd like to be able to do is take a light curve like the one that you see in the bottom and then infer the properties of the solar system that we're looking at, in particular how far away is the planet from its host star, and what is the mass of that planet. And so these are the kinds of things that we'd like to know. It's a very well-posed problem, but the problem is that it's actually fairly computationally expensive to run through these forward models. And getting the masses from one individual event is often this kind of a challenge where experts have to get involved in doing the computation and deciding what part of the large parameter space these computational models should start in. And so when you're thinking about doing something like Markov chain Monte Carlo to be able to get posteriors, it's fine in the context of having one person working on this one data set, but when we think about having lots and lots of these things it gets really scary.
Now why are we excited about this? It's because there are new facilities which are coming online, not just Rubin LSST but also a new space based facility called Roman, which has as one of its key projects to open up this white space that you see at the bottom right hand corner, where we get to find objects using exoplanet microlensing that are far away from their parent star, so that's farther to the right, and then also lower mass. And you can see some of the planets from our own solar system depicted there, and you can see all the other planets in this somewhat busy plot that have been discovered by other techniques. So this is a really exciting moment when we're about to start getting this onslaught of thousands of potential solar planets, and as we want to characterize them we need better and yet precise techniques. So this calls for automated and more efficient inference approaches. What we did, on simulated Roman light curves like you see on the left hand side, is use an autoregressive flow that allowed us, in something like one or two seconds as we get a new light curve, to produce a realistic and asymptotically correct posterior. And instead of having humans in the loop and deciding what part of parameter space to go after, we were able to show in this paper from 2001 that we're able to get, something like 100,000 times faster than traditional MCMC techniques, posteriors on realistic light curves that were going to be credible. So this unlocks a whole possibility of looking at real data.
And one of the things that we wanted to make sure that our system was able to do was to be able to recover these known degeneracies in the kinds of inferences that we'd want to make. You can see these two degeneracies that have been well studied for the last 40 or 50 years. One's called the inner outer, the other's called the close wide, and that's just a deep mathematical degeneracy that exists in the gravitational lens equation, where you don't know from your light curve whether you have for instance on the location of your planet either inside or outside of the so-called caustic, or whether it's in one of these two different configurations. So these were known degeneracies and we wanted to make sure that we could reproduce them, and indeed we could. These are the posteriors that you see on the left hand side, the corner plots that showed the covariances between parameters, but also as you can probably see if you can see my mouse, these two different islands of high posterior probability that are separated by a known amount that we know a priori from the lens equation. And we're able to show with the same exact light curve that you wind up getting essentially the identical answer whether the planet is near the host star or far away from the host star. So we're pretty excited about this. This means that as these observatories come online and we get more data, we're going to be able to do these very fast inferences.
But then my student Keming Xing started noticing that we were seeing these other degeneracies that were popping up when he was just randomly selecting events from the prior space, and we got curious about it, and it turns out those turned out to be real and we were able to reproduce them, but they hadn't been looked at before because this doesn't often happen in nature in these configurations. And what we wound up realizing, or recognizing, or maybe we just say hypothesizing, was that the degeneracies that we were seeing looked to be both new but also maybe kind of more ubiquitous, that as we moved in one direction or another they started to look like the other two well-known degeneracies. And so we made this claim that there was an undiscovered mathematical degeneracy in the gravitational lens equation that could be described by the simple equation that you have at the bottom here. I won't go into the details of it, but what it allowed us to do is go back and look at the 23 previous exoplanet events that had been looked at before, and in most of those the authors would say well this degeneracy doesn't exactly predict the location of this other degeneracy, but that must just be some unknown systematics in the data that we didn't understand. But when we went back and applied our new equation, which was an ad hoc equation that we wound up surmising from the data, we wind up realizing that indeed this ubiquitous degeneracy could actually very well describe the data that had been seen in the past. So we suggested that there was this previous degeneracy that had been missed by the theory world.
And what's exciting is that my student and collaborator Scott Gaudi at Ohio State went off and then proved that this degeneracy actually existed in the gravitational lensing equation. And so this had been kind of hiding in plain sight and we were able to uncover it, not because we were looking for it but because AI was able to give us this acceleration coupled with domain knowledge to be able to understand something deep about the universe that we didn't understand by other means. So very excited about this. And I can't say that I know how to reproduce that workflow that led us to this insight, but I think it is one of the moments in astronomy meets AI where AI has really taught us something fundamentally new, or at least helped us learn something fundamentally new about how the universe works. We're trying to now generalize the code base that we used for that work to allow others to be able to use off-the-shelf data featurization to be able to do this neural-based inference as a replacement for different types of MCMC activities that people might want to do. So this is an area of active work, and those that are interested are happy, please contact me offline and happy to connect with you. Just as an example that we were able to use that codebase for a completely different technique, a postdoc that was working with our group was interested in being able to do inference on spectra of stars and was able to use the initial versions of the code base that we were building to do inference in a very fast way, and we were able to
show without a lot of extra work we're able to sort of use this code base out of the box.
In the little time that I have left, I wanted to talk not so much about in some sense what the bread and butter is in this Midstream, where you've got lots of data and you want to do interesting science with it, but what comes sort of Downstream from that. And this is motivated in part by this quote from Garry Kasparov on this idea that the best chess player is a good human plus a machine, and that is the combination of those two that can do just incredible things. It's not going to be replacing people with AI, it's going to be augmenting them to do their very best. We've been trying to take little stabs at that using an application that I and Matthew Graham and Michael Coughlin and other collaborators on this grant that you're all part of have been doing, which is called SkyPortal. And this is an interaction platform where people come to talk about objects, to organize their followup of those objects. And we've been working with this application at scale for more than three years, with several hundred users working with hundreds or even thousands of events a night. And we're starting to get a very large corpus of data of the kinds of things that people say about them, the annotations, not just from people but then also bot annotations of lots and lots of data. We've got lots and lots of ML that's been baked into this. But the thing I wanted to talk about was how we're helping reduce the cognitive load on people who are trying to make all of these decisions with all this data, albeit in one place. How do you think about and how do you organize over hundreds or thousands of events a night?
And so there we're starting to explore the idea of taking in all the data that we have on a given object and asking different commodity large language models to be able to give us summaries that can be used and useful for humans as they're sifting through lots of data and as they're deciding for instance what they're going to be observing tonight. Without having to go through all of the data, can you condense it down to the most interesting thing? So asking something like ChatGPT through the API infrastructure, or we're also working with Claude more recently, in one single paragraph give us in a third person a statement about what's interesting about this source. So we're summarizing this for people, which is helpful. And then because we're taking the results of those summaries and embedding them in a large dimensional space with a different embedding model, we can now for a given source search for other sources that are like that source and suggest to people that hey, if you like this source maybe you'll like this one, because it's actually similar in this abstract embedding space way, different than saying it's close to it on the sky or it's got similar characteristics in things that we can measure. Now we can do this in a sort of broader but somewhat more black boxy way, and so that's actually very exciting for us. And I think there's going to be a whole interesting world where people will be able to potentially query large amounts of papers and data and start rapidly iterating through hypothesis generation and testing, even without acquiring new data.
I want to just quickly talk about the upstream and the role of AI before data. One of the things that we did is, before we even look at data, we've got sort of raw data in the can, and one of the challenges that we have is oftentimes the data that we get is corrupted by cosmic rays, so charged particles that are hitting our detectors. Instead of us seeing the beautiful galaxy that we see there, we see all these sort of corrupted pixels. And so we're using autoencoders to find these corrupted pixels and produce masks of these cosmic rays, and then taking that in a separate task and then inpainting to get a sort of prettier picture of what this image would look like if it didn't have these cosmic rays. The idea of finding cosmic rays is part of the data workflow that we have to do to get to the kind of light curves that we were showing you earlier on in the talk. And this is sort of an example of having done well. What you see on the left hand side is a raw Hubble Space Telescope image, on the right hand side is a cleaned up version of that where we found the cosmic rays and then inpainted over. And interestingly and importantly, on real data, compared to the state-of-the-art approaches which use a Laplacian transform to find the sharp edges of these cosmic rays, we're able to achieve state-of-the-art in the discovery of those cosmic rays, but then also were able to with GPUs do this much faster. So as we think about the implementation of data workflows on real data, oftentimes it's not just important to be better, but to be as fast or faster. And so we're hoping that ideas of using these neural techniques for the data workflow itself is going to start taking root. Interestingly, the thing that we were pretty excited about is as we looked at the convolutional layers that were learned in some of our networks, it was able to pick up something that looked a lot like a Laplacian kernel, which was the handcrafted idea that people had a few decades ago for being able to find cosmic rays. But of course all the other kernels were learned from this specific data that gave us better answers.
Another kind of real image that you see on the left hand side is a blurry image of the center of our galaxy that you get because of atmospheric blurring and other detector effects, from everywhere down from the telescope all the way through the detector, through all of the optics. But I'm sure many of you are familiar with the idea of adaptive optics, the idea that you can correct some of the atmospheric blurring to be able to hone in on, this is now the center of our galaxy where there's a large black hole that people are very interested in. And that idea of using corrective optics has been around for a long time. You take light that comes off the telescope, split off a little bit of that light, look at the blurry image, and then in a controlled loop be able to correct what is actually observed from your camera by changing the mirror that the light actually bounces off of. And being able to do that in real time is what gives us these very clear images that you see on the right hand side. Putting this in the context of machine learning, you might think of this as a reinforcement learning problem where we're measuring the current wavefront, and what we have, our current state, is that wavefront, but all the other actions that we've had in the past and what the wavefront measurements were in the past. And what we want to do is deform our deformable mirror, where we're basically just deciding on our piston voltages to change that mirror to do this correction, and that's our action space. But unlike going up down left right, fire no fire on Atari games where RL became probably better known to the broader community outside of CS, we have some really interesting hard problems in astronomy. That action space, if we're thinking about pistoning every single one of these pieces of the mirror, can be something like 10 to the three, 10 to the four in size, not just a dimension five. And also we want to be able to piston at potentially kilohertz. We also have interesting questions like can we learn an offline policy with a simulation of the blurriness of sky and how adaptive optics would work, and then maybe do a downstream task where we're in real time actually updating our policy model that we would wind up learning to be able to do better. And can we actually beat out the current state-of-the-art in the ways in which this is done without machine learning? So this I think is a really exciting question.
There are many other places, I'm sure Chris Stubbs has seen a lot of this before, there's thousands of subsystems inside of LSST Rubin, huge amounts of data is being generated in each of those, there are control loops there that, while they're doing well and are well positioned to make this facility the world's premier data taking facility not so long from now, there are interesting questions about whether we could actually do better with AI and the sorts of RL policies that might be learned offline and then applied in the real-time loop. And while LIGO, this is one of the pieces of LIGO that you see here, is acquiring not a lot of data about the gravitational wave sky every day, it's the kind of amount of data you might be able to pass around with a single disk, it's acquiring a lot more data, of order terabytes of control data every day, and none of this as far as I know has sort of advanced AI techniques with RL in these real-time control loops. So this is going to require CS people, hardware people, and domain experts working together to find places where there are computational bottlenecks, or in places where we have an intuition that we think we might be able to do better, and actually start testing these out and putting them into a real time setting.
The last thing I just wanted to say in the Upstream sense is something that I think is extremely low hanging fruit, and that's doing sensor fusion for a prediction of transparency on the sky. We have lots of data that we acquire about the sky. You see a little movie of the sky as the Milky Way moves through it over the night, and you can also see these low-level clouds coming through. We also have satellite imagery, we have other sensor data, and one of the things that I'm interested in and started thinking about is the idea of bringing all of this data together to be able to predict in the next minute, the next hour, maybe the next few hours, what the transparency is going to be in various parts of the sky. So that as we try to make predictions about where we're going to observe next, we can be smart about being in places that are going to be the best places for us observationally and give us the highest signal to noise. We want to do this not just in the context of generic planning, but we want to be able to plan out our observations of real objects, of real events, to be able to do better science. And what you see here on the left hand side, coming a little bit full circle, are the actual observations of neutron star merger events, of a single event that happened in 2017. And this is the actual data across multiple different bands, and you see the time scale is just a few days here. We want to be able to find more of these things. We haven't really been able to find any more after a gravitational wave event has happened on the sky, and one of the big challenges there is that the gravitational wave localizations of these events are very large, in these sort of bananas on these skies, these green bananas that you see on the sky. There are billions of galaxies that we could potentially look at with our optical telescopes, and if we knew exactly where to look we could try to obtain light curves like you see on the left hand side there, but we don't know where to look, and we have to tile across all the different localization possibilities to try to find this new object, as you see depicted on the right hand side, of a new object that appeared next to a galaxy. We want to find these needles in the haystack in this massive search space.
So I've been working with a student here at Berkeley on using a reinforcement learning policy and learning it on a graph neural net to be able to optimally observe with LSST Rubin these large bananas on the sky. And you can see a depiction of those probability regions on the sky here, and then the learned policy of where we point our telescope over time with our different filters. And we're showing that we're beating state-of-the-art, which is a handcrafted set of decision rules about how to observe. And importantly, what we're trying to do is we're trying to craft these RL policies with loss functions or reward functions that are not just sort of the generic ones of did we detect it or not, but we want to say that if we don't detect one of these really interesting events, we want to be able to rule out the largest part of the parameter space that you see here on the right hand side. And so we're couching these reward functions not in the context of heuristics but in the context of science.
So with that, I'll end by saying I'm very excited by what I've been hearing about A3D3. We have a new proposal in for an NSF AI Institute, and many other groups other than ours that have also proposed, so there's about to be a huge amount of infusion of energy and work into the AI meets astronomy world, and I hope I've motivated some of that for you here today. So I'll stop here and happy to take any questions if there's time. Thank you, that was great. That was an amazing tour of many exciting things. I think we're open for questions. Don't see any online, so can I ask, I guess I have maybe a sociological question. How are these concepts for real time followup, what kind of reception are you getting in the astronomy community? Is it being received enthusiastically by all, or is it a struggle in some or all aspects in trying to bring in new ideas for how to coordinate the followup?
Yeah, it's a good question. I think the answer is it's mixed. It depends upon the kind of sub questions that we're asking, and I think the time domain community in particular has been very receptive to any ideas around optimization. So a lot of work that went into what you've seen here has nothing to do with AI or ML, it has to do with the tooling. Just the programmatic API access to telescopes itself was a huge amount of effort. The robotization of telescopes has happened over the last 20 years, and people being open to the idea of robotization was itself a big sociological hurdle. And now that we have that, I think we're seeing more reception to the idea of let's try out this new idea. For instance, in the LSST Rubin community there's about 3% of the telescope that's been allocated at what's called Target of Opportunity. This is where a new event happens and the community writ large makes a decision it's worth going after and stopping the regular scheduled program of observing the sky in the way that they were intending to for that night, and giving up the telescope to a group to be able to observe the sky. And most of the time when we talk about this we're talking about either following up on neutrino events or gravitational wave large bananas on the sky. And so there's lots of debate about exactly how we should do that, but until we can sort of prove offline that we can get more efficient discovery of these kilonova events, we're going to be implementing other policies. And so I think my goal in the next couple years is to make sure that we're at least aware that there are these RL learned policies that could be used that may be 10% more optimal than the current techniques, and then it will actually be convincing the community that this is the kinds of things we want to do.
The last thing I will say is that the stakes tend to be a little bit lower the smaller the aperture size, and so on meter class telescopes there tends to be more interest in experimentation. One of the things that I didn't have time to talk about was the fact that with SkyPortal, that application that I showed as part of the kind of central repository of new events coming in, ML inferences happening within our SkyPortal universe and then triggering telescopes, is that we can do all of that without any people in the loop. And there's now hundreds of supernovae that have been discovered by the Zwicky Transient Factory, observed, and then discovered by real bogus, then inferred that these are probably type 1a supernovae, and then sent off to robotic spectrograph telescopes. Spectra have been taken, that data has been reduced, fit to real models, infer that they're type 1a supernovae, and then sent out and broadcast to the world. That's happening on a regular basis, on a nightly basis, without any humans in the loop. And people are doing this because the apertures are small and the cost for being wrong and observing this part of the sky incorrectly, or you did it because you made a bad inference somewhere up the chain, is not actually that big. But as we get to the larger apertures, the sociology and the pushback becomes certainly stronger.
Thanks, Joshua. Question. Nice talk. I was looking back at your slide, and I just want your perspective on using large language models, because I assume you probably face a lot of the similar problems that we face in Material Science where large language models solve everything and they solve nothing. And then I saw that you were trying to merge that information with latent descriptors. I'm just curious how that has worked, and if you think that there's a better way to maybe combine some of the benefits of language models when you need high precision sort of tasks like you do in astronomy.
Yeah, so to be clear, the LLM stuff that I showed is A, early days, and B, the cost of being wrong is not that big. And so we're not trying to go for high precision here. This is really supposed to be AI assisted, UX guided discovery and exploration. That's the downstream task where it involves a human kind of saying yeah that's interesting, I'll click on that object and see what comes with that. We're trying to measure people's interactions with those types of models. But LLMs in particular, and transformers, are the kinds of things that we're exploring with our foundation models. We're finding that it's obviously hard to do, we have lots of data to learn from, but just even coercing our irregularly sampled noisy time series data in multiple bands from multiple telescopes with lots of different types of signal noise into a single model itself is really challenging. But the idea of kind of predicting ahead of what is the next data point, which is what LLMs do, what is the next token, that's very amenable to time series. We're trying to predict what's the next flux point. And we're not trying to do that because actually prediction in astronomy is not all that interesting, it's kind of more of what can you learn from it. So what we're finding is that with our use of an LLM type of technique is that we're able to get really interesting embeddings that are quite not necessarily predictive of what's coming next, although the loss function is predicated on that. But we think we're going to be able to use those embedding spaces for downstream tasks with great precision, maybe even say of the yard precision. But the actual outputs of those, of like what is this light curve going to evolve to be, is kind of more of a party trick than it is the actual core science that we're trying to get to.
I see a question from Merlin. Hey, mine was also a question on the Netflix recommendation system for supernovae. I just didn't quite get how, could you just run me through how you were generating the text and exactly how the similarity thing works, because I just didn't quite take it in at first.
Yeah, so there's two components. So one is all the data that we have about a given object, so this is light curve data, this is spectra, this is people's comments about those objects. We have bots that come in and just dump a whole bunch of data about that object as soon as it happens, and then we have people who are actually writing summaries of it, and we even have other kind of publications about those objects. All of that data is fodder for throwing it into summarization. So you can basically put it all in a large XML file and ship it off to Claude or ChatGPT-4 and just ask it to summarize that data in human readable form, and that's sort of the first task. The second task is taking that summary and then using an embedding model, which also is provided by all the large API companies at this point, and taking this natural language and embedding it into a 1600 dimensional space, which in principle carries all the concepts of what's in that paragraph, and then saving that into a vector database. We use Pinecone, but there's plenty of others that you could use. And then when a new object is entered into that database and we look at that object, we can query that database for all the sources that are within some cosine distance of that object. So we're finding things that are like it in that space. So it's not exactly a recommendation engine, it's more of like a ranking engine. In principle though, it'd be an interesting idea of people giving thumbs up or thumbs down and eventually we wind up learning that you love supernovae and RR Lyrae like Chris Stubbs.
So I guess my question then is, how come you go from all of that metadata via the LLM's generated description and back to an embedding rather than just simply generating the embedding from the metadata? I mean, just going through that step seems like a potentially very lossy and counterproductive thing to do.
Yeah, you're totally right on that point. That's why, so we're doing that because that is the only thing that we have available to us at this moment, but the work on these multimodal foundation models will exactly allow us to get to what you said, where you take the data itself, you learn your own embeddings, you're not trusting ChatGPT who doesn't really know anything about the context of astronomy in the way that we think about it, and then learning our own embeddings and then be able to do our own embedding in our own vector database. So you're absolutely right, the roundtrip nature of that doesn't make sense for the long term, it's just the only thing we have available to us now. What we want to be able to do, what you see up on the screen, is we want to be able to learn that ourselves from our own data.
Okay, awesome, thank you. Yeah, good question. Right, more questions or comments? Okay, we're at the top of the hour, so thank you again very much Josh, that was really nice, and thanks for everybody for attending. Bye Josh, thanks.