How machine learning became central to discovery and inference in time-domain astronomy (real-bogus supernova discovery, self-supervised variable-star models, and simulation-based inference that uncovered a new microlensing degeneracy) and where AI goes next: downstream in AI-guided exploration, upstream in telescopes run as robots.
Second of three Tuesday lectures in “Berkeley in Space”, a series organized by Donald Mastronarde for the UC Berkeley Retirement Center's Learning in Retirement program (Oct 29 – Nov 12, 2024). Given over Zoom on Election Day; the recording ends before the Q&A.
“We don't have personally identifiable information in our data. It's not that valuable. And if you make a mistake, you don't make a billion-dollar error. Maybe you're embarrassed in your career or something like that.” – Joshua Bloom
“It's fast, parallelizable, transparent, deterministic, and versionable — pretty much none of which is the characteristics you would describe of a person.” – Joshua Bloom
“I, for one, don't like to do machine learning just because it's fun. It is fun, but it's actually quite painful and hard, and to do it right requires a lot of computation and a lot of careful and expert knowledge. We do machine learning in astronomy because we're trying to get at new science.” – Joshua Bloom
“This degeneracy that we discovered using machine learning is actually deeply embedded in the math of the so-called gravitational lens equation, and it had really been hiding in plain sight for the last 50 years.” – Joshua Bloom
“It's not just about using AI to accelerate discovery of the known knowns, or even help us find anomalies of actual objects. This is now giving us a deeper theoretical view of how the universe works.” – Joshua Bloom
MASTRONARDE: Welcome, everyone. This is the second lecture in the LIR series “Berkeley in Space.” Last week, we heard from a mechanical engineer, Panos Papadopoulos, about the challenges and prospects of sending humans far from Earth into a hostile environment where they will need to be more or less self-sufficient. Today and next week, we are hearing from scientists who use a variety of tools to study distant parts of the universe where presumably no human will ever travel outside of science fiction.
Joshua Bloom is Professor of Astronomy here at Berkeley. He was educated at Harvard, Cambridge University, and Caltech. At Caltech, he received his Ph.D. in astronomy and astrophysics in 2002. Some of his specialty areas are black hole transients, gamma-ray bursts, optical/infrared instrumentation, and supernovae. He has published over 300 articles and is author of the book “What Are Gamma-Ray Bursts?” in the series Princeton Frontiers in Physics. He has also been involved in AI applications from before the time that AI became so prevalent and sexy, having been co-founder and chief technology officer of the startup Wise.io a decade ago, and that expertise will be apparent in today's talk. Among the honors Professor Bloom has garnered are being a Sloan Fellow, a Junior Fellow at the Harvard Society of Fellows, and a Hertz Foundation Fellow. He has also received the Data-Driven Discovery Prize from the Gordon and Betty Moore Foundation and the Pierce Prize from the American Astronomical Society. I'm delighted to welcome him here today to speak to us on “Transforming Astrophysics with AI.” Josh, go ahead.
BLOOM: Thank you, Donald. Thank you so much for having me. It's very nice for all of you to join today. I know this is a very interesting and special and potentially anxious day for many. So we have, I hope, a learning treat for you all today to get to know some of the astrophysics that I'm doing with my group, and more broadly about how AI or machine learning is transforming the ways in which we do astronomy.
It's been a very exciting time. I'm going to be focused mostly on, as I said, my work, which tends to be in time-domain astrophysics, but what's quite apparent to people that have started paying attention to these new methods as they creep into astronomy is that they're having impact really across the astronomy landscape. As we try to understand, for instance, the nature of dark matter, as we try to use things like very distant supernovae to understand what the nature of dark matter is and the Hubble constant, and do cosmology more broadly, there are components of machine learning that are being used to do models, for instance, of these very complex galaxy systems that you see in front of you.
People are also using models from machine learning to be able to do image reconstruction. You may have seen this beautiful image of the first picture of a black hole that we have. And it is not an actual image that's taken natively. It's something that has to get reconstructed with a tremendous amount of work. And what's very exciting about machine learning is that people have started realizing that they can do better than using traditional computer vision techniques to be able to make these exciting images, and then the discoveries that follow from them. People are starting to study, using, again, various components of machine learning, the gravitational-wave background left over either from the Big Bang or from the inspiral of black holes as they wind up perturbing spacetime as they get closer and closer together and leave an imprint on something that we can actually measure here on Earth.
And then last — and these are all things that are of great interest to me, but not the core of my own science — people are starting to now get spectra of the atmospheres of planets outside of our solar system. These exoplanet atmospheres are very complicated, and you see basically absorption from the atmosphere from the parent star as a function of wavelength in the infrared here. And turning these observations into inferences about what the nature of these atmospheres are is a very computationally complex problem. And people are able to accelerate that inference using a type of machine learning, which I'll describe briefly towards the end of the talk today.
My agenda here today is to introduce you to some of the challenges that we have in time-domain astronomy. I'll also give a little bit of a deeper intro into what I work on and what I'm interested in by virtue of this introduction. And then show you how machine learning — and I should say I use machine learning and AI interchangeably here — is already central to the way in which we do discovery and inference in astronomy. And I'll end my presentation today, before we kick off the Q&A session, with what I think are some exciting new opportunities that I'll call the upstream and the downstream, and that become more apparent as we go through the talk here.
As you heard, I came to Berkeley about 20 years ago. I'm coming on my 20th anniversary, and it's been just a fantastic place for me to develop my career and work with, I think, the best colleagues in the world, not just in astronomy, but in computer science and statistics. I've had an opportunity to teach a wide variety of classes, the kind of traditional grad classes in astronomy like radiative processes and high-energy astrophysics, but also had a chance to develop my own interest in teaching to a much wider audience various aspects of data science and even Python programming. It's been a wonderful place for me to grow as a teacher.
I've obviously been doing a bunch of research, and as you heard from Donald, I work on extragalactic transients. These are explosions that happen outside of our galaxy. I'm also very interested in time-variable objects within our own galaxy. And then I've really started getting into ways in which we can use computers and advanced statistics and more recently machine learning to be able to power through and do more with the data that we have coming, as more and more of that data is acquired. And you also heard very briefly that I did do a stint at a company that I started, which was called Wise.io, started out of Berkeley, and it was acquired by GE in 2016, and I got to spend a little time there as well. So I've also had the chance to spend some time in industry and to get to use some of the techniques and ideas and know-how working across lots of different types of interest and capabilities and domain knowledge with large teams, and it's been just a fantastic ride for me.
I worked on a phenomenon, for those that may know about this, called gamma-ray bursts. That was sort of the early part of my career, and started spending more time on other types of phenomena like supernovae and so-called tidal disruption events, when a star is swallowed up by a black hole at the center of a galaxy, and that's something we can detect for days, months, and years afterwards, the aftermath of that. Also worked on building telescopes and automating telescopes and building instruments as well, for us to be able to do time-domain astronomy in new and innovative ways. And then I've also been spending a lot of my time now, almost 15 years, in working on different ways to bring statistics to the fore for us to be able to do our astronomy better and more wisely. And about 10 years ago, I created the Center for Time Domain Informatics, which is in Evans Hall, which is a connection between astronomy, statistics, and computer science. And we had a number of great successes out of that project.
I thought I might kick off setting up the problem that we have in time-domain astronomy with what I'll call this Rumsfeldian challenge, where we have lots of knowns, we have lots of unknown unknowns, and then we have everything in between, the known unknowns, for those that remember Donald Rumsfeld and his famous quote. What you see here is something that astronomers look at quite often. These are so-called light curves. So we have time on the x-axis, and we have some measure of brightness on the y-axis.
And what you see depicted are a number of different types of light curves of different explosive events that happen generally outside of our own galaxy. Type Ia supernovae are incredibly important objects because — I don't know if you can see my cursor here — at peak, they all appear to be about the same brightness, and the differences between them can be calibrated, which means we can use them as very exquisite measures of distance. And that turns out to be one of the premier ways to be able to measure the fundamental parameters of the universe, i.e., do cosmology. So studying Type Ia supernovae and finding them is incredibly important. Type IIP supernovae are a different type of star which explodes. And those are the most common types of supernovae in the universe. So if you want to understand the death of stars that are not too different than our Sun, but a little bit more massive, these are the types of supernovae that you want to study.
And then these other types of objects, which are much more rare — and I should say that these are largely theoretical curves that we see for most of these other ones. We've never seen what's called a pair production supernova. It's been theoretically discussed and fleshed out for decades, but we've never seen one of them, so we don't know whether nature actually makes them. We have things like compact objects, like neutron stars, running into red supergiants. That's the idea of what that might look like, much fainter than some of these other ones.
And then there's this little blip here, which is the afterglow effect after two neutron stars inspiral because of the loss of angular momentum and energy in the orbit due to gravitational radiation. Eventually these neutron stars smash into each other and produce a burst of gravitational waves, and then leave behind some dying embers that in principle give off light that we can see at optical and infrared wavelengths. And this was a theorized light curve until 2017, when it was first detected. And remarkably, what people had been thinking about, a lot of them here at Berkeley, turned out to be largely borne out by observations.
So these are the kind of known knowns and the known unknowns, but of course, by definition, I can't plot on this curve the things that we don't know about yet, that we haven't even thought about. And so when we are taking data over the whole sky — and we do this with very large-scale surveys; we're imaging large swaths of the sky every night, repeatedly — we're trying to find all the things we know about, all the things that we're excited about, but we're also trying to keep our minds open to the kinds of things that we may not have seen before, and make sure that we're not just throwing those into the dustbin because they don't look like things that we know about.
So how we discover these is really critical, and how we do this at scale without, generally, people in the real-time loop, because there's just too much data. And how do we then make priorities of what it is that we're going to do once we have a big basket of these? Say I find 30 or 40 candidates of these objects a night, how do we go after these and go after those? What I mean is follow-up resources. We want to get more images. We want to take spectra of these things. Maybe we want to trigger satellites to be able to point at these very special positions in the sky and get more data. And so the optimization over this entire sort of teeming universe of explosive events, let alone variable objects in our own galaxy, is a very hard problem. It's largely intractable. So I've been trying to take little pieces of it throughout my career and try to figure out ways that we can do it better.
And in some sense, we're in this kind of embarrassment-of-riches phase in astronomy, where we just have too many of these things, too many of these transients. It started off many decades ago, but really got going in earnest, I think, with the Palomar Transient Factory, which was run out of Caltech. And I was a part of starting in 2009, getting about a gigabyte of data every 90 seconds, which was of order about 40,000 new transients every single night that were in this data set. And that was followed on by this Zwicky Transient Facility, also out of Caltech, something I'm also part of, increasing that number by an order of magnitude. And we're right on the edge now of the beginning of the Large Synoptic Survey Telescope project on the Vera Rubin Observatory, which is going to produce even more. Two million objects a night is what we expect as this thing kicks off.
We don't have nearly enough people nor follow-up resources to be able to find these things. So we have to take what is, I would say, cheap discovery, like lots of availability of transients, and make our decisions about how we use the world's most precious resources for doing the expensive follow-up, like the Hubble Space Telescope, James Webb Space Telescope, which launched a few years ago. And at some point in the future, we hope, is the Thirty Meter Telescope and other even larger facilities at the billion-dollar scale. So this is maybe an articulation of the bottleneck that we face, and hopefully gives a little bit of impetus to the kinds of problems that we're going to be asking ourselves how we deal with it.
So how do we go from the sort of older mode of having people looking at all the data, which is maybe the traditional way in which you think about how astronomers do their work — going out to the telescope and communing with it and coming home and sleeping all day and then looking at the data the next day? We just can't do that anymore. And we certainly can't do it for these very large-scale surveys that we're producing. And so I really consider myself quite fortunate, having been at Berkeley, to have started interacting with computer scientists and statisticians in around 2008, 2009, starting to articulate the challenges that I had, and people are starting to say, well, there are these techniques that you might be able to make use of.
And it's not so crazy, by the way, for astronomers to see that there's a new emerging tool set that is arriving and arising in other sciences or in other realms, and then we wind up making use of it. There's a famous guy named Galileo, who you may have heard of, who heard about this invention of this thing that was supposed to look for ships over the horizon. And he did this incredible thing. He just went like that, and he looked up. And all of a sudden, the birth of modern astronomy began as we co-opted this technology that was invented for military purposes for our own sake, and in some sense, the rest is history.
And then likewise — and this is perhaps less well known — is that astronomy has, for many years, many decades, served as an interesting test bed for statistical and computational exploration. Jim Gray, who spent some time here early in his academic career and then went on to Microsoft, has this great quote where he says, “I love working with astronomers since their data is worthless.” He was a fantastic data scientist and computer scientist. Some of you may have actually known him. And he convinced Bill Gates and the others to invest in astronomy, not because he was necessarily that interested in our outcomes, but he saw the large amount of data that we're acquiring is a wonderful test bed to test out parallel computing and different ways of doing visualization on large scales. And importantly, unlike the other large data sets that are out there, we don't have personally identifiable information in our data. It's not that valuable. And if you make a mistake, you don't make a billion-dollar error. Maybe you're embarrassed in your career or something like that. But the stakes are pretty low, yet the data is very rich. And so there is this kind of rich symbiosis between astronomers using tools and people who are building tools starting to see astronomy more and more as a wonderful place for them to develop those toolkits.
So I, through my career here, really laid out this vision for doing a huge amount of what we do scientifically in an automated way. And I'll call it automated data-driven discovery and inference, but it's really even beyond that. We started off, as you'll see in just a couple of slides, figuring out ways to do machine learning for discovery, and then moving on to classification. But then upstream from that, there's all the other interesting things about how do we schedule telescopes. How do we even build telescopes? Can we aid our construction and our design of new facilities with these new tools? And then downstream is how can I take what I've learned, how can I take all the data that I have available to me, and start to do some novel inference.
I don't think we're quite there yet in being able to have AI write all of our papers for us, but it shouldn't be too much of a stretch, given what I know you've all been seeing in the news about these ChatGPT-like tools, that we are getting to the point where large amounts of not just science can be done and accelerated with AI, but some of the ways in which you can communicate with each other — what our results are — are also fodder for the next generation of AI tools.
So let me transition a bit now just to talk about the machine learning centrality in the work that we do. But before that, I wanted to just quickly provide a refresher of what it is when we talk about machine learning or AI. It's a “field of study that gives computers the ability to learn without being explicitly programmed.” So that maybe makes some sense, but it was a definition that came out in 1959. A probably easier way to think about it are the places where we can make great use of it. I won't go through all the different areas. These are the non-nefarious things that people can use AI for, by the way. There's a whole other list that you may know of.
Classification, for instance, is really important in the commercial setting for finding fraudulent activity, for analyzing text. Regression analysis, again, in the commercial world for doing sales predictions or forecasting. We can use AI for imputation, for data cleaning, recommendation engines — as you're well aware of from things like Netflix and Amazon, they're making heavy use of recommendation engines — the ability to cluster, the ability to find outliers, the ability to take very, very large dimensions of our observations and turn it into a smaller number. This is dimensionality reduction, so that people can look at that and try to gain some sort of insight. If I have many, many dimensions over which I'm looking at something, it's very hard for the human brain to keep that all in one place, but if I can visualize it in a smaller number of dimensions, that often can be very useful. And then more recently, there's things like generative, or creating text, images, or creating chats or code completion. So it's being widely used outside of astronomy. But as you'll see, there are many places where a lot of these ideas are actually starting to make great progress for us in astronomy.
Great. So one other set of terminology here before I go on is the different types of machine learning that exist in the world today. These are not hard-and-fast boundaries, but I find them to be pretty useful because you may have heard some of these terms before. The first one I want to highlight is this idea of what's called supervised learning. That is where we have a large corpus of data, and we have some input data and we have some output data that we already know about. So in the context of astronomy, this may be lots and lots of pictures of stars, and then the output might be our classification of those stars, or maybe it's different types of galaxies and we want to know what those classifications are. Here somebody's already done the hard work to tell us what those answers are. This is what's called labeled data.
And now when we build a machine learning model, we're teaching a machine to take all that input data and make a prediction about the output. And if you boot up a machine learning model and haven't done any training, just like a little kid, it's going to make a lot of mistakes. But there are techniques that have been developed in the computer science realm that have allowed us to get better and better answers by learning from the data itself, and the idea is that we can build models that work extremely well on our training set and then apply it to other data that we haven't seen yet.
There is also another type of learning, which is called unsupervised learning, or sometimes called self-supervised learning. And this is where we don't have any labels. We don't know what the answer is. We just have the data. And what we're trying to do there is a little bit more nebulous. We're not trying to make predictions on new objects to get exactly the same distribution of those data. Instead, what we're often trying to do is to find hidden structure. We're trying to find clusters that we maybe didn't know about in the data. Or we're trying to find weirdos that are not in the locus of where all the other data lie in some representational space. And this is something that I've been spending a lot of time with, and you'll hear more about it later on, but I just wanted you to see those two.
And then last — I'll spend very little time on this; I have one or two slides towards the end on this — something called reinforcement learning. This is a type of machine learning model that takes input from the real world and, instead of making a prediction about what's going to happen, decides to take action in this world. It's more about making decisions. And here our job is to come up with a reward system that says that if I'm thinking about a self-driving car, for instance, that I can make my way through a very complex environment, having learned on lots and lots of examples. And now, instead of making a prediction of what's going to happen in the future, I now have to make a decision about whether I'm going to turn the steering wheel left, right, or press on the brake or press on the accelerator. So there's a huge world of robotics where reinforcement learning is becoming incredibly important in lots of fields. There's really only one or two ways in which I think we can make use of this in astronomy.
And just to put a finer point on the supervised versus the unsupervised distinction in the type of machine learning that we do: here on the left you have an example of the projection of these three different classes of objects, distributed by the two different dimensions. I'm not even telling you what those dimensions are, what their units are. And what we might try to do is to make a model that says, if I have a new object that has a position in X and Y around here, is it likely to be a green star or a blue square or an orange circle? And your eye already tells you it's very unlikely to be an orange circle because it's not near that cluster. But you want to build these models that are robust, recognizing the fact that we're training on not a lot of data. And so whenever I make a statement about whether something belongs to this class or this class, that statement necessarily needs to be probabilistic in nature.
And then on the right-hand side is supervised. It's a bunch of data that are now projected in these two dimensions. And we might ask questions about how many clusters there are. Or I might decide, because these two objects at the top look like they're a little bit farther away than these other two, that these are objects worth exploring because maybe there's some weirdos — and it may be just bad data, or maybe that physically these are different types of objects than I've seen before. There's some sort of rarities.
So where are the places in my work where AI is really helping? One is on discovery. So we like to take images of the sky en masse, as I've said before, and what we have here on this three-panel is an image in the middle of a small little part of the sky, and you can see a little galaxy in the center here surrounded by a bunch of stars, and then a shallower image, so not as high of signal-to-noise, that I maybe just took last night. And I want to compare these two, and there's nothing different between these two to the eye, except for the fact that one is higher signal-to-noise than the other. But when I subtract them, I wind up seeing a little new object here. And this is indeed a supernova. I want machines to be able to find these things for me.
Once I have something new, I want to ask the question: is it worth spending my precious resources to follow this up? So here's another light curve. It's not nearly as beautiful as the ones that I showed you at the beginning. Here's a little variable object. Brightness actually goes up on this y-axis here. For a while it's sort of burbling around at some certain values, and then every now and then it deviates and it gets much brighter. If I saw this in my database, is this something that I care about? Is this something I'm excited about? And oftentimes the first entrée into the answering of that question is coming up with some sort of classification. How is this like the other things that I've seen? And how is this unlike all the other things that I've seen? And then I can make some decisions from there.
And then the last thing, which is becoming really exciting for me and those in my group, is something we call surrogate modeling or simulation-based inference, and that's used for parameter estimation. These are places where we have physical models of how things work, and what you see plotted here are physical models of supernovae light curves. So this is the brightness of supernovae as a function of time for different inputs, values of the explosion. So what is the explosion energy? How much mass is involved? What were the characteristics of the star that blew up? And it turns out with just a couple of different parameters for this input, you can run through a supercomputer amount of computation time and get out these light curves after you follow all of the physics from this exploding thing. And so the forward model, as we call it — when you start with a couple of numbers and then you get your observables — tends to be extremely expensive.
So if I now have observables — I have something I've observed, which may be these actual data points here — and I want to fit it and find what the parameters are of the physical model that I have. If my physical model takes milliseconds to run, I can just keep on moving over it and use traditional Markov chain Monte Carlo techniques or other gradient descent techniques, and I can get a good measurement or a set of estimates of my parameters of my model. But if it takes you a month to run one model, you need ways to accelerate your ability to either do the forward model or what we call the inverse model, which goes from data back to model. So these are all the really exciting places, and I'll try to touch on all of these here.
This is what I call midstream. So this is the place in astronomy where I'd say most people in AI are working, where we have lots of data in hand, and now we want to make use of it. We want to learn something about it, or we want to make some decisions about what to do next. There's some other really interesting things about the interaction downstream from this, of how you get people generating hypotheses and testing hypotheses with the data that's there. This isn't looking at individual objects, but maybe looking at large-scale numbers of objects and allowing people to do what they're best at, to make these associative decisions across lots of different domains and see patterns that we can't ask machines to learn because we really don't even know how to ask them to do that. But people, when they see it, can come up with these hypotheses and come up with these great ideas. And are there places where AI is going to help us here? And I think the answer is yes.
And then I'm also getting very interested in what I'll call the upstream. These are places before data is even taken, or before we've even looked at some of this nicely cleaned-up data: what can we do to get better data, and what can we do to make the midstream inferences even better? And so as I think about the whole life cycle and the whole time that we spend as astronomers, we all spend time across this entire landscape, but we've really only been doing AI in this middle part. So I'm very excited for the future of these other components.
I said that it may be kind of quaint to think about astronomers looking through telescopes, but when we look at our data, we often also just do what you're seeing here. We take out our little microscope or zoom in on our computer, and we start examining these images. This is a picture from the 1890s of the Harvard College Observatory, where the Harvard College Observatory had a big data problem. They were getting more photographic plates from the Southern Hemisphere than they knew what to do with. So they hired a number of people to compute. In fact, they called the people that they hired, all of them women, “computers,” who would go through and look at the data and pull information out of that.
Surprisingly, or maybe not surprisingly, this is kind of the way in which time-domain astronomy existed when I first started here at Berkeley. My colleagues and many others around the world were asking undergraduates to be looking at data every night to try to find new types of supernovae, for instance. And it makes sense, because it's a great way to do training for undergraduates. But as we started getting more and more data, beyond what you're getting from just one telescope, now from many, or from telescopes that had much larger cameras, this became an intractable problem, became a real, significant bottleneck that I got worried about.
So one of the first forays into machine learning that I had was in what we'll call a discovery engine, which looked at those image subtractions that I showed you earlier and sifted through all these — what kind of looks like reasonably good subtractions, but these are all artifacts of bad subtraction techniques — to try to find the real new objects in the sky, so this discovery bit. So I have all this new data and I want to find where all the new things are that are varying on the sky. And this turns out to be, at the time, about a thousand-to-one needle-in-the-haystack problem. For every one of the real new objects that we had in a data set from the Palomar Transient Factory, there were about a thousand of these bogus ones. So our cost of being wrong was actually pretty high. If we just more or less labeled everything bogus, we would never have anything real. If we called everything real, then we'd spend way too much time doing follow-up on things that just were not worthy of follow-up.
So we built this — not just something that worked offline and worked on paper, but something that we were able to put into a production environment that was part of the actual real-time data stream, that allowed us to do this classification of: is it real or is it bogus? It's a zero-one classification problem. And it's fast, parallelizable, transparent, deterministic, and versionable — pretty much none of which is the characteristics you would describe of a person. Even if they are experts in this problem, it's really hard to do all of this with people. And we're very proud this engine ran for over a decade, and new versions of it are still running today. We were able to find the nearest Type Ia supernova in over three decades using the real-bogus discovery engine that suggested to ourselves and our colleagues which were the most interesting objects on the sky to consider.
And as you can see here, this is sort of a classic discovery image of a supernova. It's in the Pinwheel Galaxy, by the way. On August 23rd, 2011, there was nothing there. And then there was a little bright light that showed up, and then it got brighter and brighter. And in fact, this actually got bright enough to observe with binoculars or a low-power telescope. It's the only supernova since 1987 where that's been true. And so what's important to recognize here is that this object — which turned out to be very, very important, because understanding the nature of Type Ia supernovae is absolutely critical for us to understand how we use them in cosmology — we would have discovered this as a world, as a community, without our machine learning system, but it would have happened a couple of days or weeks later, as this thing got so bright that all the amateurs in the world were able to see this.
But because we were able to detect it only about 11 hours after explosion, and within a couple of minutes we were able to make a machine learning statement about its realness and the fact that this is an interesting part of the sky, we were able to get follow-up resources on this extremely quickly, and we were able to get the Hubble Space Telescope and the Chandra X-ray Observatory looking at it. And because of that — this is a very complex slide as part of one of the papers that we were able to produce after the discovery — we were able to rule out for the thing that exploded, the so-called progenitor, all the areas that you see in color. And because we were able to get early X-ray limits or Hubble Space Telescope limits, we were able to rule out these different things.
And because we were able to observe early on with lots of telescopes, we were able to rule out that the thing that exploded was different types of stars that are similar to our Sun — that would be these right here — or other types of variants that are a little more exotic here. These are all sort of nuclear-burning types of stars. The only thing that was allowed and remained are objects in effective temperature and its average density. They're so-called compact objects, namely white dwarfs or neutron stars. And so while it's not a big surprise that Type Ia supernovae come from white dwarfs — that's been surmised for a long time — now we have a new line of evidence, because we did machine learning, that allowed us to say unequivocally it can't be the sort of ordinary-type stars that was the cause of the explosion. That was fantastically exciting for me to see the science get done because we were able to do machine learning at scale.
Now real-bogus is actually a cottage industry. You really can't have an imaging survey any longer if you don't have your own real-bogus system. And so any new survey that comes up is making use of this. And one of the ones that I'm actually most excited about in the future, that you may have heard about, is something called the La Silla Schmidt Southern Survey, or LS4, which has a major partnership component from Berkeley and LBNL. It's down in Chile on a one-meter telescope. And we're going to be observing 20 square degrees every 60 seconds and getting over, sort of, two gigabytes a minute. So we're going to be competitive with this Zwicky Transient Facility, and now doing this in the Southern Hemisphere opens up a whole lot of really interesting science goals that we have here on the left. And one of the things that my group will be involved in is helping develop real-bogus for us to be able to make use of all of that data as it comes off the telescope.
I talked about transients, extragalactic transients, already, but I'm also very interested in variable stars. And this is maybe one of the highlights I wanted to present to you of how we go through the mechanics of machine learning in our own work to be able to do some interesting inference. So here we've got a light curve of an object, and it's just one of 50,000 on the sky that were observed and known to be variable. This is a projection of the entire sky, just in two dimensions here, but the center of the galaxy is right here, and the plane of the Milky Way is right here. And so this object just happens to be above the plane of the Milky Way. And unless you can do a Fourier transform in your head, which I never met anyone who can, you won't know that this is actually a periodic variable that varies at about a half-day period. And it's a certain type of object called an RR Lyrae star.
We wanted to take this entire catalog and apply classifications for all of these objects. Fifty thousand is really just too much for even a group of experts to go through and do a credible job. And even if they could, it's hard to assign probabilities. But we only had something like 810 known classes, or known objects with class identifications, over this whole catalog. So we had to build this ability to do classification on this type of data, to be able to make statements about what type of class of object this belongs to. And it turns out it's really hard to do if you do it in a purely supervised context, but in a self-supervised or unsupervised context, we were able to do and achieve state of the art.
And here's a place where we were able to innovate a little bit on the machine learning side for us to be able to get innovations on the astronomy side. This is a paper from just a few years ago where we made use of something called an autoencoder (AE) that does something very, very simple. It takes the data that we have. Here's just a small representation of the light curve. And we have a machine learning network that takes all of this data and compresses it down into just a small number of values. This is called a bottleneck. And then we decompress that bottleneck. And the goal here isn't to produce a classification; it's to just reproduce the original data.
And this seems like a kind of silly exercise, but if I have hundreds or thousands of data points, and those data points can be represented by some sort of functional form that requires only a couple of input parameters, then this bottleneck layer, as I wind up learning over lots and lots of objects, should wind up learning that representation. And so we can use that representation, even though these individual values that show up for any object are not all that interpretable, to be able to do downstream tasks with that, to be able to do things like classification. And we were able to do this self-supervised feature learning for the first time in astronomy that allowed us to leverage not just the 810 objects that we have to do training of our network, but a much, much larger corpus of unlabeled objects. And because of that, we're able to achieve state of the art.
And what's exciting is that this idea of compression using autoencoders is not new to astronomy now and is being used in lots of other fields. But unlike other fields, we also know a lot of the physics of the objects that we care about. We know, for instance, if we have periodic variables, that after they've gone through one whole cycle, they come back to the same flux value that they had before, and they repeat. And so one of the things we're interested in doing is finding ways that we can build our machine learning networks to be aware of the physics that we're aware of, so that we can do better learning and the predictions that we make are physically motivated and are physically valid.
In computer vision, you can imagine that if I'm trying to take an image of something in front of me and I want to make an inference about what it is, if I rotate that thing in front of me, I should get the same answer. And there are ways of actually baking into the way in which we learn and the architectures of these neural networks, as part of machine learning, that allow us to guarantee that we get the exact same answer regardless of the rotation. This is happening in high-energy physics and quantum chemistry. There was a postdoc up at the lab named Tess Smidt, who is working on so-called Euclidean neural networks, where she's trying to predict for complex molecules what the interatomic forces are. And clearly, if I rotate that atom by some arbitrary set of angles, those forces are still identical. Their vectors have changed, but the exact forces are identical. So people have been trying to figure out ways to build our knowledge of the symmetries and the physical properties of our systems into machine learning.
So this is really where ML meets physics. So we're trying to find architectures in machine learning that allow us to conform to the known taxonomies and conservation laws and symmetries. In the interest of time, I think I'm going to skip over this one thing that I did with my student along these lines. Needless to say, we were able to find a symmetry in the way in which we do machine learning on variable stars, and we were able to achieve state of the art.
Now, one of the critical things that I wanted to say, which maybe wasn't apparent throughout the talk, but I wanted to make explicit, was that I, for one, don't like to do machine learning just because it's fun. It is fun, but it's actually quite painful and hard, and to do it right requires a lot of computation and a lot of careful and expert knowledge. We do machine learning in astronomy because we're trying to get at new science, trying to push the envelope in astronomy. And when we can produce these large catalogs of classification over variable stars, our endpoint is not a published paper on that, although that's maybe one of our milestones along the path. It's to say, now that we have this, what can we do with it?
And one of the things that I was very proud of — as we build these catalogs of probabilistic objects, we're able to start querying it and start asking interesting questions and start getting more data. We were able, for instance, to find an extremely rare class of so-called R Cor Bor/DY Per stars. Taking spectra of just our 20 top candidates, we found eight new discoveries, which tripled the number of these known stars in our own galaxy, just because we had made this catalog. One of these stars was as bright as what would have been seen by the Babylonians. So this is not like this is some super-faint star. This has been around for a while, and nobody noticed it was one of these weirdo type of stars. We're able to find the masses and radii of isolated stars in detached, highly eccentric eclipsing binaries, which was extremely interesting. And on and on and on, these have become these really great starting points for us to do a lot of novel astronomy because we were able to do some great ML.
Now, state of the art for us isn't just taking light curves that I showed you before, but now getting spectra and getting other sorts of metadata and combining what's called a multimodal data set in a self-supervised context into a large-scale machine learning model that allows us to localize objects in a smaller-dimensional space. And this is also turning out to be tremendously exciting. What you see here is a projection of our bottleneck space, essentially, into only two dimensions, just so us humans can look at it and try to parse it. And it's been colorized by the 10 or so different classes that were known from a different catalog of these objects.
What's amazing is we didn't tell the machine learning model about these classes. It wound up discovering that all these things called SRs are all clustered in the same place. And it also shows us that other types of objects — in this case M; it stands for a Mira variable — are also really, really close to each other in this sort of abstract representation space. And those that know variable star astronomy can start to look at these different objects, and it starts to make sense why they're actually put close to each other. And it's actually remarkable that we can effectively rediscover classes.
It might be a little hard to see on your screen, but there's a little sort of subclass right here of other types of Miras that are here and here. These have just been labeled in our data set as Miras, but when we looked more closely at it, it turns out this is a known subclass of Miras, and this is the other dominant subclass of Miras. And likewise here, these are two different types of rotational variables. They very cleanly separate out into two clusters. And then when you look at their spectra and you know something about the physics of them, it turns out that physics is actually different. And somehow these models have started to learn some of this physics, which has been really remarkable.
You might also notice that there are some colors here that are showing up in places where they probably shouldn't. And a lot of these turn out to be mislabeled, incorrect classifications that we're able to just identify and pop out. And then there's some other objects that are kind of odd and weird, ones that don't appear in the locus of any of these classes. And these are the ones that we're now studying and following up and asking questions: how different are you, and why are you weird? So this is just kind of a new playground for us as we've started making these models with this new data.
I know I'm running a bit short on time. So I wanted to highlight one of the exciting things that's happened more recently in my group, which is a place where we learned something fundamentally new about the universe, not just found new objects like we've seen before, using something called simulation-based inference. And this was done in the context of so-called microlensing for exoplanets. Those that know a lot about exoplanets know there's lots of different ways to discover them. This is one of the more innovative and clever ways that I know of, which is when you have a star that moves in front of a background star. Because of general relativity, the mass of the foreground star winds up bending the light of the background star and causing a temporary magnification that you see here. If it's a star just by itself, you get a nice symmetric curve. But if it has a planet that is dragging along with it, in a solar system, it causes a little perturbation that's in principle detectable.
And this is great, but when we see one of these events, we need quite a lot of computation to be able to discover and uncover what the mass of the planet is and its distance, for instance, from its parent star. So if we want to study lots of planets en masse, we want to find a whole bunch of these different types of events. And this is very timely, because there's a new satellite which is launching very soon that's going to be pushing our ability to do the discovery of planets in the Earth-mass regime, in sort of the Earth orbital radius, in a way that we haven't been able to do from the ground. All these other dots you see here are the discovery of exoplanets from other sorts of techniques. Direct imaging is only the very big, bright ones that are far away from their host galaxy, and then there's other types of techniques that have been used. But now we're about to open up this whole space using these new satellites.
The problem is we're expecting thousands of these events per year, and for us to be able to do the kinds of complex inference techniques to figure out the distribution of the masses of these exoplanets, it really calls for a lot more computation than is really feasible. And so as we were thinking about this problem, my student and I started asking: could we come up with a way to take those light curves that you saw of these microlensing events and directly infer on the masses and the distances of the planets from their host stars? And so I won't go into the details of the actual method that we used here. Suffice it to say that we train on a lot of simulated data, and that allowed us to be able to get something like a 100,000-times speedup in our ability to make measurements of the parameters that we care about if we're handed one of these objects.
And one of the things that we wanted to make sure worked through this whole process was our ability to recover two types of so-called degeneracies. And this is where you have light curves that look very similar, but they have very different interpretations of where the planet is relative to its host star. There's the so-called inner-outer degeneracy and the so-called close-wide degeneracy. I won't go into all the details of the configuration here, but what we won't know — because it's a mathematical degeneracy in the way in which we solve the general relativistic equations — is, if we see one type of light curve, whether the planet is at this distance or this distance. And we won't know, in the other type of degeneracy, if the planet is here or here. And so it's not a big deal if we get this one wrong, but it starts to be a big deal if we get these wrong here. And so one of the things we want to make sure is that our ability to make measurements of these important parameters still showed these two types of degeneracies. And indeed, they did. In the mass degeneracy and the distance to the host star, our system was able to show that if you're given one of these light curves, you can't tell the difference between whether the planet is close or far from the host star.
So we're really excited that we had a technique that looked like it was going to work. And then my student started playing around with some of this data and the new machine learning model we had. And because we're able to do inferences really quickly, he started seeing weird types of degeneracies that should manifest as many different modes in these posterior spaces. And we got excited about it. First, we thought it was wrong. Then we realized it might actually be a new type of degeneracy that we just happened to uncover. We wound up coming up with a very simple heuristic equation that described this degeneracy. And if you believe what we were saying, it says that the two known degeneracies, the inner-outer and the close-wide, are actually limiting edge cases of something we're now calling the offset degeneracy, which is a ubiquitous degeneracy. And we just saw this manifest in the model. It appears to be right, but we don't know why. And when we look back at old data, it turns out that we were able to better describe the old data than the original papers could, just using the two sort of antiquated degeneracy formalisms. And so we're extremely excited about that.
We published that paper in Nature Astronomy a few years back. My student and our collaborator then did the hard work to see if they could uncover why this thing was, and if this thing was actually real. And it turns out it is: this degeneracy that we discovered using machine learning is actually deeply embedded in the math of the so-called gravitational lens equation, and it had really been hiding in plain sight for the last 50 years. And so the person that wrote the News and Views piece about our paper said that while AI is unlikely to replace scientists in the foreseeable future, this work demonstrates that it can be harnessed to help us understand deeper mathematical patterns in the underlying theory. And this, I think, is one of the more important milestones in astronomy and AI, because it's not just about using AI to accelerate discovery of the known knowns, or even help us find anomalies of actual objects. This is now giving us a deeper theoretical view of how the universe works. And we really stumbled upon it, and obviously we're tremendously excited about it, but it was something that we really couldn't have done if we hadn't done these AI models.
And maybe, if it's okay, if I can take another three or four minutes, I just wanted to briefly highlight a couple really interesting ideas in the downstream area. And this uses the previous result as its point of departure. And the idea here is that, as Garry Kasparov, the famous chess player, said, the best chess players in the world are not going to be humans or computers. It's going to be humans plus machines. And it's that notion of where we may go in astronomy that gets me tremendously excited. I've been building a software system to allow astronomers to interact with new types of supernovae, new variable stars, et cetera. But it's a lot of data that's flowing through, and we've been using machine learning in various parts of it. But we wanted to figure out ways in which we could use machine learning and AI to help reduce the cognitive load for people to try to remember all the things that are happening in the sky at once that they have to act upon, and to maybe even start using AI to do some self-guided observations and hypothesis testing through this very large database.
So one of the things that we released: I think it's the first kind of use of large language models that I know of in astronomy, where we take all the data for one object, and instead of somebody having to assimilate all that for the first time, they can just read a very short summary of that object, which was generated from the input to, effectively, ChatGPT of all the data we have, asking it to summarize for us. We can also, based on that summary and based on a so-called embedding of that summary, ask for objects that have similar summaries, and also allow people to make queries on the data. So that is, I think, sort of Wild West. It's sort of new for us. We're extremely excited to see where that goes.
Upstream, it's one of the things I'm also very excited about. It's the role of AI before data is acquired. Here's an example where we have very raw data that we want to process. We oftentimes get images from the Hubble Space Telescope that have lots of scruff on them. These are so-called cosmic rays. These are charged particles that have hit our detector. And we want to find those cosmic rays. And then we want to mask them out and make beautiful pictures of what we would have seen if there hadn't been cosmic rays.
And these are the types of things that we've been able to do with the method that we built. We take the thing on the left and, just out of the box, produce a picture of all the galaxies that are there. And this achieved state of the art over all the other techniques of how people were able to process data to find these cosmic rays. I won't go into the nitty-gritty of this. But again, suffice it to say, we're able to use this technique to improve upon the previous state of the art, not just in its quality, but also in the speed at which we could find these cosmic rays. And interestingly, the way to do this — this is the old way of doing this — is just to do a Laplacian transform over the whole image. One of the layers that was discovered by our neural net looked a whole lot like a Laplace transform, which got us pretty excited that this kind of computer vision in a box was working extremely well and doing the things that human-guided intuition would tell us it shouldn't.
So that's something we've done, but something we haven't done yet is to think about adaptive optics. What you see here is a blurry image of the center of our galaxy, and one that's been de-blurred by using deformable mirrors that operate in real time to take out the constantly changing atmospheric blurring, and to try to sharpen up those images. These are very complex systems that are perhaps most easily understood as light coming in from the sky. And we have this deformable mirror, which tries to fix the effects of the atmosphere and then shoot it down to the thing that we're actually going to observe. But for it to do this kind of deformation, it needs to pick off a little piece of the light. We need to do a whole bunch of math and figure out what to change on these little deformable mirrors.
This is where I think reinforcement learning may be very interesting, and we started up some projects both here at Berkeley and with our collaborators in other places, where now we can think of this whole problem not in traditional control theory terms, but using reinforcement learning, where we can build a system that says, “I want to measure all the things that I can measure, and I want to make a decision about what to piston on my mirror.” And so we're trying to couch this problem in reinforcement learning terms. And it's incredibly interesting from a CS perspective, because this action space isn't as complex as even self-driving cars. Self-driving cars — there's really only a few things you can do in a car. You can hit the brake, you can hit the accelerator, you can turn left or right. Here we have action for every single location on this deformable mirror, and that could be of order 10,000 actions that we want to take every hundredth or thousandth of a second. And so it's a very, very hard and interesting problem, and it's something that we're just starting up, but I think we'll be able to get a lot of traction on.
And there's plenty of other places where we have very complex machines that we want to be able to control in new ways. And I have a student, my first-year graduate student, actually working with the Vera Rubin LSST system, in their active optics system, to try to see if we can improve upon the ways in which they use machine learning to get better, more clear images. And if that's the case, then we'll be able to deploy something onto what is a billion-dollar facility and make the science that much better for everybody.
There are other things that I'm also excited about. I don't think I'll go into all these, and I'll wrap up here. But one thing that's very clear, and it seems like very low-hanging fruit to me, is that while astronomers still go to telescopes, and sometimes still go outside and look at the beautiful night sky, we're often looking at satellite images of what's coming at us. And we're also looking at the horizon and looking for clouds. And based on those clouds and based on what's coming at us, we will change the way in which we decide to observe the sky over the course of the night. But right now, machines are kind of blind to all this. We don't actually give them the ability to make predictions about what's happening in the sky. And so I think by taking satellite data and these fisheye images from observatories and training on them, we'll be able to predict what the transmission is as a function of position on the sky, so that we can allow these large-scale surveys to be able to do better science at higher efficiency.
I'll skip over this problem here and just get to my summary. In the midstream, ML is already very central to astrophysical discovery and inference at scale. And I'm getting more and more excited about self- and unsupervised, and some so-called semi-supervised, approaches that are ways in which we can get through what we call the small label problem, where we don't have a lot of objects that have been classified, but we still want to work with a large amount of data for us to get the best answers. I'm interested in those downstream ideas, and also the upstream ones, where we can take better data and even inform the way in which our telescopes take data without us in the real-time loop, by doing the kind of weather sensing that I noted.
The last thing I'll just say, and then we can go into Q&A, apropos of where we are today and what's upon us for this next week, is something from Oscar Wilde, who said, “We are all in the gutter, but some of us are looking at the stars.” So with that, I'll say thank you.