Faculty talk in the opening session on time-domain astronomical data and anomaly detection — finding rare and novel transients and variables in large survey data streams with machine learning.
Day within June 5-6 approximate.
“The onus on astronomers when working with data in the context of anomalies is not for us to write papers about how to find anomalies, but to actually find them.” – Joshua Bloom
“If we're cursed by dimensionality, we're really damned when we have real data.” – Joshua Bloom
“We don't really care about anomalies. What we care about is uncovering new physics, and anomalies are just really a stepping stone, a means to an end.” – Joshua Bloom
“Way out into the distant future, like in 2027, we're probably going to be in this world where this is the query that we all learn as grad students: please develop a plausible physical model using MHD, nuclear astrophysics, GR, that explains all the data we have on this collection of sources.” – Joshua Bloom
Thanks everyone. So I think it should be obvious by now that the onus on astronomers when working with data in the context of anomalies is not for us to write papers about how to find anomalies, but to actually find them and then show that the physics of the objects that we are identifying as anomalous from the data are actually different than some common objects that we already know about. And so it's beyond the methodology. And when we actually go from applying these methodologies to data, you wind up seeing just how absolutely hard it is to be working in the anomaly space in astronomy. So my talk hopefully will give some context to that.
This is one of the ways that we can think about the explosive universe in the time domain. On the x-axis is time, on the y-axis is terrible astronomer units for brightness. And the stuff in the middle, type 1a supernovae, type 2P supernovae, very common. You heard from Ashley, once every second or two in the universe these things happen. They're very interesting for some types of astrophysics. People have made a cottage industry out of finding these things and studying these things, but they're very boring because they happen all the time. This is the haystack that people talk about when they say we're going to classify the haystack of transients so that we can get rid of it and look for all the needles.
One way to think about those needles is physically driven models and theories of interesting objects that we haven't seen yet. This slide was made before 2017, when the actual first kilonovae were identified — in this very short and very faint transient, neutron star, neutron star mergers produced this little blip of light. But these other things that you see up here are theoretical, have not yet been identified. And it's pretty clear based on just the light curves alone that when they happen, they're going to hit you over the head, right? So if we're able to build filters against the things you already know about, when these things happen, if they even happen in nature, we will find them. By definition in the Rumsfeldian sense, what we don't know and haven't thought about we can't even put up on this slide, but that is the underlying current of the things that people are thinking about when they are trying to work in the anomaly space.
More commonly found — and this is what Ashley alluded to in things like the changing-look AGN — is that what happens is we see these light curves, they look very similar to objects that we've already seen before, but then when we get follow-up data at other wavelengths or we take spectra, we wind up noticing that they're actually somewhat different, and then upon lots of digging wind up realizing that there's new physics. Some examples of new subclasses: we think things look like gamma-ray bursts, but then if you dig in a little bit more and get some other data, they appear to be something completely different in terms of their origins. Relativistic tidal disruption events, for the astronomers in the room, may be one example of that. And then there's also other types of weird supernovae that look one way in their spectra early on and then they transition into another type of spectra. And there's lots of different types of examples of that. But what's very clear is that for us to have a full complete picture of the physics of these anomalies, we often need to get follow-up that doesn't just come from one telescope in a few bands, but we need to get this at multiple wavelengths. And we need multimodal observations to be able to truly understand if we are looking at something novel.
So in my group we've been working with multimodal foundation models for a lot of different reasons, but I'm going to index here on what we've been able to do in the context of anomalies. This is a bit of a complex slide, but the idea is that we have a bunch of different modes. So we've got time series, we've got spectra, we've got metadata. They each have their own encoding. And then through a contrastive learning process, we wind up comparing these different modes to each other and wind up forcing the embedding spaces of all these to be the same, which allows us, as we learn on a very large unsupervised dataset, to be able to build up this kind of representation, a learned representation across all these different modes, to give us a more holistic view of these different objects. And then when you compress this 512-dimensional space — just for visualization purposes, not for analysis purposes — into two dimensions using something like [UMAP], you wind up seeing very naturally, even though we didn't give the different classes of these objects to the learning process, they very naturally wind up clustering based on their actual known or believed-to-be classes.
And what's kind of exciting here is not only are we getting that kind of clustering very naturally — so we're effectively learning in the representation space that there are these different clusters — is that we have found, somewhat surprisingly to us, there were two different clusters just in this two-dimensional space of Mira variables, even though in the original training data these were only labeled as just Miras. But then upon inspection we saw that those two different clusters very naturally spectroscopically split up to two known subclusters of Miras. So this is kind of a nice demonstration that in a very unsupervised way, and just even visually going in, not even with clustering methods, we can go in and start finding some really interesting new subclasses. So that's kind of a nice demonstration of that.
But what's also kind of interesting is that we can look for those point anomalies in places where, for instance, we have an object that shows up in an embedding space that's not where it's supposed to be according to these learned representations. And these could be objects that are incorrectly classified. Usually it's because they have things like their periods are wrong, or actually the classification catalog that we used itself was wrong, in a few cases. Or we have things that are actually correctly classified but they're weird, maybe in one of the modes. And so we have examples of those, of what we call in-class outliers, where we have a kind of normal-looking light curve in an eclipsing binary system, but then we have something anomalous where we have a whopping line there that's usually not there in most eclipsing systems. There's some sort of accretion disc process. This has X-ray associated with it, which was not part of the original data. And this is kind of a known subclass of detached eclipsing binaries that are transferring mass and producing these X-ray lines. So that's kind of a nice proof of concept as well.
But the thing we got really excited about is because we have this embedding space, we can do searches in, let's say, cosine distance of everything to everything else. And we wind up seeing this really interesting stellar-like spectrum with a massive absorption line right here. And I got really excited about this because I had no idea how to do this physically. And we got really into the idea that this may be some sort of interesting accretion disc. And then we wound up querying to find other objects in just the spectral dimension that looked like it. And there were 10 or 20 or 30 of these things, all with different light curves. And we wound up concluding in the end that what we had was a bad spectral reduction. And so, before we wrote the Nature paper, for those that remember this movie, instead of bad dates it's just bad data. And so the anomaly detection worked, that's a good thing. The problem is it wound up showing us some bad stuff in the data. And this is in a very large dataset. I don't know what went wrong in that reduction.
But this is going to happen. This is going to happen a lot. And it makes sense that we should start thinking about other ways to be finding these anomalies. In part because, as you start thinking about the curse of dimensionality, when we have these large-dimensional representation spaces, which we kind of need to do when we're looking at these complex physical objects, we're not just looking at four or five numbers that can adequately represent one of these modes. We're going to need dozens, maybe even hundreds. In that context, the distance between the nearest points in a very large-dimensional space winds up getting large, but the difference between the largest distance and the smallest one is actually getting smaller and smaller. And so what this says is that over time, in these representation spaces, if they have to be large to describe the data, one of the challenges that we have is that it's just going to be harder and harder using clustering techniques to find these outliers.
And so one idea is to use not the representation space, but use the representation alongside classification, to be able to come up with distance metrics. This is something we developed in 2012 actually, in the context of random forests, but it can be done with other types of learned representations, where very simply we're just asking, at least in this context, as we have a new object and we query a new object that's unlabeled, where does it show up in all of the different trees in the random forest relative to other types of objects. It's different than isolation forest because it makes good use of the classification. This avoids these kind of distance approaches in the feature space, and it is semi-supervised in that it learns from the actual labels. So we're actually imbuing some of our knowledge of the physics of the systems that we're looking at, to be able to get this distance metric.
And this also worked. And we had an undergrad in 2012 looking at our best anomalies, and she found this amazing object which looks kind of like a Cepheid that's extremely long period, 250 days. And we had emails back and forth saying, well, should we call Leslie Sage at Nature, is this something that we should get excited about? Adam Miller, my student, said, “This looks like a Cepheid, but the period is way too long.” And it turns out there were a bunch of these things in our dataset. We got super excited. And then we're like, we should probably look at the images. And indeed, there's like a big red Mira right in the middle there. But then we realized the photometry we had was from a large aperture. And what was happening is these Mira variables, which are largely sinusoidal, go up and down. When they went too far below, the light from these other two stars here basically dominated inside the aperture. And so we found an amazing set of anomalies. It just turns out it was bad data. So you see the theme here. Importantly, to emphasize, is that domain experts had to be in the real-time loop before we actually started going farther down the road. And it's that kind of imprecise and bad data, that you didn't usually take yourself but somebody else took for you and put in a database, that becomes a major bugaboo as we try to actually find new physics.
And so if we're cursed by dimensionality, we're really damned when we have real data, right? So if we're getting just to the edges of our very large-dimensional volume very naturally, if we now have noise and improperly characterized noise, we're going to always find these objects that are beyond the edges of what we've already seen before. And this is a real problem for us. I think Tolstoy said it best: all data anomalies are anomalous in their own way. And for those that have looked at amazingly important events, there was a massive glitch right as the most important gravitational wave event of our life happened. The fine guidance sensor on Hubble fails sometimes and produces beautiful images, and then unfortunately we have to contend with satellites that streak through our images. When you look at data you find a lot of anomalies. And so the question now is, can we start thinking about embracing anomaly detection to operate our telescopes better and produce better-quality data downstream for the end users? And so maybe we should be thinking not so much about anomalies on this sort of final resting place of data that's been carefully curated for us in databases, but doing it upstream, running it on the telemetry of logs from these telescopes, running it on raw science images to catch those kind of weirdos, and to find the distribution shifts that those that are running these large facilities can identify before they wind up putting it into a database and cause people to write papers they shouldn't be writing.
So it should also be clear, in just the last minute or two that I have, that we don't really care about anomalies. What we care about is uncovering new physics, and anomalies are just really a stepping stone, a means to an end. And so if we think about using algorithms on large databases to find anomalies, what we should be thinking about that as is a stepping stone in a much larger data flow, where we can do maybe a better job upstream. But downstream from that, we might also be thinking about ways that we can help ourselves and protect ourselves against going down some path. So how do we automate novel physical insight on this kind of autonomous scoring that we're going to be building? Well, one could be to build in these physical gut checks. So we know there's things like brightness temperature limits and Eddington limits and things like that. So maybe we have some crude physics that we fit to all of our anomalies. Or maybe we build models with inductive bias from the beginning, so our representations don't allow us to go off into La La Land. Or maybe we use simulation-based inference on some of these things, where we have good physical models that are expensive to run on lots of data but we can do it in a surrogate quick way.
So that's where we should be now. We're getting to the point in LLMs where maybe we're going to start asking agentic Occam's-razor questions like: here's some class of objects that I'm excited about, can you go off and look at the literature for me and tell me what these things might be, more mundane or otherwise? I'm assuming that there's something wrong with the data, but let's figure it out. Rank-order the ones that might actually be truly physical. This could be one of the filtering techniques that we'll get to very quickly. And then, way out into the distant future, like in 2027, we're probably going to be in this world where this is the query that we all learn as grad students: please develop a plausible physical model using MHD, nuclear astrophysics, GR, that explains all the data we have on this collection of sources. What other data should we get to test that theory? And by the way, just go get the data. So with that, I'll put up my summary, and happy to take questions. [Applause]
QUESTION: Josh, I liked your curse-of-dimensionality slide. Is there a way to work around that with dimension reduction followed by nearest neighbor and tessellation methods?
BLOOM: It is a good question. I think the way we have been working around it, and why anomaly detection has actually been doing okay, is because some of the things that we look at and we care about en masse can actually be reduced down to just a few dimensions, and then we're only working in 10 or 20 or 30 dimensions. But my supposition here is that as we start knocking off most of the objects that we're going to be seeing in LSST Rubin, for instance, we're going to have to get to more and more subtle physics, which means for us to create distinguishing characteristics, distinguishing representations, we're going to naturally be forced to have larger numbers of dimensions. But yeah, for sure we should be thinking about creating sparsity and rewarding sparsity, even at the loss-function level, in the creation of our representations. Because we just picked 512 for our thing, but maybe we would have done just fine with 128, and then you start being able to do better and better with the clustering.
QUESTION: Hi, I have a question regarding the anomaly definition. I just wonder whether, because of these many variables, we could end up with — for each, I just pick one star from the sky and I can find a definition, make it weird — and with this, a concern of how we can avoid this situation.
BLOOM: Yeah. I mean, I think that gets to that dimensionality problem, right? Everyone in this room is like the most amazing person in the universe in some axis, and let's pick that axis very carefully. And yet at the same time, we all can be well characterized by a smaller number of parameters and approximated, not with an LLM, but something that's kind of finite with a finite number of bits. So it is a problem. It gets to that dimensionality issue, but fundamentally in the end it gets to whether that star represents new physics, right? And yes, that star is going to be weird in some very special way. But does that matter, right? And does that matter question gets to: does it represent physics that we think is fundamentally different, or an improvement upon the physics that we already know? And for most of the cases the answer is going to be no, it doesn't. It's just sort of a stochastic implementation of something that we already know, those processes. And so you're kind of hinting at one of the biggest challenges, which is that if everything can be anomalous if we slice it the right way, then how do we ensure that if it truly is anomalous it also maps to truly new physics? Okay.
MODERATOR: Let's thank Josh again.