Research Exchange lecture on taking the astronomer out of the real-time loop: robotic telescopes that talk to each other over VOEvent, discovery engines like the Palomar Transient Factory that will soon outrun follow-up resources, and machine-learned real-bogus scoring and classification of transients and variable stars.
A noon talk in CITRIS's Research Exchange series (Sutardja Dai Hall), introduced by Masoud Nikravesh, CITRIS's Director for Computational Science and Engineering, and webcast to the other CITRIS campuses. The title shown is descriptive: the announced title was not preserved, and the slide deck survives only as an empty file. CITRIS's recording is linked above.
“You have to look at this data and realize what you're seeing is something moving, and what that thing is is actually a very important object. So taking the data doesn't mean that you actually understand the data.” – Joshua Bloom
“It used to be said that people would go up into these telescopes with a thermos full of coffee, and they'd ride around in these telescopes all night, and they'd come back down with a thermos full of something else. That doesn't happen anymore.” – Joshua Bloom
“All of a sudden it inverts the entire problem we had: where before we had a lot of follow-up resources and not a lot of discoveries, all of a sudden we're going to have a whole lot of discoveries and not enough follow-up resources to go after them. So we have to be even more clever about how we actually use our follow-up resources.” – Joshua Bloom
“Without telling these off-the-shelf classifiers that these objects are actually related to each other, in some sense, those classifiers are telling us that those objects are related to each other.” – Joshua Bloom
“What we're doing, really, is taking the traditional role of astronomers as being intimately involved in the data collection and the inferences, and we're trying to remove the astronomer from that in a real-time sense.” – Joshua Bloom
NIKRAVESH: This is Masoud Nikravesh. I'm the Director for Computational Science and Engineering at CITRIS. I would like to welcome you. First, I would like to thank Infineon for sponsoring our research series lectures and also the luncheon. I would like also to welcome our viewers at the other three CITRIS campuses: UC Merced, UC Davis, and UC Santa Cruz.
And a few announcements. We'll have the Cal Day on April 17 — that will be a talk with Greg Niemeyer at 1 p.m. here, at the community sciences. And then also on April 22, they have a poster and judging for the Big Ideas of the students.
Today, it's a great pleasure to have Josh Bloom, our faculty at the astrophysics group, astronomy. Josh graduated from Caltech with a PhD. Prior to that, he was at Harvard and also Cambridge University. He spent three years at Harvard as a fellow there. He is the PI of the largest infrared telescope, and I think they are published over hundred of the refereed journal papers and others. But there are a few significant things that I would like to mention today. So he finally finishes a new book on gamma-ray bursts, which is the area of his expertise. But also, recently they discovered the farthest, or the far distance, star from our Earth. It happened 600 million years ago. Apparently, no. It took 13 billion years that that signal comes to the Earth, but it happens on 600 million years, when the Earth was 600 million years old.
Josh also, he's the PI for the CDI NSF project that is actually focusing on the streaming of the largest-scale astrophysical data and also looking for the transient sources — one of the challenging programs. Today he's going to give us an overview of some of these projects, and it is a great pleasure to have him here. Josh?
BLOOM: Great, thank you very much. Well, what you're looking at, of course, is a composite of Galileo's notebook from observations that he took basically 400 years ago today, plus or minus a couple of weeks. It's a revolutionary set of data, and it shows, of course, the discovery of the large moons around Jupiter, which until that time were not known. I show it here today because I wanted to emphasize a couple of points, which I think this does very well for me. One is that using new technology — in this case, using a telescope and, instead of pointing that way, pointing that way — can herald revolutionary discoveries that change our worldview.
And in this sense, if you'll remember back to your history books, the Ptolemaic worldview — that the universe was fixed, and these perfect orbs which we called planets were indeed perfect without any blemishes — certainly there was no room within that worldview for moons which orbited these perfect orbs. And so, although there was some evidence before this time that this worldview might not be correct, this is one of the nails in the coffin against that old worldview, and foisted in a modern view of how we see ourselves in the universe today.
The second thing to note, of course, is that this is a time-domain observation. That is, the data were taken in time. And it's the motion in space that actually led to the inferences of what these things actually were. Just getting one snapshot, you might be convinced, as other people had been in the past, perhaps, that what they were looking at is just stars. But instead, they saw these — what looked like stars — basically moving around this planet. Having the time domain actually helps a great deal and is actually crucial in astronomy.
And the last is — and this is a bit more of a philosophic point — that this picture illustrates the crucial role that humans have in discovery and have had in discovery. Not only in taking the data, which of course Galileo did, and he preserved and recorded that data to great accuracy that we obviously look at today and marvel, but it's the collection of the data, it's the inference about that data, and it's the deep insights about the universe that come from that data, have always had humans in the loop.
So now, 400 years later, this nature of astronomical discovery and inquiry is only now just undergoing a fairly substantial phase transition. Not only do the objects of interest require larger and more sophisticated telescopes — it's the dynamic universe, it's the changing universe, that sometimes demands reactions by us humans and the resources that we have at our disposal on timescales that are faster than we can actually collect information and respond if it was just us in the loop. So if we couple that now with all the massive data sets and massive data rates that are now coming online, it becomes very clear that traditional role of astronomers is changing too, by necessity.
What's not really changing, however, is this pathway to science. Getting the data, or just observing the data with whatever instruments you have at your disposal, and then finding, say, individual objects, say, in an image, and recording the properties of those objects — where they are on the sky, what their brightness is, et cetera — then leads, if there is a keen mind looking at that data and recognizing how that data perhaps fits in or doesn't fit into a current paradigm, to discovery.
Discovery, say, of a new object — say, one of the Jovian moons or some of the other types of objects that we'll be talking about in a minute — then isn't the end, of course. It's saying, I now have something new and I'd like to know a lot more about it. Maybe I only have an image in one filter and I want to see what color it is. Maybe I want to disperse the light and get a spectrum of it and see what particular elements might be in that object. That's what we call follow-up, and that's where we basically bring more resources to bear on the thing that we wound up discovering. And then after that, after the follow-up, you take the totality of the information — and obviously you're helped by the theory of the types of objects that you're looking at — and you create some sort of scientific inference.
What's obviously very interesting, of course, is where discovery happens, when discovery happens, and how it happens, and how we then use those discoveries to decide what sort of resource allocation we want to bring to bear on the problem. Before getting into the hardware, the software, and some of the new technologies that we've all been thinking about, I thought it would be useful to introduce you all to what I call the dynamic universe. This is the changing universe. This is things that move on the sky and things that go bump in the night.
Here is perhaps the most important three-frame movie you'll ever see in your life. You may not know that now. You may not know why, but I'll hopefully press upon you that that's true. Here's this little object here, which appears to be moving. This is data taken, I think, something like 60 seconds apart over the course of a few hours, and you wind up seeing this little object moving with respect to all these background stars. You can also see that there's some noise in the image and things like that. Does anyone have any idea what that object might be, for the non-astronomers in the room? It's not an asteroid. I heard a planet. Sort of. This is the discovery images of Eris, the object which is now called a dwarf planet, which is larger than Pluto and caused all this brouhaha over the last couple of years that caused all these kindergartners to cry at nighttime because Pluto wasn't a planet anymore. See, it is important.
So motion in space is important. There's another much more slowly moving object, which was just announced basically last year, year before, called Fomalhaut b. It moved by a small amount — a tiny fraction of, say, what the full moon subtends on the sky — over the course of just a few years. And these are images taken with the Hubble Space Telescope — it looks like you're actually looking at things twice, but we actually have an image overlaid with another image taken in 2006. And you see that from 2004 to 2006 this little thing — which is moving around a very, very bright star, which is off screen here, and you can't see it — is actually the first image of an extrasolar planet ever obtained.
Just acquiring this data — and this is one of the important points I wanted to tease out here — just acquiring this data and looking at it doesn't inhere any discovery. You have to look at this data and realize what you're seeing is something moving, and what that thing is is actually a very important object. So taking the data doesn't mean that you actually understand the data.
In addition to motion in space, this dynamic universe also means changing of brightnesses. Here is what astronomers call light curves, taken over the course of a few years in four different filters. So you have colors that you basically put in — or filters you put in front of your detector — and you take a bunch of data. And you see that this source appears in any one of these colors to be essentially constant over several years. But there were two times, or what we say, two epochs, when it looks like that source got fainter by a little bit in all these different filters. And this is one of the few objects in a set of something like 20,000 that we searched that had this property.
These are what are called eclipsing M-dwarf binaries, and these are very low-mass stars, and the utility of finding such things in data that was taken not for the purpose of finding such things, but for the purpose of something else, was that we then could apply lots of resources to understand what these objects were and get some fundamental properties of these things. In particular, we were able to determine the mass of this, now in units of the mass of the Sun, and the radius of these objects in units of the Sun as well. And this whole space — these fundamental properties of stars, mass and radius and also temperature — you would think would be well populated by astronomers these days. There are multiple theoretical curves that explain what these different relationships should be. But you can see just with a few data points, you can even start ruling out some of these various theories. This is essentially uncharted territory that got started because of this data-mining exercise.
So here's this example path, in reference to the first slide that I showed you, of an observation and finding of these various objects on the sky. This was taken with the Sloan Digital Sky Survey, the second incarnation of that, on a 2.5 diameter telescope. And there's basically a very small fraction of sky, 250 square degrees, covered in five filters. And the data were taken every two days or so over the course of about four years. That led to something of order 800 million detections of different objects. Then discovery — well, that was basically just brute force, specialized codes that my graduate student, Cullen Blake, wound up writing, inspecting these light curves, figuring out what was spurious data, et cetera — a fairly tedious process. But sifting through that needle in the haystack, we found one of these interesting objects. Then we decided to follow this up with some of the larger telescopes available to us, and that led us to, after a lot of analysis, come up with this inference of what the mass was, the radius, et cetera, et cetera.
In addition to these nearby objects in our own galaxy, of course, as Masoud was hinting at, there are very violent massive explosions in very distant galaxies. Here is what will be a movie of a supernova going off on the outskirts of a very old galaxy, which is something like 10 billion light-years away. This right now is just an artist concept. There's the supernova going off, and now we zoom out, and we actually start looking at real data from the Hubble Space Telescope. We go farther and farther out, just giving you some context to what this needle-in-the-haystack work might be. And while it looked like that supernova got bright and faint over the course of just a few seconds, the typical timescales associated with the rising and the falling of this brightness is of order weeks. So you actually have a while to find some of these things.
In the universe, there is about one supernova every second. But with our current technologies, the state of the art is that we find about a thousand supernovae a year as a human population. It's obviously a very difficult thing to do because you are trying to find brightness changes over the course of the entire sky, and you don't know when one of these things is gonna happen in which galaxy, and it's a significant challenge. Supernovae, of course, are some of the bread and butter of the dynamic universe and the science that we derive from the dynamic universe. There are different types of these supernovae. Obviously, it's not important that you know these types, but they're very common in the sense that we have been able to characterize and classify a bunch of different types of supernovae over the last several decades and put them into classes that help us actually derive different types of science from them.
But there are other types of supernovae, so-called pair-production supernovae, which may be much brighter. The brightness here, by the way, is in a logarithmic scale, with higher brightness up on top and fainter on the bottom. These typical pair-production supernovae are thought to be as bright as the Milky Way — all the light of the Milky Way — for over a year, probably several years. They would be brighter than an entire galaxy. We've never seen one — or at least we're not sure whether we've seen one or not; there's some indications that maybe a few have been seen. But then there's some other things which are just basically theorized. You don't have to know all the details of these things. But these are extragalactic explosions that probably happen. We think we have an understanding from a theoretical standpoint of how they might manifest themselves in visible light.
And the other thing to recognize, of course, is that some of these events, which might be of great interest to astronomers, actually only last for a few days, unlike traditional supernovae. Collisions of neutron stars, for instance, might produce a very short-lived transient event. So finding these objects — again, this needle-in-the-haystack work — and then discovering them, and then also using the available resources to follow them up, becomes even more pressing than it is just through traditional supernova science.
So how are we getting at this dynamic universe, this changing universe? Well, there's a number of different technologies, and there seems to be some confluence of change that's happening pretty much in all the different realms that astronomers have been working with traditionally. Here's a very classic picture of one of the classic astronomers from the 20th century, Edwin Hubble. We might have heard of him before. He has a famous constant named after him. He has a telescope named after him in space. A lot of your tax dollars went to pay to promote his name. He was a classic astronomer, a classic observer in the sense that he spent a lot of time at telescopes looking through eyepieces. In this case, he's sighting where he's going to wind up looking, and then he's going to wind up collecting data with the main telescope down here. He's also very famous for never being photographed without having a pipe in his mouth.
But this also, again, just like the Galileo picture, emphasizes that traditional role of humans in the data-taking process. One of the major improvements from the days of Hubble was the evolution of the technology of how we actually even acquire the data. Forget about making inferences about the data. Here is somebody sitting at the top of a telescope, basically putting in film into the focal plane of that telescope. These are called photographic plates. And in the mid-1980s, there was a revolution in astronomy where we were replacing photographic film with these charge-coupled devices, or CCDs. And remember that the winner of the Nobel Prize in 2009, Boyle and Smith, actually went to the invention of the CCD technology, and is obviously very pervasive in many of the things that we do today.
It's the digital imaging and the digital storage that replaced these photographic plates that basically brought about a tremendous revolution in the way in which we acquire data. It used to be said that people would go up into these telescopes with a thermos full of coffee, and they'd ride around in these telescopes all night, and they'd come back down with a thermos full of something else. That doesn't happen anymore.
In addition to just the technologies that allow us to take the data and store the data, there's also been a bit of a revolution in the relationship that people have to how the data is taken in a given night. Here we are sitting in a control room of a telescope, essentially telling a telescope where to go, how long to sit there, how to take the data — essentially opening up the shutter, closing the shutter, and moving on — inspecting the data in real time. This human operation is the way which astronomers have been operating for years, and only really in the last two decades has this concept of automation or robotization of telescopes started to catch on.
One of the most successful scientifically robotic telescopes is the KAIT telescope, operated by Professor Filippenko here in the astronomy department, at Lick Observatory, just about two hours from here. And this is a supernova discovery machine. It takes data without any humans in the loop in the given night. It opens up the dome at the beginning of the night, closes it at the end of the night, and actually slews around to different places on the sky to take data throughout this night. The autonomous data taking was a major change. Obviously, it becomes a lot less taxing for our poor graduate students, who don't have to stay up all night anymore.
But beyond robotization, there's also this sense — and where we're all heading — of this intelligent data collection. Here I am standing in front of a telescope that I automated as part of my postdoctoral career, and I still operate today. It's called PAIRITEL, the Peters Automated Infrared Imaging Telescope.
And beyond this robotization, we want to have the telescopes do a bunch of things. We want them to not only take the data without any humans in the loop, we want them to figure out actually what they should be observing based on some complex optimization over all the priorities of what people have said they wanted, say, during the daytime. We also want them to be very careful about weather, so that when it starts snowing or raining or locusts come, they will close up. So detailed weather sensing and reactions to the environment is an important component of all this. And perhaps most important in the context of the dynamic universe is these quick reactions. Having no humans in the loop means we can essentially send an email to the telescope. It can start slewing onto a position on the sky and start taking data in a matter of seconds.
And so this allows us to basically have these now intelligent data collection agents. We're somewhat cheeky on our website in saying that we're the robotic transient astronomy for the 21st century. You might scoff at that — that's fine. But we are one of the first telescopes to ever Twitter itself while it's observing. So after it starts observing, it starts sending out Twitter messages. And then people who are following this Twitter stream can actually go and see what the telescope is looking at while the telescope is looking at it. So the hardware — these intelligent robotic agents — are basically now how people think about building new telescopes and retrofitting old telescopes; it has also become in vogue. And it's gotten a bunch of people in the astronomy world very excited about what happens now that we have these intelligent collection agents. These smarter telescopes will show you things you've never seen before. And those of you who haven't seen this down here, you might note this amateur astronomer cowering as this large, very unwieldy-looking telescope comes to take over its livelihood.
So one of the big challenges now — now that we're able to do this automation at the hardware and software level — is how do we get these smart telescopes to talk to each other and even to federate themselves, that is, make sure that they're optimizing what they observe on the sky without any humans in the loop during the course of a night. One of the main problems is that the way in which astronomers announce what they're doing or what they've seen — how they announce discoveries — is nothing much more than a telegram. It's a modern telegram. It comes in the form of basically an email. But you notice that these emails — this is from The Astronomer's Telegram, which is sort of state-of-the-art for how discoveries are announced — have some hypertext markup associated with them. There's some references to other telegrams. And you see it's just written basically for a human to be able to understand. So whenever you start writing in natural language, of course, it means that you're obscuring the underlying data that, say, robots who might want to be parsing these messages might want to respond to certain types of events.
So one of the things that I took upon myself starting in 2004 was to help the international astronomy community come up with a messaging scheme that would allow us to announce and talk about events in a machine-readable way. This is what's now called VOEvent, and has a large group of people now working on this messaging scheme, and has now been vetted by what's called the International Virtual Observatory Alliance. The IVOA is sort of the IEEE of standards in astronomy. And it's nothing more really than an XML message that describes who, what, where, when, and how something happened on the sky. But because we now have this common language that robots can talk, it allows these robots to actually start creating networks. And over the last few years, we've been building up something called the VOEvent network, which is also related to another network called the Heterogeneous Telescope Networks, where we have what looks sort of like the prototype of the Internet itself — these major nodes, these backbone nodes that actually are passing messages around. And this is actually a working network, and it's growing every day. Understanding how to get these robots all talking to each other, how to federate and get better data and make scientific inferences, how to basically do better follow-up, has been a major thrust in the community for the last couple of years.
I don't know why we have these apocalyptic views of robots, but here it is from a conference that I organized about a year ago down in Santa Cruz. The hardware and software for doing this is now coming online, for doing this good follow-up. But as you remember from that first diagram, you can't do good follow-up without good discovery. And until recently, discovery in many ways has been basically the bottleneck. Finding new and interesting objects for this willing and able network to react to has actually been a rarity, and it's actually been fairly difficult.
So where do these discoveries come from? And the very exciting thing about the epoch that we live in now is that it truly is the dawn of the time-domain discovery engines. One such engine that I'm associated with, the Palomar Transient Factory, as it's called, is actually operating on that same telescope that you saw Hubble standing at. He's not there anymore, of course, so we kindly took all the equipment that he had used before out of the way and put in monitor detectors, roboticized the entire telescope, and now we're taking data nightly. The goal of the Palomar Transient Factory is fairly simple but obviously fairly ambitious. It's to characterize the entire transient and variable sky, looking at things that go bump in the night. We have a 100-million-pixel camera that's now mounted on the telescope, and we get something of order — that should say gigabytes a night — a few gigabytes per night.
But this is only the precursor. What comes next is this 800-pound — or 800-ton — gorilla in the room, the so-called Large Synoptic Survey Telescope. It's got a number of partners throughout the United States and is slated to begin at the end of this decade. It will be collecting one gigabyte every two seconds, and it will be finding something like a million supernovae a year, something like 100,000 eclipsing binaries like the ones that I showed you before, and something like 10 million asteroids. It's going to be collecting of order 800 million light curves every three days and updating them. It is truly a behemoth. And this is the thing that we're all looking towards in the future and trying to make sure that we're able to deal with the data rates.
This overwhelming amount of data that we're expecting to get from the LSST — the Large Synoptic Survey Telescope — has got a number of us scared, because all of a sudden it inverts the entire problem we had: where before we had a lot of follow-up resources and not a lot of discoveries, all of a sudden we're going to have a whole lot of discoveries and not enough follow-up resources to go after them. So we have to be even more clever about how we actually use our follow-up resources. This really begs the, I think, fundamental question: how do we do discovery, follow-up, and inference when the data rates — and also the timescales that I mentioned before, requiring very rapid follow-up — preclude any human involvement?
This problem that we're looking towards in just the next couple of years is obviously not unique to astronomy. The Economist just had a big section on the so-called data deluge. And I think this picture is actually very useful in understanding that we're getting just so much data and we're trying to get something very fruitful out of the data we have. And getting something fruitful out of the data we have basically means we have to make very good sense of it and do it very, very quickly.
So how do we handle this data deluge? The approach that we've taken within my collaboration, which we call the Transients Classification Project, is one that will use machines and software and intelligent software to help us with the discovery process. And not only the discovery process, but the process of actually doing scientific inference. Again, it's this abstraction of the traditional role of astronomer into this modern era where humans are not in the real-time loop. There's obviously a number of people, as you can see here, that are involved in this project; those in yellow font are students. Actually, it's a little bit hard to see, but we have a number of students involved here, and this is a growing project. And through Masoud and the CITRIS program, we've been very thankful to get some time from Google, Yahoo, and Amazon on various parallel computing clusters. And what we're trying to do is get all of our codes up and running in a fully parallelized way.
Getting a little bit ahead of myself, I should say the top-level goal of this project is the autonomous creation of new knowledge that itself spurs further follow-up and inquiry. And what we're really trying to do here is take all of this data that's coming from, in some cases, not just one source of a discovery engine, but maybe multiple catalogs that are local or on the web, and generate probabilistic statements about what those things actually are. And that, obviously, for many of us, means classification — putting those into certain buckets of things we know — but it even means putting these various objects which might go bump in the night into buckets of things that we don't know. What do we do if we find an event that we've never seen before? How do we even know that we've never seen it? How do we even discover it?
We want to provide this access to all the people that we're working with to the current events and the past events. We want this framework that we're hoping to build to be able to learn from its mistakes. So as people say, “Now, I just spent a whole lot of my follow-up resources chasing after something you told me was X, but it turned out to be Y, and now I'm mad at you,” then we want the ability for this framework to be able to adapt and change. And perhaps most important, we're hoping this framework is going to be able to operate at sufficient and scalable rate, so that if we triple — or add by something like five or six orders of magnitude — the amount of data that we get on a given night, we're not going to be completely swamped.
Already with the Palomar Transient Factory, we're sifting through gigabytes a night, and as I said before, with LSST, it'll be up to terabytes of imaging data nightly. This is just one frame taken every minute with the Palomar Transient Factory, and you can see the full moon here to scale. We're getting a tremendous amount of the sky in any one go. To give you a sense of how this discovery actually happens, we get in all these different images something like 1.5 million candidates per night — that is, things that appear to be changing, either getting brighter or fainter. Forget about moving for now. And from that, there's probably something about 1,000 bona fide sources. Most of those turn out to be asteroids, about 300 variable stars in that entire haul, and maybe only 10 new transients. So this reduction by something like a factor of 10,000 or 100,000 is obviously a major issue for us, and it's already happening at the gigabyte-a-night rate.
So you can imagine a couple of approaches here. One is you just get 10 to the three undergraduates looking at every single one of these candidates, and make sure you crack the whip, and they tell you what the objects are that appear to be good. Or you can start using machine learning to help you sift through this data and drill down into the most likely candidates. So we've done that. As part of the project, we built what we're hoping will be an open crowdsourcing platform for all scientific research, where you can basically upload a project where you have a bunch of questions that you want to ask a bunch of people, and then get back all the results in database form.
And so what we did is we exposed to the experts in our collaboration some images. Here is a new image of the sky, which is called the postage stamps, just to zoom in around a single object. And here's the reference, that is, a stack of the static sky around that object, and a difference. And you see that this looks like a pretty lousy subtraction. And so the human here has said that this looks like a pretty bogus subtraction. I don't think that looks real. And then you have one here, a subtraction that looks maybe like it's real. Who knows? And here's one that actually turns out to be real. Now, those of you looking at this may say this is a bit of a black art, and in some sense that's the point. There are a number of us who are used to looking at these subtraction images and making these very complex inferences with our neural networks that we have in our brain, and deciding whether this thing looks real or not.
But by marking up a number of these, and because we have metrics on these various objects that we derive from our images — say the total brightness or the total ellipticity of the residual here, et cetera — we can then use just classic machine learning techniques to figure out what the important parameters are and get into the objects that are most likely real. We've created this metric called real-bogus, which is basically a number going from zero to something like one, and we apply that to all of our candidates before any humans even look at any of the data. This is all just done in software. And we get something very interesting and obviously not very surprising: most of what we look at is completely bogus. Most of the apparently changing parts of the sky turn out to be artifacts of the detector, or they turn out to be just poor subtractions. Essentially the software algorithms aren't perfect at doing these subtractions. But we get this long tail of astrophysical good stuff, and that's the stuff we then really start scrutinizing and really try to understand what this thing is. This is basically where the discovery process — noting that this truly is an astrophysical, good object — starts happening.
So we have something of order 10,000 variable stars and transient objects now projected onto the celestial sphere here. This is the location of Andromeda, this red dot, which is circling around. And we have something of order 200 to 250 nights of data that we've been able to collect and get to these 10,000 sources. So this is a fairly major triumph for the Palomar Transient Factory. In some ways, this is just telling us where things are on the sky, where there is a very interesting object that appears to be changing in that part of the sky. But what is it? This is where we now get into classification and scientific inference. And obviously we're now down into the 10,000-object level. You can imagine start having humans back in the loop at this point, but we're interested in going beyond that, because we are thinking ahead to LSST.
So what do you do? Well, it's not just the place in the sky, it's how that object, or that source, is changing as a function of time. We get back to these things called light curves. And here we have four different types of objects, four different classes of objects. And you see the very different time scales here on the x-axis. These are very short eclipsing systems, where the eclipses and the overall period of this binary system is much smaller than a day. Events and objects that are happening, and binary systems that are happening and changing on order of a day. And then we have supernovae, and then we have long-lived variables. And we have this huge classification tree of objects: pulsating stars, eclipsing systems, cataclysmic systems — that is, systems that blow up.
And you would think that all we need to do is just watch basically all these different objects that we found and identified as being interesting or potentially interesting for a long enough period of time to then come up with an inference and just get these very well-sampled light curves. The problem is that with astronomy, we have this problem of day and night, so we can't just be collecting data all the time. That's one thing. But the other thing is we're never collecting data that looks nearly as good as this. We're always collecting data that's noisy and irregularly sampled in time, and that turns out to be an important thing when we start thinking about applying machine learning techniques to understanding what we have.
In addition to that, perhaps even worse, we have spurious data. So these are bad data points, essentially, that creep themselves into these light curves. And in many cases, the event of interest hasn't even happened yet. This is a theoretical fit, this green line, to this event, which is called a microlensing event. And we're really interested in getting lots of data around the peak of that event, but it hasn't happened yet. And we don't know exactly when it's going to wind up happening, so we have to have some predictive power when we actually make these classifications.
Now, as I said, humans are very good at looking at this data and making inferences of it. If you showed this to most experts in the field, they would say this looks like a very good microlensing candidate. And indeed, in the past, when people have done synoptic surveys with far less data in hand, they've been able to build some very nifty web pages, get this sort of visual analytics approach, showing a human, on basically a given web page, all the data that one would need to know to make some inference about what this thing is. Here's some spectra here. This is the equivalent of what I showed you before of these subtractions from the reference and the new, and a whole bunch of other data. So a human could look at this, an expert trained in this could look at this, and then decide what to do next. But this works well when you've got 100 objects a night, maybe 10 objects a night, and you can actually have humans looking at everything. But it clearly doesn't scale to LSST rates, and it's already starting to break down at the rates that we have in the Palomar Transient Factory.
So there is this idea that's been around in our community for the last few years of basically using the citizen scientist, using what's called crowdsourcing, to push out all of these various candidates to the world and ask people to look at this. There's been some great successes in outreach, in having people looking at a number of different galaxies and classifying that. Many of you may have heard of the Galaxy Zoo. There's also something called Stardust@Home. But there's a problem with crowdsourcing in this — not just now, the terabyte level, but beyond, the petascale era — is that there's really two issues here. The more people involved in this sort of crowdsourcing, on getting their opinions about what's happening in that part of the sky, the less their expert opinion can be, right? If you need a million people looking at something, the chances that you'll get a whole bunch of experts start diminishing very rapidly. And the other point is that if you can use just a few humans to train on a training set, and then you can apply machine learning techniques to actually get similar answers — maybe they're not exactly as good, or maybe it gives you a slightly different answer — you should do that.
One of the problems that we've had in the astronomy communities: I think we've been so excited about outreach at some level that we wanted to show the world what we've got in terms of all this great data, and we've been pulling back all this information from them, but it hasn't been, in the end, all that useful. And indeed, as people have started looking at the results, say from the Galaxy Zoo, they've been able to train very simple classifiers to do just as well as millions of people and millions of people-time hours spent on these projects. So we really are interested in pushing to the limits this machine learning approach to classification. And there we have two main topics of data that we can use — main sources of data. One is the time series itself — it's the color, it's the changes of brightness — and we can use the time series and try to compare it to the other light curves of other sources and all the different classes out there.
We can also use context. That is, we can actually say, well, where did this object actually happen in the sky? If you came up to me and you said, I just found an object last night, I wouldn't really say, well, what was its brightness and what its light curve look like? I would say, well, was it in the ecliptic plane of how the Earth moves around the sun? Because that was probably an asteroid. You say no. And I'd say, well, was it near the galactic center or in the galactic plane? You say no. Well, okay, it's probably not a star. Was it near a galaxy? And you say yes. And I say, what color was that galaxy? And you say red. And then I say, was it at the center of that galaxy or on the outskirts of that galaxy? And you say on the outskirts. I say, okay, it's a Type Ia supernova, and I'm right. So without knowing anything about the time series, but just knowing where the thing happened, actually goes a very long way in helping us get a classification. And in the context of not a lot of data, context sort of reigns supreme, and then as we get more data, the actual light curve itself winds up becoming more and more important in making a classification statement.
So how do we do this? We've got this heterogeneous data, and we now need to somehow ram it into these traditional machine learning frameworks. And the way which we are currently doing this is taking all these different light curves and basically crunching as many numbers out of that, turning these light curves into real number line — what are called features. So we've got all these variability metrics — metrics about periodicity, shape, et cetera — and all the context metrics, like I said before, the distance to the nearest galaxy. And it's very clear to most astronomers that if you can do this and you can do this very quickly, you can pretty quickly put different objects into different classes. Here's a plot showing a bunch of different classes of objects. It's not important that you know which ones are which, where I've plotted now period and log period, in units of days, so this is 100 days, this is one day. Again, some sort of principal component analysis applied on the light curve itself. This is very well-sampled data, and you can see that you get some great distance between these various classes.
One of the issues, of course, is that we don't always have great data on all the different classes. And in fact, somewhat surprisingly when we started off with this project, is that if we're trying to actually learn from what's already out there, the classes that already exist, we actually couldn't find a lot of data that was in a regularized, easily digestible format. So we wound up creating a site called DotAstro.org, which became a repository for all the publicly available light curve data on all the different classes of objects that we could get our hands on. And it now allows astronomers to go through and browse and find different light curves of different objects, and look at objects by classes or look at objects by other properties, and allows us to then visualize those light curves as a function of time, and, most importantly, pull over some sort of homogenized, regularized markup of that data. So that's been incredibly useful, because it allows us to now train off of that data.
And the way in which we do this, just schematically to show you, is we have an instance of some object of, say, class blue triangle — that is, in this case, maybe a supernova. We've got its time series data, and we know where it was on the sky. And then we derive all these features off of those — all these different extraction modules that turn all this data, this heterogeneous data, into something on a real number line, in a multidimensional real number line space. And then we can do that for all the different classes of objects of type blue triangle and produce distributions of all these different numbers in this different space. And then we can look at other types of classes, in this case light blue triangle — should have used a different color, maybe a different shape — and we can basically compare these different distributions. And you see that in some of these different spaces here there is no distinguishing power between these guys, these two different classes. But in some cases, you get a lot of distinguishing power, these examples here. So if I had a new instance of an object who I thought might belong to this class or this class, I could do the same technique, and I could actually extract out this information and figure out which one fits better.
So we're now applying a range of standard machine learning techniques to this data and trying to come up with probabilistic views. In addition to that, we also have knowledge, of course, of how different types of sources, specific types of sources, change, and so we can create traditional fitting, basically algorithms like chi-square fitters, that tell us how well we're doing. Here's the light curve of this microlens that I showed you before, this event where it gets really bright and goes faint, and you see this little spurious data here. And you see two classifications. Basically, what's the probability that this object or this event belongs to this class of microlensing? And you see that we have this thing called J48, which is just a machine learning algorithm, that after something like 150 epochs winds up realizing that this event that happened is really not happening again. And so what happened was actually probably a microlens. And then we have other examples where we have this fitter that says, oh boy, I really look like a microlens up here — it rails to a probability of one. And then it winds up seeing all this spurious data and says, hmm, maybe that wasn't a microlens, that was something else.
So the idea, of course, is that we can use all these different machine learning techniques and combine them in the form of, say, plugins that are good at fitting specific domain-specific events with machine learning things, and then we basically have all these different probabilities that we're going to wind up mixing together to come up with a final answer. There's a very complex framework, as you can imagine. It is indeed complex, and getting more complex as a function of time. But under the hood, what we're basically doing is coercing all of this heterogeneous data into a homogeneous multidimensional number line space. And we're using, more or less at this point, off-the-shelf tools that allow us to come up with some classification statements. And this summer, we were able to parallelize most of the stuff that we've been doing onto the Yahoo Cloud and the Amazon EC2 cluster.
Here's a depiction actually using open source genomics code that allows us to show relationships between different objects. You won't be tested on this, don't worry. And you see that we have this existing classification structure. The problem is that the taxonomy of variable objects is actually a pretty big mess. And so if we're trying to come up with probabilities, you can imagine we'd love to add all the probabilities on the outskirts — if you think of this classification as a tree, on the outskirts of the tree — so that we wind up coming up with really strong statements like it's eclipsing or it's pulsating with 100% probability. But the taxonomy that we have available to us, which really comes from a historical understanding of how the night sky changes, is based both on a phenomenology and sort of the physics. So something is red and pulsating combines both the idea that I understand what pulsation is, and I can look at an object and know it's pulsating — which, of course, you can't. You have to apply physics to come to that conclusion. This hybrid technology — or topology, excuse me — complicates the machine learning, and it's also inaccurate and incomplete.
But a positive view of this, of course, is that when we're trying to do machine learning on all of this data, we've started realizing that perhaps the learning process itself can start teaching us the relationship between these different types of objects. So in just the last slide or two, I'll show you what we call a confusion matrix, which basically tells us, for a given class of variable object, basically applied against itself, how well do we wind up doing in coming up with the correct class. And if we had a perfect classifier, we would have all our power in the diagonal element here. But you notice that there's many cases where you wind up having some off-diagonal power, which means that we've misclassified that object. Well, very interestingly, the places where most of the misclassification happens, happens in places where we have types of objects that we know physically — because we've been studying these for a long time — are actually fairly similar to each other. So without telling these off-the-shelf classifiers that these objects are actually related to each other, in some sense, those classifiers are telling us that those objects are related to each other.
So let me end with a couple of outstanding issues that we have going forward for our community. One is something that really is unique to, I think, our problem thus far — but I think will be very useful to have in general machine learning world in the future — is how we account for uncertainty and error in the learning and the classification process. There are uncertainties in all these real number line values that we wind up deriving, and there's really no good framework now to be able to account for those. Instead of coding up these various feature extraction algorithms, can we actually have those features be discovered autonomously, say through genetic programming? How can we actually take these off-the-shelf classifiers and imbue our sense of physics into those classifiers so that they do better? This is what's called using domain knowledge. And all these classification algorithms that people are using on a daily basis — how do they parallelize? And does it really matter? Can we do all this stuff in sort of an embarrassingly parallel way? And last, do we need to build algorithms that make classification statements that already know what's on the other side of their answer? That is, if it says that this is the most unique supernova ever, it knows that it's going to wind up having a lot of resources allocated to that. But maybe it winds up ratcheting down that statement because it realizes the implications of being wrong about it.
So summary of my talk today is that the automation of data acquisition and resource federation is actually happening now. It's sort of the new revolution in astronomy. But what's coming is a huge glut of data. This data deluge is going to really fill up the pipes of all the different possibilities of doing follow-up. So knowing what to follow up is actually key. And here, the autonomous classification of events that happen on the sky is obviously very important. And what we're doing, really, is taking the traditional role of astronomers as being intimately involved in the data collection and the inferences, and we're trying to remove the astronomer from that in a real-time sense. And what should be very clear, of course, is that what we've stumbled upon is this rich interface between machine learning, computer science, statistics, and astronomy. And what we're very excited about, of course, is that some of the tools that we'll be building as part of this project are very likely to have applicability in other data-driven fields. Okay, so I'll end with that. [applause]
NIKRAVESH: We have time for a few questions or comments.
BLOOM: Completely crystal clear. Everyone's ready to be gung-ho astronomy data collectors.
NIKRAVESH: Any questions? Okay, well, thank you, Professor Bloom. Thank you.