Talks & Media | Joshua Bloom

Foundation Models and Agents for Physics

Thu, 11 Jun 2026 00:00:00 +0000

Panel on foundation models and AI agents for physics, with Mariel Pettee and Ioana Ciuca.

Key Quotes

“The only real test set is data that hasn't been observed yet. The only real test of the efficacy of foundation models is when they're applied in production to data that wasn't in the can at the time those models were created.” – Joshua Bloom

“The idea of using a very large foundation model that's good at looking at images from telescopes and then somehow plugging that into some little downstream task and then actuating a billion-dollar facility is kind of preposterous. It's even more preposterous when you realize we only have access to one CPU and a tiny amount of RAM at runtime.” – Joshua Bloom

“Now we're in this mode where it's not just you and your grad students and your collaborators putting your names on papers; it's you and your collaborators and your students and your agents putting names on papers. And so the trust still ultimately comes back to us.” – Joshua Bloom

“It is hard to see how any of us really go back to the way that we worked before LLMs came into our lives… It's changing and upending how we live and how we work professionally.” – Joshua Bloom

Transcript

MODERATOR: So our first panelist will be Josh Bloom. Josh is a professor of astronomy at UC Berkeley and he's a highly decorated pioneer at the intersection of AI and the physical sciences, and I heard that with his talk he's going to give us a very unique perspective on foundational models. Please welcome. [applause]

BLOOM: Hi all. So hopefully this will be both unique and somewhat spicy. That's what I was told to do: inject some spice into the room. All right. So we've heard a lot about foundation models today and the construction of them, the utility of them. And I had a bunch of rants. I have rants about other things as well; you can ask me later. But this is mostly framed in the form of questions for people in this room. We know about scaling is all you need. But when we're building foundation models, adding more modes — so you have spectra and images and other metadata, more training data, more diverse data sets — we have to ask ourselves, are we guaranteeing that we're getting better and better results on all the downstream tasks? And I think oftentimes we don't ask ourselves what actually makes a better foundation model. There was a question that was asked earlier in the day today about how do you know, when you're building a foundation model, that the downstream tasks that you're building them for have actually improved. And part of that comes deep inside from how we actually do representation learning. And if we're doing this in a self-supervised way, which you pretty much have to if you're using lots of data that doesn't have a huge amount of labels associated with it. This is an example from a multimodal foundation model that I built called Astro M3, where we were trying to build a foundation model around time series data, so photometry and spectra and also metadata. And we wound up realizing that if we're doing this in a CLIP way, contrastive way, where you're trying to align embedding spaces across these different modes, oftentimes you can get modal degeneracy where the same light curve, as you see here, actually gives rise to two very different spectra. And that's because you could get a light curve from two different sets of stars that could give you the same light curve but actually are very different types of stars. So as we're building these models, the thing that we're baking in are assumptions about the underlying physics and the connections across these different modes. Oftentimes when we're building these models, we're building on very high quality, high signal-to-noise training data, but we're applying it downstream in application to lower quality, different types of data from different instrumental setups. And inherently, at least in astronomy, we're probing different populations, right? If I build something on nearby galaxies and I try to apply it to galaxies that are farther away, there's different distributions of those galaxies in all of their properties, in their numbers. And those sorts of biases are not generally contemplated inside of foundation models. And so, the thing that people have for a while heard me say, the only real test set is data that hasn't been observed yet. The only real test of the efficacy of foundation models is when they're applied in production to data that wasn't in the can at the time those models were created, and preferably by groups that didn't themselves create those models, which is kind of the state-of-the-art of where we are right now.

And so when groups are building foundation models applying it for themselves, they're hoping other people will use it. And the question is, can we just throw that over the fence and hope those are used appropriately? If those models are getting bigger and bigger, there's questions about is it big enough, and do we stop caring about accuracy and do we start caring about other things like inference speed and actual cost and who gets to host these models and who's going to pay for these models in production. These are questions that are not often asked by many of us who are working with foundation models. And importantly, how do we convey those costs in a rigorous way, or at least in a non-implicit way? And the last thing I was just going to say is I have a deep concern about the role of foundation models in doing what I guess I would call citation capture. We rely in astronomy, and through all of academia, on citations, and if our work is getting subsumed into foundation models and then used for downstream tasks, oftentimes those downstream tasks are not citing the original data from whence those models came. And so we have a serious problem there. And we're also not really working with raw data. We're working with corrupted data and trying to fix that data. And that corruption is exogenous to the physics of the things that we care about. So that's it for now. You'll hear more of my spice later on. [applause] MODERATOR: Thank you very much, Josh. The next panelist is going to be Mariel Pettee. And Mariel is an assistant professor of physics, and she just started and leads a new group. They're designing new AI methods to enable discoveries across physics and astrophysics. [applause]

PETTEE: Hi, great to be here. Dennis, I was counting on 10 minutes. I hope that's okay. I will move efficiently.

MODERATOR: Sure.

PETTEE: So it's great to speak with you all today. I'll talk briefly about foundation models and then I'll transition over to agents, since this panel is covering both topics. So first I'll talk very briefly about foundation models in physics, which is a field I've worked in for the past few years. So I think it's a fair question: will foundation models revolutionize physics? And I think what I can say today is that it's clear that foundation models, especially in industry, are catalyzing incredible progress. But I'd say that their potential to yield really surprising discoveries in physics, we have not fully understood today. However, I still think that the pursuit of foundation models, as Josh just alluded to, encourages us to explore these questions of multimodal representations and self-supervised learning strategies for physics. And I do think that what we learn in that pursuit is going to be critical for the future of physics analyses. So I've just flashed a handful of examples of these types of projects that, at least with AstroCLIP and AION, have integrated multiple modalities of astrophysics data — either two modalities or three modalities or, on the order of tens or 30 modalities — and we're starting to see more efforts of this in other fields like high energy particle physics too, where I just think that there's ripe potential to think of our detectors as highly multimodal sources of data. So in the process of this, we're essentially distilling a lot of our data into a different form. So just like maybe ChatGPT is sometimes analogized as a blurry JPEG of the web, answering this question will boil down to understanding what makes our data special. How is fundamental physics data distinct from, say, industry standard machine learning data or even internet data? And how do we wrangle that data? How do we turn it into a meaningful representation? And what does meaningful even mean?

So in the process of this, I think there are some interesting questions around methodology. Which self-supervised learning method do we choose? And then also, what goes into a good pre-training data corpus? There are all kinds of different properties that we might imagine emphasizing in curating a data set for a foundation model. And maybe that includes geometric properties, statistical properties, semantics, shared information, complexity, or also some sense of data set diversity where there's an interplay between the size of the model as well as the heterogeneity of the data sets that you're including. So there are a lot of open questions here, and I don't think scale alone will erase the potential of some of these questions. And I'll also say that I continue to think that one of the values of foundation models in physics, and one of the impacts of it, could be how it forces us to maybe rethink the structures of our social alignments in the sciences. We tend to organize socially around these traditional academic disciplines that were decided decades ago. And I think it's certainly possible that we can recall that particle physicists helped bring us into this era of big science, from a time when science was done quite differently with much smaller resources and collaborations. And I think we can, as fundamental physicists, be potentially part of a rethinking of our paradigms of how we do physics. And this is not a new problem. And I think it's been argued pretty well that AI for science can be seen as a social problem. It's a question of data access and fragmentation. It's a question of resources and allocation of resources, and it's also just a question of how do we communicate across disciplinary boundaries.

Okay, so switching over to agents and physics. This has been such a remarkable year for changing the way that I think about my own workflow in physics for sure. So I think agents are incredible, and I also think that there's potential to think about agents with some more imaginative power as we deploy them in our work. So first off, I love to think about not only, once we have this interesting tool, how are we using it, but rather can we take a step back and just remind ourselves, what are agents even for? Why incorporate new technologies into our workflows? I like this quote from an essay about agency, but I think it applies just as well to the concept of agents in physics. Without emphasizing its purpose, maybe it allows us to ignore some harder questions about what does a good life look like? What does good research look like? What does impactful research look like? So I really encourage us to think about using agents and some of these new tools to really raise the bar on what scientific work can look like in ourselves and in others, to bolster a sense of trust and rigor in our field. And I think that historically we've enjoyed a lot of respect and trust from the general public, but this is changing. And moreover, forget the public — just among each other, I think it's hard enough to find trust in other people's work, especially from people who we don't know personally, and this is being exacerbated by LLMs. But the paper mill and the concept of an onslaught of new work was not caused by LLMs. It's just being exacerbated by LLMs. So I worry sometimes that this is going to encourage us to really just fall back on personal connections and make access to doing research and science even harder if you're not already well connected.

So how do we push back against this? We've seen some evidence of this today thinking about reinforcing signals of high-quality work. And I tend to believe that these things should be opt-in as opposed to enforced. These are things that authors can choose to do to add extra validation onto their work. So for instance, agentic reproducibility of results and assessments of whether the code reflects what's actually stated in the method sections of the papers, for instance. And I think we've been having some super interesting conversations about this on the editorial board of PRX Intelligence, which was mentioned yesterday by Anatol, thinking about how do we talk about LLMs in the review process, right? And I think that LLM use is not just a checkbox to indicate. I think we should encourage people to really describe in depth how LLM use is a part of their process, just like we would expect for other parts of the scientific method. Finally, LLMs can suggest improvements clearly, but I don't believe that they can be held responsible for decision-making. And there's a quote that sort of summarizes this: LLMs can make decisions, but only people can choose. So I really think that expert, insightful peer review and curation is still going to be a critical part of our scientific process here.

Now I want to say a couple things on the idea of craftsmanship, skill atrophy and joy. So cognitive offloading leads to skill atrophy. I think this is clear, and yes, this is still true even if you have a PhD. And I also emphasize that this is not necessarily a bad thing. I don't think anyone really misses the days before having maps on your phone or memorizing phone numbers. So cognitive offloading is something that can really benefit us in our work. But personally — I don't know about you — my happiness is also tied in part to a sense of craftsmanship and mastery in my work. So very quickly, as we saw yesterday, we get into this territory of why do we do what we do? What is your craft? And I think that as scientists we should have some agency to choose what parts of the process we really take pride in, what our sense of craftsmanship is. But I also think that there's some external metrics that can guide this around the sense of high complexity versus low complexity tasks. It's evident to me that coding has more structure than free form writing, for instance. And so I see free form writing as a higher complexity task than coding in some senses. And so that informs a little bit of my choice of where I put LLMs into my workflow.

And I just encountered this idea recently, just this past week, of adding an egg. So it comes from this maybe apocryphal anecdote of Betty Crocker in the 1950s, where we were first seeing evidence of pre-processed foods in the supermarket, and you could buy a cake mix that had everything included. Just add water and you make a cake. And they found that sales were really not doing so well when this was first introduced. And a lot of this was attributed to our psychology, a sense that just mixing water with the mix and the cake is ready — it kind of depressed people, or it made them think, what is my role in this process? And so, as the story goes, marketers on the team decided we should actually encourage consumers to add an egg. And that's supposedly still why that's part of cake mixes, because it makes you feel like you've really contributed to the process as you've baked a cake. So I've been thinking about this recently in terms of how do I — what egg am I adding, what is that in my own workflow? And I've seen some — I'll just flash some quick quotes that summarize this. This is from someone who I don't know on social media who was talking really enthusiastically about agents in their workflow. And then they followed up with this statement saying, actually, my goal with agents is to keep cognitive traction, and if I lose that, my internal sense of aliveness with the project fades too. So paradoxically, agents materially forwarding the work and removing myself as the bottleneck chokes the source of project emotional investment. So I resonated with this, and I imagine some of you might too. And from another perspective, choreographer Martha Graham talked about how learning is achieved through practice. It is the performance of a dedicated, precise set of acts, physical or intellectual, from which comes shape of achievement, a sense of one's being, a satisfaction of spirit. So this is elusive. It's hard to find in any practice. But it's something I'm still working through in this field, of how agents intersect with this desire to feel mastery of my craft. Okay, final thoughts. I think we should make agents weirder. I think agents and agentic ecosystems are often — and I understand the temptation — we anthropomorphize them and we think of them as maybe one-on-one replacements or analogues to individual scientists or even to scientific labs, and they are sometimes pitched as products. But I think that we could instead think about embracing their wildness and weirdness and see them as more reflections or distortions of our own thought process. So sometimes LLMs are described as mirrors, right? They give you what you give them. If you give them higher quality material, they'll give that back. I think really agents are kind of more like disco balls that reflect some parts of you and then also lots of other parts that maybe you didn't say explicitly, as well as lots of other sources. So I think potentially, imagine this is a disco ball collabial manifold. I think if we really thought of these as quite different from our own intelligence and not trying to directly analogize them, there are some cool possibilities here. And this is a slide with some advice on how to do this in practice. This is a slide I made, I think, eight years ago. And this was in my other life, thinking of integrating AI into my art practice as a creative, and in the course of this early development many years ago I thought of interacting with these models that I had developed for my own creativity along these different axes, thinking about who has the creative power, who is making the creative decisions in my practice, versus do I think of the model as my friend or as either my enemy or a critic, or at least someone distant from me. And there's a lot of possibilities here in reimagining the relationship between you as the scientist and an agent or a set of agents.

And I'll conclude with just a couple quick cautionary tales. As I mentioned, I've been involved in the AI and creative space in the early days before ChatGPT. And so I was interviewed recently as part of this kind of review paper reflecting on, how do people who were starting out in this field think about the state of AI and art now? And this might be familiar to a lot of us, right? The introduction sort of describes how a small group of people began to experiment with artificial neural networks. They created conferences and exhibitions, and they talked about machine learning and how it was shaping their field. But it really concludes with this current state of AI and art where a lot of these early practitioners don't really recognize the field anymore. They feel that their legitimacy has been affected, and there's a big mismatch between their creative process and the way that the technology has gone. And I think that potentially the way that we think about agents in our work might not only look like this, it might also look, for instance, like the music industry. So there's this quote that I've heard in the context of tech: whatever happens to musicians happens to everybody. And I worry that this might be increasingly true, where a lot of musicians, with some dark humor, describe their practice as t-shirt salesmen and less of practicing musicians. They go on tour and they sell t-shirts and that's how they support themselves. So this is kind of a sad story, but I see it as a real state of the music industry now, that we kind of have titans at the top, and then we have a number of people who make a living from their music practice — and I would say that that's on the order of a thousand t-shirt salespeople, as they might jokingly call themselves — and then there's kind of everybody else, creatives with side practices who are teachers. Maybe they're lucky enough to be affiliated with the university. They might get small grants. They might have side jobs. They might be independently wealthy. Sadly, this is kind of how music works nowadays. And I would really want us to avoid a future like this in the sciences, where only a small few get to practice the work that we do.

Okay, so this is my last slide. It's a quote that I love from an artist who is talking about the integration of technology into her work, and she encourages other artists to think of working with technology like riding a bull, where you want to have some sense of control but you also want to be surprised in the process, and I think that there's something for us to learn from that. So she emphasizes we really need to move beyond illusions of control. We need to build instruments that transform us and allow us to hear different structures and sounds from unknown spaces. Inefficient and ineffective instruments which we learn to ride like wild bulls on a limitless plane. Thanks. [applause] MODERATOR: Thank you very much, Mariel. And for our last speaker in this panel, I can introduce Ioana Ciucă. Ioana is a research scientist here at Stanford University and she describes herself as always human, astronomer and world traveler, and she told me before I should introduce her as well as a hacker of the universe.

CIUCĂ: Thank you so much. [applause] So hi everyone, my name is Ioana Ciucă but everyone calls me Joe, and I am a researcher at the Center for Decoding the Universe at Stanford. And today I want to share with you very briefly a reflection on what happens when AI meets science, and what are some of the considerations and the questions we have to take into account when these artificial entities enter our individual private research lives and also the lives of our scientific teams. So to start with, I'm going to show this big sentence here. So this is the current narrative, especially here in the Bay Area, that AI will accelerate science, and I'm just a bit curious how many folks in the audience today actually agree with this. Please raise your hand just to give us some sense. Thank you so much. How many folks in the audience today don't agree with this? Yes, we have a few strong, courageous folks here. And how many folks in the audience today think that we need a little bit more information before we can make this assessment? Please raise your hand. So I am also in that third bucket, and I believe that in order for me to make an assessment on this statement, we first need to spend a bit of time meeting each other and asking each other hard questions to define, in the first place, what we mean by accelerating science. And once we get there, through many meetings and through many debates and being skeptical and maybe shouting at each other with love, once we get to that definition of what it means for science, at that point you also have to become aware of whose voices and whose values are reflected in that definition. But I think it's going to take a little bit of time until we get there, and I think that's okay.

And until we get there, we have to ask a question: how can AI actually help us do the science that we do today? So where do we start when we have to answer this question? So here at the Center for Decoding the Universe, led by a brilliant junior researcher called Christine, we did this project called ReplicationBench, where we pretty much talked to a lot of our human expert astronomers here, and we asked them to take tasks from the papers that they knew deeply and curate a set of 120 research tasks that were either very simple tasks — like this one, that would take minutes, pretty much just look at a very big data set and select a sample — a bit more convoluted tasks that take hours, where you have to fit a statistical model to your data, and tasks that were really really hard, where you have to implement some really big algorithm, like a force solver, an N-body solver like this one, and this would take an expert a month of some sort. Oh yeah, where's the mic? Oh, this mic — yes, sorry. I survived a tornado yesterday, so that's why I'm shouting. I'm very excited. So yes, just want to put out there, these tasks, some of them will take many weeks and months. So once we actually curated these tasks, we then asked AI agents to try to solve them. And one thing to mention is that each of these tasks had a verifiable, concrete answer at the end. We have the number of stars. We have the number of peaks. We know exactly the accuracy that we want to achieve with this solver. And we found that in October 2025 the models at the time were actually getting around 20%. But the more recent models are showing improvement. So I feel, with time, that this replication task will be solved by AI agents. But more importantly, what we found in that process were the failure modes of these agents. And why are failure modes interesting? Because then we can share with the community those failure modes, and the folks who are building these systems, which could be open-source systems — and I actually really encourage folks to think about that — can then improve them. And then maybe, just maybe, we can then use these methods and integrate them in the research workflows that we have, for example, in cosmology.

These are really advanced tasks that I believe AI can help us with. And in many conversations with Marcelo here, we're thinking all the time, how can AI actually help us develop these pipelines or change some components of the pipelines in a meaningful way, and not just in cosmology but across astrophysics? I think one thing that doesn't get talked about enough is that we can use these methods to develop methodology, and we can draw insights from different fields and make really good methodology. Okay. So here we're still in the domain where we can verify that the models are doing the right thing. And once we get that signal that they're not doing too badly, we can only then think of releasing these AI systems at discovery scale, at the discovery scale of Rubin. But what I've mentioned so far is only on the level of the AI capability. So we need to do something more than just creating very capable systems. We have to create systems that are deeply collaborative and that empower the teams that collaborate with these systems. So on the capability scale, before we release these systems at the Rubin scale, we need capability. We need these systems to create reproducible analysis. We need them to learn from experience and have a sense of memory and persistence. But not only that; on the human side, we need to make sure that the human is engaging with these systems actively, if the system can ask the human questions and so on. And finally, we have to create systems that enable team science. Why should we bring an artificial system inside the research team if that research team feels disempowered, that they're not doing really cool things, not learning and so on?

So just on my final slide before I go there, where do we go from here? So far I told you a little bit about the requirements, the design requirements that I think these systems should have before we release them at scale for discovery purposes. But what are some of our own principles that we should follow as we integrate the systems into our research life? So I think we should really care deeply about our curiosity. We should care deeply that once we are curious about something and we point a telescope somewhere and we find some really cool thing out there and we understand what's happening, the next thing that we can do is, one, we want to share that with our community. So I think we should care deeply about sharing our understanding with the folks around us, and I think we should care deeply about responsibility. So that means training a junior researcher into doing science is valuable in and of itself, and we should keep track of that. And more concretely — and I think this is a conversation I've had with Dennis a little bit — concretely, what can we do to really see how these systems work with us while following these big principles ahead? I think we should just explore. We should get in rooms. We should make mistakes. We should see how they respond to us. And at the end of the day, we should imagine — science has been done in many ways over time, and now we have an opportunity to really think deeply, how can the research team of the future look, and what is the role that AI should play inside our research team? Thank you very much. [applause] MODERATOR: So let's go to the first question. We have a lot of people working on foundation models currently in many different fields and disciplines and so on. I would like to ask you, do you think, looking into the future, we will have one foundation model for all of the physical sciences or for all of science, and will it come from an academia person or from an industry person? Maybe, Josh, you can try to answer first.

BLOOM: No. [laughter] Thank you. Hello. Yeah, that works. Yes, I will answer the question. The answer is no to the question. People that are practitioners now have been using bespoke models on bespoke training sets for very specialized problems, and I think that persists. We're working on a problem right now with Rubin Observatory to do the active optics system using AI, and the idea of using a very large foundation model that's good at looking at images from telescopes and then somehow plugging that into some little downstream task and then actuating a billion-dollar facility is kind of preposterous. It's even more preposterous when you realize we only have access to one CPU and a tiny amount of RAM at runtime. And so there's a very long tail of problems that foundation models can't and probably shouldn't touch. There's obviously a large class of problems where it probably will be extremely helpful. I guess my question to people in the audience — and I'd love to hear how people think about this — is the difference, as we try to make an analogy with foundation models in language, is that language seems to have this really nice, almost bijective mapping back to how we think and how we construct thought and how we think about the world. But I don't think it's the same thing when you look at images of galaxies, right? In every pixel, there's a lot of physics in there. And if you knew all the physics, you could generate the pixel and the flux from that pixel in all wavelengths. But going the other direction and trying to get back to that physics is not clear. So I'm not sure that even the data that we're using for foundation models is the right language of physics. Now that may be true in astronomy. It may not be true, and in fact it may actually be really exciting, in particle physics. It may be exciting and actually true in chemistry, where you have a one-to-one mapping between a SMILES representation and the actual state of the atom. But I think a lot of the data that we use now in foundation models, in astronomy at least, are not the level of the language that we need to be able to solve all problems.

MODERATOR: I saw a lot of head nodding. Do you both agree?

PETTEE: Yeah, I'd say I think it's clear that physics data is different from language in ways that we don't fully understand yet. I think some of that might be related to how efficiently language embeds a world model, or a sense of the world that it lives in. And physics data does that differently. I think my answer to your question is I hope that there's competition on all sides, and I think they have different purposes. I think a very multi-purpose foundation model could serve as its own object of study. It's a modification or reflection of the universe. And the fact that it might have diverse applications is interesting and valuable in itself. But if all we want is to really use a foundation model for one or two tasks, I can almost imagine another paradigm that's like disposable foundation models, where you understand very clearly the qualities of the downstream task that you want to target, and then you can quickly generate simulated data that captures those intrinsic properties well, and then you make your customized foundation model. But right now, I guess we have some evidence that even very large-scale generic image foundation models are not able to outperform domain-specific train-from-scratch foundation models if you have access to high quality data. So I don't think that adding more data in a generic sense is going to solve the problem.

CIUCĂ: I'm not very informed in this respect, but I want to say something. I'm kind of a scrappy person. So we have an opportunity now to train these very large-scale statistical models on our data sets. And I'm just generally curious to see how these large systems actually represent our data. And I think it's important to look a little bit into this mechanistic interpretability aspect, look a little bit into the representation of these data and look at their geometry, look at how they organize information, look how different modalities show up in those spaces — and there's a lot to explore. So I would say let's build foundation models to really understand a bit more about our data, and also the tools to probe their internal representations and things like that. Yeah, but I'm not an expert, so I don't want to talk more outside of my [lane].

MODERATOR: Thank you. MODERATOR: You already touched on the amount and the modality of data. One thing which I think a lot about is, if you for example have a fractal and you train a generative model to learn this fractal, you can train an infinite-parameter model to learn this fractal. However, the underlying structure is super simple. So when we train these foundation models — and currently all scaling laws seem to suggest that they get better and better at describing the underlying physics — are we just learning the universe by heart, or are we really discovering some underlying structure? Maybe start from the other side this time.

CIUCĂ: Oh gosh. Oh no, me. Oh god, oh lord. Okay. So I feel like, how can we even tell, right? I have a counter question to your question: how can we tell? Where do we look to make that distinction? Do we look in the — yeah, where do we look? And maybe start there. Sorry. [laughter]

MODERATOR: Good. I was not prepared to get a question back. But let me say that during lunch I had an interesting discussion with some students, and we were talking about whether the universe is intrinsically infinitely complex, which would mean you need an infinite amount of parameters to describe what is going on, or if there is indeed some underlying structure. When I started studying physics some years ago, I thought there would be an underlying structure, but maybe that's not true. And if there is an underlying structure, can that be encoded inside a very large statistical model? I don't know the answer to that. Maybe Josh has one.

BLOOM: Well, all I'll say is that we saw a talk yesterday about cellular automata, right? A very simple set of rules and you get this emergent phenomenon. So perhaps there is hope in being able to describe something that visually is as infinitely complex as you can imagine, but has simple sets of rules underlying it. That need for us as physicists to kind of reduce everything down to the simplest number of equations and appeal to Occam's razor is still very much not only implicit — I think it's quite explicit — in the way that many models are trained and built. In the architectures of models, when we think about sparse autoencoders and things, we are explicitly penalizing for complexity in representation space. And so we as physicists still believe that the right way to do things are with bottlenecks, and the bottlenecks with physical equations or bottlenecks in latent space have some deeper mapping with each other. But it doesn't mean it's the right representation.

PETTEE: Yeah, foundation models are only one part of the ecosystem, and that's why physicists steering them are critical, I think, because there wouldn't be constraints if there were infinite resources and we could train infinitely large foundation models. So I think having human understanding as a bottleneck is a choice that we've made in the field, and we've seen from history that that's a pretty effective prior to put onto the process. And also cost and computational limits are going to come into play too. So I think it's our job to help place constraints onto the process, because we believe that that will yield better results later.

BLOOM: Yeah, just to add to that, if we still also believe in the scientific method, then the falsifiability of predictions is a central part of what we're trying to do here. And so if you have a great representation and it can summarize all the data that you've seen but can also make predictions about the future on unseen data, and particularly in unseen regimes, then we would ascribe a lot of power to that, and I think there is still a lot of potential for that to actually be borne out. I don't know how much research is going into that in astronomy. There's some of it in the physical world of trying to do out-of-distribution testing on some models. The challenge we have in astronomy is we only have one universe, right? We have like 10 to the 10, 10 to the 11 pixels if every pixel is one arcsecond on the sky, and that's it, right? And we have a bunch of wavelengths and we've got time, but it's still a finite amount of photons that we could collect. And one of the problems that I have with foundation models in astronomy is that oftentimes if you're trying to do downstream tasks, you're not quite sure if the things that you're trying to do downstream tasks with are actually in your training data. If I'm trying to generate what would this part of the sky look like, it may have already been in the training data. If I'm trying to do shape analysis for dark matter, we've trained it on the data that we have — and yes, you could train it on simulated data, but if we're training on real data, then we have this challenge of just the finite nature of the amount of Shannon information bits that exist in the universe.

MODERATOR: Thanks. MODERATOR: With this I want to pivot a little bit away from the foundation model and go to the second topic of this session, which was agents. So if I walk through San Francisco and I talk to some people on the street, everyone seems to have enormous trust in scientists and especially in physicists, and people really believe that everyone here knows what they're doing and we are super sure about it. When we start using agents for our work, what do we have to do, and how do we have to make sure that we trust the agents, to keep the trust going? Mariel?

PETTEE: Yeah, you and I have had some pretty interesting back-and-forths on this. I don't have easy answers for it. But I think it's a little bit uncomfortable to realize that there's actually no way to have total trust in another person's work, let alone our own work. It's so easy to fool ourselves in our own process. So there is a bit of faith that comes into the research process, and that's why we encourage many people to be part of the field — is that we hope that even if individually we're fallible, in aggregate eventually the truth will prevail. And so I think the way that I think about trust with agents is similar to how I think about trust in my own work and in new collaborations that I've started, where if I feel like I can poke and prod the results and ask for new plots and ask for validation and verification, if I can read through the code myself, if I can compare that with writing about it too, eventually after some number of months or weeks of working with the process, I have to admit that I trust the work. And that's important, because as scientists I think we really have to advocate for our particular perspectives out in the courtroom of science. And so we need to feel very deeply that we believe in the quality of the work. So I think ultimately that's the bar — our own sense of taste and rigor, and we are ultimately just in control of that.

MODERATOR: Let me ask a follow-up question to you. I have the feeling that the way agents think and humans think has a very different regularization to it. Are the questions that you ask a human and the agent different, and in which way are they different, before you trust?

PETTEE: It's so true that the types of mistakes an agent would make are very different than the types of mistakes I would expect working with a student or a new collaborator. I had the experience the other day of realizing that an agent, when it was writing code, was explicitly optimizing for the thing that I was trying to measure. And if I hadn't looked through the code carefully, I could imagine missing something like that. And that's not something I would expect a student to try to slip into the process. So I don't think I've mastered the right questions. I think I have just tried to move very carefully.

CIUCĂ: I have a lot of thoughts on this one. Okay, let me start. So through the wonderful project we've had with Owen, who is today in the audience — Owen is building an amazing system that can help us implement methods that we care about deeply in astrophysics. And through the interaction with that system, we've learned that we have to reflect a little bit on ourselves in the way we even pose these tasks to the model, in such a way that the model can understand what we expect from it and proceed to implement the solution. So I think that is step one, just thinking deeply, how do we specify these tasks in such a way that an AI agent can act on it meaningfully? And then the second thing, something that I've learned working quite closely with Marcelo, and Marcelo's been thinking deeply about, is the context. When the agent goes and solves this task, it has its internal reasoning, it uses some tools, it makes some decisions, and we have to have a way to interpret what the agent is doing along the way, and we have to have some visibility into that process. So I think from a pure engineering perspective we need to have visibility into the process, and also, one thing that would help with the trust element would be for the agent to surface assumptions. I learned that from talking to Risa a lot about this, surfacing of assumptions and things like that. I think that's super key, and also, to not forget that these are artificial entities. They don't necessarily think about the problem like we do, and just being a bit more understanding that sometimes — should we even apply the same understanding of trust to an AI agent as we do to a human? That's a question in and of itself.

PETTEE: Thank you. Yeah, I wanted to add quickly to that that agents are fickle, and as soon as you push back on them, they'll say, “You're absolutely right. I was completely wrong. Now I've totally changed my opinion.” And fortunately, that's something that students are more hesitant to do. Real humans have an actual grounded sense of what is right and what is correct, what they understand. And that's why I sometimes chafe at the comparisons between agents and students, or agents and researchers, because we're not only measuring the quality of their code and their papers, but we're also thinking about their internal sense of research taste, and that's just, in my experience working with agents so far, you can't compare them.

BLOOM: I guess I would add, I agree with everything that's been said. We also need to think about community standards around this, and this is again where people in this room should have a big voice. We all have our own trust harnesses and our own developed practices. What's very strange, of course, is that what we didn't trust 6 months ago because everything was hallucinating, maybe now it's a lot better. And so we're also in a very strange moment where we're learning to interact with these artificial systems as if they were co-authors, as if they were collaborators. And that interface is the only thing we know how to do as people. But there are things we can do in code, right? There are community standards that we can implement. I hope many of you saw this, but arXiv, I think last week, said they're banning people for a year if they find that you have a hallucinated citation in your submission. And that's pretty harsh, right? So there are carrots and there are sticks. That's a pretty big stick. But what's interesting is that they're not penalizing people because there are hallucinations in the paper; they're penalizing them because they use that as a proxy for, you didn't write this paper and you didn't read it. And so it comes back to, do you stand behind, and does your reputation stand behind, the things that have your name on it? And now we're in this mode where it's not just you and your grad students and your collaborators putting your names on papers; it's you and your collaborators and your students and your agents putting names on papers. And so the trust still ultimately comes back to us. But I think it's up to us as a community to decide where we want to land on how much affordance we can have around using this kind of new set of tools.

MODERATOR: Thank you very much. MODERATOR: Now we have 15 minutes for questions from the audience. So please line up behind one of the two microphones and we'll get your question.

AUDIENCE: Hi guys. I just want to mention one thing. My partner actually works at arXiv, and that policy is apparently not official policy. Just one of the editors tweeted it and now everyone knows about it. So you're still okay for now. I guess I have a question: we've seen some breakthroughs recently, especially in math, that agents are now starting to discover things that we didn't discover. I guess a question I have is, in particle physics or in astronomy, what do you think the first ML-driven or agent-driven discovery would look like, and how do you think the community would receive that?

BLOOM: I think we'd embrace it with open arms, because it would be an exciting watershed moment for our field. I think it's more existential in math, because a large part of the math endeavor is thinking, right? But in astronomy, it's thinking and code and computation and confrontation with data and getting more data. It's a much more complex, much more messy scientific process. And frankly, we've all been kind of waiting for that moment to happen. There's been a lot of promises made for many years that we're going to discover new physics or something new is going to happen. There have been little tiny pockets of some lights that have happened, but one of the big challenges in astronomy is that we don't have an AlphaFold kind of data set. When you ask an astronomer what would be the best things to know, it's like, what's dark energy? What's dark matter? Is there life on other planets? Okay, we're not going to think our way to any of those answers. We're going to probably have to build billion-dollar facilities and be really smart about developing new algorithms over decades. And so, some of the astronomers in the room, your job is secure. But we would love, for the thinking types of axiomatic problems in astronomy, like solving something on the theory side — I think we're ripe for that, and there's a push that some of us have been thinking about, of trying to articulate what are those Erdős-like problems that, if we knew the answer to, that'd be pretty great.

AUDIENCE: Okay. So I'm wondering about the framework in your head for thinking about what science will look like 20 years from now, particularly with respect to the AI agents side of things. And so I'm wondering, what do you think are the qualities of humans as contrasted with the AI agents, for which they will maintain a fundamental advantage over the AI agents and will prevent the world where the agents replace huge amounts of the scientific labor? And then what are the implications of that for what things will look like 20 years from now?

PETTEE: I think there's a lot of value in the fact that humans are stubborn and curious and also are not limitless. I think sometimes agents are touted as saying, “Oh, it's amazing, they never sleep, they never get tired.” But I think a lot of the understanding of physics comes from the fact that we need things to be compressed and reduced, and we can't work infinite hours. So I think there's a sense of judgment that comes from humans having limited time on earth and resources and wanting to make something valuable with that time. And I guess I would also say that human collaborations, for the same reason that I just outlined, that it's hard for humans to change their minds, humans have a special value in collaborations that I think can't always be reflected with agents, even if we solved the idea that agents are less fickle.

MODERATOR: Let me ask back, why are you so sure that those are properties that cannot be modeled with an agent, like a sense of urgency, pushing back on thoughts?

PETTEE: Definitely we can imitate it. But I think that there's a special quality to a person choosing their life's work. And I think an agent doesn't care about its work in that way. And I do think that that's like a secret sauce of being a person.

CIUCĂ: I've wanted to share this quote forever, so thank you for your question, because now I get to share this quote. So there was this student who went on one of the open forums — he was in maths — and he asked on the open forum, hey, what can one such as myself contribute to mathematics when there are so many smarter people out there now, probably even agents in the future? And then what happened is that a Fields medalist actually jumped into the open forum and answered the student with the following. He said, it's not mathematics that you need to contribute to. It's deeper than that. How might you contribute to humanity, and even deeper, to the well-being of the world, by pursuing mathematics? So I think one thing that I think is deeply human is our desire to pay it forward, to sort of have this so we persist through time, and then maybe many generations from now we'll build on what we did. And I don't know if we can replicate that in an artificial system yet, but I think that's a deeply human value, at least.

MODERATOR: All right, thank you. MODERATOR: Then we have one more question on the right side.

AUDIENCE: Yeah, so thanks so much for all of that. That was great. I have a little bit of a philosophical question, but I think it's fair because some of the stuff you guys have been talking about has been pretty philosophical. So, as people who train and evaluate younger, earlier-career scientists, I wanted to get your opinion on whether or not you think there's value, and there will continue to be value, in having an identity as someone who is an AI-plus-physicist versus someone who just uses AI in their physics workflow. Do you see that continuing to be a difference? Because there's a lot of overlap between people who train models and people who develop architectures and people who think a lot and use agents. I'm wondering what the value of maintaining that identity, versus just a physicist that happens to use AI sometimes, is now and will be in the next 5 years or so.

BLOOM: I have a pretty rosy view of that, in the sense that I think AI just subsides into the background. We don't consider ourselves — I'm an astronomer who uses COBOL, right? Or I'm a physicist who knows how to use a database. AI is just a set of tools. All the companies that have AI in their last name — okay, you have a technique. AI is going to be pervasive in every single thing that we do. And once that is, then it ceases to be novel as a way to stand out. One of the things that I'm kind of excited about — maybe it's one of the only upsides of the disaster we have on the federal funding side of things — is that we get to reimagine where the money flows for doing science. And my hope is that it becomes less siloed than it is now. If you're an early-stage career scientist, or actually any career scientist, you are thinking about which are the agencies that you're applying to and what are the sub-agencies that you're going to get your money from. There's a hope, because now we don't have to necessarily say I am an astronomer working on this type of thing, but I'm somebody who thinks deeply about these types of problems, where we may also have enough of a training to be able to work across multiple domains, that we will break those silos, and now one can think of oneself as a scientist who's interested in this class of problems, whether it's in astronomy only, or astronomy plus physics, or astronomy plus physics plus bio.

PETTEE: Yeah, I think it's useful to worry less about labels and let the work speak for itself. So I try to practice that myself and not worry too much about how I'm framing my own work. But framing is important, because it conveys to other people what your context is, where your intuition comes from, who you admire in terms of their processes, and what biases you might bring to your work too. So I think if you have that larger framing, it's something you can take pride in. And I think we need physicists who happen to use AI, and we also need AI physicists.

MODERATOR: All right, then we have one more question on this side. AUDIENCE: Yeah, thanks for the nice discussion. I have two questions; you can answer any of them, I guess. So particle physics today has large global collaborations. So my question is, if we are thinking of integrating agents into how physics is done, then there might be two considerations. First is, how much would be dependent on the corporate token pricing policy? And the next one would be, HEP collaborations today keep pre-publication material private, and also the data is private. Unlike astronomy, it's open data, but particle physics, it's private. So I would just like the panel's opinion, because there were some conversations in our collaboration regarding, how really private are our conversations with these agents? Let's say you take some internal collaboration data, paste it to an agent and say, hey, do something — and now is that public? [laughter]

BLOOM: I am an astronomer talking about identity, not going to particle physics, but maybe — I'm happy to answer, but yeah.

PETTEE: I think they're both really cool questions. I think the particle physics community is actively wrestling with the question of open and closed data and how that interfaces with foundation models, and does that mean we pursue practices that keep the data closed, but can sort of preserve the use, like privacy-preserving methods or other techniques to train without having direct access to the data. Obviously astrophysics is taking a different approach, although there are still big questions around data attribution that Josh mentioned. So I don't think we've aligned on where we are. And it's challenging because, particle physics, all of that closed data is very valuable. There are a lot of companies that would love to have access to it, I think. And so I guess that leads to your first question about, how closely should we integrate our work with industry advances, and do we depend on tokenization strategies from companies? And yeah, it's a hard question.

BLOOM: I'm announcing here that I think we should have universal basic tokens, UBT, and so every year you get a billion tokens as a scientist, and it comes from the frontier models, and they should just give it to us, and we can sell it or use it or burn it. But you're asking a very deep set of questions about who funds science going forward, and how do we do this in a way that benefits the most number of people, not just close collaborations. I think it's still TBD, and again, we come back to where things are at with the federal funding situation in the United States. It's kind of wide open what the future looks like. But we will almost certainly be seeing other types of interaction models between industry and what we all do in this room.

AUDIENCE: Thank you.

MODERATOR: I think we have time for one more question on this side. AUDIENCE: Okay, sure. Thank you, and thank you to the panelists for your discussion so far. So I come from an area of physics and astrophysics where conceptual understanding of ideas and what is happening is preferred, or at least it's the state that we're trying to go to, more so than quantitative accuracy. And so in my field very few people use AI at all, except for fixing bugs in their code and whatnot. So I'm wondering what your opinion is on the relationship between using AI to understand physics concepts, like novel physics concepts, as opposed to using AI just to do quantitative data analysis and that sort of thing.

PETTEE: You're saying its value as a learning methodology, or a tool —

AUDIENCE: Including doing novel research on physics phenomena using AI, even if you're not trying to quantitatively match data exactly.

PETTEE: I feel like I can only really speak from experience about using it as an interlocutor for trying to process new ideas. And I think if you already have a well-honed sense of when you really understand something, it can be quite effective for that kind of thing. It's similar to talking with a colleague, but I think more dangerous than talking to a colleague, because a colleague can push back more easily on misunderstandings. But I think you're asking something deeper about how do you even maybe engage in research in that direction, and I can't speculate.

CIUCĂ: Yeah, me neither, but at least when it comes to conceptual understanding — the way I use these models, because I had to teach the transformer yesterday, and the architecture is pretty theoretical in the way I had to teach it, and I asked the models to push back. I instructed them to push back really strongly on my understanding of the transformer, and I found them to be quite good at that, to be honest with you. And because they're not human, if they say, “Hey, you're kind of silly,” right, it doesn't hurt me. So I can go quite a long time with them. So there's this sense of also psychological safety, in some weird way, with these models. You're not afraid to ask silly questions to them, and ask them to ask you silly questions. I think that maybe there's something in there about not being afraid to take the riskier conceptual jumps as you approach a problem. Yeah.

MODERATOR: Thank you very much for all the super interesting questions from the audience. To close this, I would like to have one statement from each of you. It is very hard for people to give some estimate of what will be going on in two years or five years or something. So I would like to ask you a pretty simple question. What will be one thing that will be true about your work 10 years from now? Can be a small thing. You guys start.

BLOOM: That's tough. I think it's fair to say that it is hard to see how any of us really go back to the way that we worked before LLMs came into our lives. I mean, I've been using AI in my research for two decades. It's been called different things, different methods, but we've been using these techniques to further science. And what's happened in the last 18 months has been remarkable, because it's not about the specific methods, or pulling this thing over from industry and pulling it into this way of changing our scientific workflow. It's changing and upending how we live and how we work professionally. I don't know what the details of that look like, but I'd be shocked if it's just not pervasive.

PETTEE: I feel pretty confident that in 10 years I will still be giving talks, because I think that is part of the fundamental duty of scientists, is to communicate their research and their discoveries to the relevant communities that can actually make use of it. That's my definition of what impactful research is — the type of research that inspires someone else to make new work on top of that. And something funny I realized about that recently is that that also means that actually doing bad work is potentially impactful research, because that can also inspire people to correct your mistakes and build on top of that. So yeah, I think I will still be giving talks, talking to people face to face. Maybe it'll be on Zoom, but I think it'll be largely in person. And I really hope I won't have an email address.

CIUCĂ: I think for me, in 10 years — hard one. My dream is to have this very nourishing, incredible, exciting interdisciplinary research collaboration. So I hope that in 10 years from now a few of us find a problem that we're really interested in and decide as a team to pursue it, working with AI agents or the systems at that time, and us as the humans, we can think slower. We can take our time deeply to think about the problem, to think about the approach, and then maybe have these AI systems act on our thinking and then help us implement and verify and all that stuff. And together we can maybe say things that we cannot say now about the evolution of our universe or the fundamental nature of the universe and so on. So maybe it's something about being together, working on problems that seem impossible today, and being even slower in our thinking, maybe. Yeah.

MODERATOR: Great vision. Thank you very much for your insightful answers. [applause]

Time-Domain Astrophysics, Accelerated with AI

Mon, 27 Apr 2026 00:00:00 +0000

Inaugural KAAI colloquium: simulation-based inference, self-supervised multi-modal foundation models, Rubin/LSST active optics, and reinforcement learning for gravitational-wave target-of-opportunity scheduling.

Circumventing Bottlenecks and Taking Better Data in the Time Domain

Thu, 16 Apr 2026 00:00:00 +0000

ML across the time-domain pipeline: relieving discovery and follow-up bottlenecks and actively taking better data with Rubin/LSST-era surveys.

The video link is a Zoom recording; enter passcode $50FQK3L when prompted.

AI in Astronomy, a Primer

Thu, 11 Dec 2025 00:00:00 +0000

A primer on how astronomy came to lean on AI/ML for discovery and inference at scale, for a multi-speaker Berkeley roundtable.

AI Accelerating Astrophysics

Tue, 30 Sep 2025 00:00:00 +0000

How AI/ML tooling is reshaping astrophysical workflows swamped by the data deluge, from discovery to inference at scale.

Eurekaizing Anomalies

Thu, 05 Jun 2025 00:00:00 +0000

Faculty talk in the opening session on time-domain astronomical data and anomaly detection — finding rare and novel transients and variables in large survey data streams with machine learning.

Day within June 5-6 approximate.

Key Quotes

“The onus on astronomers when working with data in the context of anomalies is not for us to write papers about how to find anomalies, but to actually find them.” – Joshua Bloom

“If we're cursed by dimensionality, we're really damned when we have real data.” – Joshua Bloom

“We don't really care about anomalies. What we care about is uncovering new physics, and anomalies are just really a stepping stone, a means to an end.” – Joshua Bloom

“Way out into the distant future, like in 2027, we're probably going to be in this world where this is the query that we all learn as grad students: please develop a plausible physical model using MHD, nuclear astrophysics, GR, that explains all the data we have on this collection of sources.” – Joshua Bloom

Transcript

Thanks everyone. So I think it should be obvious by now that the onus on astronomers when working with data in the context of anomalies is not for us to write papers about how to find anomalies, but to actually find them and then show that the physics of the objects that we are identifying as anomalous from the data are actually different than some common objects that we already know about. And so it's beyond the methodology. And when we actually go from applying these methodologies to data, you wind up seeing just how absolutely hard it is to be working in the anomaly space in astronomy. So my talk hopefully will give some context to that.

This is one of the ways that we can think about the explosive universe in the time domain. On the x-axis is time, on the y-axis is terrible astronomer units for brightness. And the stuff in the middle, type 1a supernovae, type 2P supernovae, very common. You heard from Ashley, once every second or two in the universe these things happen. They're very interesting for some types of astrophysics. People have made a cottage industry out of finding these things and studying these things, but they're very boring because they happen all the time. This is the haystack that people talk about when they say we're going to classify the haystack of transients so that we can get rid of it and look for all the needles.

One way to think about those needles is physically driven models and theories of interesting objects that we haven't seen yet. This slide was made before 2017, when the actual first kilonovae were identified — in this very short and very faint transient, neutron star, neutron star mergers produced this little blip of light. But these other things that you see up here are theoretical, have not yet been identified. And it's pretty clear based on just the light curves alone that when they happen, they're going to hit you over the head, right? So if we're able to build filters against the things you already know about, when these things happen, if they even happen in nature, we will find them. By definition in the Rumsfeldian sense, what we don't know and haven't thought about we can't even put up on this slide, but that is the underlying current of the things that people are thinking about when they are trying to work in the anomaly space.

More commonly found — and this is what Ashley alluded to in things like the changing-look AGN — is that what happens is we see these light curves, they look very similar to objects that we've already seen before, but then when we get follow-up data at other wavelengths or we take spectra, we wind up noticing that they're actually somewhat different, and then upon lots of digging wind up realizing that there's new physics. Some examples of new subclasses: we think things look like gamma-ray bursts, but then if you dig in a little bit more and get some other data, they appear to be something completely different in terms of their origins. Relativistic tidal disruption events, for the astronomers in the room, may be one example of that. And then there's also other types of weird supernovae that look one way in their spectra early on and then they transition into another type of spectra. And there's lots of different types of examples of that. But what's very clear is that for us to have a full complete picture of the physics of these anomalies, we often need to get follow-up that doesn't just come from one telescope in a few bands, but we need to get this at multiple wavelengths. And we need multimodal observations to be able to truly understand if we are looking at something novel.

So in my group we've been working with multimodal foundation models for a lot of different reasons, but I'm going to index here on what we've been able to do in the context of anomalies. This is a bit of a complex slide, but the idea is that we have a bunch of different modes. So we've got time series, we've got spectra, we've got metadata. They each have their own encoding. And then through a contrastive learning process, we wind up comparing these different modes to each other and wind up forcing the embedding spaces of all these to be the same, which allows us, as we learn on a very large unsupervised dataset, to be able to build up this kind of representation, a learned representation across all these different modes, to give us a more holistic view of these different objects. And then when you compress this 512-dimensional space — just for visualization purposes, not for analysis purposes — into two dimensions using something like [UMAP], you wind up seeing very naturally, even though we didn't give the different classes of these objects to the learning process, they very naturally wind up clustering based on their actual known or believed-to-be classes.

And what's kind of exciting here is not only are we getting that kind of clustering very naturally — so we're effectively learning in the representation space that there are these different clusters — is that we have found, somewhat surprisingly to us, there were two different clusters just in this two-dimensional space of Mira variables, even though in the original training data these were only labeled as just Miras. But then upon inspection we saw that those two different clusters very naturally spectroscopically split up to two known subclusters of Miras. So this is kind of a nice demonstration that in a very unsupervised way, and just even visually going in, not even with clustering methods, we can go in and start finding some really interesting new subclasses. So that's kind of a nice demonstration of that.

But what's also kind of interesting is that we can look for those point anomalies in places where, for instance, we have an object that shows up in an embedding space that's not where it's supposed to be according to these learned representations. And these could be objects that are incorrectly classified. Usually it's because they have things like their periods are wrong, or actually the classification catalog that we used itself was wrong, in a few cases. Or we have things that are actually correctly classified but they're weird, maybe in one of the modes. And so we have examples of those, of what we call in-class outliers, where we have a kind of normal-looking light curve in an eclipsing binary system, but then we have something anomalous where we have a whopping line there that's usually not there in most eclipsing systems. There's some sort of accretion disc process. This has X-ray associated with it, which was not part of the original data. And this is kind of a known subclass of detached eclipsing binaries that are transferring mass and producing these X-ray lines. So that's kind of a nice proof of concept as well.

But the thing we got really excited about is because we have this embedding space, we can do searches in, let's say, cosine distance of everything to everything else. And we wind up seeing this really interesting stellar-like spectrum with a massive absorption line right here. And I got really excited about this because I had no idea how to do this physically. And we got really into the idea that this may be some sort of interesting accretion disc. And then we wound up querying to find other objects in just the spectral dimension that looked like it. And there were 10 or 20 or 30 of these things, all with different light curves. And we wound up concluding in the end that what we had was a bad spectral reduction. And so, before we wrote the Nature paper, for those that remember this movie, instead of bad dates it's just bad data. And so the anomaly detection worked, that's a good thing. The problem is it wound up showing us some bad stuff in the data. And this is in a very large dataset. I don't know what went wrong in that reduction.

But this is going to happen. This is going to happen a lot. And it makes sense that we should start thinking about other ways to be finding these anomalies. In part because, as you start thinking about the curse of dimensionality, when we have these large-dimensional representation spaces, which we kind of need to do when we're looking at these complex physical objects, we're not just looking at four or five numbers that can adequately represent one of these modes. We're going to need dozens, maybe even hundreds. In that context, the distance between the nearest points in a very large-dimensional space winds up getting large, but the difference between the largest distance and the smallest one is actually getting smaller and smaller. And so what this says is that over time, in these representation spaces, if they have to be large to describe the data, one of the challenges that we have is that it's just going to be harder and harder using clustering techniques to find these outliers.

And so one idea is to use not the representation space, but use the representation alongside classification, to be able to come up with distance metrics. This is something we developed in 2012 actually, in the context of random forests, but it can be done with other types of learned representations, where very simply we're just asking, at least in this context, as we have a new object and we query a new object that's unlabeled, where does it show up in all of the different trees in the random forest relative to other types of objects. It's different than isolation forest because it makes good use of the classification. This avoids these kind of distance approaches in the feature space, and it is semi-supervised in that it learns from the actual labels. So we're actually imbuing some of our knowledge of the physics of the systems that we're looking at, to be able to get this distance metric.

And this also worked. And we had an undergrad in 2012 looking at our best anomalies, and she found this amazing object which looks kind of like a Cepheid that's extremely long period, 250 days. And we had emails back and forth saying, well, should we call Leslie Sage at Nature, is this something that we should get excited about? Adam Miller, my student, said, “This looks like a Cepheid, but the period is way too long.” And it turns out there were a bunch of these things in our dataset. We got super excited. And then we're like, we should probably look at the images. And indeed, there's like a big red Mira right in the middle there. But then we realized the photometry we had was from a large aperture. And what was happening is these Mira variables, which are largely sinusoidal, go up and down. When they went too far below, the light from these other two stars here basically dominated inside the aperture. And so we found an amazing set of anomalies. It just turns out it was bad data. So you see the theme here. Importantly, to emphasize, is that domain experts had to be in the real-time loop before we actually started going farther down the road. And it's that kind of imprecise and bad data, that you didn't usually take yourself but somebody else took for you and put in a database, that becomes a major bugaboo as we try to actually find new physics.

And so if we're cursed by dimensionality, we're really damned when we have real data, right? So if we're getting just to the edges of our very large-dimensional volume very naturally, if we now have noise and improperly characterized noise, we're going to always find these objects that are beyond the edges of what we've already seen before. And this is a real problem for us. I think Tolstoy said it best: all data anomalies are anomalous in their own way. And for those that have looked at amazingly important events, there was a massive glitch right as the most important gravitational wave event of our life happened. The fine guidance sensor on Hubble fails sometimes and produces beautiful images, and then unfortunately we have to contend with satellites that streak through our images. When you look at data you find a lot of anomalies. And so the question now is, can we start thinking about embracing anomaly detection to operate our telescopes better and produce better-quality data downstream for the end users? And so maybe we should be thinking not so much about anomalies on this sort of final resting place of data that's been carefully curated for us in databases, but doing it upstream, running it on the telemetry of logs from these telescopes, running it on raw science images to catch those kind of weirdos, and to find the distribution shifts that those that are running these large facilities can identify before they wind up putting it into a database and cause people to write papers they shouldn't be writing.

So it should also be clear, in just the last minute or two that I have, that we don't really care about anomalies. What we care about is uncovering new physics, and anomalies are just really a stepping stone, a means to an end. And so if we think about using algorithms on large databases to find anomalies, what we should be thinking about that as is a stepping stone in a much larger data flow, where we can do maybe a better job upstream. But downstream from that, we might also be thinking about ways that we can help ourselves and protect ourselves against going down some path. So how do we automate novel physical insight on this kind of autonomous scoring that we're going to be building? Well, one could be to build in these physical gut checks. So we know there's things like brightness temperature limits and Eddington limits and things like that. So maybe we have some crude physics that we fit to all of our anomalies. Or maybe we build models with inductive bias from the beginning, so our representations don't allow us to go off into La La Land. Or maybe we use simulation-based inference on some of these things, where we have good physical models that are expensive to run on lots of data but we can do it in a surrogate quick way.

So that's where we should be now. We're getting to the point in LLMs where maybe we're going to start asking agentic Occam's-razor questions like: here's some class of objects that I'm excited about, can you go off and look at the literature for me and tell me what these things might be, more mundane or otherwise? I'm assuming that there's something wrong with the data, but let's figure it out. Rank-order the ones that might actually be truly physical. This could be one of the filtering techniques that we'll get to very quickly. And then, way out into the distant future, like in 2027, we're probably going to be in this world where this is the query that we all learn as grad students: please develop a plausible physical model using MHD, nuclear astrophysics, GR, that explains all the data we have on this collection of sources. What other data should we get to test that theory? And by the way, just go get the data. So with that, I'll put up my summary, and happy to take questions. [Applause]

QUESTION: Josh, I liked your curse-of-dimensionality slide. Is there a way to work around that with dimension reduction followed by nearest neighbor and tessellation methods?

BLOOM: It is a good question. I think the way we have been working around it, and why anomaly detection has actually been doing okay, is because some of the things that we look at and we care about en masse can actually be reduced down to just a few dimensions, and then we're only working in 10 or 20 or 30 dimensions. But my supposition here is that as we start knocking off most of the objects that we're going to be seeing in LSST Rubin, for instance, we're going to have to get to more and more subtle physics, which means for us to create distinguishing characteristics, distinguishing representations, we're going to naturally be forced to have larger numbers of dimensions. But yeah, for sure we should be thinking about creating sparsity and rewarding sparsity, even at the loss-function level, in the creation of our representations. Because we just picked 512 for our thing, but maybe we would have done just fine with 128, and then you start being able to do better and better with the clustering.

QUESTION: Hi, I have a question regarding the anomaly definition. I just wonder whether, because of these many variables, we could end up with — for each, I just pick one star from the sky and I can find a definition, make it weird — and with this, a concern of how we can avoid this situation.

BLOOM: Yeah. I mean, I think that gets to that dimensionality problem, right? Everyone in this room is like the most amazing person in the universe in some axis, and let's pick that axis very carefully. And yet at the same time, we all can be well characterized by a smaller number of parameters and approximated, not with an LLM, but something that's kind of finite with a finite number of bits. So it is a problem. It gets to that dimensionality issue, but fundamentally in the end it gets to whether that star represents new physics, right? And yes, that star is going to be weird in some very special way. But does that matter, right? And does that matter question gets to: does it represent physics that we think is fundamentally different, or an improvement upon the physics that we already know? And for most of the cases the answer is going to be no, it doesn't. It's just sort of a stochastic implementation of something that we already know, those processes. And so you're kind of hinting at one of the biggest challenges, which is that if everything can be anomalous if we slice it the right way, then how do we ensure that if it truly is anomalous it also maps to truly new physics? Okay.

MODERATOR: Let's thank Josh again.

Foundation Models in Production

Thu, 15 May 2025 00:00:00 +0000

Deploying astronomical foundation models in production settings, at the Flatiron workshop on foundation models for astrophysics.

AI Accelerating Inquiry and Insight in Astrophysics

Fri, 09 May 2025 00:00:00 +0000

How the astrophysical data deluge has swamped traditional workflows and driven AI/ML tooling: real-time telescope control, survey optimization, simulation-based inference, multimodal foundation models (AstroM3), active optics, and neural compression (AstroCompress).

Key Quotes

“When people talk about astronomy being a big data place, it's true we have a lot of data, but we also have small numbers of labels, and we are label starved.” – Joshua Bloom

“The end goal of doing AI in our work is not to do AI in our work — it's to enable novel science.” – Joshua Bloom

“We're not just looking at simulated data to find simulated results. We're actually stumbling upon something that's actually fairly deep in nature.” – Joshua Bloom

“The best chess player is not you anymore as a human, nor is it a computer; it's you plus a computer. And I think that's where we're all hoping to go with this AI-assisted science.” – Joshua Bloom

“Ultimately I think we need to stop building foundation models for foundation model building's sake and start putting them into the hands of people that don't build foundation models.” – Joshua Bloom

Transcript

All right, we're going to go ahead and get started. Hello everyone, thanks for joining. We're very excited this month to be hosting Josh Bloom for our IAIFI colloquium. Josh Bloom is the Miller Professor at UC Berkeley. He has been a pioneer in the use of and development of machine learning for time domain astrophysics. Broadly speaking, you can probably think of a time domain topic; he has probably written several papers on it. He's done work in SBI and optimization of telescope resources and is now a key player in the rapidly evolving field of multimodal foundation models. Hopefully, we'll hear a little bit about that today. He also founded a startup in AI over a decade ago, before that meant writing a wrapper around API calls to OpenAI, when it actually meant something. Today he's going to be talking a lot about the various projects that he and his group have been working on in accelerating inquiry and insight in astrophysics through AI. Great. Go ahead, Josh.

Thank you. Thanks for the introduction. All right — I've been told to mention that if you have a question, then please raise your hand and I'll run the mic over so that the folks on Zoom can hear you. Thanks for the introduction and thanks for the invitation to be here. I realized when I was getting this together that the last time I gave a colloquium at MIT was the day they announced the Nobel Prize for the discovery of the accelerating universe. And I don't know what's going to come from the rest of the day, but hopefully something amazing also will happen. I also realized after I submitted the title of this talk that I just inadvertently created a really cool acronym — which maybe is not advancing. Let's try this. There we go. Maybe that's the sound of all of us screaming out into the void or something. But I'm happy to be here. It feels a little bit like coals to Newcastle, because a lot of what's happening in the machine learning world is happening in and around this institute, in astronomy. And I couldn't be more pleased to meet all the people I did today. For those that are on the Zoom, happy also to take questions by email later on.

I'll be talking today in some sense about the things that are near and dear to my heart, and a lot of the people that you see up here have worked on various aspects of what I'll be presenting today. But I hope you'll see that we're telling a broader story about where AI fits into astronomy. I thought I'd use this slide as the point of departure to help orient everyone around the grand challenges that we have in front of us. This is a light curve of brightness as a function of time for a bunch of explosive events that are outside of our galaxy. Some of these things are incredibly important and very common: Type Ia supernovae, if you want to understand something about the evolving universe on a large scale. If you want to understand the death of most massive stars, you want to understand Type IIP supernovae. But then there's some other weirdos here. This plot was made before the famous event where a kilonova was discovered — what happens after two neutron stars wind up merging, a little blip in the sky that lasts a short amount of time. And then there's these other light curves from other weirdos that have been theoretically proposed but never seen. And then of course there's the things that we don't even know we don't know, so we can't even put them up here.

We're trying to devise systems that can be looking at lots of data to be able to discover these things, to be able to characterize them, to be able to find the things that aren't on this slide. And we're trying to do this in a way that optimizes our scarce resources for follow-up. We're going to be getting — and already have really started getting — large amounts of incoming data for being able to do discovery and initial characterization, but getting spectroscopy, getting more data from other wavelengths, is going to become increasingly hard, and there's going to be increasing competition to do this. What we'd like to do obviously is write down some single optimization metric that says if we did this right — and then we could somehow train some large network to figure out how to do all of it — then we get the best answer. But we don't have those metrics. It's not number of Nature papers or negative number of Nature papers. It's something else, and we all have different metrics, and we're all competing for the same resources. So this is a very vibrant and lively and really timely problem that we have in front of us.

And this is the telescope for many astronomers that looms very large. It's been decades in the making and is seeing first light approximately this month, maybe even this week. And so we're about to get this torrent of several terabytes a night of raw imaging data, hundreds of thousands of event discoveries that are going to be coming, and we're going to have to be sifting through. The size scale of this data obviously pales in comparison to some of the data sets that some people see in physics or in radio astronomy. But for many of us, this is the thing that we're all looking towards and have been working towards, building machinery to answer some of the questions that we have in front of us. And so there's this growing need for ML imbued into the way in which we do science, into the entire data flow. Some of the things that are obvious to the people in the room, but may not be to others, is that we have a challenging problem when we take lots of data on large parts of the sky. We have a new image here you see on the left, and we have a reference image taken from many images taken over a long period of time before this new one, and we subtract the two, and we're looking for a new supernova or some other sort of transient in there. This is a challenging problem not just because of the size scale of the data but because the subtraction process itself can wind up introducing lots of spurious artifacts. I'll get a little bit more into that later on. So are we looking at something real? Are we looking at something bogus? Once we found something, we look at its light curve and we ask questions like: who cares? What is it? Do we spend precious telescope resources to follow this thing up? And then if we're trying to do inference on the things that we are observing and have gotten a lot of data for, oftentimes the forward model — the forward physical model that we have, if we even have one — is extremely computationally expensive to implement. So if we're trying to do parameter inference, we need ways to make that faster, and surrogate models and simulation-based inference are emerging as some of the tools of the trade. I'll try to touch on all of these throughout the talk today.

This is — and I'm sorry for those that are conscious about how the oil and gas industry is screwing things up these days — a terminology from the oil and gas industry called midstream: all the things that happen in the middle of the data flow. But there's some really interesting stuff downstream, after the data has been taken, after we've done the analysis. This is where the human-computer interface comes in; this is where the AI plus scientist becomes a really interesting question. And then, as you can imagine, some of the places that I'm most interested in these days is in what I'll call upstream: before data is even taken, where are the places where AI can help us accelerate and do better? I'll start off with my introduction to machine learning, which was this world where we're getting lots more images coming off of the Palomar Transient Factory telescope circa 2008, and instead of asking undergrads to look at data and find whether something was real or not — which at the time was really state-of-the-art for grid computing, and kind of still is in some realms — we wanted to create this real-time framework that could act as that surrogate. And we get some other benefits from building these machine learning systems. They're fast, they're parallel, transparent, deterministic, versioned, in a way that even with crowdsourcing you don't get all of these sorts of things. And we're talking, at the time, the subtraction capability was producing, for every one of these real subtractions of real new objects, about a thousand bogus ones that were above some sort of detection threshold. So this was a massive needle-in-a-haystack problem that we solved at the time with random forest plus handcrafted decision rules, and we put this into production. I don't know why that's dropped off here. We put this into production, and it's led to about a thousand papers so far from the PTF and now ZTF collaboration.

One of the things I'm most proud about is that we discovered the nearest Type Ia supernova in 30 years. And because this was in the front of this galaxy and not behind it, there wasn't a lot of dust, and we're able to do some really interesting things. This was found about 11 hours after explosion, and it was assisted with this real-bogus technology. We also had in place this thing which we called Oarical, which was a hierarchical classification — not just on the discoveries but then also what is this thing — and this was happening in a real-time environment. Oarical, because I also can't spell, was supposed to be cheeky: when you're up transients creek without a paddle, you need an oar from Cal. That's where that came from, just in case people are interested.

But here's the important thing. This supernova, over the next couple days after discovery, wound up getting bright enough that amateurs could see it with telescopes, and eventually you could see it with binoculars. It was so bright and so close. This would have been discovered by lots and lots of people had we not discovered it when we did. But because we were able to do this quickly, without people in the discovery part of the loop, we were able to get on this object very quickly with the Hubble Space Telescope and with Chandra. And because of that, we were able to rule out large parts of parameter space of the possible progenitor of the thing that actually exploded, invoking some theory — but we needed observations in the first couple of days. Without going into all the detail, we were able to rule out all the places in color here, leaving behind only compact objects. It's no surprise to the people who study Type Ia supernovae that we think they're probably white dwarfs that are blowing up of some sort, but it was really nice to show that we were able to find this and infer that nothing else ordinary was able to do that explosion. And again, this is because the AI system was in production at the time.

As I hinted at, this has become a bit of a cottage industry, and now instead of random forest plus handcrafted decision rules, people are using lots of deep learning techniques. And pretty much at this point, this is part of that pipeline where we can't even imagine having people in a real-time loop. We just have so much data coming in. That second question, though, is still an interesting one: what happens after we know that there's something new in the sky?

We started asking this question a few years back, looking at existing catalogs of not explosive transients but variable stars, the benefit there being that after we make an inference, even if it's been 10 days since the last data point was taken, we could actually go and get spectra and get some assurance that we were right. The goal is to take light curves like this and infer what those objects are across a hierarchy of classifications of variable stars. And unless you're good at taking Fourier transforms in your head, you don't know that this period is about a half a day, and this is a typical so-called RR Lyrae star. The catalog that we spent a lot of time with had 50,000 variables in it — the ASAS catalog from the Southern Hemisphere — but it only had 810 labels over 26 different classes. One of the things that you should know, if you're not in this game, is that when people talk about astronomy being a big data place, it's true we have a lot of data, but we also have small numbers of labels, and we are label starved. And so it demands from us a lot of attention in thinking about how we can do some sort of representation learning over much larger corpora of data sets when we don't have a lot of these labels, when ultimately it's determining the labels for all these objects that we're really interested in.

In our first foray into the deep learning world, we started building autoencoders using RNNs — there's now better techniques — but the simple idea was to take that light curve, go through a network that creates a small bottleneck that compresses this entire light curve information into a small number of numbers, and then decompress it to try to reproduce the original data. And then what we did is use this bottleneck to do a downstream classifier — in this case, we were using random forest. This is one of the first examples of using self-supervised feature learning in astronomy that allowed us to build great representations of what was happening in this variable star world using a massive number of unlabeled data sets. And it turned out it did pretty well — achieved state-of-the-art. But we're still interested in trying to figure out ways in which we can do inference with less data and less training, because there's just not enough to train these extremely large billion-parameter models for most of the problems that we're interested in. And here is where some of the work that's happened in Jesse's group — and Tess has been involved in this as well — is trying to figure out ways, like has happened in other fields, where they've been able to introduce some sort of inductive bias in the shape of the network and the way the network's trained, to be able to make use of known symmetries in these systems, so that we could train with less data and get potentially more meaningful answers out of that. And so that's the question that we're asking: can we find ways to do embeddings? Can we find new network architectures that allow us to conform to what we already know about the systems that we're trying to infer more about?

One of the ways in which we attacked that was recognizing that with variable stars, many of them are actually periodic. And so that means that if you phase-fold them with the correct period, they keep repeating themselves over and over again. The typical thing to do in a convolution, as you're trying to get knowledge of what's happening on large distances in time or in phase, is you go to larger and larger receptive fields. Typically, though, what happens in most networks is that you wind up having to pad out beyond what you've observed with your actual data — you pad out with zeros. Zero padding is a very common way of being able to infinitely scale your receptive field. We did something really simple, which was, instead of padding, we just did what we call symmetry padding, where we took the answers from the far right side, from the farthest part of the phase, and appended it over here. So we were guaranteed that when we did convolutions on these systems — instead of doing convolutions on a sinusoid and getting this out of some part of the network, where depending upon where we started our phase we get different answers — here, regardless of where we wound up starting our phase, which is an exogenous thing to the actual object (the RR Lyrae doesn't care if we started observing it around max or around min), we wanted to make sure that we get the same answer out regardless. And so by just doing this very simple trick and introducing this invariance to this wrapping, we created convolutions in a polar context, and that allowed us to tack that on to a whole bunch of different types of networks. Essentially, over a couple of different data sets, we were able to achieve again state-of-the-art in the classification accuracy, just because we made this little simple trick where we introduced a symmetry that we knew exists. So we get these probabilistic catalogs of variable stars, and the question is: so what? Here's a great paper — referee liked it, he published it — but the end goal in this work, and I would hope in all of our work as we think about doing AI in the context of astrophysics and physics, is asking the question: how are we doing novel physics or astrophysics with the result of the AI that we've built? And so I see this really as a critical place where we inform the humans to use the precious follow-up resources that we have available to us.

With a student of mine, we wound up querying our large catalog, and we were able to triple the number of very weird so-called DY Per stars, and we got a couple of new so-called R Cor Bor stars. Some of these stars were so bright the Babylonians saw them with their own eyes, and they weren't known to be of these classes because they didn't match the exact hard cutoffs that people had made to match the canonical versions of the ones that we already knew about. Instead, we're getting these fuzzy examples where we get a ranked list of objects that could be this. We took spectra of a bunch of them, and a bunch of them turned out to be right. We're also able to find detached eclipsing binaries, which, with a large amount of spectroscopic follow-up, allows us to place those objects on the mass-radius relation and compare that directly to theory, and then on and on and on. So again, the end goal of doing AI in our work is not to do AI in our work — it's to enable novel science.

Another place, which Alex alluded to, where the community is getting very excited is in these so-called multimodal foundation models. And here's another place where we could in principle train on a large corpus of data across not just light curves but maybe spectra, maybe images, maybe metadata, maybe comments about things, and hope to get out a better answer for some other so-called downstream task. And so here is one of the simple ways to do that, where you have two different modes: here you've got words and here you've got images. This idea called CLIP — contrastive language-image pre-training — allows you to do a data-appropriate embedding and then map that to an embedding space, and then a similar sort of thing. And when we know these two objects are from the same tuple of image and word, we hope that we get a large amount of diagonal power here, which allows us to align the embedding space, and then, over a large corpus, without any other knowledge of what the classes of these things are, one would hope that you could create an embedding space that has meaning across these different modes.

And so this has been done in astronomy. Some of the people in the room here have been involved in the one in transients. There's also one that was done using galaxy images and then galaxy spectra to try to coerce this raw data into a shared embedding space, with the idea that both spectra and galaxies, and the images of those and the colors of those, are telling the same story. Same thing in the transients world: training or pre-training these embedders in simulated data space, and then taking the results of that, and then taking observed data and fine-tuning that to get a good embedding of the system, and then applying that to downstream tasks.

So this is the point of departure for our own work, where we said: what if we wanted to do more modes? We want to do not just images and spectra, but we want to do other things. And here we were able to do light curves and spectra and other sorts of metadata. And we extended the CLIP idea to a multimodal setting beyond two, where, using again data-appropriate types of embedders up here, we're able to coerce that into a shared embedding space that allowed us to build this interesting model on a fairly large corpus of data, to do interesting things, we hope. So the question is: what were we able to do?

Well, first of all, we wanted to see whether this was even worth doing at all. So we checked to make sure that if you trained a data set over one of these modes — let's pick spectra for now — how well would you do if you trained using CLIP for the pre-training and then did a downstream classification task? Here we're actually just trying to classify a bunch of variable stars. And you see we did, over the entire data set, about the same pretty much through all of this, which means this didn't help that much if this was the only data you had and you already had all of the labels that you had in hand. But where things get really interesting is where we start reducing the amount of data that we have available to us, and we train in this case with no pre-training, maybe a classifier just based on the spectra. Well, if we pre-train with all of the data and we just use the spectra with only 50% of the data, now we're starting to see the big improvements. And as we go down to smaller amounts of data, we can see the improvements get very dramatic. So this is a surrogate for the world where you have lots of data, but you wish you had a whole bunch more. You could potentially train on all of these different modes and apply it just in a single-mode context, or apply it across all of these different modes. And so this is giving us some hope that these foundation models across multiple modes are actually sharing information, learning a reasonable representation, and those can then be used for at least the classification downstream task.

And what's the first thing you do when you have an embedding space? You plot it. But you can't do it in 512 dimensions, so you use a way of bringing that down to just a few dimensions. So with a UMAP projection of those 512 dimensions into two-dimensional space, we have all the different classes of variable stars that we put into our system. And you notice, for this class called M, which are Mira variables, it looks like visually there are these two different clusters that are showing up. This is not something we trained the system to do — we didn't ask it to produce two different clusters of Miras. And in fact, during the whole training it was agnostic; it didn't even know about what the classes were for these objects. So we're only colorizing them just to guide the eye. But it turns out those two different classes are actually two different types of Mira variables, which weren't in the classification training set but showed up very obviously: these C-type (carbon-type) or O-type Mira stars really show up very obviously in that separation of cluster.

There are other minority classes that we didn't even feed into the system when we were doing the pre-training of our embedding, and then we ask where they show up once we put them in after we've built this model. So these are unseen classes. The colorization might be really hard to see: there's these certain kind of RR Lyrae stars, RRds, that are showing up in the locus of where the other RR Lyrae stars are. And so again, it's giving us some indication that it's learning something about what it means to be those types of stars, or variants thereon. You can imagine using these for outlier detections. Either finding objects that have been incorrectly classified — or they maybe have incorrect periods in their metadata or something like that — or objects that have the correct labels but look weird compared to the other ones in that class. And indeed, a bunch of these things start to show up. This is an object that's a known semi-detached binary, and yet it shows up right next to where all the RR Lyrae are, and that's obviously a problem. You probably can see a bunch of these colors up here where there's obviously some misclassifications. By the way, these classifications came from a different group who use their own machine learning classifier. It's very natural for everyone to get label problems at the order of a few percent, but this is at least surfacing a bunch of the objects that may be more likely to be mislabeled.

And then there's other types of objects, which we call in-class outliers, where the classification is correct. This is indeed an eclipsing binary, but the spectra doesn't look at all like eclipsing binary stars usually do. In fact, this one has an emission line; I think it also has X-rays associated with it. It's some sort of interacting system where there's actually an accretion disc. And so this just pops straight out of looking at this embedding space and doing something like a cosine distance of an object relative to all the other objects in the space. And we have other examples of these as well, where the classification is correct — it's an eclipsing binary system, but you see weird sorts of bumps and wiggles when it's out of eclipse, maybe due to some sort of rotation or some other type of effect which isn't usually seen.

Something that I'm excited about, when we start thinking about allowing the human to go through their own journey of exploration and asking questions of large data sets, is now being able to take a query spectrum and say: I want an object that looks like this weirdo. It's an eclipsing binary system, but I want to find other systems that look like it in spectral space. And then you can find the nearest spectra in this very large corpus and ask what are their light curves. These are different types of variable stars, but you can see just visually that these are extremely similar to each other, both in morphology, fluxes, etc. But their light curves are actually quite different. And then you could imagine cross-modality searches, where we want to find things that are the farthest-away spectrum from this object, where they have photometry that's similar to each other but the spectrum is very different from each other. And you get to play all these games essentially for free once you've built these embedding spaces. So this is an enablement, I think, of a broader set of questions that we as astronomers can start asking of this data.

I want to take a different track here. But before I do, were there any questions about the first part of this talk? Yes.

I was just curious, given that you're combining all of this different survey data, if the discoveries that you've made so far are biased towards any particular survey, or do you find that by combining them you're mitigating the bias that surveys tend to introduce into these kinds of systems?

Yeah, that's a great question. I think we're heavily biased towards the data that we actually use in this case. The goal, I believe, of foundation models broadly in astronomy now is to be able to use lots and lots of data sets — spectra from multiple different instruments, light curves from multiple different instruments with different bandpasses, different detection capabilities — so that you can become more and more immune to that. I heard today about work that people are doing here where they're trying to separate out what is the underlying physical representation going on and what is the representation of the instruments themselves, so that when you apply this to a different data set, you can have some assurances that you're going to wind up getting good answers.

The thing I didn't put up here, because I was a little uncomfortable, is that we found a really, really weird spectrum, and I got super excited about it. It essentially has very broad calcium lines, 25,000 km per second, in an eclipsing binary system. And then we queried and asked, are there other ones like that? And a whole bunch showed up — some are eclipsing systems, some are not. And what I started believing, because we're just on the cusp now of getting follow-up spectra to confirm these things, is that either we've hit a very specific moment in the phase of one of these events, or, probably more likely, we've uncovered a bad reduction pipeline in the spectra. It wasn't our reduction pipeline — somebody else's — but they were very nice to put the data out for us. So it's an interesting generic problem that we're going to run into as we use these larger models, especially on new data that we haven't seen yet: when we find anomalies, are we essentially just finding interesting clusters of bad data? And unfortunately, I think the answer is going to have to be that we're going to have to start getting follow-up data to confirm whether these are real or not. Yeah.

My question actually relates to that a little bit. Can you say a little bit about how you're quantifying uncertainty in the predictions?

Yeah, so we're not — but it is an interesting question, where for a given object, given that you can see that there's noise in the data, where should it show up in this space? Is the error bar this big, or is it actually really honed in? In this case, what we have tried is the example where you bootstrap resample the light curve and move the metadata slightly around where it is — and you can do the same thing with spectra as well, because we have the noise properties — and it doesn't move that much. Most of the noise, I think, is just because we are not training on an infinitely large corpus. But there is that difference between the model noise and the data noise. But it would be an interesting question of how little data do you need to get a good enough localization in this classification space. That might get into interesting questions as you're planning surveys: for these types of objects, I need this amount of data to be this level of confident of where it shows up. Yeah, good question.

So if you use UMAP, which is typically a tool for visualization and not outlier detection — every time you run it, it's sensitive to hyperparameters and it can create a different representation. So if you use UMAP to claim a—

Yeah, we're not using UMAP at all. We just use it so we don't have to all think in 512-dimensional space. We're doing all of that work — all the math is happening in the cosine distance space.

Okay. Right. And in the full embedding space.

Yes. Yeah. I didn't show it, but you can look at in-class distances, and they follow a distribution in the cosine of their vector angles between them. And then what we're doing is essentially fitting a Rice distribution to that and then looking at the top 10% or 5% that are just on the edge of that distribution. And so the ones that are farthest away — that's for looking for in-class weirdos, but it's a similar type of idea for looking for out-of-class objects.

This is a question more on the contrastive learning thing. I assume that you have pretty good cadence in light curves and probably spectra. Do we need to assume that these two modalities have to be equally informative to the thing, so that you can actually tell they are the same thing? Rather than, for instance — I can imagine you have very low cadence photometry, or it's very early phase, so that you don't have a lot of measurements — will contrastive learning actually miss something?

That's a great question. I don't have a great answer to that, other than saying that there is this implicit notion that the data all should be pushed into the same embedding space, and we know, in particular in variable source space, that that's not true. I guess you'd call it modal degeneracy, where you could also have the case, slightly different from what you're proposing, where you have a spectrum that is from two very different types of variable stars and they look almost like identical spectra, or you have one type of variable star and it produces a bunch of different types of spectra — and it could also be as a function of phase as well. And so the idea that they should be embedded in the same space is actually wrong. But the fact that this seems to work, even though we know this happens sometimes, is some indication that the way in which these models are being trained is somewhat immune to some of those issues. My hunch is that all you're adding by having less data is more noise in the learning process. But you're absolutely right. There is a mode — like if I took an image, for instance; let's take the image mode of these objects: they're all just going to look like a star. It's a point source. So we know that would be an example where that mode should have a very hard time getting embedded into, and sharing, the same space as the others. But it would be an interesting, I think, follow-up research problem to figure out how you could potentially learn what the relative weightings are when you actually start to figure out where a new object lives. Yeah, good. Thank you. I want to turn my attention to another type of time domain event, which — those that have worked in time domain before know — is somewhere in between an explosive event and something that's happening in our galaxy around stars, and that's microlensing. You get a typical microlensing curve, which is a nice symmetric magnification due just to the bending of light around a foreground star as this background star is passing by. And these blue images are what you would get if you were able to resolve this scene on the sky as a function of time, put in these arbitrary Einstein unit times, which is related to the mass and the distance of the object and the background star. But if you have a planet that this foreground star is hosting, then you get perturbations on those nice little, otherwise smooth curves. And so the challenge here is to be able to measure the masses of these planets and the separations from their host star by looking at a whole family of these types of events.

The problem is, doing this on an individual event in the past has been very labor-intensive and very compute-intensive. It's a very large and pernicious posterior space, and even just deciding where to hone your MCMC so you don't go off the rails is actually a bit of an art and generally requires experts in the loop. And that's fine. But as we start thinking about what we could do if we could push all the way down, as some of these space-based microlensing systems are going to get to in the future — with Roman, for instance, we expect thousands of these planetary microlensing events — we just don't have enough compute and enough experts to be able to run these inference systems on every single object that we detect. So this is a place where we have these bottlenecks in people and computation, where it calls for an automated and more efficient approach.

And this is where the simulation-based inference comes in, using a so-called neural density estimator. Again, we have our light curves. We do some sort of encoding here, and we use a bottleneck around that encoding to train, in our case, something called a masked autoregressive density estimator that takes a Gaussian in the number of dimensions that we're interested in measuring, parameter-wise, and winds up producing a posterior output. And this goes back the other direction. And so with all of these neural density estimators, the goal is to be able to train a model that can, given data, go directly to parameters. So rather than try to make fast the computation, which is a surrogate model for the forward model, we're just trying to go directly to the posterior space. This field's evolving fairly rapidly; there's lots of different approaches to it. But essentially, we're able to train over a large class of these planetary microlensing systems and get, at inference time, something like 10 to the five times faster than you get for just producing the posterior space — in part because, even though it's just GR plus whatever stars look like and following light paths, the computation for just a single set of parameters can be several seconds on a reasonable machine, and if you have to do this millions of times, that starts to add up quite a lot. One of the things we wanted to make sure of when we were producing these posteriors was that we could reproduce a well-known degeneracy. There's actually two different types of degeneracies in the system that just come directly from the gravitational lens equation. You have the so-called inner-outer degeneracy or the close-wide degeneracy, which is that you can't tell from the light curve mathematically whether the planet is here or here, or whether the planet is here or here, relative to its parent star. And so one of the things we were really excited to see when we started building up these posterior systems with these neural density estimators is that indeed, for these two different types of configurations — this is what the lensing looks like in the lensing plane; this is what it looks like as a light curve — you can see by eye there's basically no difference at all between these two very different configurations of whether the planet is close to its parent star or far from its parent star. And in fact, you might be able to see in some of these places — in this case, this is the distance of the planet to its parent star — there's actually two islands of degeneracy. There's two islands of power, sorry, and this is a degeneracy. And these inner-outer and close-wide degeneracies have been studied for many years. What's known is that you can approximately, if you know the location of one of these peaks, predict the location of the other peak just mathematically. So we're very excited to see, on simulated data from Roman, we were able to do this kind of inference very quickly and get back reasonable posteriors.

And then something strange happened. My graduate student Keming Zhang started just pulling events from the prior space, just to see what those posterior spaces would look like. And it turns out that there were some weird degeneracies which weren't supposed to happen given the set of priors. And after thinking about this for a while, he realized that it looked like there was a continuous set of degeneracies that went all the way from these inner-outer to close-wide, and there was this large gap that hadn't really been studied much theoretically. And what's also really interesting is he went back to the known 23 or so systems where you have a measurement of the location of one place of the posterior and you want to predict the location of the other one. Many of these other papers would find that they were off by a few percent and would chalk up that difference to systematics which weren't well understood in the data. But instead, what we found is that we can exactly predict the location of one posterior peak to the other one. And he came up with this ad hoc equation that allowed us to do that prediction from one to the other. And so we suggested in this paper that maybe there was this deeper symmetry in these degeneracies, and published that. And then he and our colleague Scott Gaudi went off and found that indeed it exists in the gravitational lens equation and — while people had been talking about potential unification before — hadn't really been explored nor found. So this was super exciting to us, and in the News and Views it was heralded by Mróz, who wrote about this, that while this isn't going to replace people, the fact that AI is in this place now, to accelerate our theoretical understanding of the universe, is actually pretty exciting. This came out the same week — in fact, I think the same day — that the DeepMind math-proving paper came out. So it was a big moment for us, and I think a big moment also in astronomy, where we're not just looking at simulated data to find simulated results. We're actually stumbling upon something that's actually fairly deep in nature.

So this is very exciting to us, and we've been trying since to build a capability for time domain astronomers to very quickly use nbi in their own work. And so for those that are interested, I encourage you to take a look at this repo. We were trying to — I think in industry they used to say eat your own dog food, but now you say drink your own champagne — so we were trying to use this ourselves, and I had a postdoc that was working with me in my group who said that he was working on doing a fitting of APOGEE spectra with a neural model, and we said we'll just try to use our package. My student Keming said, well, the package is so good, it's just going to work out of the box. They said it's not — and it did. And they wrote this paper in a day, because they were able to get really, really good answers, where they basically had a good simulation of real spectra, doing this advertised neural posterior estimation using the codebase. So I think we're on to something, and I think for sequential data that you have, it's probably worth trying out to see if this can also work for you.

I want to just spend the last time that I have here talking a little bit about some of these other parts beyond the bread and butter, where I think a lot of people are working, in that midstream. I'll first go downstream and then I'll go upstream, and then I'll end. And this is that recognition — I'm sure all of you have started to realize — that the best chess player is not you anymore as a human, nor is it a computer; it's you plus a computer. And I think that's where we're all hoping to go with this AI-assisted science.

I run a very large software application that allows people to upload transients and visualize them and do follow-up with telescopes, etc. But there's a lot happening even before LSST Rubin gets going — tens of interesting events per night, and oftentimes it can be hard to sift through all of that. And so what we wanted to be able to do was reduce the cognitive load on astronomers as they go through this data, which many people do. We have about 100 active daily users looking at this data and trying to decide what to do next with it. So a day or two after OpenAI came out with their APIs, I wrote into this system the ability, using this prompt, to take all of the data we have about an object and produce just a human-readable summary of that. We ship off all of the data, package it, and just shove that into a vector database. So we get LLM summarizations, which is helpful to run through for a summary of a night. But we also get distances between these objects in this abstract summary space. So now we can say, if you like this object, you may like these objects. We're creating recommendation engines to allow astronomers to click on this without reference explicitly to their classification — just looking at the data and what people are saying about it. So I'm excited about that. And then there's a lot more to say obviously about all the downstream effects, but I wanted to focus a bit on the role of AI before we even start taking science-grade data. And one of the places I'm very excited about is in optimizing, again, our precious resources, both for follow-up and even survey design. Many of you are aware, of course, of that neutron star-neutron star merger that led to the discovery of the first bona fide kilonova following that event, observed in multiple different bandpasses. That itself was a needle-in-a-haystack problem: a very large swath of the sky for where the gamma-ray burst was, and then the LIGO-Virgo localization was even better. And then a number of groups semi-simultaneously wound up finding this new event. And that was a huge triumph for the gravitational wave community, for the high energy community, and for the ground-based community as well.

This is a very hard problem. We have large-format telescopes that can look at large swaths of the sky. And the goal when a new event goes off is to very rapidly find a new object like this one somewhere over a very, very large swath of the sky, covering these large bananas that we wind up getting out of our localization error boxes from LIGO and Virgo. So we've been working on a problem where we're trying to optimize the discovery of kilonovae using the Large Synoptic Survey Telescope, so that we can collect rewards for this discovery by learning a policy that, over different filter choices — and given the fact that the sky is moving and maybe our object is setting really quickly — we can catch this event by observing it in one of the possible places in its localization error box. And what I'm excited about here — and this is a paper that's still in prep — is that we're placing the reward functions not in heuristics. It's not like Atari games, whether you get the highest score or not. It's whether we can knock out the largest part of parameter space of the physics of the objects that we care about. So you can now define a metric in this, saying maybe the largest amount of volume that we're going to wind up being able to say we didn't see an object in that volume. And so that's our reward function. So placing RL in the context of the science you want to do directly, I think, is a broadly applicable way to start thinking about our use of RL. And this is for a specific type of object, for a specific type of follow-up. But now you can imagine articulating a reward function over many different science objectives.

So LSST Rubin is going to be very helpful, as hopefully we get more gravitational wave localizations from LIGO-Virgo. But it also has to work — and it is taking first light. But one of the things you should know about LSST is that it has a lot of moving parts, and because of gravity, because of temperature fluctuations, etc., we don't always know exactly the right configuration of where everything should be, to the sub-millimeter level. There's of order 100 degrees of freedom of all the different things that can be actuated in the telescope. And what they want to do is take an image of the sky, use these so-called wavefront sensors — which look like essentially images of the primary, which we call donuts — and infer what the wavefront errors are from the last image, so that when we observe that same part of the sky, we'll get a better, higher-quality image the next time. The state-of-the-art, which is now running on the telescope, is to find the donuts, represent the wavefront as a linear combination of Zernike polynomials, and then solve the so-called transport of intensity equation that allows us to figure out then what the actual actuator changes are. The problem is that at current state — as in the last week or two — it's running too slow, and the two algorithms that are running don't meet the spec.

And so we were asked a couple of months ago if we may be able to use a neural approach to this. And this is a challenging problem: we've got overlapping donuts, and some of the donuts are cut off because of vignetting. And we were able to, on simulated data at least, through basically three different networks that we're able to pre-train with data from the pre-training data set, create something that finds the donuts and figures out how we can center on those; then does inference on those donuts, for each of them, to produce a Zernike polynomial; then, with some augmented data from the locations of where those donuts are and the bandpass we're observing, uses another aggregator to wind up giving us back the results — all of this being differentiable, so that when we get new data from the telescope itself, we'll be able to learn from that.

And to cut to the chase: the two systems that have been tested and are running now — this is the spec, with this vertical line here, below about 0.1 arcseconds — you can see that something like only 2 to 5% of the data from the current system is running within spec, let alone it's running too slow. And at least now we're getting about 50% of our predictions within spec. So we're hoping to deploy this onto the telescope in the next week or so and actually get some on-sky engineering time to make sure this works. But one of the things that we think is really critical, obviously, is that it has to be in production for it to be, I'd say, considered a success — because it works well on paper, but until we actually improve how LSST actually functions, I won't consider it one of those successes.

I'll skip this here, but you can imagine extending this to active optics, where now you're getting predictions about actuation not every 30 seconds or so, or every 10 seconds, but at the kilohertz level, where you have to now move deformable mirrors potentially with 10,000 degrees of freedom — so an action space that is much larger than the types of RL places where other groups are working. So I'll end, so that we have a couple more minutes for questions, by just saying there's a lot going on both here and elsewhere in that midstream space, where we have data and we want to do inference on that. We want to do classification. We want to decide how we're going to do follow-up. And doing inference at scale absolutely demands ML — I don't think anyone would wind up arguing with that. And here the semi-supervised and self-supervised approaches and the foundation models are helping us work with this small-label problem, and showing a lot and bearing a lot of fruit. I'm excited about the HCI, AI-guided part of that, and I hope people here can continue to push on that. But it's very clear also that there's a huge white space in the upstream, before data is even taken. Can we do better? And then that means we can't just write papers about how we could do better — we actually have to put it into production. So putting AI into production is one of those big themes: we shouldn't be doing AI unless we're actually enabling great science downstream from that. So with that, I'll say thank you, and happy to take some more questions.

AI in Astrophysics

Thu, 07 Nov 2024 00:00:00 +0000

AI across astrophysics, at the cross-disciplinary workshop on AI applications to science and engineering.

The Real AI Revolution in Astronomy Hasn't Happened Yet

Mon, 13 May 2024 00:00:00 +0000

Astronomy has embraced AI for data analysis, but the real revolution will integrate AI across the full scientific workflow — telescope design and real-time control upstream, hypothesis generation downstream — via simulation-based inference, self-supervised learning, foundation models, neural compression, and RL.

Given as both the A3D3 seminar and UW Physics colloquium the same day.

Key Quotes

“We can think of astronomy as this sandbox where computer scientists and astronomers can work together to build new algorithms and to learn new things.” – Joshua Bloom

“If we think about the north star of why we're doing AI in astronomy, it's not because AI is fun or easy… it's because we're trying to do novel science. And when we stop at things like making catalogs and we don't actually get to new insights about the universe, that's where I think we fall short.” – Joshua Bloom

“I can't say that I know how to reproduce that workflow that led us to this insight, but I think it is one of the moments in astronomy meets AI where AI has really taught us something fundamentally new, or at least helped us learn something fundamentally new about how the universe works.” – Joshua Bloom

“It's not going to be replacing people with AI, it's going to be augmenting them to do their very best.” – Joshua Bloom

Transcript

And he's going to tell us about the real AI revolution in astronomy that has not happened yet. So take it away, Josh.

Thanks. Hi everyone. Well, thanks very much for the introduction and thanks for the invitation to be here today. I wanted to just show a couple of faces and names of some of the early career people that have been working with me on some of the topics that you'll be seeing here today. I wanted to present this with somewhat provocative title, not to say that machine learning isn't already a part of a lot of what we do in astronomy, but there are I think fairly large white spaces that I'll try to cover in parts of this talk.

And I hope to leave you with this sense that while we have been doing a lot and a lot of progress has been made in service of science using AI, there's just a lot more to do and a lot more that we're all very excited to be doing. So thanks again for having me here. I know that there's a wide variety of interest and backgrounds on this call, so I wanted to maybe start off with a motivating quote, if you could call it that, from a I guess prototypical data scientist. Jim Gray used to work at Microsoft, formerly at Berkeley for his PhD, and he has this quote. He says I love working with astronomers since their data is worthless.

And many of us in astronomy take this in a very positive way, recognizing the idea that the data that we have, while not backing a trillion dollar industry like e-commerce or medicine, still is of great interest for us to understand. We're asking some of the most important fundamental questions about how the universe works, its origins and its fate and all the constituents in it. And we get to try that out and try to explore the universe using whatever tools we have available to us, not just from a hardware perspective but from a software and algorithms perspective. And Jim realized the importance of this in part because our data doesn't have this intrinsic value in a dollar sense. And if we make a mistake with it, we don't leak PII, we don't start wars, there aren't crashes of self-driving cars. So that's the way perhaps that computer scientists look at us and our data. But we also look back in the other direction with an understanding that we have this great ability to bring in new algorithms and new approaches that are being developed elsewhere to improve our own science. And we can think of astronomy as this sandbox where computer scientists and astronomers can work together to build new algorithms and to learn new things.

My own science is in the time domain fundamentally, and we look at these sorts of plots of time on the x-axis and brightness over a very large dynamic range on the y axis. And some of these so-called light curves represent the bread and butter of time domain astronomy. For instance, if you're interested in studying the origin and evolution of our universe, you might study type 1a supernovae, and you see their very specific light curves as they evolve over time over the course of many months. The most common explosions in the universe are type 2P supernovae, so if you want to know about the death of ordinary stars, or ones a little bit more massive than our own, you might be looking at things like type 2B supernovae and studying those en masse. But then there are other types of objects in the universe that have either been theorized or maybe only been observed once or twice, the so-called pair production supernovae. There's some indications we've seen those before but not for sure, and so we live in sort of this theoretical world where these events should happen. We don't know exactly how often they should happen, and we have some ideas about what they might look like.

Then there's these very faint, just barely visible events that were theorized when this plot was made, the result of a merger of two compact objects called neutron stars. And when that happens the very faint blip lasts for just a couple of days, and if we could see that coincident with a gravitational wave event, that would be a fantastic discovery. So there are these known knowns, there's these known unknowns, and then there's all the things that by definition we can't put on this plot, the unknown unknowns. So we have this Grand Challenge in time domain astronomy where we're trying to not just find these things that we know about and don't yet know about, but we're also trying to optimize our followup of these with other telescopes. We might find them on one telescope but then want to observe them on another telescope with different capabilities, perhaps at higher signal to noise or with different instruments. But these are scarce resources that we look at. And if we had this magic wand and could own all of the telescopes and all the instruments simultaneously, we might have a hope of writing down some optimization metric that we could solve and then decide how optimally to observe the sky. But unfortunately we don't control all the telescopes and all the instruments, and oftentimes we're competing with each other to look at the same objects or to look at different objects, and so we can't write this down. And oftentimes we're just trying to make our most informed choices we possibly can of how we can maximize our science.

And a lot of what I just said has been going on for many years if not decades, but this whole urgency in being able to optimize, to discover, and then make decisions about what we observe in the sky is becoming really more poignant as we get closer to the turn on of the Large Synoptic Survey Telescope on the Rubin Observatory. And you see some of the numbers here. We're expecting this massive torrent of not just images of the sky but these movies of the sky where we're going to be finding tens of thousands if not hundreds of thousands of events per day. And trying to decide how to follow those up in an optimal way is the sort of Grand Challenge. So we're going from hundreds of potential events of interest a night to thousands or tens of thousands, and the community is doing everything it can to try to get ready for that.

So in this interesting space where we're getting lots of data and we need to make sense of it, I break down some of the tasks that we have into questions around discovery. So for instance discovery in images, we're trying to find a new thing. You've probably seen this before in some of the previous talks. What you have in the center frame there is an old image of the sky taken by making many images in one place and then adding those together or meeting them, and then you have a new image on the left hand side that was just taken last night, and you want to find the new object that's embedded in this galaxy. Well, the state-of-the-art is to just subtract those two images, and if you're lucky you wind up finding the supernova that's sitting in the outskirts of that galaxy. That's what you see on the right hand panel. But you can also probably notice that there are some other little white dots there. These are spurious, you could call them, detections due to the imperfect nature of the subtraction of these images. So finding just new things in the sky, discovering them, is this a real or a bogus object, is actually a non-trivial exercise, an important gateway into us being able to do science.

Once we found something, we get these ratty light curves like the one that you see here, taken over days or weeks or even years, and we have to ask ourselves this question, what is this thing that's just been found or was found recently? And given its metadata that we have about how its light is changing and maybe where it is on the sky, is it worth following up, is it worth spending our precious resources to look at it with other telescopes? And then I think the last major task in the context of time domain, although this is shared across lots of different parts of astronomy, is doing model-based inference. So we have a theoretical model that we can potentially put into either a large supercomputer, or maybe we can write down the effects of a set of hyperparameters that describe this model, and we wind up getting out potential observations. So oftentimes we go the other way, where we have new observations and what we want to do is infer what are the parameters that describe this model. So what you see here are some example light curves from supernovae, and these are theoretical light curves from large scale supercomputer simulations where we put in not a large number of parameters, something like the mass of the pre-explosion object, the energy, etc., maybe a couple of other parameters, and with some random seed and throwing it through all the physics that we know, we wind up getting out these families of light curves. We want to be able to do the inverse problem of take data and go backwards and get the values of those parameters.

So this is what I call the Midstream. But one of the things that's really interesting, and when I say Midstream the way to think about it is after data has been taken it's in the can, we want to do something with it, we want to do inference, we want to understand what telescopes we might trigger autonomously or semi-autonomously to do the observations. But there's this whole sort of Downstream thing of the data's now been taken and now we want to involve the human in the real-time loop, or in the part of the loop that we're potentially best at, of coming up with hypotheses and testing those hypotheses. And I'll talk more towards the end of the talk about what I think are the opportunities in this Downstream. But as you might guess, Upstream from the Midstream is this idea that before data is even taken there are opportunities for machine learning and advanced techniques with computation and data to be able to do better, to take data in a more intelligent way, to optimize the kinds of things that we're interested in. And I'll also talk about some of those opportunities in the Upstream towards the end of the talk.

Focusing on the Midstream for now, I mentioned this idea of trying to do discovery on lots of images. This became an imperative about 15 years ago as these large scale survey operations were coming online, and Matthew Graham was heavily involved for instance in this. From the onset we wanted to be able to find these needles in a haystack, and what you see are some examples at the bottom of real detections. And you see things that if you just were counting five sigma detections above some noise floor, those are actually bogus detections. And this is sort of, you can see that the real ones already sort of look similar but the bogus ones all kind of look different from each other. We wanted to create a real-time framework that would allow us to find and discover these new objects in the sky as quickly as possible without humans in the loop, because humans were becoming the bottleneck. We literally couldn't find enough graduate students and undergrads to look at data as it was coming off of telescopes circa 2008, 2009. So we built this machine learning system that we put into production called real bogus, and that was at the time about a thousand to one needle in a haystack problem. So for every thousand bogus subtractions or artifacts there was one real one. The subtractions have gotten better and the techniques have gotten better so it's now down to maybe one in 100 or maybe one in a few hundred, but it is still something that is absolutely essential that we have.

And one of the things that we're very proud of is all the discoveries that happen across lots of different subdomains in astronomy. The one that I'm most excited about and still most proud of is that we were able to create a ranking for astronomers of the most likely new objects, crossmatching that with nearby galaxies. Our colleague Peter Nugent and company were able to identify this new very young supernova that occurred 11 hours after explosion in our first image in 2011, and that turned out to be the nearest type 1a supernova in over three decades. Now what's interesting about this is that while amateurs would have wound up discovering it a few days later with their small scopes, and actually people could see it with binoculars when it reached its peak brightness, this object would have been discovered. But what's important is that we were able, because machine learning was in production and looking at real data in real time, we were able to get on this source with our precious resources, triggering Hubble Space Telescope or the Chandra X-ray Observatory. And it's because we were able to get there so quick that we're able to do novel science. I won't go into the details of what you see on this plot here, but all the regions in color were able to be ruled out, in some of which because we were able to get on this source just within a few hours after explosion. And that allowed us to rule out a whole bunch of possible ideas of the progenitors of these type 1a supernovae. It didn't come as any surprise that the only white space that was left over in the bottom right were compact objects. But it was sort of a I think a great vindication of the idea that we're not just doing machine learning for machine learning's sake, and we're not just doing discovery on things that eventually we would wind up discovering, but because we had ML in the real-time loop we were able to do novel science. And that I think that's for many of us

the north star of why we're putting ML into production environments. Now real bogus has become a cottage industry, and our old school computer vision techniques that we were using with handcrafted features have now given way to a whole deep learning set of approaches. This is largely a solved problem, A, and B, this is also I think a reasonable place where we can't have surveys anymore, at least at the optical infrared domain on wide swaths of the sky, where we're not using some types of machine learning. It's just too much of a bottleneck to involve humans in that part of the workflow.

Now once we do discovery, lots of science. I mentioned the type 1a supernova science that we're excited about, but the other types of things you might want to do is understand these periodic variable stars, again sticking with the time domain that's nearest and dearest to my heart. And what you see here is a depiction of 50,000 variable stars projected on the sky and their locations, and one of the light curves of those taken over many years in the top left. And unless you're good at taking Fourier transforms in your head, you probably don't know that this source actually has a period of about a half day. And if you could see the period fold of this light curve and you saw the peak to peak amplitude, you would probably know that this is an RR Lyrae star, which is a very common type of variable star. What we wanted to do is take this database of light curves and ask the question, what are these things? So classify a large data set. But the interesting thing, and this is potentially lost on those that haven't been working deeply in the context of AI and astronomy, is that when we think of astronomy we think of these petabyte scale data sets, that's a large data problem for sure, but we actually don't have a lot of labels of the things that we're interested in. I think astronomy has what I would call a small label problem.

And just to give you some details here, for this one data set there were 26 different classes of sources of variables over just 810 of those sources. So we had some of these variable stars in this data set where only three or four were known to be of this minority subclass, and so trying to build a classification model on top of this data turned out to be incredibly hard, and we had to build some Active Learning techniques for us to involve human experts to quote unquote buy more labels. But we eventually wind up getting into some of the neural techniques. The previous work was using random forest and handcrafted decision rules, but somewhat early on, right as recurrent neural nets were starting to hit other parts of computer science, we brought that into our field. And we're able to use what I'm sure is very familiar to most of you, a somewhat off the shelf auto encoder, although we had to change some of the loss functions and other types of approaches to handle the irregular sampling of the time series, to build these self-supervised networks that would allow us to learn features without the handcrafting of the features themselves. And the simple idea, as I'm sure you see here, is that you take these light curves, you create an encoding network that creates a small compressive bottleneck layer of maybe 64 numbers, and then you create a decoder, and your job is to just reproduce the outcome. Now this is self-supervised in the sense that we were able not to use just the 810 of our labeled sources, but we're able to use all of the 50,000 to learn these features. And then once we had this bottleneck layer we could use traditional classifiers like random forest to be able to do better, and we achieved state-of-the-art on this data set when this paper came out.

Now one of the other things that we started asking questions in the context of this time series data was, instead of just building these generic auto encoders that didn't know anything about astronomy, could we make use of some of the symmetry that we know exists in the data that we're looking at? And the idea of imbuing and imposing physical constraints and symmetries into neural nets in the context of science is itself not a new one. It's been done in computer science, high energy physics, quantum chemistry, and beautiful depictions of these rotation and translation equivariant convolutional neural nets that operate on the intermolecular forces in these different particles. The idea is that when you rotate something, even if you go into a different direction, you should still get the same forces out if you're doing convolutions on these objects. And so nothing should change just because you rotate something around. And so we had the same thing we could exploit in variable stars, and that is the periodic nature of many of the types of objects that we're looking at. And so we did a very simple thing, which is to take a set of different types of neural architectures where, as you look forward, as you look backward towards these sort of deeper layers, oftentimes as you go farther and farther back you end up doing zero padding. We replaced the idea of zero padding with symmetry padding. So as you go farther and farther back for periodic variable stars, eventually you wrap around and you want to get the data that you had in the previous period. And so just by doing that and using the convolutional layers, we created this idea of convolutions in polar coordinates and not in Cartesian coordinates. And the result of that, without going into all of the details, was being able to do convolutions where regardless of the starting point of where we are in time, or I guess more precisely in phase, we essentially ensure that we get the same exact results out of each layer. And so just by the symmetry padding we're able to add this back into a whole bunch of different types of networks and achieve, over a couple of different types of data sets of different sizes with different numbers of classes of objects, state-of-the-art for pretty much all of the approaches. So here we're making use of our knowledge of the data, our knowledge of the physics, exploiting that, and achieving very good results from that.

Now what's kind of interesting, and it's maybe a bit provocative to say, is who cares, right? You have a probabilistic catalog that you have state-of-the-art classification on. What does it matter to astronomers that you now have a big catalog of stars that have some potentially right and potentially incorrect labels? What do you do with a probabilistic classification of variable stars, or more broadly probabilistic classification of astronomical objects? The important thing to recognize is that all of that work that we do in AI is in service of doing novel science. And so because we had these probabilistic catalogs, we were able to ask interesting questions. We had a ranking question where we said give us your most likely of the unlabeled sources, the ones that were going to be of these very rare objects, and we got something like 20 or 25 ideas. And we took spectra of all of these with our precious resources, and we were able to triple the number of very rare types of what are called dipper stars in the galaxy, because we were able to just rank order the classification probabilities. Not all of them were right, but enough that we were able to on these very bright stars add significantly to this subfield of astronomy. We're able to identify highly eccentric detached eclipsing binaries, which allowed us with lots of follow-up observations to put a number of sources on the fundamental plot of mass versus radius, which is something that others had done before, but we were able to do this very quickly and identify the objects and the candidates because of the existence of this catalog. And then a number of other different science results came out of this work. But again, if we think about the north star of why we're doing AI in astronomy, it's not because AI is fun or easy, because it's kind of not either of those, it's because we're trying to do novel science. And when we stop at things like making catalogs and we don't actually get to new insights about the universe, that's where I think we fall short.

So we have lots of data, we have some labels, we're trying to now use all of that data. And you've already had a really nice talk from Kramer who talked about this new project called Polymathic where they're trying to build up these foundation models to take in lots and lots of data, do some fine-tuning for downstream tasks. This is a question that we have and many others have, like that group, in whether taking in lots of data and then fine-tuning it for a specific task are actually going to pan out and do better than if you did direct training for those tasks. We're now getting interested in this idea of kind of a kitchen sink where we're throwing in not just time series data but also source metadata, like locations of where it is in the galaxy or what its nearest neighbor is like, and even comments about the sources that people have made about some of these objects, creating these large multimodal foundation models for us to be able to do these downstream tasks. But it really remains to be seen how well this is going to work for the types of science that we want to potentially enable with it. So this is an open question and I hope you'll be seeing more about this in the coming months and years.

Another thing that I got interested in is in the context of simulation based inference and just doing model-based inference, is could we use ML in a way that would allow us to unlock some of the bottlenecks that we saw emerging. I talked a little bit about the computational bottleneck of being able to model supernovae with supercomputers. That's maybe you've got one object and you really want to know what its parameters are. But we're also entering this world where we're going to be getting so much data on so many different types of events and objects that even if we have not super compute amounts of compute that are needed for us to produce realistic light curves from models, we may have so many of these objects that even if it takes a few minutes to do some sort of model-based inference on one object, if we have tens of thousands or more, it starts to become computationally uncomfortable if not intractable. So I started getting interested in a field that I'll try to motivate some of the science of here very quickly, in exoplanet microlensing. So the basic idea is that we have a light curve on the bottom left hand side, it's varying in time, it's a magnification that happens when a mass moves in front of some background light source. If that mass also has a planet next to it, that is it is a solar system, you can get these sort of very fast blips that you see happening a little bit after time T equals zero, because the mass of the planet itself sort of perturbs the lensing that you wind up getting from general relativity. The light would be bending around this single mass, it's also bending around in complicated ways, a little bit like the bottom of your pool, around these solar systems that are the kinds of things that we're interested in. And so what you'd like to be able to do is take a light curve like the one that you see in the bottom and then infer the properties of the solar system that we're looking at, in particular how far away is the planet from its host star, and what is the mass of that planet. And so these are the kinds of things that we'd like to know. It's a very well-posed problem, but the problem is that it's actually fairly computationally expensive to run through these forward models. And getting the masses from one individual event is often this kind of a challenge where experts have to get involved in doing the computation and deciding what part of the large parameter space these computational models should start in. And so when you're thinking about doing something like Markov chain Monte Carlo to be able to get posteriors, it's fine in the context of having one person working on this one data set, but when we think about having lots and lots of these things it gets really scary.

Now why are we excited about this? It's because there are new facilities which are coming online, not just Rubin LSST but also a new space based facility called Roman, which has as one of its key projects to open up this white space that you see at the bottom right hand corner, where we get to find objects using exoplanet microlensing that are far away from their parent star, so that's farther to the right, and then also lower mass. And you can see some of the planets from our own solar system depicted there, and you can see all the other planets in this somewhat busy plot that have been discovered by other techniques. So this is a really exciting moment when we're about to start getting this onslaught of thousands of potential solar planets, and as we want to characterize them we need better and yet precise techniques. So this calls for automated and more efficient inference approaches. What we did, on simulated Roman light curves like you see on the left hand side, is use an autoregressive flow that allowed us, in something like one or two seconds as we get a new light curve, to produce a realistic and asymptotically correct posterior. And instead of having humans in the loop and deciding what part of parameter space to go after, we were able to show in this paper from 2001 that we're able to get, something like 100,000 times faster than traditional MCMC techniques, posteriors on realistic light curves that were going to be credible. So this unlocks a whole possibility of looking at real data.

And one of the things that we wanted to make sure that our system was able to do was to be able to recover these known degeneracies in the kinds of inferences that we'd want to make. You can see these two degeneracies that have been well studied for the last 40 or 50 years. One's called the inner outer, the other's called the close wide, and that's just a deep mathematical degeneracy that exists in the gravitational lens equation, where you don't know from your light curve whether you have for instance on the location of your planet either inside or outside of the so-called caustic, or whether it's in one of these two different configurations. So these were known degeneracies and we wanted to make sure that we could reproduce them, and indeed we could. These are the posteriors that you see on the left hand side, the corner plots that showed the covariances between parameters, but also as you can probably see if you can see my mouse, these two different islands of high posterior probability that are separated by a known amount that we know a priori from the lens equation. And we're able to show with the same exact light curve that you wind up getting essentially the identical answer whether the planet is near the host star or far away from the host star. So we're pretty excited about this. This means that as these observatories come online and we get more data, we're going to be able to do these very fast inferences.

But then my student Keming Xing started noticing that we were seeing these other degeneracies that were popping up when he was just randomly selecting events from the prior space, and we got curious about it, and it turns out those turned out to be real and we were able to reproduce them, but they hadn't been looked at before because this doesn't often happen in nature in these configurations. And what we wound up realizing, or recognizing, or maybe we just say hypothesizing, was that the degeneracies that we were seeing looked to be both new but also maybe kind of more ubiquitous, that as we moved in one direction or another they started to look like the other two well-known degeneracies. And so we made this claim that there was an undiscovered mathematical degeneracy in the gravitational lens equation that could be described by the simple equation that you have at the bottom here. I won't go into the details of it, but what it allowed us to do is go back and look at the 23 previous exoplanet events that had been looked at before, and in most of those the authors would say well this degeneracy doesn't exactly predict the location of this other degeneracy, but that must just be some unknown systematics in the data that we didn't understand. But when we went back and applied our new equation, which was an ad hoc equation that we wound up surmising from the data, we wind up realizing that indeed this ubiquitous degeneracy could actually very well describe the data that had been seen in the past. So we suggested that there was this previous degeneracy that had been missed by the theory world.

And what's exciting is that my student and collaborator Scott Gaudi at Ohio State went off and then proved that this degeneracy actually existed in the gravitational lensing equation. And so this had been kind of hiding in plain sight and we were able to uncover it, not because we were looking for it but because AI was able to give us this acceleration coupled with domain knowledge to be able to understand something deep about the universe that we didn't understand by other means. So very excited about this. And I can't say that I know how to reproduce that workflow that led us to this insight, but I think it is one of the moments in astronomy meets AI where AI has really taught us something fundamentally new, or at least helped us learn something fundamentally new about how the universe works. We're trying to now generalize the code base that we used for that work to allow others to be able to use off-the-shelf data featurization to be able to do this neural-based inference as a replacement for different types of MCMC activities that people might want to do. So this is an area of active work, and those that are interested are happy, please contact me offline and happy to connect with you. Just as an example that we were able to use that codebase for a completely different technique, a postdoc that was working with our group was interested in being able to do inference on spectra of stars and was able to use the initial versions of the code base that we were building to do inference in a very fast way, and we were able to

show without a lot of extra work we're able to sort of use this code base out of the box.

In the little time that I have left, I wanted to talk not so much about in some sense what the bread and butter is in this Midstream, where you've got lots of data and you want to do interesting science with it, but what comes sort of Downstream from that. And this is motivated in part by this quote from Garry Kasparov on this idea that the best chess player is a good human plus a machine, and that is the combination of those two that can do just incredible things. It's not going to be replacing people with AI, it's going to be augmenting them to do their very best. We've been trying to take little stabs at that using an application that I and Matthew Graham and Michael Coughlin and other collaborators on this grant that you're all part of have been doing, which is called SkyPortal. And this is an interaction platform where people come to talk about objects, to organize their followup of those objects. And we've been working with this application at scale for more than three years, with several hundred users working with hundreds or even thousands of events a night. And we're starting to get a very large corpus of data of the kinds of things that people say about them, the annotations, not just from people but then also bot annotations of lots and lots of data. We've got lots and lots of ML that's been baked into this. But the thing I wanted to talk about was how we're helping reduce the cognitive load on people who are trying to make all of these decisions with all this data, albeit in one place. How do you think about and how do you organize over hundreds or thousands of events a night?

And so there we're starting to explore the idea of taking in all the data that we have on a given object and asking different commodity large language models to be able to give us summaries that can be used and useful for humans as they're sifting through lots of data and as they're deciding for instance what they're going to be observing tonight. Without having to go through all of the data, can you condense it down to the most interesting thing? So asking something like ChatGPT through the API infrastructure, or we're also working with Claude more recently, in one single paragraph give us in a third person a statement about what's interesting about this source. So we're summarizing this for people, which is helpful. And then because we're taking the results of those summaries and embedding them in a large dimensional space with a different embedding model, we can now for a given source search for other sources that are like that source and suggest to people that hey, if you like this source maybe you'll like this one, because it's actually similar in this abstract embedding space way, different than saying it's close to it on the sky or it's got similar characteristics in things that we can measure. Now we can do this in a sort of broader but somewhat more black boxy way, and so that's actually very exciting for us. And I think there's going to be a whole interesting world where people will be able to potentially query large amounts of papers and data and start rapidly iterating through hypothesis generation and testing, even without acquiring new data.

I want to just quickly talk about the upstream and the role of AI before data. One of the things that we did is, before we even look at data, we've got sort of raw data in the can, and one of the challenges that we have is oftentimes the data that we get is corrupted by cosmic rays, so charged particles that are hitting our detectors. Instead of us seeing the beautiful galaxy that we see there, we see all these sort of corrupted pixels. And so we're using autoencoders to find these corrupted pixels and produce masks of these cosmic rays, and then taking that in a separate task and then inpainting to get a sort of prettier picture of what this image would look like if it didn't have these cosmic rays. The idea of finding cosmic rays is part of the data workflow that we have to do to get to the kind of light curves that we were showing you earlier on in the talk. And this is sort of an example of having done well. What you see on the left hand side is a raw Hubble Space Telescope image, on the right hand side is a cleaned up version of that where we found the cosmic rays and then inpainted over. And interestingly and importantly, on real data, compared to the state-of-the-art approaches which use a Laplacian transform to find the sharp edges of these cosmic rays, we're able to achieve state-of-the-art in the discovery of those cosmic rays, but then also were able to with GPUs do this much faster. So as we think about the implementation of data workflows on real data, oftentimes it's not just important to be better, but to be as fast or faster. And so we're hoping that ideas of using these neural techniques for the data workflow itself is going to start taking root. Interestingly, the thing that we were pretty excited about is as we looked at the convolutional layers that were learned in some of our networks, it was able to pick up something that looked a lot like a Laplacian kernel, which was the handcrafted idea that people had a few decades ago for being able to find cosmic rays. But of course all the other kernels were learned from this specific data that gave us better answers.

Another kind of real image that you see on the left hand side is a blurry image of the center of our galaxy that you get because of atmospheric blurring and other detector effects, from everywhere down from the telescope all the way through the detector, through all of the optics. But I'm sure many of you are familiar with the idea of adaptive optics, the idea that you can correct some of the atmospheric blurring to be able to hone in on, this is now the center of our galaxy where there's a large black hole that people are very interested in. And that idea of using corrective optics has been around for a long time. You take light that comes off the telescope, split off a little bit of that light, look at the blurry image, and then in a controlled loop be able to correct what is actually observed from your camera by changing the mirror that the light actually bounces off of. And being able to do that in real time is what gives us these very clear images that you see on the right hand side. Putting this in the context of machine learning, you might think of this as a reinforcement learning problem where we're measuring the current wavefront, and what we have, our current state, is that wavefront, but all the other actions that we've had in the past and what the wavefront measurements were in the past. And what we want to do is deform our deformable mirror, where we're basically just deciding on our piston voltages to change that mirror to do this correction, and that's our action space. But unlike going up down left right, fire no fire on Atari games where RL became probably better known to the broader community outside of CS, we have some really interesting hard problems in astronomy. That action space, if we're thinking about pistoning every single one of these pieces of the mirror, can be something like 10 to the three, 10 to the four in size, not just a dimension five. And also we want to be able to piston at potentially kilohertz. We also have interesting questions like can we learn an offline policy with a simulation of the blurriness of sky and how adaptive optics would work, and then maybe do a downstream task where we're in real time actually updating our policy model that we would wind up learning to be able to do better. And can we actually beat out the current state-of-the-art in the ways in which this is done without machine learning? So this I think is a really exciting question.

There are many other places, I'm sure Chris Stubbs has seen a lot of this before, there's thousands of subsystems inside of LSST Rubin, huge amounts of data is being generated in each of those, there are control loops there that, while they're doing well and are well positioned to make this facility the world's premier data taking facility not so long from now, there are interesting questions about whether we could actually do better with AI and the sorts of RL policies that might be learned offline and then applied in the real-time loop. And while LIGO, this is one of the pieces of LIGO that you see here, is acquiring not a lot of data about the gravitational wave sky every day, it's the kind of amount of data you might be able to pass around with a single disk, it's acquiring a lot more data, of order terabytes of control data every day, and none of this as far as I know has sort of advanced AI techniques with RL in these real-time control loops. So this is going to require CS people, hardware people, and domain experts working together to find places where there are computational bottlenecks, or in places where we have an intuition that we think we might be able to do better, and actually start testing these out and putting them into a real time setting.

The last thing I just wanted to say in the Upstream sense is something that I think is extremely low hanging fruit, and that's doing sensor fusion for a prediction of transparency on the sky. We have lots of data that we acquire about the sky. You see a little movie of the sky as the Milky Way moves through it over the night, and you can also see these low-level clouds coming through. We also have satellite imagery, we have other sensor data, and one of the things that I'm interested in and started thinking about is the idea of bringing all of this data together to be able to predict in the next minute, the next hour, maybe the next few hours, what the transparency is going to be in various parts of the sky. So that as we try to make predictions about where we're going to observe next, we can be smart about being in places that are going to be the best places for us observationally and give us the highest signal to noise. We want to do this not just in the context of generic planning, but we want to be able to plan out our observations of real objects, of real events, to be able to do better science. And what you see here on the left hand side, coming a little bit full circle, are the actual observations of neutron star merger events, of a single event that happened in 2017. And this is the actual data across multiple different bands, and you see the time scale is just a few days here. We want to be able to find more of these things. We haven't really been able to find any more after a gravitational wave event has happened on the sky, and one of the big challenges there is that the gravitational wave localizations of these events are very large, in these sort of bananas on these skies, these green bananas that you see on the sky. There are billions of galaxies that we could potentially look at with our optical telescopes, and if we knew exactly where to look we could try to obtain light curves like you see on the left hand side there, but we don't know where to look, and we have to tile across all the different localization possibilities to try to find this new object, as you see depicted on the right hand side, of a new object that appeared next to a galaxy. We want to find these needles in the haystack in this massive search space.

So I've been working with a student here at Berkeley on using a reinforcement learning policy and learning it on a graph neural net to be able to optimally observe with LSST Rubin these large bananas on the sky. And you can see a depiction of those probability regions on the sky here, and then the learned policy of where we point our telescope over time with our different filters. And we're showing that we're beating state-of-the-art, which is a handcrafted set of decision rules about how to observe. And importantly, what we're trying to do is we're trying to craft these RL policies with loss functions or reward functions that are not just sort of the generic ones of did we detect it or not, but we want to say that if we don't detect one of these really interesting events, we want to be able to rule out the largest part of the parameter space that you see here on the right hand side. And so we're couching these reward functions not in the context of heuristics but in the context of science.

So with that, I'll end by saying I'm very excited by what I've been hearing about A3D3. We have a new proposal in for an NSF AI Institute, and many other groups other than ours that have also proposed, so there's about to be a huge amount of infusion of energy and work into the AI meets astronomy world, and I hope I've motivated some of that for you here today. So I'll stop here and happy to take any questions if there's time. Thank you, that was great. That was an amazing tour of many exciting things. I think we're open for questions. Don't see any online, so can I ask, I guess I have maybe a sociological question. How are these concepts for real time followup, what kind of reception are you getting in the astronomy community? Is it being received enthusiastically by all, or is it a struggle in some or all aspects in trying to bring in new ideas for how to coordinate the followup?

Yeah, it's a good question. I think the answer is it's mixed. It depends upon the kind of sub questions that we're asking, and I think the time domain community in particular has been very receptive to any ideas around optimization. So a lot of work that went into what you've seen here has nothing to do with AI or ML, it has to do with the tooling. Just the programmatic API access to telescopes itself was a huge amount of effort. The robotization of telescopes has happened over the last 20 years, and people being open to the idea of robotization was itself a big sociological hurdle. And now that we have that, I think we're seeing more reception to the idea of let's try out this new idea. For instance, in the LSST Rubin community there's about 3% of the telescope that's been allocated at what's called Target of Opportunity. This is where a new event happens and the community writ large makes a decision it's worth going after and stopping the regular scheduled program of observing the sky in the way that they were intending to for that night, and giving up the telescope to a group to be able to observe the sky. And most of the time when we talk about this we're talking about either following up on neutrino events or gravitational wave large bananas on the sky. And so there's lots of debate about exactly how we should do that, but until we can sort of prove offline that we can get more efficient discovery of these kilonova events, we're going to be implementing other policies. And so I think my goal in the next couple years is to make sure that we're at least aware that there are these RL learned policies that could be used that may be 10% more optimal than the current techniques, and then it will actually be convincing the community that this is the kinds of things we want to do.

The last thing I will say is that the stakes tend to be a little bit lower the smaller the aperture size, and so on meter class telescopes there tends to be more interest in experimentation. One of the things that I didn't have time to talk about was the fact that with SkyPortal, that application that I showed as part of the kind of central repository of new events coming in, ML inferences happening within our SkyPortal universe and then triggering telescopes, is that we can do all of that without any people in the loop. And there's now hundreds of supernovae that have been discovered by the Zwicky Transient Factory, observed, and then discovered by real bogus, then inferred that these are probably type 1a supernovae, and then sent off to robotic spectrograph telescopes. Spectra have been taken, that data has been reduced, fit to real models, infer that they're type 1a supernovae, and then sent out and broadcast to the world. That's happening on a regular basis, on a nightly basis, without any humans in the loop. And people are doing this because the apertures are small and the cost for being wrong and observing this part of the sky incorrectly, or you did it because you made a bad inference somewhere up the chain, is not actually that big. But as we get to the larger apertures, the sociology and the pushback becomes certainly stronger.

Thanks, Joshua. Question. Nice talk. I was looking back at your slide, and I just want your perspective on using large language models, because I assume you probably face a lot of the similar problems that we face in Material Science where large language models solve everything and they solve nothing. And then I saw that you were trying to merge that information with latent descriptors. I'm just curious how that has worked, and if you think that there's a better way to maybe combine some of the benefits of language models when you need high precision sort of tasks like you do in astronomy.

Yeah, so to be clear, the LLM stuff that I showed is A, early days, and B, the cost of being wrong is not that big. And so we're not trying to go for high precision here. This is really supposed to be AI assisted, UX guided discovery and exploration. That's the downstream task where it involves a human kind of saying yeah that's interesting, I'll click on that object and see what comes with that. We're trying to measure people's interactions with those types of models. But LLMs in particular, and transformers, are the kinds of things that we're exploring with our foundation models. We're finding that it's obviously hard to do, we have lots of data to learn from, but just even coercing our irregularly sampled noisy time series data in multiple bands from multiple telescopes with lots of different types of signal noise into a single model itself is really challenging. But the idea of kind of predicting ahead of what is the next data point, which is what LLMs do, what is the next token, that's very amenable to time series. We're trying to predict what's the next flux point. And we're not trying to do that because actually prediction in astronomy is not all that interesting, it's kind of more of what can you learn from it. So what we're finding is that with our use of an LLM type of technique is that we're able to get really interesting embeddings that are quite not necessarily predictive of what's coming next, although the loss function is predicated on that. But we think we're going to be able to use those embedding spaces for downstream tasks with great precision, maybe even say of the yard precision. But the actual outputs of those, of like what is this light curve going to evolve to be, is kind of more of a party trick than it is the actual core science that we're trying to get to.

I see a question from Merlin. Hey, mine was also a question on the Netflix recommendation system for supernovae. I just didn't quite get how, could you just run me through how you were generating the text and exactly how the similarity thing works, because I just didn't quite take it in at first.

Yeah, so there's two components. So one is all the data that we have about a given object, so this is light curve data, this is spectra, this is people's comments about those objects. We have bots that come in and just dump a whole bunch of data about that object as soon as it happens, and then we have people who are actually writing summaries of it, and we even have other kind of publications about those objects. All of that data is fodder for throwing it into summarization. So you can basically put it all in a large XML file and ship it off to Claude or ChatGPT-4 and just ask it to summarize that data in human readable form, and that's sort of the first task. The second task is taking that summary and then using an embedding model, which also is provided by all the large API companies at this point, and taking this natural language and embedding it into a 1600 dimensional space, which in principle carries all the concepts of what's in that paragraph, and then saving that into a vector database. We use Pinecone, but there's plenty of others that you could use. And then when a new object is entered into that database and we look at that object, we can query that database for all the sources that are within some cosine distance of that object. So we're finding things that are like it in that space. So it's not exactly a recommendation engine, it's more of like a ranking engine. In principle though, it'd be an interesting idea of people giving thumbs up or thumbs down and eventually we wind up learning that you love supernovae and RR Lyrae like Chris Stubbs.

So I guess my question then is, how come you go from all of that metadata via the LLM's generated description and back to an embedding rather than just simply generating the embedding from the metadata? I mean, just going through that step seems like a potentially very lossy and counterproductive thing to do.

Yeah, you're totally right on that point. That's why, so we're doing that because that is the only thing that we have available to us at this moment, but the work on these multimodal foundation models will exactly allow us to get to what you said, where you take the data itself, you learn your own embeddings, you're not trusting ChatGPT who doesn't really know anything about the context of astronomy in the way that we think about it, and then learning our own embeddings and then be able to do our own embedding in our own vector database. So you're absolutely right, the roundtrip nature of that doesn't make sense for the long term, it's just the only thing we have available to us now. What we want to be able to do, what you see up on the screen, is we want to be able to learn that ourselves from our own data.

Okay, awesome, thank you. Yeah, good question. Right, more questions or comments? Okay, we're at the top of the hour, so thank you again very much Josh, that was really nice, and thanks for everybody for attending. Bye Josh, thanks.

CuratingAI Art Exhibition: Closing Panel

Sat, 27 Apr 2024 00:00:00 +0000

Founder and curator of the CuratingAI exhibition, leading the closing panel on whether artists working with generative AI tools are creators, curators, or collaborators (panelists including Fernando Perez).

Astrophysical Machine Learning

Thu, 19 Oct 2023 00:00:00 +0000

Astrophysical machine learning for a College of Letters & Science faculty roundtable.

The Real AI Revolution in Astronomy Hasn't Happened Yet

Mon, 18 Sep 2023 00:00:00 +0000

Why astronomy's AI transformation is still ahead - the earliest delivery of the argument later given as the A3D3/UW colloquium.

AI Assisted Discovery: from UX to Eureka!

Tue, 23 May 2023 00:00:00 +0000

How AI/ML has uniquely contributed to novel science in astronomy across three assistive modes: circumventing human-centric bottlenecks, accelerating physics-based computation, and hypothesis generation — including LLMs for UX and a simulation-based-inference breakthrough in microlensing degeneracy theory.

At the Flatiron Institute's Ingrid Daubechies Auditorium; video in Day 2 Session 1 recording.

A Surprising Discovery Using SBI in Exoplanets

Wed, 11 Jan 2023 00:00:00 +0000

A simulation-based-inference result in exoplanet microlensing (with K. Zhang and collaborators), at the AI-in-astronomy splinter session of AAS 241.

Accelerating Discovery and Inference with Machine Learning

Sun, 20 Nov 2022 00:00:00 +0000

Group overview of ML-accelerated discovery and inference, for the Berkeley astrophysics roundtable.

The Link Between Astronomy and ML

Wed, 14 Jul 2021 00:00:00 +0000

As Berkeley astronomy chair, with host Lukas Biewald: why astronomers were early ML adopters, real/bogus detection, uncertainty quantification, likelihood-free inference, and why ML hasn't displaced domain expertise.

Key Quotes

“Astronomers are quite good at using and co-opting tools that are built elsewhere to get our work done. Maybe the most famous example is this guy named Galileo who heard about this thing called the telescope and instead of pointing at the horizon looking for enemy ships, he pointed up that way, and the rest is history.” – Joshua Bloom

“We have objects like neutron stars, which are extremely compact stars that have the same mass thereabouts of our sun, but are the size of San Francisco. So that density, we can't reproduce that in the lab.” – Joshua Bloom

“Right now, physics, astronomy, chemistry, for the most part we're working in a world where machine learning is this really big important tool in our toolbox but it's not become the fundamental driver of how new insights happen.” – Joshua Bloom

“Doing inference and doing interpretability on the models that we build requires a fundamental understanding of the noise model of the data. And without that, nothing of what we do is going to be believable.” – Joshua Bloom

Transcript

Josh: Astronomy and physics work in a world that's sensor-based, fundamentally, in terms of our observations. Because it's sensor-based, there's noise. So, unlike in the AlphaGo-Atari world where every pixel has a perfect measurement, if you take an image of the sky or you measure some time series, there's noise associated with it. Because there's noise and because there's a finite amount of training data, if you build models off of that, you get uncertainties in the models because of its lack of expressiveness or its overgeneralization or overfitting. Then, you also have a source of uncertainty in what it is that you're trying to understand, just because fundamentally you don't have a perfect measurement, your signal noise is imperfect.

Lukas: You're listening to Gradient Dissent, a show about machine learning in the real world and I'm your host, Lukas Biewald. Today, I'm talking to Josh Bloom who is the Chair of the UC Berkeley Astronomy Department. Astronomy has been the source of many innovations in data and machine learning. It's also changed a lot due to machine learning. I'm really excited to talk to him about astronomy in general, but also how machine learning has affected the field.

Josh, thanks so much for doing this. I have so many questions about astronomy in general, as someone interested in it but not very knowledgeable about it. I'm gonna try to control myself from just going down that path. One thing I was thinking about is, it seems like astronomy has informed, or ex-astronomers have done, so much interesting work in machine learning. I was wondering if you have any thoughts on why that is, why there's such a path from astronomy into machine learning? It must have something to do with the large data sets that you all deal with, but is there something there? Even you went to a startup at some point and came back into the field, right?

Josh: Yeah. The way I put it is that astronomers are quite good at using and co-opting tools that are built elsewhere to get our work done. Maybe the most famous example is this guy named Galileo who heard about this thing called the telescope and instead of pointing at the horizon looking for enemy ships, he pointed up that way, and the rest is history. We have been co-opting tools for centuries for our own benefit. And partly that's because I think astronomers are naturally curious people, but also because we're looking for an edge, fundamentally. We are often working right at the limit of where there's an obvious answer where you have a lot of data and it's high-signal noise to where it's just complete noise. And the real discoveries are happening essentially at the 5-sigma level. We are incentivized in many ways to pull in all these different tools and toolkits from all over the place. Astronomers, obviously, aren't just using these tools, we're using a whole bunch of inference techniques and problem-solving skills in a way that I think becomes very valuable outside of the specific questions that we ask.

So, for sure, when I started a company in the machine learning space — and we can talk about the origin story of that if you're interested, and how we came to ML — we started hiring. And while we certainly weren't looking to hire people that had a similar background to us, oftentimes when we got into coding exercises and we got into solving problems, a lot of the people that were making it through that we were excited about had a physics, more broadly, an astronomy background. They were people that could work with something that they had potentially never seen before, analyze it in a way an engineer might to get it down to its constituent parts, and then innovate on top of that.

But I think you're right, the other big component, at least in these days, is the availability of just so much data, and our need to do something with that data in real time with limited resources is a natural entrée into where machine learning comes in.

Lukas: From your perspective, what do you feel like the big interesting questions are right now in astronomy? What do you feel like you might learn in the next, I don't know, a couple decades that would really change this field?

Josh: Well, it's all over the place. First of all, one way to think about astronomy is as a great laboratory for physics. So if we start there, and I think it's maybe somewhat apocryphal but Einstein really didn't like astronomers, it turns out most tests of general relativity happen in the astronomy context. There are some terrestrial ways in which we can test GR, but most of the really interesting tests of GR these days, and has been for 100 years, is by looking at the skies. Specific events and specific large-scale structure of the skies gives us clues in some of the very basics of how the universe works at not just global scale, but at a microscopic scale. So we're also testing our understanding of how atoms work and understanding even what's going inside of the nucleus of atoms by looking at what happens on extremely large scales, which is just mind-blowing to think about.

So, if we think about astronomy as that laboratory for physics, another way to ask that question would be “What are the really important physics questions that we have?” One is, “What is the nature of matter at extremely high densities and temperatures beyond nuclear density?” We have objects like neutron stars, which are extremely compact stars that have the same mass thereabouts of our sun, but are the size of San Francisco. So that density, we can't reproduce that in the lab. We need to look at how those stars behave when matter hinges upon them or just even what their static distributions are in radius and mass, to learn something about what's happening with nuclear matter at those really high densities. We don't know whether general relativity is right. It looks like it's really, really good on a lot of different scales, a lot of different mass scales and a lot of different length scales. But we're constantly testing this hypothesis that is general relativity, of whether it is a perfect description of how matter moves in the universe and how the universe is shaped by matter. We know it can't be perfect because it breaks down at the quantum mechanical scale, and there are things that happen in astronomy that allow us to test some fundamental precepts and hypotheses that come out of general relativity.

In the gravitational wave world, which is essentially the ripples of space-time due to the changing locations of matter around other pieces of matter, we've had massive breakthroughs in just the last couple of years observationally, where we've seen the inspirals of black holes, and potentially the inspiral of neutron stars, smashing into each other. In the last few seconds, there is a huge burst of gravitational wave energy which we can detect on Earth, but we can also now start to see glimmers of the idea that we can start testing some basic ideas of general relativity in those last even milliseconds. So as instrumentation gets better there, I suspect our understanding of where GR is working and where it potentially breaks down will become really interesting.

We're also interested in, at cosmological scales, understanding the expansion history of the universe, the origin of the universe. Why did the universe appear to inflate and exponentiate so rapidly in just less than a millisecond, 10⁻⁴³ seconds? Why did it grow so quickly? We know it had to based on observations at later times. What's absolutely remarkable right now is that when we look at the constituent parts that drive the dynamics of how the universe we think changes — as in how fast it grows and how fast it appears to be accelerating in its growth — ordinary matter, as I'm sure you and your listeners know very well, makes up only a few percent of that recipe. Dark matter is a quarter of it and then dark energy is the other part of it. We really don't know anything about dark energy. We don't know whether it's a particle. We don't know whether it's something even deeper than that. We don't know whether dark matter is a particle on a tiny scale that isn't predicted by the standard theory or whether it's large clumps of black holes that were left over after the primordial expansion of the universe.

So the biggest breakthroughs may come in a deep and fundamental understanding of what are those constituent parts. It may also come with a recognition that the framework that we have for understanding how the universe unfolds is right now fundamentally wrong and we'll look back on this in a couple decades and say, “Boy, we were only looking at just part of the elephant, and now when we have a bigger picture of it, things become more clear.” There's more obviously. The last thing I'll just say because I'd be remiss not to, is understanding the origins of life and the prevalence of planets that can sustain life outside of our solar system. There is a huge push, both at Berkeley where I am and then across the world, in building new instrumentation and new theory that helps us understand how planets evolve, where habitable planets could be around sun-like stars, and how we're actually going to find them, characterize them, and potentially even understand what primitive forms of life there are in those atmospheres.

Lukas: So I have a feeling this is probably an annoying question, but it comes up a lot when I talk to ML people just in casual conversation who don't really know astronomy, so I'll just ask it because I hear it a lot and I'm curious. When I hear about dark energy and dark

matter, I wonder do you really… is that just a fudge factor that shows that we don't really understand what the physical laws of the universe are? Is there a reason to call it matter and energy? Is there some sense that you're sure that it is matter?

Josh: In some sense, there are two fudge factors. Fudge factor A which we'll call dark matter and fudge factor B which we'll call dark energy. Dark matter is much better understood in how it behaves than dark energy. There's a lot of evidence that this stuff actually exists. I won't go into all the details here, but on many different scales, we have observational evidence that shows that while there are some people in the theory world that feel like they can explain away some pieces of that evidence, there is no successful alternative theory for explaining away this fudge factor with just a different way of thinking about the universe. It looks like it's actually stuff. We know it interacts gravitationally and we hope that it interacts weakly in other ways. There are lots of endeavors actually looking for dark matter within a lab or within a cave and there's some other ideas of how astronomers could actually find the details of how dark matter interacts with itself, maybe with ordinary matter. So yes, it's a fudge factor in some sense to explain the overall evolution of the universe, but it was originally discovered to explain the anomalously fast motions of galaxies and clusters of galaxies. You just add up the total mass associated with the light of galaxies, because we know how to roughly map the light of a star to its rough mass and the distribution thereof. There just wasn't enough mass, so there was this missing mass that's associated with galaxies. It turns out there's also missing mass associated with our own galaxy. We've been able to systematically rule out ordinary matter like electrons and protons, but I think the best bet is that it is some other series of particles that we haven't yet envisioned, but one day we may be able to find. On the dark energy side —

Lukas: Do you know the distribution on a scale of a solar system? Can you tell where it would be from gravitational effects? It sounds like it follows a similar distribution to matter we can observe.

Josh: That's right. Actually, we think within our own galaxy, the dark matter, which is all around us, either as, essentially, a fluid — there's particles of dark matter running through you all the time — or in extreme clumps in the form of primordial black holes. That's the other extreme that has the mass of a comet or a mountain, there'd be dozens of those flying through our solar system. There are potential ways in which we could actually discover these dark clumps and we have a whole series of observations looking for the particle version of that. It behaves a lot like ordinary matter, but in our own galaxy, while gas and stars — at least in the local solar neighborhood — are moving of order something like 200 kilometers a second around the galaxy, we think that there is a fluid, or these large clumps of it, which are moving in slightly different ways than the ordinary matter. So the ordinary matter and the dark matter, by definition of the gravitational interactions, actually do talk to each other and they do influence each other. But because the dark matter is non-compressive and unlike gas, when you smash it together you get heat, this stuff, this fluid, sloshes around back and forth. I don't know the way in which we'd be able to detect the amount of dark matter that we think must be, let's say, in the sun because there's almost certainly some amount of dark matter that's been captured in the sun. It's such a small fraction compared to ordinary matter around us that… There are plenty of ways in which it could be hiding in plain sight.

On the dark energy side, that is very much more of a fudge factor to explain the dynamics of how the universe expands. In fact, again, going back to Einstein, when he was working out some of the dynamics of the universe, he had this thing that he called his biggest blunder, which was coming up with this fudge factor constant to make the left-hand side and the right-hand side of the equation work. Then when it was found in the ‘30s and ‘40s and ‘50s that there wasn't any of this accelerating expansion, he thought it was a big blunder, but ironically we actually needed that fudge factor. What's interesting is that we have that as a constant. It's got a constant amount of energy per unit volume, that's the simplest way to think about it. But we also don't know whether it's constant in time. It could actually be changing its constant. So there could be a temporal dynamic.

Lukas: Wouldn't you see that in different rates of expansion?

Josh: Yeah. So there would be different rates. There's already different rates of expansion just because in the early history of the universe, ordinary matter and dark matter dominated the expansion, as in it was slowing up. As the universe became more tenuous and this material basically lost its dominance in these equations, there was a time several billion years ago when dark energy took over and is now the thing driving the dynamics. If dark energy is a constant and we measured it well enough, then the universe will just continue to exponentiate and grow, and will be this big… it won't be exactly an evaporation, but it will be called the Big Rip. It will basically all just rip apart from each other, and cosmology in the next hundred billion years will stop being about the observations of 40 billion galaxies and turn into just observations of stuff in our solar neighborhood because all the other galaxies will run far away from us. But we don't know that that's the case. It could actually… that constant could turn off for some reason. It could have other terms that haven't yet expressed themselves.

Lukas: Cool. Well, I have to also ask you about the other topic that you brought up on finding so many habitable or seemingly possibly habitable planets in the universe, at least that we can see. Do you have any thoughts on that? Are there theories why we don't see life on… If there's so many planets out there, why we don't see other life?

Josh: Well, I think we know now that life, at least intelligent life, is not teeming, right? Enrico Fermi had the Fermi Paradox, “If life is so ubiquitous, why aren't they all around?” It's pretty clear that it's not as ubiquitous as “Every solar system has intelligent life”, that's not a big surprise. What we haven't yet nailed down in the overall demographics is “What is the exact set of conditions that could give rise to any sort of life?” We have a reasonable understanding now that of order one solar system around a sun-like star will have of order one habitable planet. Maybe it's two or maybe it's a half, but it's not zero, and it's not ten. Then getting into the actual chemistry of what leads to — and biology of what leads to — life that's sustainable, that's really where the cutting edge questions are on the theory side and obviously we have some great laboratories in our own solar system to ask those questions — in the form of atmospheres of other planets — and we're just now entering an era where we have sensitive enough equipment to be able to measure detailed chemical properties of atmospheres of other planets and other solar systems. What I think will become clear over the next, let's say, two decades is exactly what the rate is of planets that are in habitable zones around their sun-like stars that appear to be in some sort of disequilibrium when you look at the overall chemistry and the temperature profile of those atmospheres. How is it that we have something that is a volatile element that is still around? It means that there's something else on the surface that's producing it. It won't guarantee that it's life. The question about finding other intelligent life that we could potentially interact with in some sense is beyond the horizon of modern astronomy, but there are groups, as you know, that are using modern astronomy tools to do those sorts of searches.

Lukas: When you say disequilibrium, is that something that we would notice about Earth if we were far away from it and looking at it?

Josh: Yeah. It's a little bit outside of my field, but if you took a spectrum of the Earth's atmosphere… and people have done this by looking at the earthshine. So, right around the time of the crescent moon as it's setting right after the sunset, you often can see the un-illuminated part of the moon and that's because what you're seeing is the sun's light reflecting off of the Earth's atmosphere, bouncing off the moon, and coming back to your eyes. You can take a spectrum of that earthshine and there are signatures in that that if we saw that in other planets, we would say, “Aha, there is something that…” — and I don't know the details of which element does what — “…that is not in an equilibrium given the temperature of the Earth.”

Lukas: Oh cool, interesting. It seems like astronomy has had so many advances in my lifetime, which is so cool. Do you think that that's mostly due to better equipment to see more or do you think it's better use or figuring things out from what we're seeing? I guess it must be both, but some of the astronomy experiments that I read about just seem totally brilliant of synthesizing… it seems like we could get one snapshot of the world around us, and it's incredible to me how much physics, or how many things we discover from just looking up into space.

Josh: I think a big part of that is indeed the 20th century was the opening of our eyes beyond visible wavelengths. X-ray astronomy really only started in the ‘50s, gamma ray astronomy around that time as well. Once we get above the Earth's atmosphere, which absorbs a lot of the light thankfully, at other wavebands we just see a whole universe that we either didn't imagine or only had a vague idea that could be out there. So a big part of the 20th century was just opening up our eyes to new wavebands and understanding the connection between different objects and events like supernovae, how they are connected to each other across different wavebands, and what their role is in driving the dynamics of a specific galaxy and what the role is in the creation of elements. We didn't really even know how to ask those questions I think, properly, until the last several decades. So part of it's that, and that opening of the eyes is driven a lot by technology. But then it's, “Okay, well, I have my eyes open, but they're blurry. So how do I sharpen them?” There are plenty of things again back to the original conversation at the beginning around co-opting tools. Astronomers learned about adaptive optics being used for military purposes and were able to get much clearer images of the sky because we're now pointing lasers up into the atmosphere and exciting a sodium layer high up in the atmosphere, which acts as a temporary star. We have corrective lenses that at many, many times a second are correcting the waveform errors that come as the star's light comes through the atmosphere and gets blurred. So we have all that kind of technology. Of course, digital technology starting in the early 1980s meant we were taking very high-dynamic range images of the same parts of the sky we were looking at before. So we were able to see farther away, see fainter objects, and at the same time there were a number of innovations in the telescope

world even on the ground that allowed us to build bigger and bigger telescopes. In the end, we haven't gone that far from Galileo's telescope to the world's largest telescopes, 10-meter class telescopes today. That's just bigger and bigger, collecting light. But the innovations that it's taken for us to get there have been real, and have been driven by the need for seeing fainter objects, seeing with greater clarity, seeing across more wavebands. Some of the biggest discoveries in some sense happened outside of the electromagnetic band, over the last many years. The observations of very high-energy cosmic rays — so these very high-energy charged particles moving very close to the speed of light, understanding the origins of those is still an active topic — and the discovery of gravitational waves directly using interferometers on the ground is a massive innovation that took arguably 40 years for us to get there technologically and several billion dollars of taxpayer money that went into that. It took a large number of people to be very, very convinced that the physics was right and they'd be able to get there. So the fact that they were is one of the great crowning achievements of our field: a recognition that, driven by theory, we were able to invest billions of dollars to get to a set of discoveries that we could have only dreamed of 10 years ago.

Lukas: Do you think that gravitational wave sensor was more of an engineering feat? Because it just seems so incredible to be able to send something so small, or was it more of a theo-… what was the hardest part of that?

Josh: Well, the early days — that predated me — where theorists were in active discussion about whether you could even use these circle interferometers with lasers to look for this deformation. Once people became convinced that the theory was right, you're exactly right, this became an engineering feat, which — to maybe more of an interest of your listeners — is about project management and about people management, and bringing the right people to the table with the right skill set. And recognizing, of course, that the entire endeavor doesn't need to be one big innovation, right? There are places where you absolutely need to innovate and create new things that don't exist for you to get to your goal. But to do this essentially on time and on budget on a 20-, 30-year time scale is just mind-boggling.

Lukas: Is the achievement of that just verifying that gravitational waves exist or do we have a new type of sensor that might somehow find interesting stuff in our world?

Josh: Well, without trying to prejudice where things are going, I will say that the history of astronomy — in that context of opening up your eyes to new things — invariably leads to discoveries that were unexpected. So far, I'd say the only large unexpected thing that's come out of the gravitational wave set of observations is the sheer mass, the enormity of the individual black holes that are colliding. There wasn't a lot of great motivation to say that we'd be seeing 100- and 200-solar-mass black holes that were colliding into each other. In some sense, it comes back to the astrophysicist to ask the question, “How do you even make 100-solar-mass black holes and then put them in the vicinity of another 100-solar-mass black hole?” We were thinking in the end it would be 10-solar-mass and 20-solar-mass black holes, that was the best bet if you asked most astronomers. So there's a little bit of surprise on that. None of us were surprised of the existence of gravitational waves. There had actually been Nobel prizes given out for the indirect discovery of the existence of gravitational radiation by looking at the orbit decay of neutron stars in a binary system. We had known that it was very likely that this existed, but the direct detection of that was a very beautiful vindication. Now that we're there and we're having to grapple with understanding the demographics of the black hole population, a real interesting question is how, as I was saying earlier, how can we use our observations going forward to test general ideas about general relativity?

Lukas: When I was a kid, I remember learning about the Hubble Telescope and the excitement around… in general putting telescopes into space was this big exciting project that seemed really cool. Have we gotten so good at signal processing or undoing the effects of the atmosphere that that's no longer such an important thing to do to get our telescopes up in space? It also seems like when I was a kid I had this sense that telescopes were getting bigger and bigger and we were seeing more and more things, but has that maybe stopped? Do we still aspire to make even more gigantic telescopes to see deeper in space? Where do you think that's going?

Josh: It's a great question. It depends on who you ask. There isn't a general consensus of the right answer for that, which is good because the right answer is you do what the science demands. There is a very successful satellite that was launched into space called the TESS satellite, whose sole purpose was to look for Earth-mass planets around sun-like stars and to find those just using the so-called transit technique, where one planet moves in front of its parent star and that star slightly dims. To do that, you need to see the dimming of a star in one part in 10⁵ or one part in 10⁶, which you just can't do from the ground. There's just too much atmospheric flickering that you just can't correct. We can get down to one part in 10³ or maybe one part in 10⁴ from the ground, but that's pretty much as far as we're able to go. So for finding exoplanets of Earth-mass size, you pretty much have to go into space. Rather than build a huge telescope, what they did is mount the equivalent of a bunch of glorified cannons and a bunch of glorified iPhones to look at a very large swath of the space so they could study many, many stars simultaneously. There, they weren't all that interested in looking at stars that were faint because once you discover one of these planets, you want to have lots and lots of photons with other follow-up facilities to actually do all the work. So they actually needed a very wide field to get very bright stars. But there are other people who are launching large satellites with large mirrors because they want to look at very faint explosions, supernovae in very, very distant galaxies. Yes, you can do that from the ground. It just turns out from a price perspective, there are some types of science that are actually easier and cheaper to do from space. My own interest depends upon, “Can I do this from the ground? If not, what's the simplest and cheapest thing that we can do from space?” One of the things I'm really excited about, which you may not be aware of, is there is quite a big and interesting push now towards smaller format satellites, i.e. CubeSats, in part because if you have a very dedicated science goal, and you need to look at, let's say, one object for just a month but you need to do that at one second cadence, that's really hard to do from the ground. But you could potentially do it from space very, very cheaply now because the actual parts are largely commoditized, and the launch — which is a very dominant cost for heavy space vehicles — is more or less zero because there's so many launch vehicles going up in space for all these different reasons. You can piggyback a whole bunch of these small satellites more or less for free. So, what I think you'll see in the next 10 years or so is a Renaissance not so much at the large-telescope level, but at the small-telescope level in space. And the last thing I'll just say is that we sometimes have to go to space because the Earth's atmosphere blocks certain wavelengths. So if we're interested, for instance, in phenomena at the ultraviolet sky, because of our ozone layer, we'd block off most of the UV light. So we couldn't do anything from the ground.

Lukas: Cool. Well, I guess I want to make sure I ask you some questions about machine learning also. I wanted to ask you about… so you have this group ML4Science, right? I'm curious, what inspired you to put that together and what kinds of stuff you work on there?

Josh: Well, it might be worth talking a little about how we stumbled upon machine learning in my research and where that's led to. About 12, 13 years ago, we were actually dealing with lots of images coming off of telescopes from the ground. The normal behavior when you get lots of data had been — and in many circles still is — just hire more grad students to look at the data. I was looking for ways to scale our way out of what was a very repetitive inference task, which was the discovery of new events in the sky. What we typically deal with is a new image that's taken of the sky, and you have a template image of that same part of the sky taken in the months prior where you've stacked up a whole bunch of really good images, and you subtract the two off. That subtraction process is imperfect because of the atmosphere, because of instrumentation effects. What people would do is look at postage stamps around all the five-sigma positive signals, but most of those are actually spurious. The first place where we landed in the utility of machine learning for my own research was creating what we call a real bogus detector where we trained off of good subtractions, i.e. of real objects, and bad ones because of all these different detector effects and instrument effects. We were able to build something with good enough false positive and false negative rates that we were able to put that into production and reduce the amount of time it would take a person to look at a whole night's worth of candidates from hours down to minutes, and still keep a person in the loop. At the time, I had the conceit that if we can do this, it means then we don't need people to look at the follow-up data. We can actually just get to the point of almost writing a paper without any people in the loop. But as you know well from your current work in your previous company, people in the real-time loop is still important and can be very important even when it's machine learning-assisted. So, that was very successful, and that was back in the old days of random forests, before deep learning had its Renaissance. Now, this idea of real bogus discovery happens pretty much in every project going way beyond where we were a while ago, now using modern deep learning techniques.

Lukas: Before you go further… in my previous work, I always admired the site Galaxy Zoo where they got lots of people to crowdsource some of the labeling of these images. Did you look at that at all? That always seemed like such a cool project.

Josh: I did look at that. Yeah, I did look at that. I think crowdsourcing in astronomy has been really wonderful as an outreach tool and there certainly have been some scientific papers that have come out of that. In particular, there was the discovery of a weird class of gas around certain types of galaxies that was made by somebody looking at images of galaxies. But a lot of the labeling, if I'm being really honest, by people in the Galaxy Zoo world could have been done and ought to have been done by a machine learning classifier. Is this a spiral galaxy? Is this a red galaxy? The questions that generally are asked in that world… I've done this in classes that I've taught, we have a student for a final project

try to reproduce the ROC curves of people in classification, and they can do well. We actually showed for the supernova classification challenge, we were able to build a machine learning classifier off of the original training data from Galaxy Zoo and outperformed Galaxy Zoo in a false positive, false negative sense. One of the challenges that I think all of us have in employing people to do repetitive inference tasks is to ask ourselves the hard question of “Can I have a machine do it?” If the goal is to involve people so that they're involved in research and they're helping, that's fantastic. If the goal is to get people looking at data because maybe they'll also see something and answer a question that we didn't even ask, that's fantastic as well. But for the specific task that a lot of crowdsourcing questions have asked, especially with where computer vision has arisen, we're able to do that better. Moreover we're able to do it faster and we're able to do it in a repeatable way. So one of the other challenges, of course, if you ask somebody to label a bunch of data and then you ask them to come back tomorrow after they've had a beer and label the same data, you'll get a different answer. From understanding the demographics of everything we see, I think it becomes a lot harder when you have people that are deeply part of that process.

Lukas: Got you. So, I cut you off though. You were doing this quite a while ago and especially vision techniques, I think, have massively improved. I don't know even “especially vision techniques”, but there's this moment where vision got quite a lot better. Did that affect the way you used machine learning in your work?

Josh: Yeah. We always are careful in a sense to try not to look around in astronomy and say, “That's a computer vision task. That's clearly solvable now by CNNs, so let's go work on that problem.” There is a little bit of “everything looks like a nail because we have this really cool hammer”. That was a computer vision task, this real bogus detector that we had to solve if we were to break this grad student bottleneck. There are plenty of tasks that people are doing asking questions of images that were around before, but perhaps weren't as interesting because we had no way of solving those problems and now we can do those at scale. I actually focus less on images now and focus more on irregular time series data. But I think one of the important things to recognize about where astronomy is, maybe relative to many of the other fields of the people you've talked about on this podcast, is that we haven't had that moment that maybe existed in NLP where Jelinek said “Every time I fire a linguist, my language detector gets better.” The idea that if you start removing domain experts out of the loop and you actually start building language models, just learning off of data, and it gets better and better — we don't have that moment in astronomy. Computer vision is the same thing too. You fire a bunch of old-school computer vision experts that learned about Hough transforms and stuff, and now you just throw it into a big CNN with lots of training data, and you get better answers than what you were ever able to do in the past. That hasn't happened in astronomy. We've used ML in lots of different places as accelerants and as surrogates to harder computations so we can get faster answers. We can do inference at scale in ways we were unable to do before. But it's the same thing in biology, right? ML didn't invent CRISPR and Katalin Karikó — who was this person who toiled away for decades trying to understand how mRNA could lead to a vaccine — she was actively denied tenure and actively denied grants. She had nothing to do with ML, but if it wasn't for her we wouldn't have vaccines for COVID. Biology, I think, also hasn't had its ML moment where you can start firing domain experts and start doing things. Right now, physics, astronomy, chemistry, for the most part we're working in a world where machine learning is this really big important tool in our toolbox but it's not become the fundamental driver of how new insights happen.

Lukas: I guess one key difference here though is the work product of astronomy is explaining the world that we live in, right? Whereas… first of all, I'm not sure if I agree with the comment about linguists. I don't want to go on record —

Josh: It's not my quote.

Lukas: No, I know, I know what you mean. I think definitely linguists still do the best job of explaining language in the… I think linguists would probably say we use ML techniques to understand language better, not that we've replaced ourselves… although, it seems like modern translation techniques are less informed by linguistics than I might have expected when I was younger. I wonder if it's a function of a domain being more “trying to engineer a certain solution to a specific problem” versus “do some kind of explanation”. We've actually talked to a whole bunch of biologists and it does seem like some of the processes around drug discovery are starting to be more and more informed by ML, and moving in that direction.

Josh: I think that's where we exactly are right now: there is a huge amount of ML that is informing astronomy, but I don't think we're anywhere near where the NLP world is. In part it's because, to your point, we haven't really been able to articulate a set of outcomes that are comparable or have as much weight or as much import as an NLP task like translation. You can directly correlate, I assume, the quality of a translation from language A to language B to some dollar outcome. And in astronomy, we don't have the ability to do that. So our loss function is a little bit more complicated. As we're learning these various different tasks as part of our workflow, we don't have the ability in the same way many other fields do to articulate that loss function in terms that have this monetary value. When you ask this question about “What is the nature of dark energy or dark matter?” or “How many exoplanets are out there that host life?", those are in some sense quantifiable answers. But as you're saying, that's where more of the explainability has to come in. I certainly don't think we're even trying to get to the point where we fire a physicist so that I can hire a computer scientist. It's going to be the marriage of those two people, or as an individual and their skill sets, who are going to make a lot of progress. I think the really exciting place where we could get to — and there are little tiny pieces of this starting to happen — is whether an application of ML to a bunch of data can be something that leads to a discovery on a bunch of questions that we didn't even know how to ask. That would be a real hallmark moment in our field. Right now everything is done in largely a supervised context. Obviously, we've had some semi-supervised and unsupervised ways of looking for anomalies and outliers, and things like that. But even that, it becomes a guide to a domain scientist looking at this and saying, “Oh, yeah. Of course, I know what these things are”, or “This is because the data is spurious.” Maybe what's really fundamental, if I think about it, is that the job of these ML pipelines that we build on different parts of our data isn't so much about prediction in the same way that if I need to predict what the next word is or I need to predict if this is a cat or a dog or what the best thing to show somebody is next, that is the proof in the pudding and you've done well because you can measure what the outcomes are after that. If I make a prediction in astronomy, that's really just for hypothesis testing. If I have a new theory that's gleaned off of data, the job of that theory is to make a prediction about what happens if I observe outside of the domain of the data that I already have, to falsify itself. We haven't really wrapped our head around the idea that ML in the context of the physical sciences isn't just about making predictions at scale so that we can get slightly better data farther down the work chain. If it's going to actually drive our deeper understanding of how the universe works, it has to couch itself in the terms of hypothesis testing, Occam's razor. We haven't really gotten there yet.

Lukas: I'm so surprised to hear you say that because it seems like we fund all this work to make better devices and telescopes, and it seems like

they pay us back in terms of these really awesome new understandings about the physical world. It seems like you make a bigger telescope, that's just seeing things slightly better however you put it, right? Isn't it similar? If ML can help you get better data to inform your predictions, wouldn't that be a big deal? Does it really need to… Do you really need to be completely replaced by ML for it to be…?

Josh: No, I certainly don't want to come off trying to make the argument that ML hasn't been important. We're currently working on a project where a big part of that whole data taking, data planning, data reduction, initial discovery, initial inference, initial follow-up, that all happens. There's little pieces of ML through that entire chain, that all happens without people in the loop now, which is absolutely incredible. Telescopes more or less talking to telescopes mitigated by ML. This is where we are. There's only going to be more of that going on. What we're doing is we're optimizing our resources and our resource allocation because we're using ML. But I still see that as fundamentally an accelerant and a surrogate to what we were pretty much doing in the past. I haven't seen anything that fundamentally changes the way we conduct ourselves as physicists. But again, as I said, there are little pieces starting to show up, [like] the rediscovery of the Higgs boson using pure ML without reference to the basic physics of how particles interact.

Lukas: Do you think that would've worked without knowing about it? Is that —

Josh: That's the question, right? Until we get to the point where someone says, “I ran my ML pipeline on this particle physics data and I saw this new thing. And everyone in the group didn't believe me until they got 10 times more data and it turns out it was there.” We haven't really gotten there yet. There's been a few places where people have found another exoplanet in a complicated data set that people hadn't seen before. But astronomers for the most part are still Bayesians, and we're still governed by Bayes’ rule where we come to our problem with a bunch of priors. We get data that updates our beliefs and we get slightly better, or sometimes much better understandings, in our posteriors. If we talk about inference and understanding, we need to couch it in terms of what we think are the physical properties and the physical things and the parameters that describe the object that we're looking at. We're getting better at that. One of the big things I'm doing in ML right now is trying to use different types of networks in a whole new class of approaches called likelihood-free inference to go directly from raw observations to posteriors or proximate posteriors. I think that's extremely exciting and can be applied to a whole bunch of different places.

Lukas: Cool. That's so cool. One thing that I wonder about… it must be interesting being in your shoes. Are you doing most of this with ML grad students? Are you at the point with your data pipelines where you need to pull experts from industry or maybe… it's so funny how much of our data pipeline stuff has come originally from astronomy and sciences in general, so maybe it's always at the cutting edge, or do you feel like you need to get experts in terms of just handling these volumes of data sets and building these gigantic models?

Josh: It's a great question. So again, I think the answer depends. There are lots of examples in my own research and my own work where hiring very good data engineers, and having some ML expertise on the team, suffices. It's where you actually need to innovate, create new algorithms, take some existing network, and completely blow it up and change the way that it works, that you do need somebody with deep domain background in CS, ML. One of the beautiful things about being on the Berkeley campus is how everyone across campus is looking to work with each other because, again, we all recognize, at least from the physical domain side, that there is incredible work that's happening in the Computer Science department, the Stats Department. I've just become a member of the faculty of BAIR, the Berkeley AI Research group, so I get to interact with those grad students and those postdocs. We still, I think, face the challenge that any academic arena does when crossing over into other fields of trying to make the kinds of problems we have compelling for the other side, and have the other side recognize that they're not just setting up a Spark cluster for us, and downloading ResNet. What the people in computer science and stats need to realize is that we are asking questions of data in a way that they are not — of the benchmark data sets that they're often working on algorithmically — and because of that there needs to be some real fundamental innovation. I've been really fortunate in my career to have gotten grants that have allowed me to hire people outside of the traditional astronomy background. I hired PhDs in computer science and statistics, and that's where some of the most interesting innovations happen. Where we're at, I'd say now as a profession, is really struggling with the idea of how much should our students have to learn for them to be able to work on this as their main endeavor. We don't have as part of our curriculum a deep training in stats, let alone ML, let alone software engineering. I don't know where they find the time, and where they will find the time going forward, to be able to get all of that in at a fundamental level. We're working on it, we're trying. Berkeley has started a new data science major that the astronomy department is connecting up into with their own classes. But there isn't, at the national level, a holistic understanding of how we're going to do training of the next generation of physical scientists, so they're not just conversant in ML but they can actually do a bunch themselves.

Lukas: Actually, the question I wanted to ask you — which, this is a good segue — when I was looking at your website, I found hundreds of research papers, but also mixed in some opinionated blog-like posts on programming language details. I'm wondering for you, and maybe something I'm asking myself, how do you stay current on… how did you even find time to get to high-level programming at all and how do you stay on top of that? Are you spending time writing code yourself as a professor?

Josh: Yeah, I write a lot of code. That's some of my happiest times. In some sense, that's my hobby. I came to programming early on in my academic career when I was an undergraduate, where I was basically told by my future advisor at Los Alamos I can't work there for the summer unless I've taken a class in C, and I did. That was more or less the only class I ever took in computer science. Again, it was this matter of necessity, just like it is with building better telescopes. I decided, when I was a postdoc, to automate an old mothballed telescope — which was a fairly large one-meter class telescope in Arizona — and take all the pieces that had been manual when the telescope had been run before and automate every single piece of it so it could run autonomously. I asked a friend of mine at Los Alamos, “Which language should I write in?” And he said “Python.” I said, “What's that?” He said, “Just do it. It's a cool language.”

Lukas: Wait. What year was this?

Josh: That was 2002.

Lukas: Oh, wow.

Josh: I wrote a whole telescope automation software package using state machines and connecting up to device drivers in C++ in 2002, where I was just feeling my way through it. I think I wrote my own datetime module and I didn't realize that datetime was there. So, I just stumbled upon it. And then what you do when you're an academic and you wind up realizing something is interesting, is that you feel bad that you're not teaching it to your students, so you do. So I started in 2008 a bunch of Python boot camps on campus to get people into Python, in part because especially at Berkeley, we caught the open source ethos pretty early and the kinds of languages that people around me were using — like IDL, interactive data language, and MATLAB — were just expensive. Moreover, as scientists, we certainly want to understand the algorithms that we apply and we want to at least be able to look under the hood if we need to. I started evangelizing Python around these parts and started building classes on top of Python. So, a graduate level seminar on “how do you actually use Python in the real world”, ranging everywhere from doing stats to scaling up Python programs to testing frameworks to interacting with hardware. That class

still goes on, but I've got to say I've ossified a little bit around Python. I've spent a little bit of time with a few other languages, but for me I've become conversant enough and gotten fairly deep into this scientific Python community. Jupyter, for instance with Fernando Perez here in the Stats Department, has really been a huge part of what I've used for teaching for a long time. The NumPy and the SciPy stack have a lot of activity here on campus as well. Stefan van der Walt has a huge role in that. So, it's in the water, I'd say. Definitely, the proof is in the pudding, having recognized that Python is extremely versatile as a superglue language for all the kind of stuff that we do. Yes, I still code. Last summer during the pandemic, the happiest times were me learning React, so I could build this large-scale React app that we're doing for astronomers to interact over data.

Lukas: Isn't React fun? React is so much fun for me, I thought I hated frontend.

Josh: I don't know if I would use the word fun. What I love about it, it's just so wonderfully different than the way you think about Python programming. And obviously, it's rewarding in a sense that you build it, you ship it, and users see it right away, in a way that if you build some cool Python tool, you may be the only one in the world that uses it. Just because it's on PyPI, it doesn't mean that somebody is actually going to download it, use it.

Lukas: Well, did you use TypeScript with React?

Josh: No.

Lukas: No?

Josh: We were like JSX kind of…

Lukas: I see, cool. Maybe I just like React because I think I was writing a lot of frontend stuff around 2008, and found it frustrating, and then went back to it a few years ago, and just was impressed how much things had evolved in the decades.

Josh: I love React, but I don't like testing React apps.

Lukas: I was trying to do some typing stuff recently, actually it was with your student Danny, and I was really wishing that Python's typing worked a little bit more elegantly, especially in the scientific computing domain. I felt like when I was doing research briefly, the code bases were truly messy in a way I've never experienced in industry — this may be a long time ago — do you do things in your lab to keep things maintainable? Maintainable, as students come in now and they need to do various research projects. Are you able to find time to clean up code and eliminate tech debt and things like that?

Josh: I think we're probably better than most, but never as good as I'd like to be is probably a reasonable answer. I'm at least aware of the existence of things like unit tests, unlike many of my colleagues in our field. Yeah, it is a mess. Again, it comes back down to loss functions and incentives, right?

Lukas: Yeah, totally.

Josh: When we write a grant, there is no imperative — as much as I think it'd be great — to say, “By the way, one of the outcomes if you're writing code has to be that this is going on GitHub and that it's going to have a CI/CD like a Travis attached to it so that when pull requests come in, you know whether they're going to be working or not.” There's none of that at all. So if you do any of it, you're doing it out of the goodness of your heart at zeroth-order, but as you know first order is because you're doing it to help yourself in the future. Oftentimes, in a research context — and this gets back to a question you were asking about, “Do you need to hire ML people to work with massive amounts of data?” — what I was going to say is that not all of what we do is massive data. Astronomy has a lot of data, but we have only a small number of labels for instance. It's a big data problem, but actually a tiny number of labels for the kinds of stuff that I'm interested in, or zero labels. So how you do one-shot learning is a really interesting kind of problem in a physical context in the presence of noise and uncertainty and model uncertainty. There's lots of questions that we ask in the context of ML that are actually small-ish data problems, or they're large computation problems because the forward model is extremely expensive and requires a supercomputer, but the amount of data we're dealing with is thumb drive-level. But because of that, we tend to atomize our activities around projects, around papers. I read a paper with a student, we figure out a cool new thing to do in the machine learning context, and unless that is going to be a major new widget that gets plugged into some new facility or existing facility, then it's just out there in the world and people can write papers saying their scaling curve is better than my scaling curve and we can have an argument at a conference one day. That's the end of that code base, right? Whereas, as you know, in the industry world, you're generally not writing code as a one-off and then just casting it aside. So the incentives there to keep things maintainable, keep things up to the latest versions of Python and blah, blah, blah, they just really aren't there for most of what we do. There is a subset of what we do where it absolutely has to be battle-tested because more and more people are going to be downloading it and using it. I tend to see those projects as extremely exciting, but there's not a lot of, I'd say, astronomers who have the experience with full CI/CD pipelines and in production dev ops that I've been lucky to have in my career.

Lukas: Let me ask you this, what does your lab's tech stack look like? Are you using PyTorch? What's your standard tooling? Are you on Python 3.7?

Josh: It's actually pretty agnostic as students have come in, because students tend to gravitate towards me who are interested in ML and I naturally gravitate towards them. That's how, I guess, gravity works.

Lukas: Sure.

Josh: I've been agnostic to whether it's TensorFlow-land or PyTorch-land. I think that's becoming less and less important as TensorFlow has evolved more towards the PyTorch way of thinking about the problem. If you said, “Build me an ML thing right now,” I'd probably start in Keras just based on my own past experience. But obviously I'm looking at code with PyTorch, and PyTorch Lightning I think is — from a teaching perspective — the last time that I had to teach some ML, I was doing it in PyTorch Lightning. Although, I had a notebook in Keras and I reproduced the same thing in PyTorch Lightning. Of course, we had Weights & Biases there as well for monitoring.

Lukas: Nice. That really warms my heart.

Josh: I've been introducing a new cohort of people to your product.

Lukas: Thank you.

Josh: That's obviously top of the stack. It very much is a Pythonic world now. As I was saying before in this other large project, which is called SkyPortal, that we're using as an interaction platform — with, now, hundreds of people are using it on a daily basis looking at real data as it's flowing in and interacting over individual objects — that tech stack is obviously more complicated. There is a component of it which is slightly external to the stuff that I built in my group — but is part of our project — which is more or less a large MongoDB engine that's dealing with terabytes of data, and there's a bunch of ML plug-ins to that that run in real time. And that's, I think, using TensorFlow. Then what we've built is essentially a Tornado-based, API-first backend and it attaches to a really large Postgres database, and on the frontend is React.

Lukas: Cool. Well, we're almost out of time, maybe we're even overtime, but we always end with two questions that I'd love to ask you. The second-to-last is, basically, is there a topic in ML that you think should be studied more than it is? Is there something that you would look into if you had extra time on your hands?

Josh: Yeah. There are a lot of things I wish I had more time for. Where I think there needs to be more work in the ML world is around UQ, uncertainty quantification. Astronomy and physics work in a world that's sensor-based, fundamentally, in terms of our observations. Because it's sensor-based, there's noise. So, unlike in the AlphaGo-Atari world where every pixel has a perfect measurement, if you take an image of the sky or you measure some time series, there's noise associated with it. And because there's noise and because there's a finite amount of training data, if you build models off of that, you get uncertainties in the models because of its lack of expressiveness or its overgeneralization or overfitting. Then, you also have a source of uncertainty in what it is that you're trying to understand, just because fundamentally you don't have a perfect measurement, your signal noise is imperfect. I see some of that research, again, coming out of the ML world, but I see some of the stuff I'm most interested in as coming out of the physics-and-astronomy-meets-ML world. I'd love to see more of that more broadly. I think it's partly our fault as domain scientists for not coming up with the equivalent of grand challenges like with protein folding, where if we had this, we would be able to make great strides. We need to have not just benchmark data sets for other fields to be playing with, but we also need to be really clear about some of the important questions that we're asking. I think in the end — back to a lot of what we were talking about throughout this whole interview — doing inference and doing interpretability on the models that we build requires a fundamental understanding of the noise model of the data. And without that, nothing of what we do is going to be believable.

Lukas: Interesting. I guess that's a good segue into my final question, which is, when you look at making the machine learning models actually work for you — actually do something useful — what are the big challenges that you typically run into?

Josh: Well, it is a good segue from the previous one, because we are struggling, I'd say, as a community with recognizing that there is this large algorithmic toolkit that has been developed in the computer vision / NLP world that we could just take, make a couple modifications to, and do what we're already doing better, faster, and at scale. And as I was arguing through the middle part of the interview, that isn't where I think the biggest revolutions are going to come from, or at least I hope that's not where they come if ML is going to wind up being involved. One of the harder problems is articulating what are the really hard problems in astronomy that can only be solved with new ML tools or new ML innovation. We're all working on it in different ways, we all have our different biases, I think we may wind up getting there. The other one is maybe more practical, which is that it's very hard to put machine learning into practice. It's easy to write a paper on machine learning and convince a referee that you're doing pretty well. Maybe release some code. Maybe have the referee kick the tires on that code. That's pretty much where we're at as a community. But trying to get it into a real workflow that affects real people's lives on the other side of that, there's not a lot of us that have experience with it. No one's really trained to do it well. So most of the time when it's done, it's done in an ad hoc way, leveraging some understanding of how software engineering is supposed to work, but as you know well, machine learning in production is a very different beast than ordinary software in production. I don't think as a community, we fully grasp how hard it is. The other side of that, of course, is that because machine learning is so exciting to so many, we're starting to train a number of students that have just enough knowledge to be dangerous. But because again everything looks like a nail when you've got a new hammer, a lot of people, I think, are going off hitting nails that they ought not to be. One of the things that I always say when somebody says, “What's the worst thing about machine learning?", is I always say, “It's because you always get an answer.” Especially in the context that we're looking at, if we always get an answer and we're getting data that's outside of our original domain or some notions of concept drift or something because the instrument is changing, we don't have any guardrails against that. Luckily, unlike in many of the fields that your listeners work in, if we make a mistake, people don't die, and we don't blow up billion-dollar facilities, and things like that. So we live in a little bit of a nice sandbox where the mistakes that we make may have implications for lack of good resource allocation. But we still could wind up making statements about how the universe works that is fundamentally wrong because we don't know enough about what's happening under the hood.

Lukas: Josh, thank you so much for your time. I really appreciated it, that was super fun.

Josh: This is great. Thank you, Lukas. Great questions.

Lukas: If you're enjoying these interviews and you want to learn more, please click on the link to the show notes in the description where you can find links to all the papers that are mentioned, supplemental material, and a transcription that we work really hard to produce. So, check it out.

Machine Learning in Astronomical Surveys (ML Club intro)

Wed, 03 Feb 2021 00:00:00 +0000

Overview of ML in astronomical surveys framed around Rubin/LSST: real-bogus classification, inpainting for artifact removal, anomaly detection, calibrated Bayesian classification, similarity search at scale, and MLOps for reproducible science.

Physics-Informed Machine Learning for Inference & Discovery

Tue, 15 Dec 2020 00:00:00 +0000

Physics-informed ML for inference and discovery, in the astronomy track of the Columbia-organized MLSE conference.

Physics-Informed Machine Learning in Astronomy

Thu, 15 Oct 2020 00:00:00 +0000

ML across astronomy workflows: the Planet 9 search, new variable-source classes, semi-supervised VAEs generating realistic time series to optimize observing schedules, and deep-learning cosmic-ray detection and correction.

Originally scheduled Apr 17, 2020 in person (cancelled); delivered virtually.

Key Quotes

“There's obviously a lot of interest in optimizing ad clicks, and understanding Twitter sentiment perhaps, but understanding how the universe works I think is pretty interesting and pretty fundamental.” – Joshua Bloom

“It's a little bit like Anna Karenina: all real objects all look the same. All bogus objects are all different in their own way.” – Joshua Bloom

“When it comes to machine learning, it turns out we have very few examples or exemplars of the objects that we care about. So we have a big data problem, but we also have a small label problem.” – Joshua Bloom

“With machine learning, you always get an answer. And that is a wonderful thing, but it is also an extremely dangerous thing. Always getting an answer means if you've put something into production, you're getting a result that hasn't been vetted by people.” – Joshua Bloom

Transcript

Admit all.
Pavlos, would you like me to stop sharing my screen just for a bit while we're waiting for people to come in, or just keep this?
Up to you, Josh. If you like, I usually start without it and then I share. But whatever you like. I think we can start. Natasha, can we start the course?
I'm letting people in. Whenever you guys want to go.
I'll start with this slide. I'm all set when you are.
All right. Natasha, recording?
Yes, it is.
Recording, okay. Welcome. Good afternoon, good morning, and good evening to everyone. My name is Pavlos Protopapas. I am the scientific program director for the Institute for Applied Computational Science. You're here for the seminar. We have about six, seven seminars every semester. So today, it's my special pleasure to introduce Josh. I've been working with him for a while. Josh started from Harvard College, then he went to the other side of the pool, Cambridge. He did his master's there. PhD at Caltech, and now he's a professor and chair of the Astronomy Department at Berkeley. He's also a senior fellow at the Division of Data Science Berkeley. And if I'm not mistaken, he was instrumental in the creation of that division. He was one of the first people to start data science at Berkeley. His research is largely time domain astronomy, transient events, or telescope insight automation, and AI. He's also in the cusp between astronomy and machine learning. He teaches radiated process, high-energy astrophysics and at the graduate level Python for data science. Also, he has been awarded the Data-Driven Discovery Prize from the Gordon and Betty Moore Foundation, and the Pierce Prize for the American Astronomical Society. Before that, at one point, he was the CTO of Wise.io, which was sold in 2016 to GE. And today, he's gonna give us a talk about physics informed machine learning in astronomy. Let's welcome Josh. Josh, the floor is yours now.
Great. Thank you very much, Pavlos. I really appreciate the introduction and the invitation to be here. I'm excited to talk to you today about some of the work that we and many others have been doing in machine learning in astrophysics. I thought I'd start off with this quote for a little bit of levity. Recognizing that Jim Gray, for those that don't know, who is a prototype of a data scientist, many years ago wound up realizing at Microsoft that by using data from astronomers, they could test algorithms. They could test ways to scale compute. And Jim loved working with astronomers because unlike with lots of other big data, when you make a mistake and data leaks, or you actually make an inference that's wrong, credit cards aren't leaked and people don't get actually hurt. So working with astronomers was seen, and still is seen by many computer scientists and statisticians, as a kind of safe sandbox environment to play.

But I also wanted to bring up this quote because there's something interesting about how astronomers look to computer science and statistics for the development of our own work. And while we're fairly famous for taking work that's been done in other places, not so much on the algorithmic side in the early days, think of Galileo, pretty famous astronomer, hearing about this new discovery called the telescope, which was meant to point over the horizon. And instead he said, what if I do this and point upwards? And the rest is history on that front. But astronomers are pretty well known for co-opting both hardware and algorithms and approaches computationally to our own work. So we love working with computer scientists because there's a lot of fantastic work that's coming out, especially in the machine learning world, that we can try to bring in and adopt for our own purposes.

And last, what I'll say is I think computer scientists and statisticians, with all due respect to the work and the data sets they do, I think they need astronomers. And I think they need domain scientists more broadly to ask interesting and hard questions about the physical world. There's obviously a lot of interest in optimizing ad clicks, and understanding Twitter sentiment perhaps, but understanding how the universe works I think is pretty interesting and pretty fundamental. So we're hoping that we provide an interesting set of data and questions that help push the algorithmic side as well. So that brings us to this talk, which is really, hopefully, an introduction for many into the kinds of ways in which ML is being used in astronomy today, but also somewhat forward-looking into the sorts of things that we're trying to do to push the envelope, not just on the domain-driven questions that we have in astronomy, but what we're seeing is that there are special ways in which astronomers can contribute fundamentally, I believe, to the machine learning literature itself, in trying to change the ways in which we actually do learning by informing the learning process with the physics that we know of the objects that it is that we study.

So I'll be talking a lot about the work from my group and my students and postdocs, and mention the work of many others. But I just wanted to pause for a second and say that some of the most important work in this front is happening right there with Pavlos's group. And he's really been an inspiration for a lot of the things that we've been thinking about as we move forward. I also wanted to say before I moved on how wonderful it is to be here with all of you today virtually. It is a shame that I couldn't go back to my old stomping grounds and see many of my favorite colleagues. But that'll happen at some point soon, I hope. Just as a brief primer on the place where I've come from to try to get to the kind of research that we're doing now. You already heard some of this in Pavlos's introduction of me. But really maybe starting from the right-hand side, I've been focused most of my career on studying transient events in the universe. And I'll talk a little bit about the types of objects that we're interested in, and obviously talk about how the work that we're doing, and many others, can help gain insight into the phenomenon itself. Which ultimately, from a domain science perspective, is the thing that we're trying to get to.

I've also been doing a lot of teaching. Started Python bootcamps about a decade ago on campus. And that's now grown and blossomed, and is now part of a much larger effort of data science on campus, which is now a new major and now a new minor. And as you've heard, is also a large administrative division on campus that sort of matrix into many of the other things that we do at Berkeley. And then I also have somewhat of, for an astronomer at least, a special view on machine learning in production, having founded and been the CTO of a machine learning applications company originally focused on natural language processing when we were an independent company, and then moved on over time as we wound up being acquired by GE into automated insight on machines. So the agenda for today's talk is to give you an introduction into the centrality of machine learning in the way in which we do discovery and inference today. Talk a bit about how self- and semi-supervision is becoming more important, at least in the work we're doing in astronomical time series.

And then also talk about how some of the self- and semi-supervision is being used in, or we have a paper, at least, hopefully other people will start using it soon, for actually reduction of data. So looking at raw data and improving it, and doing it in a fundamental way using neural nets. And then I want to spend some time where we talk about exploiting the physical symmetries that we know, baking these symmetries directly into the learning process. And how we can, because we are physically constrained in the sorts of objects that we study, when we actually do machine learning on them and want to do inference, how we can use variational autoencoders with physical constraints to produce realistic light curves, that is brightness versus time of the objects that we care about. And then at the end, if there is time, I want to touch on some new work that we've just put out on likelihood-free inference. I should say that I think I'm monitoring the chat room on my iPad over here. So if there are any questions that arise, please put that into the chat. If I do miss it, I'll ask Pavlos or Natasha to just speak up if there's something that I've glossed over that somebody is interested in. Hopefully, we'll also have some time at the end to chat. So this is the big overview of the time domain of the kinds of objects that we and many people are interested in.

What you see on this plot is time on the x-axis versus some astronomer units of brightness on the y-axis. And what you see overlaid here is a couple of different light curves. As I said, brightness as a function of time, of different types of objects of interest in astronomy. First, I'll point out the type Ia supernovae in the blue curve. Type Ia supernovae are incredibly important for understanding the end states of some types of mass of stars, but also more importantly, for being probes of cosmology. And the Nobel Prize in Physics, in the context of cosmology, was given for using type Ia supernovae to discover the accelerating expansion of the universe. So finding lots of type Ia supernovae in a data stream is actually pretty important. Type II supernovae are the most common types of supernovae in the universe. And those are the most common way in which large stars wind up dying and exploding. If you want to understand the creation of elements in the universe and star formation, understanding how type IIp supernovae work is also important.

So those are the objects that we know about. And those are the objects that we know we're trying to find when we look through our large datasets, but then there are other types of objects that we have theorized about. This NS-NS merger, which stands for neutron star-neutron star merger, is a very quick and faint event, at least it was predicted to be. This is a slide that's about 10 years old now. And that was thought to be the kind of signature we might expect following a gravitational wave event where these two very compact objects, very massive objects, smash into each other and produce not just gravitational waves, but this light that we could potentially see. It turns out that in 2017, one of these events was actually seen. And it looked a lot like the theorized light curve that you see here. So this is something that we were pretty sure we would see. We didn't know the exact details. And then there are these other objects that are up here, neutron stars colliding with red super giants, or these mythical pair production supernovae. These are also theorized to exist. And we're also trying to find these things in our data streams.

But the reason why this is a Rumsfeldian challenge is because we're not just looking for the known knowns and the known unknowns. By definition, I can't put the unknown unknowns up here. And so our big goal, if you think about it from a time domain perspective, as we're trying to find and study the sky and find interesting objects in the sky, isn't just to optimize on the things that we know about, but also to try to find, in the presence of really pernicious noise, a whole bunch of interesting new objects out there. So this is our big challenge. If we had the world's capability of following up every discovery that we make in one of our imaging surveys, then this wouldn't be all that hard because everything that looks interesting in the sky, we would just throw a big telescope at, and get spectra and try to understand them in more detail. But we don't have that. And there's a very strong competition, for those that don't know how a lot of astronomy works observationally, for these very precious resources to observe different parts of the sky. So even though I might want to look at every object that looks interesting to me, there are plenty of other people that are trying to do other types of science with other types of objects. And so this is indeed a grand challenge.

And what looms very large for us in the astronomy community is the Vera Rubin Observatory. And this was formerly called the Large Synoptic Survey Telescope or LSST. This is gonna be producing something like 20 terabytes a night of raw imaging, tracking 18 billion objects over the course of its survey lifetime. And it's going to require something like 150 teraflops just to produce the first data release in a few years’ time. The final catalog, after about 10 years, is gonna be 15 petabytes of just essentially metadata. So this is going to be something of a tremendous interest for us, but this is the kind of data stream that we're trying to figure out how we can make sense of it and how we can optimize our scientific return. So this gets me to the first part of the talk in the context of machine learning, which is to say that we already are using machine learning to help us power some astrophysical discovery. And we're able to do this at scale. So there are parts of the inference chain, as it were, with new data streams, where we feel fairly confident we've learned how to extract and mine interesting sources out of that.

And this gets back to an original problem that I and my group worked on, which was to do something which seems pretty simple. That is to find new sources in the sky. And the state-of-the-art way to do that was and still is taking a so-called reference image of the sky. So a deep median stack of the same part of the sky that was done previously. And for a new image, align those two and subtract them off. And when you've done a good job in the subtraction, you see what it looks like at the very bottom here, these reals. These are new sources, basically where there's a little extra brightness above where there was previously. But it turns out, doing the subtraction is not all that easy. And this leads to a lot of artifacts, these bogus detections. And it's a little bit like Anna Karenina: all real objects all look the same. All bogus objects are all different in their own way. This is actually one of the major bugaboos in us being able to find and then study these objects. Because it turns out that the number of bogus in some of the surveys 10 years ago, the number has come down a bit since then, as subtraction techniques have gotten better, is about 1,000 to one for the number of reals. So we're really doing a needle in the haystack problem here.

And so what we did is developed one of the first machine learning, not just algorithms, but computational infrastructures that ran on real-world data that learned from both bogus and real, and essentially scored every single object that came off of these new telescopes producing these images. And that allowed us to be fast, obviously, compared to people in doing inference, parallelizeable, transparent. And interestingly and importantly, for science, deterministic and versionable. Unlike when you ask people to look at data, you always will get the same answer out of these machine learning algorithms. So that helped us quite a lot. And this may seem obvious in retrospect, but when we started this work, I was reminded of this amazing picture from Harvard College Observatory over 120 years ago where astronomers, and particularly at Harvard, had a very big data problem. They were getting more images coming off of telescopes from the Southern Survey than they knew how to deal with. And so they hired a number of people, mostly women, to look at this data and try to opine on it. And the person that you see in the back, just for historical reference, Henrietta Swan Leavitt, who is considered one of the major figures in modern cosmology, discovering an important relationship between the period of pulsation of some types of stars and their overall brightness. There's also a very nice play by Lauren Gunderson called “Silent Sky”, which dramatizes the life of Henrietta Swan Leavitt. So for those that haven't seen it, I really recommend it.

But this looks old, and it is, but it's actually pretty much how most people, until very recently, were dealing with large data problems in astronomy for discovery, is essentially hire more graduate students. So bringing the machine learning component into this problem was a fairly dramatic shift away from needing experts. Because as we know, domain experts don't scale. And one of the highlights I'd say of some of the early work that we did in this space was to automatically mark up the most interesting objects in the sky. And this led to what was then the earliest discovery of an exploding type Ia supernova in one of the most nearby galaxies in the last three decades. And this ML real-bogus discovery was important not so much because we found this object. Because in the end, this object is so nearby and got so bright, you could have seen it with binoculars if you knew where to look. And it would have certainly been discovered by amateur astronomers. But we found it early. And because we were able to find it early, we were able to get the world's telescope resources pointed at it and trained at this position. And we were able to do some interesting science that we wouldn't have been able to do had we waited longer.

This is a bit of a busy slide for non-astronomers. But what I wanted to say is that all of the regions that are colored were ruled out by some of the data that we were able to obtain very, very early. And in particular, what we're trying to understand with type Ia supernova, even though we use them for cosmology and people have won Nobel Prizes using them as probes, we still don't know all the details of why they explode, and whether there's one star involved or two stars involved. And it's almost certainly two. But one question is, is one of them actually compact and so-called degenerate? And because we were able to rule out the green region, as you see here, we were able to rule out all but the most compact objects that we know about locally. And so this became very strong evidence that one of the objects which blew up was a compact object. Not a huge surprise, but it became a new line of evidence that we didn't have before. And again, we could do this because of the ML assistance.

Many of you who have been working in ML for a long time know that ML, not just on paper, but in production, is really hard to do. And if it's really hard to do, you really have to seek other reasons and other ways to do it if you're actually gonna put this into production. And oftentimes, what you see in academic settings, especially in the domain sciences, is that we try to apply a new machine learning algorithm to some existing data, write a paper about it, and say, here's my ROC curve, let's move on. The prize is really the sorts of plots that you see here. We're able to do domain science in a new and novel way, faster and better because we've applied machine learning in production. Now this whole idea of doing a real-bogus to discovery and classification is now really a cottage industry. And all significant surveys are now building their own real-bogus detectors, not just using random forest algorithms like we did way back in the day. Which seems silly to hear just because that was only 10 years ago. But now a lot of people are using neural nets and putting those into production as well. So that's fantastic. We are using this. It's become an important part of the way in which we wind up obtaining data and using data.

But one of the things we're trying to do now is to push the envelope a bit more still, on this kind of real-bogus idea, but recognizing that we need to think beyond the score, beyond the accuracy, beyond the how good is this algorithm, to something a little bit more expansive that I'll explain in just another slide. The place where we're having to push the envelope is in our search for Planet Nine. So there is a purported theorized massive object beyond the orbit of Pluto, gravitationally bound to the sun. And one of the interesting ideas is that we've already imaged this object somewhere in our archives, but what we need to do is, because this object is too faint to see in one image, we need to actually shift and add these images slightly to follow the unknown orbit of this planet. And if we do that right, then this object may wind up getting popped out of the shift and added data. So there's a student here at Berkeley named Mike Medford, working with his advisor, Peter Nugent, basically doing the shift and add. Now, as you can imagine, the space over which you have to search isn't just one direction of shifting and adding. It's all possible allowed orbital phases of this unseen object.

And so what happens is, after we do a shift and add, we wind up finding candidate objects that are a little bit brighter than the typical noise. And we have to very quickly score them to decide, do we want to even save this object? And we have to score it because it turns out we don't have enough data available to us at the nearest supercomputing center at LBL to actually save all the data that matches some sort of simple criterion above some noise threshold, which is fairly amazing in its own right. And it's because we're producing tens, or hundreds of billions of potential candidates as we run through this old historical data. And so what we had to do is build

a machine learning algorithm that could be extremely capable of deciding, is this real or bogus, based off of simulated data of what an unseen Planet Nine might look like. And then we had to distribute this over a very large supercomputing cluster. And because we didn't want this opining on do I save the data or not to be the bottleneck in the work we're doing, we had to make these predictions in 10 milliseconds. So what we're finding is that to do this right, we needed to shrink our models down to a very small size. And we needed to build an extremely lightweight infrastructure around those models that could serve a prediction at scale very, very quickly. And so what we're starting to see now in astronomy isn't just a push to get better answers, but to get better answers with other types of constraints like smaller models that can opine on data very, very rapidly.

I want to turn my attention now to self- and semi-supervision. I'm seeing some of the questions that are coming up in the chat. And I think, at least my quick view of the questions that I've seen so far are ones where I think I can address some of that through the rest of the talk. So I will do another pause when I get to the next section. And then we'll see if there are any questions based on everything that's happened up until then. So the traditional approach to classification, and this is a visualization of 50,000 variable stars all over the whole sky. The galactic plane is in the horizontal direction. And you can see, obviously, there are more variable stars around the galactic plane. And then as you go farther up, there are fewer and fewer. The typical light curve of one of these observations looks like what you see here. And unless you're very good at doing Fourier transforms in your head on raw data, you might not be able to discover that this actually has a known period of about one day. And this is an object called an RR Lyrae star.

So what we did, also about 10 years ago, was start building something that could look at raw data as it came off of telescopes or historical archive data, and build a bunch of features from this heterogeneous data that is noisy and actually sometimes has spurious detections in it. And we featurize that data in a very traditional, domain-specific way, taking all the things that we could think about from building Fast Fourier Transforms or Lomb-Scargle Periodograms, what you need to use on this sort of irregularly sampled data. And then throwing that into a random forest, and then doing some probability calibration post-processing, we were able to get some pretty interesting results, to be able to classify over something like 20 different classes of variable stars in a held-out way, in a semi-rigorous way, to know that we were able to get errors of order 10% over this very large catalog of 50,000 stars. The challenge, for those that have been working in machine learning for a while, with this approach is that we have to hand-code features. We have to actually say, okay, I can put my domain knowledge hat on, and I can write some code and I can take out the relevant features in this noisy data. That also means that at predict time, when we have to run through all of these features, we have to write and run a code that essentially scales with the total number of features. We also had a very small number of labels: with 50,000 objects in this survey, we only had about 850 known labels over 20-ish classes of stars. So this is a very small label problem.

And that's actually an interesting thing. When you talk to astronomers who talk about machine learning, oftentimes, the first thing we talk about is how much data we have. I'm guilty of that myself at the beginning part of the talk. But when it comes to machine learning, it turns out we have very few examples or exemplars of the objects that we care about. So we have a big data problem, but we also have a small label problem. And it turns out that after you build one of these models and try to apply it to a different dataset, it doesn't really work that well. So what we did is build a semi-supervised autoencoder using a recurrent neural net architecture that allowed us to do basically automatic generation of features. And for those that haven't worked with recurrent neural nets or autoencoders, the simple way to think about it is the picture that you have here. You have this raw data and you build a neural net, which learns to encode that down to just a few numbers. And that's depicted here with this B. That's what's called the bottleneck layer. So think of this as a compression. And then you uncompress the data with a decoder. And the goal here is simply just to reproduce the original data you had. And so if this is a very simple, let's say, sinusoid, we probably only need a bottleneck layer of size three because we've got to capture the amplitude. I'm getting a little bit of feedback. I don't know if somebody is off mute. We need to get the amplitude. We need to get the phase and we need to get the period. And so if you have a more complex-looking source, then you need a larger bottleneck layer to capture all the relevant data.

So the idea here is that we can use this bottleneck layer as the creation of features for us. And I won't go into all the details of what the actual architecture looked like, but what I wanted to point out, first of all, is that it wound up getting essentially best-in-class classification accuracies on several different datasets. But what I wanted to point out is we had to modify existing autoencoder infrastructure to handle the irregular sampling of the data. And also, because we understand something about our noise properties very well in astronomy, we're able to make use of that to change the traditional loss function, which is usually some mean squared error, to a weighted mean squared error. And that allowed us to not over-index in the learning process on poorly measured data. That's one of the things that we had to do, where we had to monkey around with the actual architecture itself. But the other thing that we realized during the course of this work was that even though we had a small number of labels, what we're looking at here is a self-supervised feature learning process, where even if we don't know what the answer is, that is, the classification of a given object, we could throw something that doesn't have a label into the system and actually have the system learn from that to reproduce essentially that data. And the goal there would be that something could get encoded in this bottleneck layer, that even though we didn't have that label, we'd still be able to get a better and better set of features out of that.

The extension of that work is, in some sense, not just taking what you see at the bottom here with a depiction of what I just showed you from the previous slide with the encoder in blue, and the decoder, also in blue, on the right-hand side of the bottleneck layer, but also using both self-supervised lines of thought and lines of learning, and supervised lines of thought where we actually know for some of these cases what the classifications are. So this is what I've been calling the semi-supervised kitchen sink, where we're throwing everything that we know at this problem to get better results. And a survey of not just this approach that you see here, but also different types of specific architectures is something that we put out on the archive and is now in press. And I've given you the link there. I did this with a former postdoc of mine, Sarah Jamal. I want to turn my attention now to what I'll call denoising autoencoders, which is a very similar idea as what you saw before where you wind up taking data and scrunching it down to a smaller number of bits, and then blowing it back up with an encoder. And that's in an application to raw images.

What you see on the left-hand side is a raw image from the Hubble Space Telescope, which looks kind of busy. And it turns out that most of the detections that you see there are from cosmic rays hitting the detector. These are charged protons, or usually electrons or muons, which are hitting the detector and leaving streaks in the data. What we want to see is the thing on the right-hand side. And this is a very nice image of a cluster of galaxies where it's cosmic ray free. So the way that we did this, I did this with a student, Keming Zhang, in my group, was to use a modified U-Net architecture which takes these little postage stamps that include the cosmic rays, and tries to predict the mask of where those cosmic rays are. That's, in some sense, task number one. But then task number two is to take that mask in the original data, and then in-paint with a different network over the data with cosmic rays and get a clean image after that. And as you know, if I give you an image and I give you a mask, you can pretty easily interpolate over that. And the question is, can we actually learn to do better interpretation?

One of the things that was amazing in this process of learning was that we wound up, in the initial layers of the model, discovering, because these are just convolutions in this U-Net architecture, we wound up discovering that we actually had the network find the current state-of-the-art way in which we find cosmic rays. There's another code called LACosmic, which does a Laplace transform on the data to find sharp edges. The network actually learned a Laplace transform in addition to many of the other convolutions that you can see depicted here. So while we didn't direct it and say Laplace transforms are important for edge discovery, it actually wound up finding it, given the training data we gave it. And we wound up getting some really nice results. What you see at the bottom is some zoom-ins to some especially pernicious places on the sky with large streaks of cosmic rays. And you see the discovery of the mask and the inpainting over that. So it looks visually very clean and very good. And then by all the metrics that we knew how to go through, we were getting not just better answers, but we're getting faster answers than the traditional methods that are used in this data reduction.

So these are, what good talk in machine learning wouldn't have a couple of ROC curves on this, this is two of them, false positives versus true positives. And our results are the solid lines versus the previous results from this LACosmic that I mentioned before. And we do better in some types of fields than other fields, but we always wind up doing better than the previous results. And then on the inpainting side, we're basically doing better than median masking and biharmonic interpolation, and actually able to do it much faster. So here's a place where machine learning isn't really gonna be used directly for inference or discovery, but we're hoping to see how machine learning might be part of the data reduction analysis itself, even upstream from some of these high-level discovery and inference techniques. Okay. So let me just pause for a second. Somebody asked, “Was the U-Net part supervised? Did you provide the mask of where the cosmic rays were?” Yes, it was, that was a supervised problem. We were able to determine

the mask by actually doing a median smoothing over many images of the same part of the sky. So we knew what the real view of that part of the scene should be. And so that was a supervised problem where we went through and said, we know the answers there. Similarly, on the inpainting job, we also knew the answers. And so we're able to say, hey, inpainting network, please learn what it is to paint over a cosmic ray when you're in the presence of a galaxy. So it seemed to work pretty well. It looks like I'll move on and I'll take some other questions perhaps at the end of the talk.

What I wanted to transition now into, in the time that I have remaining, is a discussion about physics informed ML. Pretty much all the things that I've been talking about so far is where we're changing some of the learning architectures, asking very domain-specific questions of these learning architectures to get better answers than what we had before. But now we want to be able to use our knowledge of physics to try to learn better and faster. And one of the points of departure for this comes from a very nice paper called “Why Does Deep and Cheap Learning Work so Well?", with the recognition that these researchers had from MIT, that while there is an extremely large space of possible images, if you have a thousand by thousand image and it's grayscale, you have something like 256 to the millions possible states of that image. We're able to learn on images of cats and dogs or galaxies and stars in a very small amount of time. And that's because there's something about natural scenes that are already naturally regularized by the physics of what it is that produces them.

So if that's the case, can we actually use our knowledge of physics to impose constraints on the architecture? And this has been done and started being done over the last couple of years. First in computer vision with a recognition that if I have an image and I rotate it, it's still the same image. Or if I blow it up, or if I add a little bit of shear, it's still the same. So why not have neural architectures that look at an image and have the same output, regardless of whether it's rotated or zoomed in? In high-energy physics, people are starting to build QCD-aware neural nets. As you're looking at particle events from these higher-energy colliders, you want it to be knowledgeable of things like conservation of energy and momentum. And people are doing this in quantum chemistry. What you see here is a depiction of, essentially, they're trying to predict the forces on a molecule. Well, the molecule is the same in the left side picture and the right side picture. You want to build networks that are invariant to these rotations, translations, and permutations. And so this is a very important part of what we could potentially do.

Well, where does physics come in and where do these symmetries come into the work that I've been talking about? Well, first of all, in asking this question, how we can actually embed in our architectures and maybe even the data itself some of the things that we already know about the taxonomy of the objects we're looking at, conservation laws, and symmetries. In the objects that I've been thinking about and working on is a recognition that with periodic variable stars, the previous approaches to looking at time series data were not very aware of the fact that in phase, essentially we end up wrapping around and getting the same observations as you go around in phase. And when I say periodic variable stars, I mean just like the RR Lyrae that I mentioned before. You have a star which is changing as a function of time, but after you go through two pi radians of phase, you get back to where you were at zero pi. And that source generally winds up repeating itself fairly regularly.

So what we did is recognized that instead of zero padding out in these recurrent neural nets, as we wind up doing these so-called dilation layers, we would basically wrap the data around to the other side. And using that we were able to get what we call invariant temporal convolutional neural nets, or ITCNs. And this invariance is a very simple change to existing architectures. But what it does is it gives you very realistic results on essentially doing convolutions not in a one dimensional direction, but in the polar coordinate direction. And so I won't go into the details of what you're seeing here. But essentially because we're doing this padding where we wrap around, we're basically teaching this network that the outputs should be invariant to wherever I wind up starting a phase. And this paper appeared in a NeurIPS workshop last year. And we're working to get this into a journal now. But needless to say, for all different types of invariance that we wound up adding to the well-known types of networks like ResNet and temporal networks, ITCNs, we're able to get better answers from that just by making this small change.

And then on non-astronomy data, it's actually kind of hard to find a good benchmark dataset on periodic time series. We did what all ML people seem to do, which is coerce the famous handwritten digit MNIST dataset into a periodic time series. And we did the following. We took the original data, and the three and the eight you see there, with a random seed essentially permuted every pixel somewhere in this 28 by 28 image, unraveled that. So we got back something that looks like a light curve that's going up and down. And then started this at a random phase. And that was our way of producing this, what we call periodic permuted MNIST. And that also produced very, very good results over the baselines that we studied. So I want to talk now a little bit about how we can start using our physical constraints on learning to be able to get better answers for the types of things that we're trying to do.

And what are the challenges that we had? What we realized is that we don't have an issue of physical models for a lot of the types of events that we look at. To produce a type Ia supernova event requires a massive supercomputer. And even then, we don't know all of the physics to produce realistic light curves. RR Lyrae, we can fit with a template and there are some physical models that give us something that looks like an RR Lyrae. But what we'd like to do is to create a non-parametric, non-linear set of models that can capture the range of physically plausible conditions for all the sources that we may be interested in. And what we built was a variational autoencoder, with my postdoc Jorge Martinez-Palomera and Ellie Abrahams, who is a student of mine, where we built an autoencoder that takes raw data, real light curves with known labels. So these are ones with known classification, and then physical parameters that we know of these objects, and then compress all the data that we have of this object into a latent space and allow us to dial around not just in random directions in the latent space, for those that are used to building variational autoencoders, but also to orthogonalize the latent space relative to physical parameters. And I'll talk a little bit about what that means in a second.

But the output could be that we get these generated light curves that are realistic. The architecture is also fairly simple in the sense that we have a generic encoder on the left-hand side and a generic decoder on the right-hand side. Except that now, we're injecting after the latent space some of the important physical parameters that we want to have as part of our output for our reconstructed light curves. And one of the interesting things is, to do this well, we built the traditional reconstruction of those light curves from a variational autoencoder. That's the first term that you see on the left. But we also had to build a so-called KL-divergence term that was regularized by a hyper-parameter beta. And also because we were trying to simulate survey data, we created another KL-divergence term to make sure that the output light curves were realistic in what their uncertainties were. And by tuning all of this up, we were able to do some pretty exciting things. We were able to produce RR Lyrae light curves realistically. And now we can produce as many as we want. Not just randomly where we generically sample the entire RR Lyrae phase space, but where we can dial up and down physical parameters of these sources and get realistic light curves out. So for instance, as we dial up the temperature on these stars, we wind up seeing they become a little bit more (indistinct) as we go from left to right.

So it's this kind of thing that we think may be very useful in creating these emulators of real physical sources. As people try to build survey cadence optimization, where now they can select from a basket of variable stars, where instead of having to run a big computation to get the result out, they can just run one of these small variational autoencoder models where they can sample from a realistic distribution of temperatures and masses and metal densities. Now, this generative modeling is not a new idea. This has been done in a lot of different places in the context of cosmology. What you see at the top is realistic simulations from supercomputers of the distribution of dark matter over a large swath of the sky. And then you see at the bottom generated samples that have been learned off of the samples that you see up top. And visually, these look very similar. But then also on the top left, you wind up seeing that the distributions also wind up looking pretty similar to the more realistic simulations. People have also started generating images of galaxies, which is extremely helpful in mocking up surveys so that you can test your sensitivity to different types of parameters.

There's something else that's interesting as well in the context of generative modeling. And now this is solving the so-called inverse problem. So instead of starting from a set of parameters and trying to create a realistic version of whatever it is you're looking at, on the left-hand side, you may want to create a gravitational wave signature. On the right-hand side, this is an image from a very nice work that studies the use of likelihood-free inference in physics. Where there, you're looking at a particle physics experiment. What you're trying to do now is, instead of going in the forward direction, where you start with parameters and you get realistic realizations out, what you'd like to do is what a lot of us do in the physical domain, where we have data. And we want to understand the physical parameters that generated that data. And in a Bayesian sense, what we're trying to do is, given data, we're—

We lost him.
I think we lost Josh.
Yeah, he got bumped out. Let me go back in and see.
Let's wait a little bit then.
I realized that my laptop died. And so now I'm switching over to my iPad. So why don't we take this occasion for me to answer some questions that you may all have while my laptop reboots, and I can finish the last couple of slides of my talk? Pavlos, maybe I can ask you to moderate this?
Yeah, there is a question by Jordan. That's, “How would your network handle any variable signal whose parameters are changing over time? For example, stellar position whose period exhibits a long-term oscillation?”
Right. That's a great question. So semi-periodic variables are one of the hardest things for us to do in this invariant network. And so what we're trying to figure out is how we might be able to extend, or at least augment, these types of networks with a more traditional recurrent neural net. The other thing that one can do, and I think Pavlos has spent some work on this, is build sliding windows where you have the same network, but over time, it winds up seeing different parts of a semi-periodic variable. And that actually can get very, very good results without having to use this strictly rotationally invariant network.
Also, Josh, Michelle is asking, “What do you think are some of the biggest challenges to the widespread adaptation of ML in astronomy?”
That's a great question. And I'm gonna try to answer that while I'm multitasking because my laptop is now back up and I can try to rejoin the Zoom. So, one of the hard parts obviously is just training. A lot of us have not gone through the formal training on the stats side and the computer science side to be able to understand deeply the tools that we're using. That's maybe okay in some contexts because we don't always have to understand the inner workings of how a computation works for us to make good use of it. So there's definitely gonna be places where astronomers can just benefit from off-the-shelf algorithms. But I think to truly innovate and to truly push the envelope of these algorithms, we're going to need to be able to have a training curriculum that starts maybe even before college, that gets people that are gonna be going into domains to be able to be functional and conversational and understand how these approaches work. That's one. And I think the other one that's almost certainly worth mentioning is that some types of learning that we do require very large computational resources. And that is already something that not everybody has access to. Now, what's good about this is that not a lot of the problems, at least the ones that I've been interested in recently, have massive computational constraints. But in ones that do, having access to big iron is going to be fairly important over time. I think I'm about maybe 30 seconds away from getting into my talk again. I have only a couple more slides left. Any other questions that I can answer before we get into it?
A couple of them more, but the sub-question, or the previous one, is any argument against widespread adaptation of machine learning besides the big iron, maybe inference is a problem?
Yeah. So that's a very good question. And I have a long diatribe on that. In part, I touched a little bit on that already, which is with machine learning, you always get an answer. And that is a wonderful thing, but it is also an extremely dangerous thing. Always getting an answer means if you've put something into production, you're getting a result that hasn't been vetted by people. And machine learning algorithms don't know whether they've made a mistake. They're just putting results out. And in the context of normal imaging tasks, if I learn on an image, is this a cat or a dog, and then I give that model a CAT scan, it's gonna tell me it's a cat or a dog. It has no idea that it hasn't seen this data before. So I think that is an extremely dangerous problem. And again, because we're not all trained in being able to diagnose and understand deeply what's happening in some of these, what seem like black-box models, we wind up having, I think, a hard time learning when we're making mistakes on that front. So that I think is the most dangerous thing, is that we could be saying things about the universe that are unvetted and unchecked by our own intuition. Putting machine learning in production that's just allowed to run and actually not just do discovery, but potentially trigger other telescopes to take data there, can be extremely dangerous unless you put a lot of guardrails on it. Okay. I'm joining the meeting in progress. Of course, Zoom is asking me to redownload another version of Zoom because why not? Any other questions?
Yes. There's one more by James. James says, “In searches for negative signal in images due to eclipses or others, are there either significant differences in techniques being applied or significant difference in the sorts of false detections on the data?” I think it's a great question.
Yeah, that is a great question. What we've done, honestly, is because a lot of the interest in what I've been involved in—it's now not letting me in. I have to go and reregister. Sorry.
[Pavlos] Sorry, Josh.
That's okay.
If it were up to me, I'd immediately permit you.
Okay, now, I think I can be let back in.
[Pavlos] You should be back.
Okay. And now it's very bizarre for me to see myself from this direction here. So let me share my screen again. And I think this might come back up. So yeah, so what we've done, as you know, just out of practicality, is mostly learn on positive images. I believe that there are some groups that are now learning on negative images. So if you have a reference image and you subtract, and a source got fainter, then if you invert that image, it will look positive. Indeed, the types of artifacts look different in those negative images. So people, I believe, are starting to do that. We didn't do that in the original real-bogus that we implemented about 10 years ago.
Now you're muted, Josh, on this machine. You're gonna be unmuted on the other ASU.
Okay. Can you see this?
Yeah, you're good.
So I already talked about the likelihood-free inference. What I wanted to say is that we're starting to do this and apply likelihood-free inference to interesting hard problems in astronomy. What you see here is from a paper that we submitted about an hour and a half ago to the NeurIPS physical science workshop. And what

you see on the left-hand side is a posterior of a bunch of different parameters that describe so-called binary microlensing events. So it's when you have a star and a planet, or two stars, moving in front of a background star. Because mass bends light, you can get gravitational magnification. And what you see zoomed in here on the bottom right-hand side is a depiction of the light curve from this event. And what's interesting, and makes this problem really hard, is that this is a really ugly landscape of a posterior space. While some parameters are very well-behaved and it's pretty clear you're in a global minimum, there are other parameters, like the second or third one down that you see there, where you have two different islands. And it's unclear whether you're in island one or island two. And just because you found a local minimum doesn't mean that you found a global minimum. And so what we're able to do is learn on lots of microlensing events, run that through the entire neural net architecture that we built, and produce now inference on the parameters that produced these light curves. And we're doing this in the context of a new satellite, which is gonna be launched, called the Roman Observatory. It used to be called WFIRST.

I think just in the interest of time, I'll skip over this bit. But just end with an overall depiction of the places where we're trying to bake physical constants and constraints into our entire learning process. So in some sense, the old-school way of doing that was featurization. You take your domain knowledge on raw data. You featurize that into a set of features that you think are gonna be informative for, let's say, a classification task. Then there's also the symmetry preserving layers. And I talked about in this talk a way in which we're using symmetry padding for periodic variable stars. This is something that Pavlos and his group had also worked on, in the context of solving differential equations, is building symmetries and conservation laws into the different layers. Then there's also the bottleneck layers and imposing sparsity. This is in some sense where we, as people working in the neuro world, get to impose an Occam's razor understanding of how the physical world works by scrunching down bottleneck layers and forcing large amounts of data to be coerced and compressed into a small amount of data before you blow it up again. This is a place for us to impose this sense of sparsity. Then there's the loss function curation where we can actually enforce physically meaningful results at the instance level. So if we're getting results out that look good, but actually violate some conservation law, we can heavily penalize that from a loss function perspective. And then something that I didn't have time to go into today, is as you wind up building these networks, enforcing distributional loss, so that you're getting realistic ensembles of how stars are over the whole sky, for instance.

So with that, I'll end. Apologies for the logistical snafu. And just to summarize here, that machine learning is already very central to astrophysical discovery and inference at scale. And what we're now starting to realize, obviously, is that it's not just the accuracy or the score. We're starting to optimize on things like deployability, versionability, understanding of what these models are doing, size of the models, and speed of inference. These are all things that are important in other realms outside of physics and astrophysics, but are becoming more and more central. If you get a good score on data that's coming in, but it's too computationally expensive, or you can't fit it in RAM on the little machine that you have, it's no good. We're also starting to recognize, because astronomers live yes, in a big data world, but also in a small label problem world, that self- and semi-supervision approaches I think are becoming very key to us to be able to get good answers. And I think the most emergent area of research for all of us is in the acceleration of learning with potentially less data on physical systems when we can figure out ways to imbue our knowledge of physics and symmetries and conservation into the learning process itself. And as I touched on, I think there is a growing symbiosis between first-principles simulations that require potentially supercomputing amount of effort to get a realistic set of observations out, and generative modeling, so-called surrogate modeling, and likelihood-free inference. So with that, I'm happy to stop and take your questions.

Thank you, Josh. (clapping) Great talk. I'll be moderating. I answered some of the questions, but a couple of them that I think it's better you answer. There is one from John Wu. John is asking, “Since the variational autoencoders have intractable likelihoods, does that make it difficult to identify out of distribution examples? Are they good alternatives to autoencoder models?”
[Josh] That's a great question. We found autoencoder models were the most straightforward to train. We certainly started off in the GAN world. And the nice thing about generative adversarial networks, of course, is that you get for free an out-of-sample discovery engine in the form of the discriminator. The discriminator, when you give it new data, can basically say, “Yeah, this doesn't look like data that I've seen before.” So there's actually some great utility in GANs, not so much for data generation, but for looking at data as it comes through and potentially discovering out-of-sample events. The whole problem in general with concept drift that you wind up seeing in industry, the idea that you built a model on how the world was, and now, if it's an NLP model, people are starting to use different words today than they were using yesterday. Your model is constantly getting out of date, and that concept drift can be a big problem. Luckily, physics isn't changing all that much that we can measure. And so when we build models on existing datasets, as long as the new data is being taken in a similar way with a similar set of instruments, we have a reasonable amount of belief that while we may get some noise that comes in that we've never seen before, for the most part, if we've done a good job in modeling the data broadly, we don't have that notion of concept drift in the same way that you have in other types of industries.
Thank you. There was one more. Someone said, I missed it, what do you mean by likelihood-free estimation? That was Shane Lake asking that question. Can you clarify what likelihood—
[Josh] Yeah, so a way of thinking about likelihood-free inference is it's very much in a Bayesian context. If you're a Bayesian, you come to all problems with a prior understanding of what the parameters are that are gonna generate your data. And then you look at your data. And the data winds up informing the results. That information is what's called the likelihood. And the multiplication of your prior times your likelihood gives us the posterior, which is the plots that I showed you for the microlensing example. Those are the parameters and the uncertainty in the parameters and the covariance between those parameters that we'd like to understand. What the neural nets that do likelihood-free inference allow us to do is skirt the need to actually build a likelihood on the data, which is generally computationally expensive if you're doing it in traditional techniques where you actually have to do lots of simulations on the fly. And instead, go directly from data to posterior. Now, in some sense, there is a likelihood that is actually generated under the hood. But likelihood-free inference is sort of a shorthand for saying directly from data to posterior.
Thank you, Josh. There's one more question. I'm not gonna reveal the name of the person.
[Josh] Is it you, Pavlos?
Yes. Your autoencoder with a kitchen sink, right? Did you use the whole data set to train the autoencoder, or you preselected variable objects, or you use everything? And I ask you because if you use everything, you have some kind of covariance shift between the unsupervised part and the supervised part.
[Josh] Yeah, so in that paper, we basically threw in everything that we had classifications for. And you're absolutely right. There is an unspoken bias in this that's possible where if you don't have classifications for an object, there may be a good reason why. The underlying assumption in this kitchen sink approach is that there is a random reason why you don't have a classification for something. But for instance, a good counter example would be if you have classifications for all the bright objects, but not good classifications for all the faint objects. Then clearly, you're not gonna learn much about what it means to be some of these fainter types of objects. And this network is not protected at all from just over-indexing on the brighter objects, and giving you more likelihood on the classification of these other bright types of objects. So it's an interesting question. I'd be curious to see how you think we should and could protect against that.
All right. Thank you. Good. Josh, excellent. Really enjoyed myself, of course. And we should thank Josh again. And I think, Natasha, the video is gonna become available, right, for everyone?
Yes, it'll be posted on our webpage and YouTube channel on Monday.
[Pavlos] Yep. Okay. Thank you.
[Josh] All right. Thank you, everybody.
[George] Thanks, Josh.
[Josh] Bye.

Towards Physics-Informed ML Inference in Astrophysics

Wed, 26 Feb 2020 00:00:00 +0000

At the Computer History Museum: building ML inference for astrophysics that respects and exploits physical models, with applications to time-domain surveys.

Day within Feb 26-27 approximate.

Key Quotes

“What could be more important than understanding the universe? And doing this with ML actually, as we'll wind up seeing, pushes the envelope of what is needed algorithmically and potentially even from the hardware perspective.” – Joshua Bloom

“What we decided to do is replace grad students and experts, because they don't scale, with ML.” – Joshua Bloom

“How do we impose our knowledge of physics into the learning process directly into the architecture?” – Joshua Bloom

“It's important for astronomers and domain scientists in general to be connected deeply with the cutting-edge work of computer scientists and statisticians, but it is also tremendously important, I think, for computer scientists and statisticians to learn from us the kinds of questions we ask of our data, because our data looks different.” – Joshua Bloom

Transcript

Well, thanks very much. I thought I'd start off with this quote from Jim Gray, who was a consummate data scientist and obviously had a very good sense of humor. He recognized something important decades ago: that it was great working with astronomers and our data, because our data is noisy, it's messy, it's streaming, it's heterogeneous, and it's dirty. And if you made a mistake as a computer scientist or a statistician as you try out new algorithms or new approaches or new hardware on our data, you're not leaking PII, there's no billion dollars of damage to your company, and nobody dies. So indeed this is a wonderful thing about working with astronomers. On the flip side, astronomers love working with statisticians and computer scientists

really for two reasons. Number one, the pace at which algorithms and new hardware are coming out that are able to directly impact what we do in our daily lives in discovery and inference — which in some sense is a big part of this talk today — is just wonderful. And astronomers are very well known for co-opting tools. So the most famous astronomer you might think of is Galileo, who heard about this thing called the telescope, which was meant to look over the horizon for enemy ships, and he said, what if I just do this? And you know the rest of the story. But also, the second reason why we like working with computer scientists and statisticians is that, with all deference to my academic friends, oftentimes in

that world we feel like they're not working on important problems. So what could be more important than understanding the universe? And doing this with ML actually, as we'll wind up seeing, pushes the envelope of what is needed algorithmically and potentially even from the hardware perspective. So this talk is really dedicated to bringing all of you up to speed on some of the really interesting domain problems that we have in astronomy that we think we can start applying ML to. And in fact, as you'll see, it's becoming part of our daily lives, and what we're doing is starting to recognize there's this very deep connection between the way in which machines learn and the physics that we actually do with these learning systems.

So for us, this is the most exciting plot I think I can show you, just to summarize why we do what we do, at least in time-domain astrophysics. In the middle what you have are basically the light curves — this is brightness versus time — of some of the most important explosive events that we know about. Type Ia supernovae, for instance, as you see in the blue curve, are used for measuring the accelerating expansion of the universe. And if you want to understand how stars die, if you want to understand where all the metals in the universe come from, you also have to understand type IIP supernovae. So there are concerted efforts for many people to be looking for these things and use them as cosmological probes and for our understanding of

the physics of how atoms are created. But then there are these other objects which are up on this curve, like the so-called neutron star neutron star mergers. And this plot was made many years ago, before we actually found one of these electromagnetic counterparts to a gravitational wave event, and it turns out that the observations that we got look very similar to the theoretical curve that you see there. But then there are these other curves that are on this plot, like the pair-instability supernova, and when a neutron star runs into a red supergiant like Betelgeuse there are some really interesting things that we might be able to see. And so these are the unknown, or the known unknowns, but obviously by definition we can't

put up the unknown unknowns. In general what we're trying to do is, in a swarm of lots and lots of data, satisfy all these different scientific constraints simultaneously. And if what you're trying to do is write down some optimization metric, you would try to say, how can I maximize the science for the discovery of these objects, but then also our efficient and effective use of follow-up machinery? Just finding one of these things in the sky doesn't really mean anything unless you can actually follow them up with other telescopes and do deeper introspection. This is obviously easier said than done in many senses. There are sociological impediments for us to be able to say that we're going to be able to do this optimization, even if we write

down this metric, because our optimization metric may be very different than somebody else's. But this is sort of the grand goal: to be able to find these things that we know about and find things that we don't know about but may actually push the envelope for science. And what looms tremendously large for us is the so-called Large Synoptic Survey Telescope, which will be acquiring more data, at least at optical wavelengths, than we've ever had before. This comes online in just a few years. It's a billion-dollar facility that all you American taxpayers have already paid for, but it's going to be producing something like 20 terabytes of raw data per night, tracking 18 billion objects on the sky simultaneously, basically observing the entire night

sky that's available from the southern hemisphere every three days, and producing something like 150 teraflops are going to be required for this first data release, and the final catalog will be of order tens of petabytes. So for us to be able to mine that data in real time and effectively use all the resources that we can get at it to find these objects of interest and then follow them up is really our grand goal. So the agenda for this talk is to introduce you to the centrality of ML in some of our everyday practices — a lot of this is supervised learning. I'll introduce you to some of the things that we've been doing for a number of years and then talk about some of

the new directions in semi- and self-supervision. In particular I'm talking about what we've started working on in the context of astronomical time series, and then also talk about an imaging problem as well that's interesting to us, and then get into how we're starting to figure out how to bake our physical understanding of these various different phenomena into the learning process itself. So one thing is using ML to be able to do discovery at scale; another is being able to imbue what we already know about the universe into the learning process to accelerate it, to do better, and lastly with a couple of statements about so-called likelihood-free inference. So as I've already said, ML in some sense powers a lot of

discovery of what's already going on in astronomy at scale. And this is one of the first, I think, prime examples of not just doing ML on offline data and saying my scaling curve's better than your scaling curve, but actually putting ML into production. This is this challenge that we have when we get lots of imaging data. We're trying to find a new object in the sky, and the way in which you do that is you take a big median stack of all the images in that part of the sky — I call that the static image — take a new image that just came in, align the two of them, subtract them, and what you get is what remains. And if there's nothing there, then

you should just see noise; if there's something new, then you should see something like the rogue's gallery at the bottom there, what we call real. But as it turns out, because the atmosphere is turbulent and we have a lot of noise in our instrumentation in general, we wind up having not perfect alignments, and so most of our subtractions lead to the so-called bogus detection. So these are not real astrophysical objects; this is just an artifact of the data processing. So what people did in the past isn't so different than what a lot of my colleagues were doing just about 10 years ago when they were looking at astronomical images, which is of course hire grad students to look at the data and decide, is this

real or bogus? It's pretty remarkable, this idea of using people, experts, to look at the data and then opine on that. It started hundreds of years ago in astronomy. We had a big data problem, as more data was coming off of telescopes even back then. And just as a side note, that's Henrietta Swan Leavitt seen in the back there. She's one of the most important figures in modern cosmology for some of her major discoveries, in this same room, and there's actually a new play about her that you should see; it's called Silent Sky. Anyway, what we decided to do is replace grad students and experts, because they don't scale, with ML. This is obvious to all of you in the room.

Obviously this allowed us to create a fast and parallelizable version of all that, and transparent and deterministic statements about all these different postage stamps that were flying off of the telescope. And this is a massive needle in the haystack problem that we had to attack. I think one of our biggest discoveries that most people outside of at least our immediate fields got to know about was the discovery of the nearest type Ia supernova — which I said before is the most important object for cosmology — in 25 years. And we did this in an ML-assisted way, where we basically presented to humans a ranked ordered list of the most interesting objects in the sky as they were flying off of the telescope. And at the

time this meant that we were able to discover this type Ia supernova about 11 to 12 hours after its putative explosion, which was days earlier than it had ever been done before. Now it turns out that this supernova got so bright that if you'd had binoculars and you looked at the right place, you would have gotten photons from the supernova hitting your eye, which is just remarkable. So it would have been discovered by amateurs days later or maybe weeks later. But what was so special about being able to recognize in real time that we had an interesting object is that we're able to throw the world's resources at it and get some novel science out of that. We learned a lot about the progenitors,

the objects that actually make type Ia supernovae, in a way we couldn't have done by any other means, even if we had hundreds or thousands of grad students doing this kind of discovery. Another place where ML is actually starting to have an impact is in the search for Planet 9. Many of you of course think that Planet 9 is Pluto; we won't get into a large debate about that. There is growing evidence that if you look at the orbits of long-period comets, there looks to be a gravitational perturbation which will be more massive than Pluto, and this could actually be the real Planet 9, and we just haven't found it yet. But what many of us think is that Planet

9 has already been imaged somewhere on the sky, because the entire sky has already been imaged, but it's too faint to see in any one detection waiting in one image. So what you have to do is stack up these images. The problem is Planet 9 is moving over time, and so for us to actually find this thing we actually have to stack a whole bunch of images where we basically shift them along the orbit of Planet 9. The problem is we don't know the orbit of Planet 9, so we have to guess it, and as you can imagine that adds to a massive complexity of this data cube that we're trying to find a new object in. So what we've

done is started a search using old data at LBL using the NERSC computers, where we had to basically look at every five to ten sigma detection in our very large data cube, and that led to hundreds of billions of possible candidates. But what we did is we trained an ML, basically zero-one, classifier on a bunch of synthetic data that we had created, that allowed us to get a very good accuracy, throwing out essentially 99 percent of the false positives so we could actually keep a good fraction of the true positives. The problem was that most of those models were too big to fit in the amount of RAM that we were allocated. We need to do this at scale, and

importantly, there were so many candidates that we didn't have enough data space at NERSC to be able to save all of the candidates to disk and then process it afterwards to figure out which ones might actually be that Planet 9. So what we realized we had to do is create a whole server farm of basically these little tiny apps that you could ask a question, send it a postage stamp and say, should I save this data or not, is it a possibly interesting candidate? And we had to get this round-trip time down to ten milliseconds. So we couldn't even save it on disk; we had to put it in a TCP/IP packet and distribute it throughout the NERSC supercomputer. And so

one of the interesting things that this brings up is the need not just for high-quality models, but, as many of the other speakers have spoken about today, they need to start thinking about the importance of fast models and ones that are deployable and ones that are versioned. So that's some of the bread and butter, in a supervised sense, of where we've been. What I want to tell you about now is some of the pain points that we've had with the traditional classification approaches. This is an example over a very large swath of the sky of 50,000 variable stars that are changing in time, and there's just one object that we pull out of that. Unless you're really good

at doing Fourier transforms in your head, you won't know that the periodicity is about a half-day; that's a classic star called an RR Lyrae. What we had to do to classify all of these stars was to be able to essentially build a whole bunch of features where we imbued our knowledge of how all these different types of variable stars change, and do random forests. This was sort of 10 years ago when we started working on this. As you know, when you're doing traditional featurization it leads to a lot of hand-coded feature engineering, and the compute generally will scale with the number of features that you have to do. And in our case we had a very small number of labels. So

one of the things, if you're ever talking to an astronomer who says we have a big data problem in the context of inference and classification, it's actually a small label problem — which, those two things are not completely inconsistent with each other. But in this case, out of that 50,000 objects, we had 25 classes of variable stars we were interested in, but we only had eight hundred and fifty or so labels for that. So we had a very small number of labels and we had to figure out how to bootstrap, and we used a bunch of active learning techniques to be able to get a bigger and bigger sample. Anyway, this is quite challenging. So what we've recently recognized is that rather than do our hand-coded

features, why don't we just throw this into an autoencoder? And for those that have seen this before, it's a classic autoencoder idea, where we take this encoding of the original light curve, we compress it down to a small bottleneck, maybe 64 numbers or eight numbers, and then we have a decoder which tries to get back the light curve that we had before, and then you build a loss function on that, you backprop, and you wind up getting auto-learned features. And what we wind up using is the features that are in that bottleneck layer, throwing that into a random forest, and we wound up getting best-in-class accuracy on a whole bunch of different data sets. So this got us

pretty excited. Now here's where the interesting thing comes in. We didn't just use an off-the-shelf LSTM; we actually had to modify the architecture, because there's no notion within an LSTM of the fact that data could be acquired at slightly different times. There's sort of a normal cadence beat that's assumed, but astronomical data is taken irregularly, so we had to figure out a way to have the network be able to handle irregularly sampled data. We also have noise on our data, so we wanted to make use of the noise properties of the data in the loss function. And importantly, as I said before, we don't have a lot of labels, so we wanted to do feature learning and we wanted

to build this network without any label. So this was a completely self-supervised way of building up labels over not just the corpus of 850 but over a much larger corpus, and as you can imagine that's one of the reasons why we did so well. And as a side note, I'll say one of the things we're exploring now is our ability to find anomalous or new types of objects in this feature space, in this bottleneck space. Another interesting problem that we're working on is denoising autoencoders. On the left-hand side is an image that you actually get out from the Hubble Space Telescope. All the scruff you see there are cosmic rays, charged particles which are hitting the detector, and what you'd

like to see is the thing on the right, which is the cosmic-ray-free image of a whole bunch of galaxies — in this case it's a galaxy cluster. So we used a modified U-Net architecture, again with a bottleneck at the very bottom there, where we take one of these postage stamps that has these cosmic rays in them and we try to predict a mask, and then we do a second task, which is, given the mask and given the original image, we want to get the beautiful inpainting final version of that. And one of the things that's interesting and exciting for us is that if you look at the activations in the top layers of this network that we were building,

it actually learned the Laplace transform, which is the current state of the art of what people do now to find cosmic rays, but of course it also learned a whole bunch of other interesting kernels directly from that data. And so we got good answers; at least visually it's very pretty, it seemed to do incredibly well, and then compared to the other state of the art — what would be a talk here without ROC curves? Here's your ROC curve, false positives and true positives — we actually bested it not just in quality of the results relative to the best in class but in speed, because as many people have said to us, well, it's great that you have a better cosmic ray detector, but unless it's faster and easier to use, I'm

not going to use it. And then on the inpainting task we did better than the traditional median mask inpainting and by harmonic interpolation. So in the last part of my talk I want to shift over a little bit to where things are going and where our big interests lie now in ML, and this is what I'll call physics-informed ML. And there's this great paper, if you haven't read it, which is called “Why Does Deep and Cheap Learning Work So Well?” by Max Tegmark and company at MIT, where he said, we have this image that's, let's say, a hundred by a hundred or a thousand by a thousand image, there's 256 possible values in a grayscale image, and so if

you're trying to determine between cats and dogs, the available state space is 256 to the million, and yet with a pretty small network, if I give you a bunch of images of cats and dogs I can actually distinguish that. And the point in this paper is to say that the reason why networks that don't have that much capacity can do so well is because they're learning what's physically plausible; it's not having to search over that massive space to get a good classification answer. So this becomes the point of departure for a question that we might ask: how do we impose our knowledge of physics into the learning process directly into the architecture? And this is starting to be done in a bunch of different places.

In the context of computer vision you have these spatially invariant transforms, so if I rotate an image I still get the same activations out. In the context of high-energy physics they're building networks that are actually QCD aware, so when you throw in data from the LHC it winds up being invariant to all the possibilities. And in quantum chemistry, like the picture you have there, if you're trying to do some notion of protein folding for instance, the same molecule you have on the left is the one on the right, and if you're trying to predict forces it shouldn't matter what that actual orientation is. So building architectures that are aware of these different kinds of symmetries is critically important. And one of the

papers that we just submitted kind of helps answer some of this question, which is whether we can find these embeddings and these network architectures that conform to our known understanding of the problem. We just submitted a paper to ICLR where we're making use of the fact that we have periodic data in the case of pulsation of RR Lyrae and the kinds of things I was telling you about with variable stars, where they repeat over time. So once you know the period, then all the data you get at another period basically doesn't add much information, and so we tried to build up an architecture that was aware of the fact that as you wind up going backwards in phase or forward in phase

eventually you wind up wrapping around yourself. So we're essentially doing convolutions in polar coordinates instead of Cartesian coordinates, and we'll see in the next couple of days if this gets accepted or not; if any of you are the reviewers, be kind. So another thing we're starting to do is ask the question, can we imbue the kinds of things that we care about, if we're trying to do simulations of variable stars, into the latent space in a variational autoencoder? And in particular what we'd like to do is start with our real light curves on the left-hand side, where we know not just the time histories, brightness as a function of time, but also the labels, and we also

know in some cases what their temperatures are, what their masses are, etc. We'd like to basically teach the network to be able to make objects that are like that, and then we'd like to sample from that latent space so that we can create lots of instantiations of it, so we can optimize our telescopes to find more objects like those. I won't go into the details of what this architecture is, only to point out that it's a traditional VAE except for the fact that we're injecting, in that yellow box there, the labels and the physical parameters both before the bottleneck and then after the bottleneck, so that at test time when we want to generate new objects we can basically just start from

a random Gaussian sample from that, add in the parameters that we care about, and actually produce realistic sources. And one of the things that we had to think hard about is how we did this, and, because we're trying to make and simulate real light curves, we had to do some architecting of our loss function and do some curriculum learning, where that last component there in the term also wound up allowing us to build a KL divergence between the uncertainties in the predicted magnitudes, or the predicted light curves, and what came out of the VAE. And as a result what we're able to do is walk around not just in latent space, like you see on the left-hand side, but also walk around

in something like temperature space, that allowed us to give these realistic light curves that came out. So this is still a bit of a work in progress, but when we get this working and we're able to deploy this, it means that we'll be able to build simulations of the variable-sky universe in a way that people have never been able to do before. Of course this sort of generative modeling isn't just in time series, and for our kind of science, many people in astronomy are starting to do this in the context of cosmology. What you see on the right-hand side are a bunch of instantiations of little slices of dark matter halos on very large swaths of the sky. The data

at the top is the real data that came from simulations on supercomputers, and the data you see on the bottom is generated data. Now, they might look sort of similar, it might sort of look like it has similar noise properties. Well, one of the clever ideas here as you're building these generators — and this was done again with a VAE — is to not just look at the images that come out but to do regularization on the aggregate properties of those images, in this case two-point correlation functions or other sorts of things like you see in the top left plot. What you want to do is not just produce pretty images, you want to actually produce realistic physical kinds

of summaries of that data, and that's what this group was doing. And then on the last note, what I'll say is we're also starting to get interested in inverse problems. So one thing is to be able to generate data; another is to say, given data, can you actually get for me the parameters that might have been used physically to generate that data? So this is an inference problem: I'm trying to take data and trying to get posteriors in a Bayesian sense on the parameters that matter to us. And one of the really exciting things that's coming out of particle physics is this deep connection between the simulations, which are extremely expensive to run, and so-called likelihood-free inference techniques, where

you're simultaneously drawing from priors on what you think your parameters are going to be, and you wind up directly building up something that can do density estimation that can give you out posteriors. So what I'll conclude with is, in some sense, the summary slide that says that where we're going now is trying to bake the physics into the entire process. In some sense, starting on the far left, you have featurization — that's what we've always been doing for the last 20, 30 years, taking our knowledge of these objects that we care about and building handcrafted features and then basically letting traditional learning algorithms learn on that. But now we're starting to build these symmetry-preserving layers — that's number two — and we're also thinking

about imposing sparsity, sort of a way of encoding Occam's razor into the size of these bottleneck layers, and then we're doing loss function curation, and we're doing distributional loss enforcement over the entire aggregate. So this is the state of where I think this field is going; it's tremendously exciting for us. So with that I'll leave up my conclusions here. What I will just say is that, again, it's important for astronomers and domain scientists in general to be connected deeply with the cutting-edge work of computer scientists and statisticians, but it is also tremendously important, I think, for computer scientists and statisticians to learn from us the kinds of questions we ask of our data, because our data looks different, and the kinds of

ways in which we want to get these answers may be very different than what you wind up seeing in industry. So with that I'll say thank you and happy to take a question or two. [Applause]

Questioner: Thank you for that awesome talk. We have a question. Thank you for your talk, it was very informative. So I do not know what your research focus is, but I know that many astrophysicists — but not only astrophysicists, also computer scientists — have worked on astronomical time series data using Bayesian nonparametrics, but it seems like they are giving up on those ideas they've been working on for 20-30 years. What is your take on Bayesian nonparametrics?

Josh Bloom: Well, so for instance in the context of time series, Bayesian nonparametric models are incredibly important. We're using Gaussian processes, for instance, to homogenize our datasets, where what I didn't tell you is that there's a variable length to a lot of these light curves that we get — sometimes we get 50 data points, sometimes you get a thousand. Doing that in a traditional batch learning sense makes it really hard to throw in different sized sources, so we're actually using these sorts of techniques to get lots of instantiations, from a data augmentation perspective, of fixed-length light curves. So that's really helpful. This certainly isn't to say that this is the only approach for doing inference; there are lots of other ways to do it. And I think one of the things that scientists tend to

be pretty good at — or they ought to be pretty good at — is using the right tool for the right problem, and we've identified a number of problems where ML and these new self-supervision techniques actually, I think, are pretty useful and pretty important.

Questioner: Let's thank Josh one more time.

Physics-Informed Astrophysical Machine Learning

Mon, 16 Dec 2019 00:00:00 +0000

Keynote on physics-informed ML across astrophysics: the Planet 9 search, variable-source detection, generative telescope scheduling, and cosmic-ray removal.

Physics-Informed (and -Informative) Generative Modelling in Astronomy

Mon, 23 Sep 2019 00:00:00 +0000

Practical generative neural models in astronomy: semi-supervised VAEs mapping physical parameters to latent space for telescope scheduling, cosmic-ray artifact detection and inpainting (deepCR), and autoencoder-based semantic indexing for compressed sensing.

Key Quotes

“The second reason why it's interesting to work with computer scientists is because they're pretty clueless about important problems to work on, and they need our help in understanding what are the important questions to be asked, rather than optimizing Twitter sentiment. They should be helping us understand how the universe works.” – Joshua Bloom

“Up until about a decade ago the state of the art was to hire more grad students to look at these images and go no no no no, yes yes yes.” – Joshua Bloom

“Imbuing some notion of the physical constraints that we'd like to have into our networks is a way of learning more quickly, which is another way of saying it allows us to learn with far less data.” – Joshua Bloom

“I don't hand you this 1.2 terabyte catalog, I hand you a 12 megabyte file, and with that 12 megabyte file you could produce whatever catalogs you want. How much compression can we have that would actually preserve scientific inquiry?” – Joshua Bloom

Transcript

Hi everybody, thanks for having me here. See if I can do this. I thought I'd put up a quote from a computer scientist, Jim Gray, who recognized decades ago that astronomers are really fun to work with if you're a computer scientist or mathematician. That's because our data is worthless, and if you make a mistake nobody dies. There's no PII that gets leaked out and causes billions of dollars of damage. He loved working with astronomers and we loved working with him. There's sort of two reasons why it's great to work with computer scientists as an astronomer. Number one, there's incredible tools that are being created all the time, and in the grand tradition of Galileo who said, oh, there's this thing called the telescope that's meant to point on the horizon to look for enemy ships, what if I just did this, pointing new algorithms and new approaches at our data with the kinds of inference problems that we have has been tremendously fruitful for us for centuries.

So we've been watching the machine learning world develop and we started taking a lot of those techniques and bringing it and making it our own. But then also the second reason why it's interesting to work with computer scientists is because they're pretty clueless about important problems to work on, and they need our help in understanding what are the important questions to be asked, rather than optimizing Twitter sentiment. They should be helping us understand how the universe works.

So that's really the point of departure for this talk, and what I'll do in the talk is help motivate our use of machine learning in more of a bread-and-butter way, traditional supervised learning, and then get into the kinds of uses of generative modeling that we're starting to see emerge essentially in real time. So it's a very exciting time for us. But to kick off, what I thought I would do, because obviously there's not a lot of astronomers in the room, I'm going to try to give you a little bit of a feel for the kinds of things that we're interested in. In my world this is one of the most interesting plots to look at because it captures both the things that we know about, like type 1a supernovae. This is what's called a light curve, so as a function of time we have these weird units which is essentially, think of it as brightness, so brighter up top and fainter down below.

Type 1a supernovae are the kinds of objects that we need to find en masse to be able to do cosmology, to measure the accelerating expansion of the universe to better and better accuracy. These are the sorts of objects we need. We want to understand how stars explode and how the elements in the universe are created, we need to understand other types of supernovae like type 2p supernovae, and we see lots of these two types of events. But there's other theorized events that, when this plot was made, merging neutron stars, which gives rise to a very bright and loud gravitational wave chirp, those hadn't been found yet, and the EM counterparts, the electromagnetic counterparts, also hadn't been found. It turns out that the one event that we've seen actually follows pretty well the theoretical predictions. But then there's these other types of interesting objects, neutron stars merging with large stars and pair production supernovae. These are theorized to exist, and so as we get data coming off of telescopes what we want to be able to do is find these things in the presence of the mundane. We also want to find some of these anomalies, these needles in a haystack.

Of course those are the kind of known unknowns, and then we can't, by definition, using Rumsfeld's words, write down and plot the unknown unknowns. The things that we don't know that exist out there that would be potentially transformative for understanding of the universe are not on this plot by definition. So the thing you would like to do if you could, for a given observatory or even the class of observatories over the whole world, is write down some optimization metric for how you do discovery, how you do follow-up. Because it turns out discovery is actually fairly inexpensive these days. It's the actual, now that I found this thing, what do I do with it. You need bigger and bigger telescopes, you need more specialized instrumentation to follow it up. This is incredibly hard to do, and in fact even if you could write it down on paper there's the socio-political reasons why you couldn't actually implement this, because there's lots of competing voices about what we should be using with our precious resources.

What looms very large for us is the advent of what's called the Large Synoptic Survey Telescope, which the American taxpayers have paid for already, and will be going for about ten years. It's a billion-plus-dollar facility surveying the entire night sky basically every three nights, capturing something like eight billion objects over the course of its 10-year history, and getting thousands of observations per object. So it's not just covering essentially the whole available sky from the southern hemisphere, but also capturing the time histories of all these objects that are changing. Now most objects of the sky are not changing in interesting ways, at least for most of us, but there are these new events like the supernovae, like these very rare things like emerging neutron stars, that we'd love to be able to find and follow up and do that effectively in real time. So the size of the data are fairly large, and if you think about this again from a needle in a haystack perspective, I'm sifting through all this data just to make the initial discoveries, and that in and of itself is a challenge.

Okay, so for the agenda, I'll talk to you a bit about how we're using machine learning today in a very bread-and-butter way, and then get into some of the versions of self-supervised and semi-supervised learning, show you a couple things that we've been doing in our group around astronomical time series inference, and then dealing with images, doing inpainting during the process of data collection and data analysis. Then I'll talk about physical constraints being built into variational autoencoders and the like and the kinds of things that we can do with it and we have been able to do with it. And last I'll end with some speculative thoughts around generative catalogs and some notion of creating compressed sensing so that we can actually acquire more data from remote sites and be able to

make really good use of that.

Okay, so to start off, bread-and-butter astronomy and ML. As you can imagine, for that needle in a haystack problem, what you'd like to do is find things that are changing in the sky. So what you do is you make essentially a median stack of a whole bunch of images to get the ground truth of what the static sky should look like, and as a new image comes in you subtract your new image from your old one. The problem is the atmosphere is turbulent and your optics are not perfect, so doing that subtraction in the presence of Poisson noise and all the other pernicious noise properties of your detector is essentially an impossible task to do perfectly. So you have to do it imperfectly, and what you wind up getting is a whole bunch of bad subtractions. These are some postage stamps of those, and the number of these in classic surveys that are ongoing right now vastly out-swamp the number of new objects. So the new object is essentially something that is brighter than the static sky, or it could actually be something that got fainter than the static sky if it's an object that's changing over long timescales.

So we have to be able to sort through this very quickly. Up until about a decade ago the state of the art was to hire more grad students to look at these images and go no no no no, yes yes yes. As you can imagine this is a great place for ML, not just for the parallelizability of it, but for the fact that it's deterministic inversion of it all. So this is stuff that I worked on about a decade ago, got it infused into a project called the Palomar Transient Factory, which is in some sense a very good precursor to the Large Synoptic Survey Telescope. And it worked, it helped us sift through the sky very quickly, and perhaps the greatest result of that is represented in this image here where in 2011 a new supernova was discovered. So we call it ML-assisted, by the work that we had done allowing us to hone in on the topmost interesting candidates as quickly as possible.

This turned out to be a type 1a supernova, which as I said before is incredibly important for cosmology, and it was the nearest type 1a supernova in about three decades. So in the digital era this was the nearest supernova, that allowed us to study it in great detail. The important thing from the perspective of where did ML actually help in this world wasn't that it helped us find this quickly, it's that it helped us find it quickly and that we were able to do something with it. This got so bright that if you had had binoculars and you were looking in the right place you would have actually seen the photons with your own eyes, which is actually amazing, that generally hasn't happened since Kepler's time since we had a supernova that you could see as bright as that. So it would have been found by amateurs, but it would have been found by amateurs days later. But the fact that we were able to find it within 11 hours after explosion, which at the time was unprecedented, allowed us to throw the world's telescopes at that position in the sky, get some interesting limits on the change of brightness as a function of time.

This is a very busy plot that captures a lot of physics that we're interested in. These objects essentially allowed us to rule out all the regions that are colored here, which allowed us to rule out the things that exploded to be not traditional types of stars, which are right here, but they had to be what are called compact objects. So a neutron star or a white dwarf were the only things that were left over that were allowed. Had we only detected this days later we wouldn't have been able to make this plot and get the first direct constraints on the nature of what it is that exploded. So that's one example of the kinds of things that we have been doing for a while and are working at scale fairly regularly across multiple different projects. Now it has really become a cottage industry.

There's other things that people are trying to do which I think of mostly as a supervised learning problem. There we have these simulations, for instance, of the distribution of dark matter in the universe as a function of different cosmological initial parameters. So these are things that cosmologists think about and we're trying to measure these parameters very very carefully. Given this few set of half-dozen or so initial condition parameters in the universe, you can simulate forward and model in a supercomputer and get these distributions on the sky, essentially in 3D space, of how matter should be distributed some time after the Big Bang. So the idea is, given an observation of this, could you go and do the inverse problem, and given that, could you actually infer what those parameters are. And so with enough simulations you can imagine using a 3D convolutional neural net idea, one could then try to back that out. So this is the kinds of things that people are trying to do.

Another interesting problem that has recently come out is the idea of looking at what are called gravitational lenses. So these are individual galaxies that have been lensed by a foreground galaxy which has been subtracted off here, and when you get this lensing effect that galaxy, through gravitational effects, winds up basically bending the light around the galaxy and you wind up seeing multiple images of that object. The objective here, given lots of observations of gravitational lenses, is to figure out the size of the radius on the sky, because that then gives you access to other interesting cosmological parameters like Hubble's constant. And so in this project here they simulated a whole bunch of gravitational lenses and then asked the question, could I make a measurement of that angular distance on the sky and recover back the thing that was in the simulation, and they said yes. And then all these different plots up here with the boxes and different colors correspond to these images right here, where they actually were able to infer what these Einstein radii are. Now compared to very complex modeling, this generally is a very hands-on, very labor-intensive, large Monte Carlo problem that you have to work on, because you don't know the properties of the galaxies that's being lensed and you don't know the property of the thing that's doing the lensing.

So this is where the state of the field is at in the context of supervised learning. But one of the things that we started thinking about in our group is how we start making use of semi-supervised and even self-supervised learning for the kinds of problems that we like to work on. So in some sense the pain point that's being captured in this slide here is the reason why we felt like we had to try something new. Typically what we have are observations, again this is a light curve, not of an explosive event but of a variable star. This is called an RR Lyrae star, and it actually

has a period. Unless you're good at doing periodograms with your eye you probably don't know this period is about a half day. And what we traditionally have to do over the whole sky of variable stars, and this is a catalog of 50,000 variable stars that we looked at, we want to infer the classes over dozens of potential classes of variables and transients, and this is essentially all the data that we have. So because we started this project about 10, 15 years ago and started working on it before the resurgence of neural nets and deep learning, we were doing the traditional learning with random forests, and there we had to build features on every single object. So we have a light curve, sometimes we have some metadata, and we want to infer what those objects are, again pretty bread-and-butter supervision.

The problem with that of course is that it requires lots of hand-coding of the feature engineering, which can be labor-intensive and actually can be pretty expensive at predict time, and of course the predict time will then wind up scaling roughly with the number of features that you have. And the other big problem is that we only have a small number of labels. So while we can produce lots and lots of features of these 50,000 objects, it turns out for this study that we did we only had about 800 labels across 26 different classes. It was a very very small label problem. And so we wanted to know how we could actually learn features in a way even when we didn't have the labels. So this is where our self-supervised approach came in, where we wound up using an autoencoder. And you've already seen autoencoders in a couple different contexts today already, but the general idea of course is that you have an encoding of your light curve, you have some sort of bottleneck layer, you have a decoder, you try to get back your light curve. And then what we did, because we're still kind of in the random forest camp, is use these features and augment them with some metadata about each object and throw that in a random forest to see how well we could do.

So here, instead of hand-coding hundreds of features, we were able to let the network essentially learn, not just from the sources that had classes and good labels but from the entire catalog, and it turned out to work pretty well. So here are some original light curves of these sources unfolded, and then when you fold them you wind up seeing the reconstruction in red compared to the blue. When you have very fast periods, in this case here on scales of days instead of scales much longer than that, it turns out when you don't fold, the network spent most of its capacity trying to learn how to do periodograms. So we just basically said let's give you the folded, phased light curves, and as you can imagine it actually did pretty well compared to some of the other results. So across three different data sets this autoencoder approach basically bested all the other feature-based approaches that had been done, across not twenty-six classes because these were smaller data sets than our original one, but across a reasonable number of classes. So we're pretty pleased with that.

But this is one of these interesting places where the computer science work on recurrent neural nets and LSTMs wound up conflicting a little bit with the way in which our data is distributed. In particular when we take data on the sky we're not taking data in regular intervals, we're taking it in irregular intervals, sometimes perniciously, where you wind up having these bad window functions. So we needed a network that could natively handle the distribution of data. So we had time and flux measurements, and then we also have uncertainty in our measurements. There's a Poisson error when we wind up making these measurements, so we needed something that could naturally account for, and actually help what we had, highly uncertain light curves. Those would count less for the building up of these bottleneck features. So we built a stacked RNN.

Was there a questioner? Yeah. No, this is not microlensing data. So MACHO stands for one of these microlensing surveys. There's other data that they had that wasn't just on microlensing events. They observed large swaths of the sky for decades and produced a whole bunch of different classes of different variable stars, so we just focused on periodic variable stars.

Was there another question? Yeah. Yes, we have uncertainties. That's the cool thing about astronomers, is we believe we know what our measurement uncertainty is, because again it's just Poisson statistics. As it turns out though, we're often wrong. So if you were doing this right from a hierarchical Bayes model, if you're doing some forward model, you'd also want to have some nuisance parameter of what's the likelihood that your uncertainty is way off at any one given measurement. So people do have to take that into account. We didn't use that, for instance, in our study. But where uncertainty comes in is basically in a modified MSE. So now the loss function of the reconstruction is basically just using our measurement uncertainties in a per-epoch way, and so if we had a data point that was very uncertain it counted less in the loss, which is just an obvious place to stick it in.

Yeah, no, this is, we're just doing an autoencoder here, so this is just, we're just doing it like an MSE or an L2 loss, but now normalized by the uncertainties. So if the uncertainty is large, then it winds up, one over the uncertainties, it winds up knocking that data point out from contributing a lot to that loss. And then the other place is we had to use the notion of, here are flux measurements, we had to use the notion of the time offsets. So this allows us to naturally use RNN type of architectures where we have these interesting two properties of how astronomy data is taken. And the nice thing about the uncertainties actually is that this also allowed us to do a bootstrap resampling. It's a very natural way of doing data augmentation. So if I have a given light curve and I have uncertainties at every measurement, as long as you assume some sort of iid between all those measurements, I'm free to basically bootstrap resample every measurement that I made, and now instead of one object where I get one pass through this network, now I can potentially have hundreds. And that actually wound up helping us a great deal.

So as you can imagine the thing we were excited about here is being able to leverage a large corpus of unlabeled data for us to be able to build features across all these different classes. And the interesting thing that we hadn't expected is that when you actually use this network on a different survey, where you take data in a different way with a different distribution of classes of objects you're interested in, the transfer learning actually worked pretty well. So that's been out in the literature for a little bit. The thing that we're starting to work on

is what I call this kitchen sink approach, which is essentially taking this blue train here where you have what we did before, but now, given that we are actually interested ultimately in the classification across all these different sources, is to use the classification and have some admixture of the loss across that. But instead of doing this all where you're using this just to build some features on the time series, now these features are actually going to be learned so that if you know the class you should be able to get that class back very well. So this is the semi-supervised approach that we're working on, it's starting to show a lot of promise. We're also thinking about co-training across multiple different surveys, and not just a single band-pass flux measurement of a given object but multiple band passes, so multiple colors. So we're basically throwing everything we know at this problem across a lot of different surveys, and stay tuned for the results on that.

Another thing. Yes, good. Yes, so the question is about source metadata. So I just showed you data on the objects, how they varied in time, but of course there's a lot of observations and information that we have about where that object is in the sky, what's the nearest object to it, what are its colors, does it have radio emission, is there x-ray emission associated with it. And this is one of the challenges, is that every object on the sky has a different catalog that you can look up from the past, and some have a lot of information and some have very little information. So the heterogeneity of our data is actually one of the most confounding, difficult problems here. What I also didn't mention is that the light curves that we're using are variable length, so it's kind of hard to think about how you might do that in a batch sense. So using masked autoencoders is possible, but it's actually pretty painful.

The other thing of course that one can imagine doing is looking at the encoding that you get. And here's just for a toy problem that we're working on, two different types of light curves, what is the distribution of those encodings. And here's across all of these classes. I don't know why this looks like a bird, but it does to me. And this is just UMAP, and then all of the ones that weren't part of the training set wind up getting a very nice separation in this space. And this is where we wind up imposing an L2 norm, so you wind up putting, over all your dimensions, you basically put something on a hypersphere, which actually helps for the separability. And then of course you also can look at reconstruction loss, and here's where we get good reconstruction. And here what we wound up putting in weird objects, these anomalous things that weren't part of the sinusoidal systems that we started off with, we get very poor reconstruction, so we have large loss. So we're trying to build anomaly detectors as well, which of course gives us access to interesting new sources should they be out there.

Another thing we're using autoencoders for is cleaning up data. And so here we have an image from the Hubble Space Telescope of a galaxy cluster. Now it's an inverted image, so the darker it is the brighter the source is, and all this scruff up here, all these little things, are what are called cosmic rays. So these are charged particles that are actually hitting the detector in space, and you want to be able to get rid of those and know what was underneath. These spurious charges, anything that was underneath that is actually completely wiped out because this basically saturates the detector, it gets close to saturating the detector. So what you'd like to do is learn an autoencoder where you have ground truth, so that you can not only detect these objects but you can actually inpaint where those objects were, and there you have to learn what it means to be an astronomical object that you image. So here we're basically just using a modified version of a U-Net. So for those that have seen this before, you basically have a bunch of skip connections, you again have some notion of a bottleneck. And what we want to do is take our original image, and we did this in small postage stamps, and you want to predict out that mask, that's task number one. And then task number two is, given that mask and given the image, do an inpainting so that you wind up getting an observation of what you would have seen had there not been any cosmic rays.

So we just published this fairly recently, and one of the things that we were excited to see is that in the top layers of our convolution, we got excited about this one right here, which if you're not used to looking at these things is actually a Laplace kernel. And it turns out that Laplace edge detectors run over our images is the current state of the art, a way in which people actually find cosmic rays. So the network actually learned Laplace, which is not so surprising, but it also learned interesting things like find symmetries on the left and the right and the top and the bottom, etc. So that was very exciting for us. Visually it turns out it works extremely well. But then of course the thing that you need to do is show ROC curves, and so compared to that state-of-the-art Laplacian edge detector in different types of fields, we wind up having essentially better ROC curves for all of that, so for our false positives and false negatives. The thing that we're also very concerned about, and we actually spent a lot of time thinking about,

is model complexity and its impact on the speed at predict time, because we're talking about taking images that have already been acquired and running it through this network. The longer it takes the slower your pipeline is, and it turns out the Laplace edge detector takes a very very long time to run even if it's been parallelized on a CPU. So we actually were thoughtful I think about the size of our network, because if we had a very big network and it did much better than the current state of the art but it took 10 times longer to run, no one would use it. So this is actually starting to creep up in a lot of the problems that we're thinking about, is not just the quality from a ROC curve perspective, other sorts of impact like RAM usage, like your ability to run even on a GPU. And then from an inpainting perspective, compared to median masking and biharmonic interpolation, we did quite well and we're much faster than the other processes.

Okay, so let me turn to something that's adjacent to that work of AEs and semi- and self-supervision, to physics-informed ML. And this is an interesting paper if you haven't read it already, asking the question, why does deep and cheap learning work so well. There it's a very nice quote making the statement that if you've got an image of grayscale of 256 and it's a thousand by thousand image of a cat or a dog, the size of the available space is, whatever, it's 256 to the million, and yet these networks which have far less capacity than that seem to do extremely well, making the case that the reason why the networks that we're using are working well is because they're very quickly honing in on a much smaller space of what's physically plausible and physically relevant. So as many of you have started thinking about, and some people in this room will be speaking about, the idea of now not just having the network learn this from scratch but actually imbuing some notion of the physical constraints that we'd like to have into our networks is a way of learning more quickly, which is another way of saying it allows us to learn with far less data, and make sure that when we make predictions out of these networks they're actually physically plausible and physically relevant.

So Tess is going to be talking about these rotation, translation, and permutation equivariance types of neural nets that, regardless of what your orientation of your molecule is, you still wind up getting out the same force vectors just rotated appropriately. But this is now happening across a number of different fields, and one of the things that I think in general what we're trying to do is find embeddings and network architectures that don't just conform to the known physics but, if we have a hierarchy for instance of classes, can we actually build that in as well. We want to make sure that our networks are physically informed and are basically constrained by the physics that we have.

So this is a new project that we've started off on recently. I'll tell you first about the application of that and then show you what we're doing and how we're thinking about that. So here, going back to the Large Synoptic Survey Telescope, what we're trying to learn before the survey gets going, next year or a year from now, is how we should observe the sky. And you think, okay, well you just go like this, but then somebody goes no, because then you're not going to find the interesting solar system objects. Over all these different science objectives there's a lot of competing needs on the cadence of how you actually observe the sky and what filter when you come back to the same part of the sky. And this for instance is one of the outputs of a large simulation, which isn't just saying where do we observe on the sky, let's just make measurements of that, but then forward folding through photons from an ab initio cosmological simulation that includes galaxies to figure out what our detection threshold is so that we can then turn that into figures of merit for how well you can detect dark matter and dark energy. So this is a very complex part of this project. This is a terrible cadence if you're

interested in periodic variables, because you've just built in all these aliasing window functions that really make it very hard to find the appropriate periods. But one of the challenges is that there's some objects that are easy to simulate that you can throw into this, but there are a lot of types of objects like the ones I'm interested in that are very hard to simulate, because there aren't these kind of ab initio models where I can just say here's all the physics, now make me a whole bunch of interesting objects of type RR Lyrae or of type Mira or of type whatever. So what people will generally do is take observations of these known classes, try to do some sort of Gaussian processes and then do interpolation so that they can do these simulations. Well, this is very difficult and winds up leading to an interesting set of biases where we don't actually have the full range of the specific class that we're interested in that we could then throw into our simulation.

So what we set out to do is try to find a data-driven, nonlinear, nonparametric model over all the different classes of variable stars that we're interested in that would allow us to walk around in some space, i.e. some latent space, and produce a whole bunch of variable stars where we wind up now not just producing things that look like the variable stars that we're interested in but are ones that are actually constrained by the physics of how we know variable stars to work. And the overall sketch of that is that we have all these different light curves, we wind up having our latent space where we have nice separability in that latent space between the different classes, different colors or different types of objects, and then walking around not just in the abstract latent space but actually now infusing into the latent space some of the physical parameters that we're interested in looking at. In this case this is called T effective, so it's the effective temperature of that object. We want to be able to produce realistic generated light curves that are constrained by that.

So this is a complicated diagram. It's essentially a VAE that all of you have been seeing before, except what we're doing is we're taking our labels of our objects that we care about and want to simulate and the physical parameters of those objects, and we're injecting them after our recurrent neural net space and then basically making sure that the latent space knows about those. But then afterwards, after we wind up sampling from the latent space, we inject it back in so that we wind up having our input back in so that we can construct our light curves and be constrained by the temperature and the size and other types of properties of the objects that we care about. Like I said, we're currently, because we have sort of a swap-in notion, now using these things called temporal convolutional neural nets, which is something that's based off of WaveNet. It has some nice properties, and they act like convolutions and they don't have some of the challenges that you get with LSTMs and GRUs in their computational problems.

Yeah, what's that? Oh yeah, these are the citations. So the citation for WaveNet is van den Oord, that's the original one. Oh no, we're putting it out soon, yeah, we're still working on it. But TCNs, it's a very nice paper that compares that to all the other sequential learners. And so what we're doing is obviously, in the loss, having a reconstruction of our likelihood the same way that we had before with the previous study that I showed you, but we're also actually, in some sort of curriculum learning sense, stepping up the amount of requirement that we have, that first of all we learn our reconstruction really well, and then we start stepping, as we're learning, through this value of beta that winds up adding more power to the KL divergence to basically regularize the latent space very well. And because we're trying to make simulations out of our data, we also want to simulate what our uncertainties in our data would be in our light curves, so we're also building in a KL divergence for the predicted distribution of uncertainties.

Yeah, when I have the uncertainties, again, so these, in some sense you can think about this as another part of the data. So we're trying to reconstruct the light curve itself, but here we're also trying to reconstruct the distribution of uncertainties in the flux measurements, and again we have those because those all come with the catalogs that we're using. So just a little movie of that. So if we don't include the physical parameters we can still walk around in that latent space for a different label that we're interested in producing, but when we have that we can actually see the effects of what temperature does on the objects that we're interested in. And for those of us that have been working with these types of objects, it's kind of remarkable, when you move temperature up and down, that you wind up seeing the effects that you might nominally expect. So it's actually learning some of the properties of the distribution of the data very well.

Yeah, so another part of the project that we're doing but it's less far along than what I'm showing you here, is, so the inspiration for this was what's called a transparent latent space GAN, where you may have seen pictures of it, where you want to walk around in the space of what you're generating where you're kind of linearized on things that matter. So in the case that this TL-GAN was using, he was trying to do face reconstruction and face construction, and so there it'd be dialing up and down the amount of hair of somebody. And so rather than walking in this abstract latent space, essentially what you do is you linearize the latent space, you make a prediction of what are my six values of my latent space and what direction do I need to head in that latent space that is parallel to the physical parameters. Here we're actually just injecting those physical parameters, but your intuition is exactly right, if you walk around in this and you move Z0 down by a little bit and Z1 up by a little bit, you're actually heading in a parallel space to the physical parameter. So what

we're doing, because we have these for a subset of our data, we're also trying to see what if we just kept this latent space like this but instead of dialing in these parameters, what if we just purely dialed in the temperature direction or the mass direction or something else. Yes, correct.

And are you disentangling, are you doing something to encourage disentanglement amongst the abstract? No, we're not, well, so yes, the encouragement of the disentanglement comes from this part right here. You wind up actually getting a nice disentanglement over time. So it's interesting, when you don't have this KL divergence and you're just trying to do reconstruction likelihood, all the classes of all the objects wind up just being jumbled all over each other. But here we're actually encouraging disentanglement of the classes, and you're using the reparameterization trick anyway as well, and so you wind up getting this to look like a Gaussian.

Let me just go on for now, just in the interest of time. How am I doing on time? Say that again. 20 minutes to lunch, okay good, yes, well I'll be done. So that's our foray into the generative modeling space. Many other people are doing this as well. In the context of cosmology here, this is again producing what are called weak lensing maps. So this is in some sense a 2D projection of the dark matter distribution on the sky. Here's the original data from simulation, and it's basically indistinguishable, at least visually, from what these GANs wind up producing. But more importantly, when you actually look at the important properties of these, like a two-point correlation function for instance, they wind up falling along very well, not just producing visual similarity but also these similarities in this distributional space. What's interesting is that I don't believe that this study actually imposed regularization that the distributions of the images should wind up having these properties, it just came out naturally from the GAN.

Related to that of course is surrogate modeling, where you have extremely expensive computational projects. So here's an example of just a two-dimensional model of a supernova, where you have a couple of input parameters like the mass and the distribution of mass in the star before it blows up, and other sorts of things like its rotation, magnetic fields, etc. You have just a few numbers that you start off with, and then you need a massive amount of supercompute to be able to produce these pretty pictures here. But of course this is not what we observe from an object which is giga-light-years away. What we observe is these really noisy measurements of the total intensity as a function of time, which is the integrated light that comes out of this. And this one simulation for instance was 360,000 CPU hours at NERSC just to produce this one object, which produces one light curve for one set of input parameters. What we'd like to be able to do is make measurements like this with these black points and figure out what are the correct input parameters to the explosion. And there, surrogate modeling has been very helpful. As far as I know that's mostly only been done with Gaussian processes, where they have a family of different explosions and then they have a way of doing interpolation using Gaussian processes in multiple dimensions to be able to back out what the parameters of interest are.

This is also starting to be used as well in gravitational wave measurements and simulation. What you see on the right-hand side is what's called a chirp

signal as a function of time in milliseconds before you have a black hole merge with another black hole. The problem is the breakthrough in making the actual exact calculations for this only happened about 10 years ago. To do the full general relativistic simulation in 3D that's numerically correct and stable, you need all these different input parameters. These are dimensionless spin parameters, this is the ratio of the masses of the two black holes, and then some orientation parameters. You need all of these to be prescribed exactly and then you wind up getting these different waveforms. What people are doing in this study is taking, across these simulations of what these waveforms would look like in these different parameters, they've got these 1500 or so simulations of waveforms, they're creating a surrogate model that allows you, if you had a measurement of this chirp signal, to be able to back out what those parameters are, and they're doing pretty well. This is this study and this is a previous one compared to the full numerical relativity supercomputing simulation. Each one of these data points is like more compute time than probably most of you have ever used. This is a massive endeavor that's been ongoing for decades to be able to generate this coarse grid, so the idea is to be able to build something that is more finely grated, and that's where the surrogate modeling was coming in.

And of course, related to that are inverse problems and likelihood-free problems where you want to do some sort of inference. And in this case, and you'll hear more about this from Ben at the end of this week, you've got simulations of a particle shower from a collider and you throw that in, including the original parameters, and you use this as a way to get confidence intervals on the parameters that you're interested in.

Last, in the time that I have left, I just wanted to give you some other speculative ideas, because we're talking about generative modeling here, on how we could actually make interesting use of generative modeling in astronomy beyond the sorts of problems that I've given you. So here's actual data, this isn't simulated data, from a satellite which is operating right now called Gaia. It's making measurements on about three billion stars in the sky, and the amount of data that's coming down from the satellite is about 200 terabytes of raw data. The final compressed catalog is 1.2 terabytes of data, and there's a lot of interest in this. By the way, this is the projection of the sky if you haven't seen it before. Usually you don't get to see the Milky Way really nicely. This is the center of our galaxy. See these Magellanic Clouds, very nice image. And this is actually an image that's now been generated not from the original 2D pixels but from the catalog itself. And you can see the distribution of colors and some interesting properties.

One of the things that people like to do with this isn't just study individual objects but be able to find large structures in our galaxies that hadn't been known about. When you look at other galaxies you wind up seeing that they've actually been eating other smaller galaxies. This is called galactic cannibalism, because astronomers like to make up fun words, and it's actually true. This is the way in which galaxies form, is they basically assemble themselves from smaller little building blocks over time. And we know that the Magellanic Clouds for instance are sort of waiting to be eaten by us eventually. These are little dwarf galaxies that are in the tidal influence of our Milky Way. But there are other galaxies that had been around that actually got totally ripped apart and are now part of our galaxy, we've eaten them. We've known about these streams for a long time, these are called tidal streams. It turns out though that there are a whole bunch of other streams that we hadn't known about until Gaia started producing its maps of the sky, because it's making not just a map in 2D space, it's making a map of the locations of these objects using parallax. And this is an amazing study where they're basically finding all these essentially old galaxies that have been eaten up by the Milky Way.

So the idea is, what if you could take this catalog and you lose information about an individual object but you could compress the catalog down to a small amount, that you could still do these aggregate studies to find interesting things about large-scale structure. And the idea is again some sort of autoencoder where now I'm actually just trying to autoencode the catalog itself. And so I have my normal latent space regularization, but for my numerical columns I do some sort of weighted MSE, for my categorical columns you do some cross-entropy loss. But then what you want to make sure is, when you wind up having your batches, the outputs from this, this is the raw catalog output, so this would be like location on the sky, proper motion, velocity on the sky, etc. Some admixture of some of these winds up basically giving us some interesting insights into the distributions of the physical parameters of these objects. This is a proxy for temperature on this axis and this is a proxy for brightness here. You notice that the universe doesn't populate this space uniformly. This is what's called the main sequence, this is where most stars lie, these are where white dwarfs lie, and these are objects that have fallen off the main sequence. What you'd like to do is ensure that you actually have some regularization for all the different physical plots that you'd like to make that come out of that catalog. And so that's that extra little piece that I think is going to be tremendously useful. And then

it gets us to an interesting place. We could use this for anomaly detection, so if we run all of our objects through our mock catalog creator we could actually probably find some data quality issues where an object is not where it's supposed to be in that space and its loss, its reconstruction, is pretty bad. We could produce thousands of simulations of the Milky Way that are all consistent with the observations that we have, which would be really nice for us to be able to compare to theoretical statements about what the Milky Way distribution properties should look like. And the other interesting thing maybe is that if you were able to compress a catalog down, you can also do some option where the people who make this catalog, who want to do the science, could release the aggregate results to the world and then they could work on the individual objects themselves, but it would allow people to basically build up their own catalogs where you're not actually leaking out the individual information about any one individual object. So one of the things that we're kind of starting to get interested in is, what amount of compression could you have, essentially I don't hand you this 1.2 terabyte catalog, I hand you a 12 megabyte file, and with that 12 megabyte file you could produce whatever catalogs you want. How much compression can we have that would actually preserve scientific inquiry?

So in some sense the way that I think we're all starting to think about where physics comes into the learning process is the fact that we have the ability to infuse physics across that whole process. We already do this in the old-school way when we do featurization, when we take our raw data and we featurize it down to a couple of numbers, we're using our physical knowledge and our domain knowledge to be able to compress that data. I mentioned to you, and you'll hear more about this throughout this week, these symmetry-preserving layers that preserve conservation of energy, that preserve rotation, etc. I talked a lot about the bottlenecks and making sure that the models are sparse enough that you're not actually having too much capacity in your system, that's a way of saying we want to have Occam's razor be in effect when we produce our models. We can have loss functions that enforce physically meaningful results at the instance level, so I talked about that as well. And then what I just started introducing is distributional losses where we make sure that when we actually throw a batch of objects through we wind up getting a distribution that falls along the lines of what we know to be correct physically.

And then, last and closing up here, where I think this could be really interesting in the context of catalog compression, is the fact that we often are taking data in remote parts of the solar system. So the James Webb Space Telescope, which your taxpayer dollars have also paid for, order of magnitude more expensive than LSST, which is going to fly in a few years from now, it's sitting at the Lagrange point, at L2. The data rate from there is not very good, it's about two megabytes a second, and you can only get about 56 gigs a day down from this satellite. Yet a single instrument, and there are multiple instruments on this, at full bore can produce essentially a terabyte a day. So we're already having to make decisions about how we should observe the sky because of this bottleneck of us getting data down from these satellites. And somebody once said, these photons took a really long time to get to us, we owe it to them to actually detect them, the fact that we're not doing that and throwing out data is absolutely ridiculous. And the state of the art, by the way, in a lot of these satellites, is they wound up taking data and then they have prescribed regions on the sky that they're interested in and they only send down postage stamps of all the pixels from around that and they throw out everything else, that's their way of getting around the data rate. So, and this is very speculative, there's no reference, don't quote me on it if it's wrong but quote me on it if it's right, the idea here would be, what if you could train a denoising autoencoder on

lots of simulations of what, let's say, the James Webb Space Telescope is going to look at. There's a picture of a simulation, people do this a lot, where you have all the data available you possibly could, you build a nice beefy enough network that given the raw data you could actually not send down the raw data anymore. What if we just sent down the bottleneck layer? And of course the ground station has the model because you flew it up there in the sky. But you can also wind up periodically updating your model on the fly on the telescope and send down the diffs of the weights so that you wind up syncing up over time, when you have free bandwidth, the current state-of-the-art model. So I don't know if this is going to work, but I think it'd be really interesting, convincing NASA that I could actually fly something like this and not send down the real data would be pretty interesting.

Yeah, the question is could you install these things remotely. There's no way that they would let you do that, right? Every line of code has got vetted over hundreds of times, which means there's probably lots of bugs. And they literally generally fly not a lot of data space, and in fact JWST is flying only about 10% more data space than they can download in a day, because if they start accumulating more data and they can't get it down, there's no reason to have it. So the amount of data that's there, the amount of code and the code that's running there is all very prescribed. And at L2 you don't have a huge energy budget problem because you're reasonably close to the Sun, but as you get to even more remote sites then you really have to start worrying about clock cycles, and so all of that generally is fairly locked down before it flies. This would have to be a totally new mission, and I think the way to do it would be to do it with a CubeSat, which is only a few million bucks, and say I'm going to take data like every half second of the entire sky and send down not all of that data because I don't have enough bandwidth, but just the bottleneck.

Yeah, real time, but we have a stream which is, yeah. I mean, yes, you could of course do that. The nice benefit of having just a model that's small is that the amount of clock cycles to do a forward pass wouldn't be all that much, but I think if you were going to do this, yes, you could totally do that, you could do a hybrid approach where you're sending down all the interesting data in the old-school way. But the problem also is that we take data in a different way depending upon what we know our bandwidth is. And so what they're doing effectively is longer integration times, which opens you up to more cosmic rays and other sorts of problems. They do longer integration times because they know if they just accumulate one image instead of a hundred images, that's much easier to send down. So we're already taking data differently because of the bandwidth issue.

You're probably taking as much data as possible, then after the fact the data is taken you're deciding what to keep or not. I haven't decided whether I want to take this on because this is going to be more of a political fight, but it'll be fun to write the paper where you say this is possible, here's the kind of losses that you get. And by the way, it's not just, your loss function now would wind up being very aware of the kind of science you want to do. You basically say I want to preserve the PSF and limit down to some brightness level, so if the sky is blank there you're not basically having to reconstruct that part of the sky.

Yeah, let's say you find an anomaly. Yeah, good question. So around anomalies, I thought about that a little bit. The idea would be, as you throw the data in, whenever you have a good reconstruction loss on board you throw down the bottleneck, when you have a bad reconstruction you say that's an interesting thing that just came through, I don't know about that, I'll use that to update my weights of the model but then I'll also throw down that full image. Yeah, so I think that's the way you might do it, yes. It's a good question.

Why don't I just summarize here, and then maybe we come back to that, we have a little bit of time. But anyway, just to give you this overall sense as I close: machine learning's already fairly central to the way that a lot of astronomers are working. We're kind of using it for the bread-and-butter inference and discovery at scale, essentially replacing grad students. But we're starting to get interested in the semi-supervised and self-supervised approaches, in large part because we have so few labels, we need to do this unsupervised or self-supervised learning of labels. But where I think there's a lot of excitement, and this is really the crux of what this workshop is about, is how in this context we can actually imbue some of the physical constraints and the distributions of our catalogs into the actual loss itself as part of the regularization. And I think what we're starting to see is that growing symbiosis between the first-principle simulations that are extremely expensive and the generative and the surrogate modeling that we're actually really interested in, getting to these fast ways of getting to inference of the parameters that we care about. And last, I noted some interesting things we might be able to do with generative modeling for compressed sensing. So with that, why don't I stop, and then we can start taking some of the questions. [Applause]

Machine Learning at Scale: Astrophysics

Wed, 11 Sep 2019 00:00:00 +0000

Plenary at the DOE ‘AI for Science’ town hall on scaling ML for astrophysics — real-time discovery, classification, and inference over massive survey data streams — feeding into the 2020 DOE AI for Science report.

Optimizing DESI for the Time-Domain Opportunities

Wed, 10 Jul 2019 00:00:00 +0000

Plenary on how the DESI spectroscopic survey could benefit time-domain astronomy — variable stars, transients, circumnuclear events, unusual supernovae, and multimessenger follow-up — with proposed fiber-allocation and data-access strategies.

Time-Domain Inference and Physics in ML

Mon, 01 Jul 2019 00:00:00 +0000

Physics-informed machine learning for time-domain inference, at the Moore DDD investigators meeting.

Astrophysical Machine Learning

Thu, 18 Apr 2019 00:00:00 +0000

How time-domain astronomers use ML on streaming, noisy, distorted images of the sky — the search for Planet 9, discovery of new variable sources, data-driven emulators — and building physics-informed learning architectures.

Industrial-Grade Machine Learning

Fri, 15 Mar 2019 00:00:00 +0000

From astrophysics to industrial ML at GE Digital: bridging physics-driven and data-driven models, bias, trustworthy deployment, and ML on private data across competing customers.

Month approximate.

An Autoencoding Recurrent Neural Network for Inference on Unevenly Sampled Time-Series Data

Wed, 17 Oct 2018 00:00:00 +0000

Autoencoding RNNs for classification and inference on irregularly sampled astronomical time series.

Autoencoding RNNs for Inference on Unevenly Sampled Time-Series

Tue, 11 Sep 2018 00:00:00 +0000

Presenting the recurrent autoencoder network for classifying unevenly sampled variable-star light curves (Naul, Bloom, Perez & van der Walt; Nature Astronomy 2018).

Deck-only; venue unknown, dated from the paper posting.

Autoencoding RNN for Inference on Unevenly Sampled Time-Series Data

Wed, 05 Sep 2018 00:00:00 +0000

The autoencoding-RNN approach to irregular astronomical time series, at the LBNL machine-learning-for-science workshop.

Time-Domain Inference, Data Science Tooling, & Industrial Machine Learning

Sun, 01 Jul 2018 00:00:00 +0000

Spanning time-domain astrophysics, open-source tooling, and industrial ML at GE, for the Moore DDD investigators meeting.

50 Years of Gamma-Ray Bursts

Mon, 21 May 2018 00:00:00 +0000

A half-century of gamma-ray burst science, honoring Neil Gehrels at the Goddard memorial meeting.

Machine Learning for Time-Domain Astrophysics

Mon, 26 Feb 2018 00:00:00 +0000

How statistical ML applies to astronomy in batch and streaming contexts: feature-engineered supernova identification and variable-star characterization, and autoencoder/RNN architectures that learn without hand-built features.

He also co-organized this semester-long program.

Key Quotes

“We're now entering this age of what I'll call cheap discovery, where we're gonna be getting literally 10 million transient alerts per night from LSST… But we have a very, very expensive follow-up.” – Joshua Bloom

“In astronomy, we don't have that many labels. For things that we're really interested in, we only have one exemplar of a real object that is a neutron star-neutron star merger. One.” – Joshua Bloom

“It turns the time-domain surveys like ZTF and LSST into these sort of low-resolution spectrographs. The way I think about that is: if you heard an opera singer on the other side of the door, you could guess their gender — that one's pretty easy, that's like temperature — you could guess their weight, and you could guess their age.” – Joshua Bloom

“We shouldn't want to win the lottery as astronomers. We want to guarantee a nightly annuity of essentially guaranteed payoff, and have that be optimized not only over our facility but globally.” – Joshua Bloom

“The only real testing data is data that hasn't been created yet… The only real machine learning system in astronomy that you can trust is one that's actually been put into production.” – Joshua Bloom

Transcript

[PETER] –CS math side of things. He is touching upon the second of our talks today in astrophysics here, machine learning for time-domain astrophysics, and the interesting part about this is it's really gonna answer one of the questions — it's an approach to answer one of the questions that Eric brought up in the previous talk, which is: what do you do when you have sparse, incomplete data from a survey like LSST when you're trying to sort through these millions of objects every single day? So, Josh? Thank you.

[JOSH BLOOM] Thank you, Peter, and thanks to the Simons Institute, and to the Moore and the Sloan Foundations for their support in the past and currently. I think it's worth saying at the outset that the fundamental reason why we do the work we do to innovate on the algorithmic side and computationally is to do novel science. In this case, it'll be in astrophysics, but what you're gonna hear throughout the week is a broad interest in using these tools and building new tools to do some novel work in the physical domains. We also use this because of the fire hose that you just heard about in the previous talk, just to be able to handle that onslaught and be able to make the best use of the data that's coming at us. And I think increasingly what we're seeing is that because this is a very powerful set of tools, i.e., machine learning in astronomy, it's also becoming a competitive advantage for the groups that know how to use it.

Let me start off with what is probably the most exciting discovery of the last decade, if not longer, in astrophysics, which brings together the physics community and the astrophysics community around the so-called merging neutron star events. You're gonna hear a lot about this from Alessandra and Mansi later today, so I don't want to harp on it, but use it really as a launching-off point and point of departure for saying that there are some very big prizes in understanding the universe and the connection of, say, the gravitational wave universe with the electromagnetic universe. And we're just starting to see this era start up. And so the discovery last summer

and the announcements that you heard about just over the last couple of months have really emphasized to many of us the importance of being able to do real-time inference and discovery — not just because if you don't find it, you don't get to do good science afterwards, but because there are many other groups working on this. And in fact, that's perhaps best shown with this paper here, which has 3,677 authors. There are a lot of people involved in this work, all taking different facets around what is fundamentally just a single event. You'll hear a lot more about that event and its importance and its implications later on.

But I think this event really is a great reason for us to recognize that we need to do discovery on images, as you heard about in the previous talk, and for those of you that were at the boot camp before. On the left-hand side: are these real or spurious images? That's an important question. And then we need to do inference. If you look at the graph on the right-hand side, this is what we call a light curve in astronomy parlance. It's brightness on the Y axis and time on the X axis, and the question is, when you look at this, is it worth spending time on this object, not just with the instrument that's doing the discovery but with other follow-up facilities that are potentially more sensitive or can look at it in a

different wavelength or take a spectrum to understand perhaps the chemical composition of this object? That's the question that we ask. And if it wasn't clear from Eric's talk, it's probably worth saying that if you have sufficient sensitivity, every single thing in the sky changes. It changes its color, it changes its brightness, it changes its position on the sky if you look hard enough. And so while oftentimes we focus on the things that are most fantastic — the biggest explosions that just become so obvious they're as bright as their whole host galaxy — there are lots more subtle objects that are only changing at the few percent level or less that have some very interesting facets to them that many groups wanna follow up.

What you see on the right-hand side is also a light curve that's pretty rich. You see it's irregularly sampled. You can't really see the fact that some of these data points are noisy, but oftentimes some of the most exciting science comes out when you only have one or two data points on a new object as it's being born or as it's just starting to evolve. And so a really important question that we have to wind up asking, again, in the real-time context is: if we only have a few data points, do we observe that object and continue to observe that object and keep burning resources?

The agenda in my talk today, really geared for non-astronomers to get somewhat up to speed on the state of

machine learning in the time-domain astronomy world, is to introduce to you the different ways in which we view the time domain and real-time types of inference, with the lens not just as astronomers but as people who are applying and potentially even innovating on some of the tools to get better science out. And I hope at the end to present some interesting challenges and maybe even some right places to start for the non-astronomers in the room from a collaborative perspective.

I'll start with just some of the constraints and some of the things that we think about when we're building some of these computational systems and working on some of our algorithms. I'll talk about discovery, and this has been discussed both in the previous talk and was also done in the boot camp, so I'll go over that part pretty quickly and focus a lot on inference — the real-time part of the inference: what is this object, I only have a few data points, what do I do next? And then the retrospective: not just looking at a single object, but looking at large catalogs of objects in the time domain to try to figure out how I get good science out. And then I'll end with some of those challenges and open questions.

Just to level set what I'll present here, something that doesn't look at all like the scientific method but is actually the way in which astronomers work: you do lots of planning and acquisition

of data. What you heard about in LSST, what you'll see about ZTF, is trying to figure out the ways in which you wind up observing the sky. This is what's often called the cadence: how often do you repeat a certain observation of the same part of the sky? And there's obviously a trade-off. If you don't go back to the same part of the sky, then you're gonna miss objects that are changing rapidly, but it means then you have the opportunity to survey a larger part of the sky. So there are always some biases, in a positive way, of what type of science people are trying to get at from that perspective. It's allocating telescope resources, it's deciding how to move the data around, et cetera.

The next part, which I won't spend much time on either, is in the cataloging and characterization. You heard from Eric's talk how LSST is going to be cataloging potentially interesting places on the sky and maybe even telling you a little bit about the properties of those objects on the sky. How do you extract from an image of the sky metadata like the brightness? That's a non-trivial exercise, and it needs to be done right and well, certainly at scale. Associating that data with other catalogs is important, and again, storage and retrieval of that and making that efficient and useful for communities is quite important.

Instead, what I'll focus

on is really the next part, the discovery, which is the question: is this source in my catalog real or not? And is the source even potentially interesting for me to spend time on? And what's the appropriate science that I could potentially get out of this if I decided to keep going and start asking more questions of this data? I make the distinction, by the way, between cataloging and discovery because if you put something into your database, it doesn't mean you recognize that it's interesting.

And for me, the best exemplar of this comes from a catalog from about 400 years ago from Galileo, who was observing and discovering the Galilean moons, and you can see the actual photograph of part of that journal. There's Jupiter there, there are three of the Galilean moons, and we now know on that day that Neptune was actually at the position that's listed as a fixed star. And about two weeks later, Galileo came back and noticed that that fixed star had kinda moved, but didn't really recognize that that was actually another planet. So it was about 150 years later that Neptune was actually discovered. I find this fascinating, and potentially the best well-known example in science of cataloging something really important but not recognizing its importance. So I try to make that distinction. I argue that if Galileo had found Neptune, he would've been very

famous. Okay, we've got some laughter, which is good.

What comes after discovery is inference. I now know this object is interesting, I now know it's real. What is the source? Is this like something we've seen before? Is it not? If we're right about the classification, what is it gonna look like the next time I observe? And then the last part of that is: what's the next action? This is the federation that came up in some of the discussion after the last talk. Is this source important enough to spend more resources on it? And what is it that I actually want to do with it? And more importantly perhaps, given the sociology of the astronomy community: can I convince my friends, and collaborators, and enemies to actually stop observing what they're looking at because my object is more important than your object? We still very much focus, from a real-time perspective, on individual objects, and I'll try to emphasize for you in just a little bit why that's so. And then the last part of that, of course, is if you then have that hypothesis about what you should be doing next, you go, and you plan, and you get more data. And so this cycle continues on and on.

All right, let me give you four different facets of the things that we're trying to optimize over, because in the end, if we're talking about machine learning, we're talking about

some sort of optimization. And unfortunately, this is not a loss function that I can trivially write down and then just throw whatever mechanics I have at it to get the right answer. Some of this stuff is a little bit softer, but I wanted to tell you a little bit about the ways that we think about this and are motivated by it.

Number one is something I think you all know, which is that experts don't scale. Haven't quite figured that out yet. This picture from the 1890s is of a bunch of so-called computers looking at lots of images: when astronomy had a big data problem, lots of images coming off of telescopes, it was hire and train people to just look at data. This is what I call the “just hire more grad students” syndrome, which is what many of us have been told for years, even in the time domain: that if you have more data, you just need to get more grad students to look at that data. That obviously doesn't scale in the sheer numbers, but if we're also interested in some of these real-time applications, where an image is taken and then 60 seconds later you need to take another action based on those images, people aren't that fast. And from another perspective, which is probably even the most damning, people are pretty subjective. And so if we want to do this in an objective way — doesn't mean that there won't be biases —

you want to be able to do what it is that these people are doing on the right programmatically, and do it systematically, at least where you can codify and you can version what it is that they're doing.

Another facet, which I think has been danced around a little bit when we talk about the LSST fire hose, is that we're now entering this age of what I'll call cheap discovery, where we're gonna be getting literally 10 million transient alerts per night from LSST, and maybe an order of magnitude less from ZTF starting next week. But we have a very, very expensive follow-up. To get the best and most novel science out of the changing sky, we have to use these facilities like the Hubble Space Telescope, like the James Webb Space Telescope. These are all billion-dollar-plus level facilities, and those resources, as you can imagine, are very precious, and they're massively oversubscribed. Everybody wants to be using these facilities, and it's extremely expensive for them to do that. So there's an opportunity cost, because when you do your science, it means somebody else isn't doing their science, and that's interesting from a systemic perspective. There's also the people opportunity cost as well. It costs energy and people to spend time getting these observations prepared. And then there are these interesting questions around the false positives. If I'm trying to do extremely novel science, how often should

I be using one of these billion-dollar facilities and just observing a place in the sky where good science doesn't actually come out? And oftentimes, you wind up being more conservative and requiring a guaranteed scientific payout before you actually do any speculative work on these big facilities. Another facet–

[NANCY] Discovery is getting expensive.

[JOSH BLOOM] Sorry, what's that?

[NANCY] Discovery is getting expensive, right, with LSST?

[JOSH BLOOM] Well, discovery per– or dollars per discovery would be an interesting number to look at. I'd argue that it's actually getting cheaper, if each one of those is equally interesting, you could argue. And again, because LSST wasn't designed to do time domain — it was designed to do other science in the static sky — I'd argue that's a very, very cheap use of resources, to piggyback off of the static sky science.

And then there's the kind of Rumsfeldian challenge, which is trying to optimize over the known knowns — so there's some bread-and-butter science that you could get out of doing time domain — the unknown unknowns, and then stuff in the middle, things that you have theoretical models for but you haven't yet seen, like the known unknowns. And what's interesting is that this plot is, I don't know, 10 years old or so. The curve that you see for the neutron star-neutron star mergers is approximately correct just from the theoretical perspective of what we expected to see following a neutron star-neutron star merger.

Now I'd call this more of a known known, but before, I used to herald that one as a known unknown. But there's stuff that are unknown unknowns, and by definition I can't put them on this plot. So you see the timescales over which big explosive events wind up proceeding. There are some other really interesting things that wind up happening even on shorter timescales than are shown here, and I'll emphasize those a little bit later on.

Another aspect of this Rumsfeldian challenge is the small number of labels. In a huge amount of the machine learning literature, there is this implicit assumption that I have lots and lots of data, at least from a supervised perspective, to be able to build these models. If you want to build a good image classifier, just get more images and get a whole bunch of people to label them, like at the Flickr level, or do it implicitly through Google searches and things like that. In astronomy, we don't have that many labels. For things that we're really interested in, we only have one exemplar of a real object that is a neutron star-neutron star merger. One. We've got theoretical models of a whole bunch of those with different inputs, but for some of these we really only have dozens or maybe hundreds at best, or maybe thousands as we're entering into this LSST era. So we're really,

even though it's a big data fire hose, from a labeling perspective it's still very much a small data problem.

And the last is that it's all well and good to have some plots that show how you have a better false positive versus false negative curve than somebody else in retrospect. These systems that involve machine learning that are gonna be put into place for astronomy have to be done with robust systems that actually work in real time. And oftentimes one of the biggest challenges we have is not on the algorithmic side; it's not on being able to produce those plots in retrospect. It's being able to stand up robust systems that work at scale that can ingest this fire hose and produce results that people can wind up accepting and using and then moving on into that value chain of inference. And for those that haven't read this yet, this is one of my favorite papers in machine learning. It has no equations in it. It just highlights all the different bugaboos of what it means to build a real, robust, functioning system that involves machine learning, and how different that is even from just best software engineering practices. So forget about the fact that astronomers are not trained as software engineers — we're certainly not trained in general as being able to build and stand up and maintain and innovate these very large-scale systems.

So that's some of the constraints and some

of the concerns that we have as we start thinking about bringing machine learning into our world.

I want to now touch a little bit on discovery — that facet of, “I've got something in my database, is this interesting? Is it real? And what should I do next with it?” You've already seen real-bogus mentioned in the previous talk. You see the bad subtractions on the top as we're trying to find new objects, and the real, good subtractions on the bottom. It's a little bit like Anna Karenina, where all the bogus detections are all different than each other, and all the real ones are all similar to each other. But unfortunately, state of the art puts us at about a thousand to one, or maybe we're getting down to a few hundred to one, of this needle in a haystack. That is, for every real object in a real image that we wind up taking of the sky, there are hundreds of bogus detections. So we have to find these real ones in the face of quite a large number of bogus detections.

And what's nice is that if we can build a real-time framework to identify what's real and what's not, we get some really nice things out of that. It will be fast, because it's just algorithmic implementation. Embarrassingly parallel, because every single object we can ask that question. Transparent, in the sense that we can know why we got the answer we got. So if

you're doing it in a random forest context, you can literally follow down the different trees and figure out why you got that answer. It's deterministic, so given the same data, you get the same answer, unlike with people. And it's versionable, so I can keep on upgrading, and if I need to go back and reproduce what I had before, I at least have a fighting chance at that.

What we found in some of the early work on building a real-bogus detector for real systems was that it was a very hard problem. It was based on features that we derived out of the images and the subtraction images — we got of order 75 of those. First of all, that's a bit of a computational challenge, ‘cause some of those features are expensive to construct. But then we also wound up realizing that there are only really a few algorithms that actually did well on the same input feature space. And in particular, we found that random forest was doing extremely well relative to all the others given the same feature set. And so we wound up implementing this and sticking this into a real pipeline that was run up at LBL as part of PTF, and what happened is every time a new transient or a new object was cataloged, it would wind up getting scored in a real-bogus sense from zero to one, and then, depending upon where you make your cut of what you call real and what you call not real, you wind up

pushing this into downstream systems to be followed up and potentially even looked at by people.

One of the things that those who were at the boot camp saw, and the astronomers already know about this, is this great discovery that was done by Peter, which I'll call ML-assisted in the sense that a real-bogus classifier was applied to this incoming data stream on new images that were coming off of PTF. About 11 hours after explosion, PTF wound up observing and cataloging, and then Peter wound up noting that one of the highest-ranked real-bogus objects from the catalog was actually a very young supernova that turned out to be the nearest Type Ia supernova in more than three decades. So only a few people in the history of mankind have observed supernovae this close by and this early and have been the discoverer, so I was very happy to see that Peter changed his business card just so everyone knew how important that was. But the thumb–

[PETER] Is covering a student to be determined. Yeah.

[JOSH BLOOM] Exactly. There's a student under that fingernail there. But what's more important — and this supernova, by the way, would wind up rising in brightness to the point where if you had binoculars you could have observed it through those binoculars, which is just absolutely stunning. So it was eventually found and could have been seen by astronomers worldwide, but because it was found so

early, we were able to do, as a community, a lot of really interesting science that we hadn't been able to do in the decades before that: get spectra, get more observations of it. And a paper that I worked on with Peter and a few others in the room was ruling out possible progenitors, the things that make those supernovae. So everything in green and all the other colors were excluded because of the lack of detections of certain characteristic signatures that we could have seen, and that left behind essentially only compact objects as the only viable candidates. It doesn't matter that you know all the different plots and what they all mean up here — just to point out, though, that because we were able to get on so early to this object and recognize its importance, we were able to do some novel science with that.

This idea of building real-bogus has really flourished, and as Danny talked about — and this is one of his slides from his boot camp discussion — this was used in the Dark Energy Camera survey, particularly around finding supernovae. And so there, they wind up building another real-bogus detector, and they set a different threshold between zero and one, and depending upon whether you're above or below that threshold, you wound up getting what is essentially world-class false positive, false negative rates. So getting down to MDR — means misdetection rate — meaning that

if you set your threshold criteria at 0.5, around 4% of the new candidates that you wind up having in your catalog you wind up not identifying as real. But that means that your false positive rate is also extremely low. So you see where this trade-off is. Only about 2% you would say are real when they're actually not. Now, the nice thing is if you keep on coming back to the same part of the sky, you wind up seeing the evolution of this real-bogus score, and objects that are getting brighter in the sky generally will wind up getting a more favorable real-bogus score over time.

So this has now become a cottage industry, and essentially every time-domain survey that is looking at images has their own flavor and their own approach and their own training data for being able to identify and do discovery effectively in real time. I'd say while there's probably still a little bit of blood to squeeze out of the stone, if you look at this plot on the right-hand side, we can't get much better than this. So maybe we get down a factor of two or something like that in misdetection rate. But just to emphasize that this isn't just a theoretical exercise on existing old data: these are being put into practice in real-time systems.

Let me talk a little bit more about real-time inference. And by the way,

please stop me if you have any questions. I'm happy to take them in real time. Yeah? I was only joking — I am not taking any questions.

[AUDIENCE MEMBER] If it's a 2% false positive rate, that still means like 30% of the things that you're saying are real are in fact bogus? ‘Cause you said there were 100 times more bogus elements?

[JOSH BLOOM] Yeah. If you do the math, you're actually saying that — and this is why you need to pull this number way, way down — oftentimes most people won't start observing and doing follow-up until there are multiple real detections in the same place of the sky. Or you're willing, if it's a nearby galaxy, to ask from a query perspective: is there something which is really new, which has just only got one detection, it looks like it's real, and it's near one of those galaxies? In that case, you might be willing to put your neck out and actually start observing with other telescope facilities. The other thing you can do is — that's just where the threshold is, and in principle, that's where the gray area is. If you want to be extremely confident in what you actually call real, even with a single detection, you can just require your tau in this case to be very, very close to one. And so that's really a measure of probability. I don't know how well calibrated that is for the DES survey —

if it's at 0.5, whether it's really 50/50 real or not. But that's one way you can gain confidence and cut down the number of follow-ups after that. Yeah.

[AUDIENCE MEMBER] This is just photometrically based? Or–

[JOSH BLOOM] This is just photometrically based as in– sorry, what do you mean by that?

[AUDIENCE MEMBER] There's no spectra. There's no spectra.

[JOSH BLOOM] No spectra yet, right. Correct. This would be the launching-off point to get spectra. These are just based on the images and the preceding images for that part of the sky.

[AUDIENCE MEMBER] Gotcha. Yeah. Quick background: spectroscopic information is expensive.

[JOSH BLOOM] Thank you for saying that. Spectroscopic information is extremely expensive. Typically, that's only done on one object or only a handful of objects, and you typically need large telescopes to do it, and it has to be a pointed observation.

[AUDIENCE MEMBER] Yes. I guess it's a more general question, so we can take it at the end if it's too much of a generic thing. I'm curious, in general for astrophysics, what is the real cost of discarding the data? If you are sticking in some — I'm not saying black box, but something where you might not completely understand how the process works — and you throw out some real signals, how bad is this for discovery?

[JOSH BLOOM] No, it's a fine thing to talk about. It's a great question. For those that didn't hear it: how

bad is it to throw out real signals that you just misclassified, effectively? Unlike in particle physics, where we really can't save all the data as it's coming off of an accelerator, and you have to make choices essentially in very real time about whether I save this event or I don't save this event, astronomers at least are in a mode, even in the LSST era, where every photon is sacred, and we save everything. Now, it goes into a database, but in principle, if that part of the sky becomes interesting eventually, we then have certainly a fossil record from our databases of what we had said before and what was there before. So we're not really throwing that out. There are interesting questions about accessibility to those databases, and whether that's really kind of a dev null in all practical terms, or whether that's something that people can actually mine. I would generally argue that most people are not mining previous surveys and their data effectively. But when you do, you often get lots and lots of prizes about what was happening in that part of the sky before you observed there.

[AUDIENCE MEMBER] So when you have something like this, you're not discarding the events?

[JOSH BLOOM] You're not discarding them, but again, it's sort of a decision process of: I've got this huge fire hose, I've gotta somehow make that into a little nice stream of things that

I can handle, and I'm not gonna throw out all the water that's in the fire hose that I'm not dealing with. I'm only gonna deal with the things that I really like. But because we're often interested in the changing sky on rapid timescales, and the most interesting observations that we can take are ones that come immediately after discovery, it doesn't help me much if I go back into my catalog a year later and say, “Oh, there was a really cool thing that was happening.” Too bad, because most of the objects — if you go back to my light curve plot of the known unknowns and the known knowns — most of those things eventually disappear. And so if you miss it, it's pretty much gone.

[AUDIENCE MEMBER] Yeah, thanks.

[JOSH BLOOM] From a real-time inference perspective, just quickly, what we were able to do is put a very rudimentary classifier in the next step, which is: this is an interesting source — what is that potential source? We had three high-level types. Is it a variable star of some sort? Is it a transient, i.e., something that's changing explosively? Or is it a rock in our own solar system, i.e., an asteroid or so? And then there are subclasses beyond that. We didn't do a great job of this — and this is a confusion matrix, for those that know about that. You'd like to see all your power up to one on the diagonal, and all the off-diagonal power is

misclassifications. We didn't do a great job with this, but we did show that you could use external databases effectively in real time to get some notion of what this object was, which gave a little bit of indication for those working on that survey that this would be worth looking at. And again, a big part of what we try to do is not just do this on paper, but do it in practice, and so we got to the point where this classifier was acting as a person and having conversations in the same effective chatroom with other people. This is what we called the PTF Robot, and in this case it was saying, “I think this is a supernova or a nova or some explosive transient of some sort.” And this turned out to be a very interesting object which led to a paper in Nature, where actually we had missed it, but there was some burbling before this actual explosive event, which was a really important find for a supernova — to be able to see some pre-supernova activity. So again, we got to the point, I wouldn't say where we had great results or it was very accurate, but we at least spent some time in trying to make sure that this got into production.

This isn't just in the optical domain. We've done similar things of being able to do real-time inference in the gamma-ray sky on objects that I've spent a

bunch of time on, these things called gamma-ray bursts. These are short-lived blasts of high-energy light, gamma rays and X-rays, that come from a random place in the sky. What you see on the left-hand side is a depiction of the static gamma-ray sky. Essentially this is the entire sky if you looked out with gamma-ray eyes, and then you see a burst that lasts for five seconds or so, and it briefly swamps the entire universe in gamma rays. So these are very, very bright events, difficult to localize on the sky, but they don't just produce gamma rays. They also wind up producing optical light after that, and these are the brightest optical sources in the universe. Much brighter than quasars. Much brighter than pretty much any type of supernova you could ever imagine. You see the relative brightness on the Y axis here in a log scale in power, so some of the brightest gamma-ray bursts — afterglows, as they're called — just swamp everything else as well in the optical sky.

And what's great about that is we can see these events to the edge of the observable universe, and if we can recognize that we have an object that can be observed at the edge of the observable universe with large telescopes, we can get spectra, for instance, of these afterglows, and they act as sort of lighthouses, and we can learn a lot about the very early universe using those events as probes, if only

we observe them in the first few hours after they've happened.

[AUDIENCE MEMBER] Mechanism?

[JOSH BLOOM] The mechanism — the progenitors that lead to these things, the ones that are very bright, are basically very large massive stars that wind up collapsing, probably into a black hole. Think of these as beyond supernovae, which wind up collapsing into other types of objects. And then the actual physics of what produces this light is basically relativistic shocks that are emitted, that are moving very, very close to the speed of light and have a tremendous amount of energy, and that energy — the kinetic energy — is released as these shocks wind up happening.

So what you'd like to do is be able to take the information not from ground-based observatories, which have a hard time observing gamma-ray light, but from the satellites that wind up discovering these things, and infer whether the objects you have are at high redshifts — so very, very high distance, very far away objects. This is also a hard problem, and what we wound up looking at is whether we could take — and you can probably see right here, there are only a few objects in red that are these very high-redshift objects that we'd like to follow up with our big facilities. And the ones in black are the ones whose redshifts we know are not very high, and the ones in gray are ones we don't know the answers to. And there are just not

that many of those, so the question is: could we take the immediately available data and make a prediction of whether something is high redshift or not? And the answer is really no. As you can probably see — it might be a little bit difficult — there's almost no way to pick the red points out of the other points, but now we have multiple features that we can try to use to try to improve it. And what we did is we said, “Well, if we can't say yes or no, this thing is definitely at a high redshift or not, can we rank a new object to say, if I have an X amount of observing time to follow up, is it worth doing or not?” — which is a slightly harder question to ask. But we wound up finding that if we only follow up 20% of the ones that we rank-order and say are the highest redshift, then 60% of those will wind up being at high redshift. So this is a big impurity, big precision-recall problem, but the area under that curve relative to random is actually non-zero. So this is actually a very useful tool that we were able to build and deploy, and we did it with only 600 events — and talk about small numbers of labels, we only had 17 objects that we could train on. So getting guarantees, or at least trying to convince ourselves that we weren't overfitting

on the data was where we spent most of our time.

Okay, let me transition now to large-scale aggregate inference — not on individual objects necessarily in real time, but in the time domain over large catalogs of sources. Yeah?

[AUDIENCE MEMBER] Was that validated with real follow-up or with synthetic retrospective?

[JOSH BLOOM] We, for a year or two, were publishing whether we think something is high redshift or not. We had a hard time actually just determining whether we were right with this calibrated curve, but certainly there were a few that we thought were high redshift which weren't, and it was the other way around. But this wasn't actually a very well-used tool in the community, to be honest, and we wound up shutting it down. So this is an example of something that worked really well on paper but potentially not very well in practice.

[NANCY] Let me ask the same question on what data's going into the–

[JOSH BLOOM] This is data that's just available, for those that know about it, from the Swift satellite. When one of these satellites winds up discovering a gamma-ray burst, they send down a whole telemetry of stuff like where it is in the sky, how bright it was, how long it lasted, and some other information about the rudimentary spectrum. I think there are eight or 10 features that went into this.

[NANCY] The host properties?

[JOSH BLOOM] No. No, no. This is literally just the first packet that

comes down from Swift — what can we say about it without knowing anything about what's detected in UVOT or even XRT?

[AUDIENCE MEMBER] Okay.

[JOSH BLOOM] This is a picture of the variable sky as viewed by the Southern Hemisphere, at least, and you notice the big hole up in the top left there. Each one of these points looks a lot like that light curve in the top left, which is basically time on the X axis and brightness on the Y axis. And the question, again, is: is this object, which looks kind of nasty and looks sort of random, worth spending time on from a follow-up perspective? There was a survey that was published a number of years ago that had 50,000 variable stars of the sky, and there were only about 800 of those that had bona fide classifications across about 25, 26 different classes of variable stars, and we asked the question: could we use that data to get good classifications that were probabilistically determined from that survey?

So we did what we generally will do when we do a machine learning approach, the old-school thing: we make this into a supervised problem. We take that very ratty time series data, turn it into a supervised classification problem where we just do features, and so we produced about 75 features from that, using unordered statistics like variability metrics, ordered statistics and doing periodograms, et cetera, and then even context metrics:

where is this on the sky, what color is it, et cetera. And with that, we were able to produce what is and was the best in class, where we wound up being able to very reliably, and with calibrated probabilities, determine over these many classes what a source was. So if you gave us a source that wasn't part of our training data and you asked us to label it, we were able to show what its class was, and then we were able to do some follow-up observations of that source to actually prove, for a minority subset of those, that the classifications were actually correct.

And we produced what I think is one of the first probabilistic catalogs of variable stars, which we made accessible to the world, where we let people traverse through the taxonomy of variable stars, click on things, and then order them by the different probabilities of whether they belong to that class or not. You can see that blue curve there is the probability, and then on the right-hand side, you wind up seeing essentially the probability that these belong to different classes. So having probabilistic– oh, then we made it social so Facebook would buy us. That didn't work either. And so what we were able to do is produce this catalog.

And what we've done since then is work on a system which we call Cesium, which allows not just us but many other people

to build their own survey classifiers over large amounts of time series data. And we did this around astronomy, but we try to make this in a domain-agnostic way so that you could actually use this on any sort of domain that had time series data. This is still a work in progress, but we're now actually starting to be able to use this for new surveys as they come online. And rather than building one-off purpose-built infrastructure, we're getting to build a whole bunch and make use of a whole bunch of subsystems in these architectures that use some of the modern software practices.

Now again, one of the things I wanna emphasize is that building probabilistic catalogs and making websites and stuff is cool, but our main focus of doing this is to be able to do novel science. And so what we try to do in my group was take that probabilistic catalog and then ask interesting questions of that and then do follow-ups. For instance, what we were able to do is look for very strange types of objects called R Cor Bor or DY Per stars, and cutting to the chase, with follow-up observations with spectra of only about 20 of our candidates, eight of them wound up being new discoveries of these very, very rare stars in a catalog that had been around for 10 years. One of the stars was almost as bright as you could see with the naked eye

and was probably known by the Babylonians. And here we were able to do this using machine learning to help us hone in on the probable objects. Very, very low purity of the sample, but very high efficiency of discovery. Yeah?

[AUDIENCE MEMBER] Going back to the catalogs and surveys, are there any sources of bias, aside from Southern Hemispheric bias?

[JOSH BLOOM] That's a great question, and something we try to grapple with. There is a bias in the sense that it was taken by a single telescope in a certain filter. And it was taken with a certain cadence, so we're certainly biased in that survey against finding objects that change very rapidly and go away. That's one bias, so you don't see any of those in the sample. They spent a lot of time in the galactic plane, and there's certainly a distribution of variable star types that is different in the galactic plane than off the galactic plane, and we're trying to figure out how you could disentangle almost a prior of what you would expect that distribution over those 26 classes to be, so that instead you could wind up producing some sort of posterior where people could then dial in whatever their biases were and get different probabilities out. Yeah.

[AUDIENCE MEMBER] I think I saw the Magellanic Clouds there too.

[JOSH BLOOM] Yeah, the Magellanic Clouds are there, and we don't have those in the Northern

Hemisphere, et cetera. So yes, certainly there are biases in that, which is why there is some difficulty, which I'll highlight at the end, of taking that entire classification machinery in that model exactly and then applying it to another survey. Another survey taken of the same part of the sky would discover, and would have in its catalog, a different distribution of variable stars.

The other thing we did is find highly eccentric detached eclipsing binaries, which allowed us, with, again, follow-up observations with high-resolution spectroscopy of the stars that we found, to put them on the mass-radius relation of stars, which is a pretty fundamental thing that you like to know. It's actually hard to find stars where you can actually do this, but we were able to find a number of those. And again, we used that catalog as a launching-off point to do novel science. So those became science papers in and of themselves.

And the last thing that we've done in this field is something that's, I think, still pretty bizarre and I'm still coming to terms with: this is the idea of looking at a light curve like the ones I've shown you and trying to infer the fundamental properties of those objects that you would only get traditionally from spectra. What is the temperature of that object? What's its surface gravity? What's its metallicity? And what we were able to show is that by just looking at the variability in time over a couple of different

colors of variable stars in a certain part of the sky, we were able to infer the three properties I just mentioned with as much accuracy as you would get from a low-resolution spectrum. So it turns the time-domain surveys like ZTF and LSST into these sort of low-resolution spectrographs. The way I think about that is: if you heard an opera singer on the other side of the door, you could guess their gender — that one's pretty easy, that's like temperature — you could guess their weight, and you could guess their age, and be accurate to about as well as you could do if you were actually able to then go and measure them directly. So this is very interesting. Like the gamma-ray burst project, this is only something that's worked so far in retrospect. We've yet to then apply this to new surveys, but this is one of the things that we're hoping to do as some of these new surveys come online.

Everything that I've mentioned so far has been using what I'll call handcrafted features. But there are some real challenges with that. Feature engineering, for those that know about that, is very expensive, and oftentimes requires a lot of domain knowledge. If you are trying to separate two very similar types of classes of variable stars, oftentimes you'll bring in the expert in that domain or subdomain or sub-subdomain, and try to build some math that encodes what their brain is telling them of

why these two things are different. That's an expensive and iterative process. It's also a small data problem where we don't have a lot to train on, and oftentimes the traditional machine learning techniques don't account for feature uncertainty, and it's often very difficult to apply one model to another survey.

One of the things that I'm excited to talk about today is where we threw out traditional feature engineering and we used an autoencoding recurrent neural net to create those features for us in an unsupervised way. That is where we don't need a whole bunch of labels. And for those that don't know about autoencoders, the idea is actually pretty simple. You create an encoding function, which is demonstrated with this E here, of that light curve, essentially that raw data — and then, it's actually probably pretty hard to see on the screen, you compress that down to a bottleneck, which is a small number of floating point numbers, like 64 numbers, and then you take that encoded small number of data points and you decode it so that you try to reproduce your original light curve. So without knowing what this object is, if I get the architecture right, I can build this encoder-bottleneck-decoder thing, and then use the bottleneck as features in a traditional classifier.

And this encoding process actually works pretty well. These are some examples. The raw data is up at the top for two different sources, and

for now just focus on the red curve. What you can see, if we fold it on the correct period of this source, you wind up seeing that the red points very well match the blue data. So the encoding and the decoding, even though it's a very lossy process, wind up producing some pretty nice-looking light curves. And what's cool about that is something we were able to demonstrate in the paper, where if we just had sinusoids — very noisy, irregularly sampled sinusoids — and we actually looked at those encoding features — essentially it's like a small N-dimensional embedding — we're able to show that those features correlate very closely with the things that we put in. So we get period out, we get phase out, we get amplitude out of these things.

And so what does that mean? It means that this network is learning, in this case here, what it means to be a periodic source, and it learns what it means to have a period of a certain sort, because we're taking what is hundreds of data points and compressing it effectively down to four. And in the context of astronomy, what we're doing there is we're not saying we know this is gonna have this period, and it's gotta have this amplitude, it's gotta have this skewness and kurtosis. It's saying: just learn the features that get me back to my original source. So cutting to the chase here, we were able to show that we rivaled the best-in-

class results from all the handcrafted feature engineering. In two of the three surveys we looked at, we beat all the other best-in-class sources and models. So we're very excited about this, and what it means for us is that instead of having to build new features for every new survey we look at, we can actually just throw it into this machinery and use that bottleneck layer as a way for us to build features.

For those ML folks in the room and online, here's the architecture we used. I think one of the important things to point out is that unlike other recurrent neural net architectures, we're able to make explicit use of the delta times between the observations. And we're also, in our loss function, accounting for the inherent uncertainties in the observations. So if you have one observation which is kinda ratty and doesn't have a very good measurement — actually has a very large error — it won't adversely affect your reconstruction. And what's also nice about this is we're able to augment our data not by, as people do in the image domain, moving images around and shifting them and changing pixels here and there. We're able to take the original light curve data itself and essentially bootstrap, resample that light curve, and that actually wound up helping the learning quite a lot. And I think the thing I'm most excited about: it means that we can do unsupervised feature learning. So instead of having to learn features and

build features by hand, we can leverage large corpuses of unlabeled light curves to build up these features. And because that bottleneck just becomes these abstract features, we can then use those and augment them with other sorts of metadata like colors, et cetera. I'll skip that for now.

Let me just, in the last few minutes, highlight some of the challenges and open questions that we have. One of the things I'd love to spend time on with the folks in the room is trying to understand how we can find new phenomena — look at a retrospective catalog and say, “Are there any clusters in some space that are different than the types of objects we already know about?” And I'd also like to, in a real-time mode as a new object is just starting to develop, not just identify this is worth following up, but identify: is this worth following up, and it's actually different than anything we've ever seen before, or different enough that it's worth spending even more resources on to learn more?

Another thing that I think is interesting is this whole small data, or at least small label corpus, problem. And the question there is how we can leverage one model built on one survey and apply it to another survey — that's transfer learning. Can we even do better than this sort of 4% misdetection rate? And there, teams like at Harvard — Pavlos

Protopapas and his group — are looking at some unsupervised and semi-supervised techniques.

What's interesting is that if you think about the architecture that we created for the classification problem, this is a very non-linear, non-parametric model that can produce realistic light curves of real objects without me having to put the physics in. So can I use this going the other way, and try to infer back the physics for some of these objects whose physics we actually know? Can I use these as surrogates or emulators? For instance, let's say in the gravitational wave world, it's extremely expensive to generate a gravitational wave signature with a few input parameters computationally, so you have these coarse grids. Can you fill in those coarse grids with similar types of networks to produce these emulators? And can we use these neural net models to be able to help the LSST, for instance, figure out what their cadences should be? If I can now produce a fake universe of transients of all different sorts, you can then actually create a cadence and optimize on a cadence to be able to find those sources.

And then systematic or systemic optimization is probably beyond the reach of this group or anyone, which is: optimize over all global resources. I'd love to be able to maximize the scientific output, but again, we're in this competing mode where different groups are all trying to get access to similar telescopes. And last, does this all

have to be fully automated, or is there still a role for people in the real-time loop of the real-time discovery? Certainly people eventually will be the ones that write the papers, at least for now, and if that doesn't happen, that'll be pretty exciting as well. But is there a role for people to be asked questions from the model and say, “I think it's one of these two, can you do a little bit of exploratory work? Give me the answer,” and then the model could wind up updating itself? I'm generally curious about that.

Let me end with a pretty exciting discovery that was announced just a couple of days ago, where an astronomer in Argentina was just turning his telescope on for the first time — and the first thing you do is you look at a beautiful galaxy — and he happened to catch a supernova that was going off right in the earliest stages, the earliest stages pretty much we've ever seen. These are the observations from that amateur astronomer. And then eventually observations were taken later, as in like a day later, and it took a while for the amateur astronomer to figure out that there was this new source, but this was heralded as like winning the cosmic lottery. You turn on your facility, you look at something, you find something that shows up in Nature a year

later, and people are very excited about this, rightfully so. This is the earliest parts of the evolution of a supernova that we very, very rarely get access to. But what I hope you've seen in the talk today is that we shouldn't want to win the lottery as astronomers. We want to guarantee a nightly annuity of essentially guaranteed payoff, and have that be optimized not only over our facility but globally. And so I think that's an important point: we're trying to systematize discovery and inference, and it's very clear that machine learning is becoming a very, very critical part of that whole process. So I'll stop there, and I think we've got time for just a few questions. (applause)

[PETER] So was that the GE in you in the last slide there? Oh no, that's the one before, about the annuity end.

[JOSH BLOOM] Oh, the annuity. Yeah, well, guaranteed payoff is much nicer than every now and then if you're lucky.

[PETER] I wanted to kick off with a question for you. Did you learn anything interesting with the things that you misclassified? Was it something missing in your model, or was the actual object that you were looking at not exactly — like, the label on it wasn't exactly right?

[JOSH BLOOM] Yeah, that's a good question. In the construction of a model, or even more broadly, a system of software that involves applying a model to data and building that model, you often wind up holding

out some of the sources that you think you know the answer to, and you apply the model to that, and then you measure how well you think you're doing. Oftentimes you're extremely confident in the model that this source is, let's say, of type RR Lyrae or a supernova, but the label says something else. That's either your model's wrong, which is often the case, or sometimes it's a cause to go back and say, “Was my label actually right?” And so it actually can be a bit of an iterative process for the missed labels during this model construction process, where you go and you say, “Did I actually label that right or not?” And if you labeled it right, then your model was wrong, and the question is: what can you do in your model so that you don't get that wrong again? And what you do there is potentially go back to the drawing board and say, “What are the other features that I'm missing?”

One of the negatives of the recurrent neural net autoencoder that I presented is that it becomes much more of a black box. You don't know what those features are, so do you add more features, do you make less? One of the benefits of the handcoded features: if you get something wrong, you can say, “Oh, I got it wrong because I forgot to take into account that some of these types of objects look like this at the beginning, even though most of

the ones in my training set looked like something else. So I'll just build an extra feature that fits for that sort of thing.” I think that's a long answer to your short question. It's an iterative process. You either learn something about how you're doing modeling ineffectively, or you could learn something fundamental about your data.

Now, the real challenge is if you're doing all this in an offline mode, because you've iteratively rebuilt your model even on held-out testing data, you haven't really held out your testing data as much as you need to to gain a real good confidence of what your ultimate uncertainties and errors are gonna wind up being once you put this into a live mode. What I generally say is that the only real testing data is data that hasn't been created yet — i.e., the universe hasn't evolved in the next second to produce those objects. So it's all very well and nice to have “here's my false positive, false negative curve” on paper. The only real machine learning system in astronomy that you can trust is one that's actually been put into production — is my contention.

[AUDIENCE MEMBER] You mentioned optimization, global optimization, of follow-up. We just, as the community, just witnessed the largest follow-up campaign ever conducted for a source. Is there any evidence, in your opinion, that it was suboptimal in some form?

[JOSH BLOOM] I think that's a great question. We just saw, for those that

didn't hear it, essentially the entire world of astronomers going after one place in the sky. It was massively suboptimal. People were observing the same part of the sky with the same sort of filter at the same time. Now, the nice part about that is that we're really, really sure that the observations at that point in time and that filter were right, because it wasn't one person saying it, it was 10 persons saying it. So there is something nice about that, but there was very little coordination, I'd say, and the next time there should be more.

Now what happens is, because these precious follow-up resources like large telescopes are indeed precious, what you wind up seeing is that the people that lord over those, like the directors of them, wind up playing a bit of referee and saying, “Well, you guys shouldn't do it,” or oftentimes more of a matchmaker and saying, “You're all asking me for the same observation; you are now part of the same group.” And so it's a very ad hoc, very real-time process of how these collaborations wind up evolving. It's hard to see how we break out of that model. Individual telescopes and telescope consortia are doing a pretty good job of that, recognizing they need to optimize, but globally I think this is largely an intractable problem — not technologically, but more sociologically and politically. Nancy, do you have a question?

[NANCY] Yeah. I just wanna say I really enjoyed your talk, Josh.

The question I had was about your encoder. You showed a slide about different variable stars. I was trying to read it quickly, but the three examples I think were all periodic. Not this one — I think the one after, where you have a table where you compare more traditional methods to this fancier method–

[JOSH BLOOM] Oh, yeah. Yeah, this one.

[NANCY] So is this– I mean, I was curious how well this works for variable stars that are not periodic, and where, even if the final answer's not 90-something percent, the traditional classifiers really do a bad job, so there could be more dramatic improvement — whether there's a–

[JOSH BLOOM] It's

a great question. We focused on periodic variables ‘cause there was the largest corpus of label data that we could find on those. We are asking that exact question on not just variable stars, but explosive sources. There's a new challenge coming out called the PLASTIC challenge, which I'm sure you know about, which is for supernovae and transients so n– particularly non-variable stars of being able to classify those. So there's a group that's gonna be producing fake versions of all these different types of objects, and they're asking the rest of the community, “Can you build a good classifier on that?” I think the sort of network that we built can be very applicable to those, but I, we haven't done the work yet to see how good

those are relative to the other classifiers. [NANCY] ‘Cause that's where you don't know a priori what the features should be, right?

[JOSH BLOOM] That's right. Unfortunately, a lot of the very aperiodic stars are ones that just have stochastic variations, and so other than fitting power spectrum density distributions of what the variability is, there's not a whole lot of other features that you can build. Certainly a periodogram doesn't really make sense in that context. But in the context of burbling quasars, we have built specific purpose-built features knowing that quasar light curves behave like damped random walks, and so you can get parameter fits out of those.

[PETER] We have one question

from the Twitter universe.

[JOSH BLOOM] Twitter universe.

[PETER] Eric Bellm, of all people, has a question for you. Is it practical or useful to apply a fine-tuning layer atop your pretrained autoencoding neural net to apply it to new surveys, à la applications using ImageNet?

[JOSH BLOOM] It's a great question. That's our supposition. I think we mentioned that a bit in the paper, but we haven't done the exercise to take a pretrained network on a large survey and just say, “This network has figured out something fundamental about how variable stars work, but it doesn't know a lot about the way in which this particular data was taken in this other survey. Let's apply it.” We have taken the network we built on one

survey, applied it directly to the other, and it worked really well, just like when you apply ImageNet or VGG-Net to images that aren't from that original corpus, you get very good answers. But you're right, what you'd like to be able to do is freeze some of the layers of the model and then retrain some of the other layers so that it learns some of the peculiarities of that survey. That was a good question.

[PETER] All right.

[AUDIENCE MEMBER] Go ahead.

[JOSH BLOOM] He's had his hand up for a while. Yeah.

[AUDIENCE MEMBER] So my question follows up from this one about the follow-up for use of telescopes’ resources. That sounds an awful lot like

explore versus exploit in behavioral ecology, which game theorists talk about quite a bit. Have you looked there for inspiration?

[JOSH BLOOM] Certainly the notion of explore/exploit is top of mind in the groups that I've worked in, where you say, “I can either go after, let's say, Type Ia supernovae, which if I get enough of those, it's a guaranteed payout on some timescale.” Exploit as in, “I have a little moment in my telescope follow-up resources where I don't have anything scheduled. Why don't I go after something where I'm not sure what the answer is?” What hasn't been done is where there's an explicit discussion and identification of what the explore/exploit metrics are, let alone the adherence to those metrics, because oftentimes when you're on a

telescope and somebody hands you a basket of a hundred objects that you could look at, you go, “Yeah, I don't like that one. That one's in that part of the sky. I don't like that part of the sky right now,” or, “There's a cloud over there.” So we haven't done a really good job as a community, and I don't know any groups that are being extremely explicit about, “We are going to explore this amount,” as in potentially waste a whole lot of resources, “and exploit this amount.” Maybe take the rest offline?

[PETER] Yeah. Okay, so let's take our lunch break now, and then we'll come back at 1:30. (applause)

Industrial Machine Learning

Wed, 06 Dec 2017 00:00:00 +0000

Keynote arguing that the real machine-learning revolution — in improved and saved lives — will come when ML automation is coupled with industrial data.

Industrial Machine Learning

Tue, 15 Aug 2017 00:00:00 +0000

Applied Data Science invited talk (as VP of Data & Analytics, GE Digital) on deploying machine learning against industrial data at scale.

Day within Aug 13-17 approximate.

Marrying Physics-Based and Data-Driven ML Models

Mon, 14 Aug 2017 00:00:00 +0000

Follow-up interview after the Wise.io acquisition: Industrial AI at GE Digital, autoencoders for building training datasets, and combining physics-based models with data-driven ML.

Key Quotes

“One of the distinctions I make between the consumer Internet of Things and the industrial Internet of Things is that when your Fitbit breaks you call up customer support and you sort of complain, but when your jet engine has a problem that can have some serious consequences. And so the stakes are a lot higher.” – Joshua Bloom

“You're in this very interesting dance where it's lots of data, yet in some sense it's a small data problem because you only get the sort of bad or rare anomaly events every now and then. And even when you get those, they're all sort of Anna Karenina, like they're all unhappy families, they're all different in their own way.” – Joshua Bloom

“I think of it as physics models only getting us 90% of the way there to a great answer, and then adding a sort of data-driven layer on top of that is the path that we're seeking.” – Joshua Bloom

“Your life as a person depends upon the industrial Internet of Things, and it's a great time to be part of that, and there's a massive amount of work to be done.” – Joshua Bloom

Transcript

[Music] Hello and welcome to another episode of TWiML Talk, the podcast where I interview interesting people doing interesting things in machine learning and artificial intelligence. I'm your host Sam Charrington. If you get my newsletter you already know this, but last week we hit a major milestone for the podcast. I'm excited to share that thanks to you, we've served up over 500,000 plays of this show. We're super grateful to everyone who's ever listening to the show, send us feedback, or engage with us via the site or social media. Thanks also to all of our guests, especially those who started out as listeners and later became guests like Evan Wright and Sarah Guo. We've come a long way in a short amount of time and we couldn't have done it without you.

Next up, we're ready to announce the first winner of our AI conference giveaway. Drum roll please. Congratulations to Shinu from New Haven, Connecticut. Shinu was one of only 5 people to complete every possible method of entry and clearly it paid off. Shinu, we look forward to seeing you in San Francisco. We're still working to finalize our second winner, so stay tuned to our Twitter feed at @twimlai for updates. If you didn't win the contest but still want to join us at the artificial intelligence conference in San Francisco, head over to twimlai.com/goaisf and enter code PCTWIML for 20% off the cost of most packages.

Next, we're less than a week away from the first ever meeting of our online paper reading group, the TWiML online Meetup. Our first discussion will be on the recent paper from Apple, “Learning from Simulated and Unsupervised Images through Adversarial Training.” If you haven't registered yet, head over to twimlai.com/meetup and you can do so there. The meetup will be Wednesday August 16th at 11 a.m. Pacific time. It will be recorded for those who can't participate live. If you've already registered but haven't read the paper yet, now's the time to get started, and if you have started reading and have questions, please post them over on the Meetup Slack channel which you should have been invited to after registering.

Before we get to the main event, I'd like to give a quick shout out to our friends over at Bonsai for their continued support of the podcast and our industrial AI series. Bonsai offers an AI platform that lets enterprises build and deploy intelligent systems for industrial applications. If you haven't investigated the company and their platform before, I think you'll find it interesting. You can find more information about them and their early access program at bonsai.ai/twiml.

Finally, about today's show. I recently had a chance to catch up with a friend and friend of the show, Josh Bloom, vice president of data and analytics at GE Digital. If you've been listening for a while you know that Josh was on the show around this time last year just prior to the acquisition of his company Wise.io by GE Digital. It was great to catch up with Josh on his journey within GE and the work his team is doing around industrial AI now that they're part of one of the world's biggest industrial companies. We talked about some really interesting things in the show, including how his team is using autoencoders to create training data sets and how they incorporate knowledge of physics and physical systems within their machine learning models. Of course, Wise.io at GE Digital is a sponsor of my industrial AI research and this podcast series, and for that we're extremely grateful. To learn more about Wise, visit their site at wise.io. To learn more about Wise.io at GE Digital visit their site at wise.io.

All right everyone, I am on the line with Joshua Bloom. Josh is VP of data and analytics at GE Digital, and if Josh's name sounds familiar it's because it should. He was our fifth guest here on the show and that was back in September of 2016. Josh, welcome back to the show, you're our first repeat guest.

Awesome, thanks for having me.

I'm super excited to have you on the show, and I encourage folks who haven't already listened to that, or even haven't listened to it recently, to go back and listen to it again. It was TWiML Talk 5 and it's been one of our most popular shows of all time, so you can really dig deep into Josh's background by going back and listening through that show. But for now, why don't you give us a little bit of your background and catch us up on what you've been up to recently.

Yeah, so I started off in physics and astrophysics and did a PhD at Caltech, went to Harvard as a postdoc, and then got really interested in robotization. It was then that I started getting excited about machine learning as I realized that we had some big data problems coming down the pike when we needed to do discovery on new images, let's say coming from telescopes. At that point my colleagues were basically saying I would just hire more grad students to look at the data. So that's how I came upon machine learning a little over 10 years ago. Fast-forward a bit, we wound up building up an end-to-end system to do some astronomy projects with data and machine learning, and then with the research group wound up starting a company called Wise.io based out of Berkeley, California. Over time we wound up building out a set of products in customer support, integrating with Zendesk and Salesforce to help support agents basically become more efficient and delight their customers even more. Last year, around the time that we had our first conversation, Sam, we had some broad interest from a number of different companies, and GE Digital wound up becoming very interesting and compelling for us, and I'm sure we'll get into that. But we announced our acquisition by GE Digital in November of last year in 2016, and since then have been working within GE Digital for a much broader GE ecosystem and GE's customers, and certainly happy to talk to you about what we've been doing and how we've made the transition from really customer facing and consumer internet to industrial internet.

Great, that's exactly what I want to focus on for this show. And in fact we can start out by me thanking you and GE for graciously sponsoring our industrial AI, both the research and the podcast.

Well it's our pleasure, and needless to say it's been fun for me personally and for those around me at Wise within GE Digital to also have you go through the same sort of process as we

wind up learning about this industrial machine learning world. In some sense we've lived sort of parallel tracks, coming to realize how important it is, what kind of value there is, and how different it is from the other types of machine learning and AI that's done still in industry but not in the sort of hardcore machine industrial context.

So maybe let's start there with the kinds of things you've been learning. I can't think of a more target-rich environment. The way I think you've once explained your role at GE is to kind of AI all the things, or at least be a part of that, and there are certainly a lot of industrial things there to be AI'd, if you will — we just verbs AI, I think you get the credit for coining that, if that's right. How's that been going, and what have you been learning?

Yeah, in some sense the practical nature of what it means to do machine learning in production at scale with fault tolerance is the same sort of thing that we took from our work in previous product sets and applied it to the sorts of problems that GE has in front of it. And I'd say the most practical thing is to say that we shouldn't be AI'ing all the things, that a lot of things don't need AI. Oftentimes when I give talks internally within GE, one of the big things that I'll challenge people with is this notion that everything needs machine learning, or if all we did is just apply TensorFlow to it then all our problems would be solved. We don't believe that. I think that people who have been working in this field for a long time understand that very deeply. And so part of our mission is to help people within GE and eventually GE's customers to understand what are the workflows, what's the type of data where advanced analytics or machine learning or even more broadly AI can be used to affect better business outcomes. It's with that lens of why are you doing this that we're able to say yes to a bunch of things but say no to a bunch of other types of projects where traditional business rules may just make great sense, or these are problems that are massively complicated, these are machines that have very good physical models that describe future behavior based on past behavior and that's good enough given the business outcome.

But I'd say one of the things that's changed a lot for us is to understand how important this is, not just at the scale, not just in the details of what it means to get a right or wrong answer. For your listeners to understand, you're always trying to optimize something when it comes to accuracy, an area under a curve or some false positive rate at a fixed false negative rate. But as we start imagining — and we'll get I think into some of the details in this interview about some of the specific projects — you can imagine that there are new places on the ROC curve where you don't ever want to be wrong, where you always need to be right. And this changes the nature of how you do data science, it's changing the nature of how you build products around it. But I'd say the thing that we've learned is that that view of not everything needs machine learning to get a great business outcome, but everything needs to work within the current context of not just an old industrial company like GE which has been around for 125 years, but with the business processes that have been built up in some of the oldest, most analog parts of the economy. Let alone not being digital, let alone not being software savvy, trying to bring products into a place that is sort of used to doing things with the status quo because they just work is a challenge in and of itself. And all of that in some sense makes great sense when you understand that what we're talking about is machines that affect our lives. One of the distinctions I make between the consumer Internet of Things and the industrial Internet of Things is that when your Fitbit breaks you call up customer support and you sort of complain, but when your jet engine has a problem that can have some serious consequences. And so the stakes are a lot higher, and because of that there's a whole regulatory environment which is something that very few people in the consumer internet have had to deal with. Yes, you sort of have to think about PII, you have to think about HIPAA compliance and things like that, but now if you're talking about a regulatory oversight body with the FAA or the FDA, there are some extra boundary conditions that are put upon us as we start thinking about bringing machine learning into those sorts of worlds.

Your team isn't really targeting or chartered with kind of making sweeping revolutionary changes at GE, but rather you're proudly taking a much more incremental approach.

You have to start somewhere, and again taking a very practical view of what it means to transform an existing workflow that involves data, that involves people, that involves physics based models, that involves decision rules, and then start bringing that into a machine learning centric workflow. There are a lot of different stakeholders involved, and many times people have already tried bringing machine learning in and failed for various different reasons. And so one of the sharp elbows that we wound up building up as an independent company, and we bring to GE now, is around that notion of what are the problems that you should be solving, and in particular should you be going after the highest value, most complex ones, or should you really just start somewhere. We really think about low-hanging fruit, and in some sense that's our lens: within the industrial context, where is the low-hanging fruit where it's so obvious that AI can have a measurable impact, not just on things like accuracy or time to make a prediction or something like that, but in real dollar terms. The way I like to talk about it is we're not trying to solve the problems that end with a B, they shouldn't end with a K. And so we're sort of in that M world where at the millions of dollars a year level, if we have impact there and we can start working with the individual business units and their customers to start helping people understand even how to structure a new problem around AI, and understand what it means to do data governance right, how you wind up basically building up an AI-first product from scratch, then we wound up winning because we get to multiplex across multiple internal and external customers.

One of the things that I specifically remember about this conversation we had on this point was you talked about kind of impact of 1% in your world.

Yeah, that's right. That really in some sense gets to the scale. If you have a 1% improvement in a product and a workflow that is making hundreds of K a year in revenue, that's not a really big deal, but if you do have a 1% improvement, let's say in efficiency, in a billion dollar product, that starts to get to be real dollars. So in some sense, what I just said before of taking the low-hanging fruit and not trying to solve the really big problems, we get away with within the context of GE just because the scale at which we're talking is just so immense. To just give you a sense

of it, for instance, when a jet engine finishes a flight, call it a five-hour flight on average, there's about a terabyte of data that's generated just from that one engine. And you can imagine even the process of offloading that data from the airplane after it's landed and then getting it into a data lake, that itself can be pretty complicated. But then doing sort of real-time analytics on that and making some decisions from a preventative maintenance perspective is one of the really important things that we have to be able to do. But now if you think about, well, there's 50,000 jet engines that are flying every day, that gives you just some sense of the enormity of the scale. So each flight is basically a day of Twitter data, and then you go a factor of a few orders of magnitude larger than that. So for us, yes, we get to work on the quote unquote low-hanging fruit with fairly large dollar numbers, in part because making small incremental improvements in the workflows that involve lots of data for very expensive, important machines is just sort of the reality of where the industrial internet is right now.

So can we talk a little bit about some of the use cases that you've seen? Are there ones that you can walk us through?

Yeah, so I can't go into all the specific details, but the one that I just spoke about within aviation is an important one, and something that's been discussed publicly is the need to have advanced analytics applied to data that's coming off of airplane engines to achieve better outcomes. One of the things that is important to recognize about many of these industrial use cases is there's a huge value to being able to understand ahead of time when something is going to break or whether something is in trouble. And that is where you get into some very interesting data science problems of, especially given a lot of these objects very rarely fail, how you build up sort of counterfactual evidence so that you can test your models offline. The easiest thing to do would be to build the machine learning model that says take every engine off the wing after every flight, and by golly you'd find every single problem because somebody would take it off, but then that whole industry would come to a halt. We would probably destroy the world's economy just given the extra latency of what it would take to retire an airplane every single day. So that obviously doesn't make sense. So there our false negatives would be basically zero but our false positives would be just uncomfortably high. The other approach is to say everything's working all the time, and for the most part you'd be right. And the number that I have in my head is that the sort of failure modes are only, a few in a million flights will there be a significant problem with an engine, which is why we have multiple engines on a plane. And so you're in very, very small number statistics land, and you can't really ever know, if I said take this engine off the wing, whether it would have failed had I not said that. There may be some diagnostic evidence you could see when you actually look at it, but you can't ever gather the counterfactual of what if I didn't do this. And I can't really run an A/B test either, where you say, well, I think you should take these off the wing but I'm not going to say anything about that. That obviously has its own problems as well. So you're in this very interesting dance where it's lots of data, yet in some sense it's a small data problem because you only get the sort of bad or rare anomaly events every now and then. And even when you get those, they're all sort of Anna Karenina, like they're all unhappy families, they're all different in their own way. That's a real challenge from the data science perspective, and that's where some of the interesting innovation has to happen from an R&D perspective, is to work in this kind of really long tail world. So that's kind of one family of use cases that we're interested in, but I imagine before we get into other use cases, I'm sure people are asking, okay, how do you address that technically from a data science perspective, what are some of the ways that you tackle that problem?

Well, in some sense it comes back down to, do we even tackle that specific one, or do we tackle ones that are adjacent to that? And getting back to some of the work that we did in customer support — and I think this is one of the really core design principles of how you think about building machine learning products — that is building assistive tools that wind up not sort of making a decision by themselves but actually provide

information and insights to analysts who are looking at the data. And there are analysts who are looking at the results of data coming off of all these engines. So instead of saying yes, this is going to fail and take it off, or no it's not, there's an adjacent problem where you can wind up saying I'll create a ranked, prioritized list of the engines that I think an analyst might want to look at, and then you let the domain experts in that world go through and make some decisions on that. So it becomes kind of an accelerant and an efficiency play, rather than a black or white, almost draconian type of thing. So we're not anywhere near the point where machines are going to wind up generating work orders without any people in the loop.

So when you wind up pulling it back a little bit from “this is going to fail” and “this is okay” to “I think this is something somebody might want to look at” — it turns out that analysts look at lots of things, and so they're often digging into a specific engine to understand what's happening with it. And so we have a lot of that data from the past. Now it's not a very long tail problem, we make it sort of more evenly balanced of X percent look like they're fine just from the very high-level overview, and 1 minus X percent look like they need to get some more work done or somebody needs to dig into the data a little bit more. And as long as X is close to 0.5, or even if it's 0.1, you're in pretty good shape, because then you can apply sort of classic supervised machine learning to that. So that's sort of one trick that we have, is to take something out of the very sort of rare regime and try to bring it back into a world where it still has value, you could still measure that value, but it becomes kind of an empowerment tool rather than something that's making absolute decisions. That's kind of one thing that we've brought from our past lives into this one.

I guess the other one is the nature of the data is very different. You've got data coming from lots of different subsystems, it tends to be time series data. In the past we had worked a lot with natural language processing, and one thing that's very powerful with this sort of data is that if you have a lot of it, there are techniques one can use, let's say within the deep learning world, where you can in a very unsupervised way build up some capability of generating features out of that and then do anomaly detection off of those features or just do direct classification off of those features. And so you get to leverage lots of the quote unquote normal data and normal behavior and then use that to be able to make inference on the things that look like they're out of band. So for us, being able to apply some of the cutting-edge techniques in, let's say, recurrent neural nets and unsupervised learning around these time series data sets is very interesting.

And I guess the third part of that, which is coupled to the other two, is trying to do this all at scale and trying to do this all as real-time as needed for the specific problem. And we've really crossed over in terms of our back end in terms of what's needed from sort of large single machines in a multi-core environment to a multi-node type of environment. So doing deep learning across multiple GPU instances is something that our infrastructure that we had built before has had to adapt to.

What I heard was you talked about using deep learning to generate features that would allow you to train more traditional supervised models, and that reminds me — you may not be aware of this, but we're starting a paper reading group associated with the podcast, and the first version of this meetup is going to be on the 16th of August, and the paper that we're going to go through is one of the papers from CVPR where a team at Apple basically used the generative adversarial network to generate data to then train a supervised model, I think supervised. Actually, I'm not clear on this because I haven't read the paper yet, but I will by the sixteenth. But the point that I wanted to pick out is I've heard this notion a couple of times around using deep neural networks to generate training data for either supervised or more traditional models a few times, and I wanted to make sure that that's what I heard you say you were doing, and also kind of get some feedback from you on how broadly applicable are you seeing that across the various use cases you're looking at. Are you doing a lot of that?

Okay, so to be clear, we're not using GANs to generate training data. There's another approach called autoencoders that allow you to generate features in an unsupervised sense. So it's adjacent, but it's quite a different problem. What I will say though in general about GANs — and there's only kind of a few papers out on this in this context — is that it's actually kind of interesting if you think about the data privacy and the sensitivity around some of the data that we have access to at GE. Passing data around, and clearly it needs to be done in a highly covered and highly regulatory-approved way, isn't always the best thing, and especially when

it's very large. So you can imagine that there are use cases where different groups within the same company may need to get access to data, but instead of sending the data itself over, you can imagine building a GAN that's able to generate data that's like the original data. So if you have very sensitive data that you don't want moving outside of your walled garden or your data lake, you can imagine building a GAN that essentially simulates that, and instead of handing somebody the keys of the original data you could hand them the keys to the GAN, which, if for some reason it fell into hands that weren't supposed to see it, could provably not be able to reproduce the original data. Yet folks who got access to this GAN would in principle be able to build machine learning models against that. And so I think it's a very interesting and clever way to start thinking about passing information about a set of data around without actually having to pass that data around. And so being able to build models that learn from different groups and their data without those groups having to share data amongst each other, or without having to aggregate it all into one physical place, is of great interest to us. And it's not just sort of an interesting thing to do, in many cases it's a necessity. If we want to build great models and we can't even see the data or we can't federate it into a single data lake, we have to have really clear paths to being able to do that. And again there's academic literature on this, but there's not a whole lot of work that's being done on this in practice. So that's kind of one interesting regime.

So you bring this up as I think an important one, and the other one that we touched on before at the top was around sort of the marriage of physics-based and data-driven models.

Yeah. Unlike again in the consumer internet, where you build a data-driven model around customer behavior or around actions or on sentiment, etc., you can try to build some sort of latent understanding about how the brain works, but there are very complex biological systems effectively that are giving rise to the data that you wind up trying to apply on. There is no physics behind recommendation engines, there's no sort of core principles there. Yet in the industrial world you've got jet engines, you've got MRI machines, you've got wind turbines, nuclear power plants, and these are all built up by physical objects that, if you knew all the physics of them, then you wouldn't need any data because you'd be able to predict exactly what's going to happen in the future. So again, for a preventative maintenance perspective, you'd be just fine. But as we all know, even in very simple physical systems we often don't know all the physics. And while GE has I think some of the world's experts in all the various different subdomains, in material science, etc., building up complex physics models, I think of it as physics models only getting us 90% of the way there to a great answer, and then adding a sort of data-driven layer on top of that is the path that we're seeking. So rather than what we did in the past, where you just take effectively a fully data-driven model to get your outcomes, we're quite interested in understanding how in a rigorous way do we combine the outputs of physical models essentially as

the inputs to data-driven ones, in addition to all the sensor data that you're getting. So I'd say that's a really critical distinction. It's also a huge amount of white space that we see in the industrial machine learning world.

And that's something that GE's been pursuing or evangelizing for a while through this notion of digital twin. Can you talk a little bit about that and the role that it plays in the work you're doing around ML and AI?

Yes. A digital twin, for those that haven't heard the term, is an idea, and an implementation of an idea, that every physical object should have a virtual version of it that could live in the cloud, or if it's very sensitive can live in an on-premise environment. And that digital version should be kept up to date with the physical version of it, and it should know about its maintenance history, it should know, in the context of an asset model, if this is a part in a large machine it should know about the machine itself. So it's a very base layer. I think of a digital twin as a digital representation of a physical asset and all the data that's available about it, both historically and then in real time. Where AI and advanced analytics comes in on top of that is to say, well, given all of this data, can I make a predictive statement about what's going to happen to the physical object by interrogating the digital version of that? So rather than having to ping a hard drive which is on the device itself and try to pull out data, we need strategies that take data from those edge devices, bring them into the cloud, and then it allows me in a more relaxed cloud environment to be able to ask questions of that, maybe take some actions based on it. And then the next step after that, of course, is to take the results of some of those predictions and push them back into the physical device itself and potentially even update things like configuration variables based on predicted outcomes. I don't think we're really there yet across a wide swath of GE assets, but my sense is that's where a huge amount of value winds up coming in, if you're able to build machine learning models now not just against this one twin but against all the twins in the same asset class, and use those models not just on one customer's data but across all customers’ data to be able to get better outcomes.

And so this is somewhat related to the role of simulation in building ML and AI systems for industrial applications in general, right? We talked previously in a conversation about how you can't just take the engine off and put it through its paces to generate data sets. You

know, what have you learned about the process of using simulation as a way to create these models?

Yeah, so to be honest I haven't learned that much. Doesn't mean I can't anticipate on it, I just can't say that I've learned a tremendous amount, just that those aren't the sorts of problems that we've been directly exposing ourselves to. Clearly there's a kind of reinforcement learning play in that conversation, about being able to simulate the environment or the results of an action that you wind up taking, and being able to build models offline before you wind up deploying it into the field. We haven't ourselves been working on sort of that kind of robotics angle, but that's obviously really important. That said, simulations get back to that physics-based model that I was describing earlier. In some sense I think of physics-based models as essentially simulation. You've got a simulation of your physical object because you think you understand most of the physics based on the whole history of what's happened to that object. If you've done your job with simulation, you should have some uncertainty bands into what state is happening next. Then again, could you build a machine learning model that's effectively taking the results of that simulation and then using that as your more or less physics-plus-data-driven model? Yes. We've got I think a sort of simpler notion of how you do that, which is just taking the base predictions out from the physics-based model and using that as inputs, in addition to all the sensor data, to build a machine learning model off of that.

Can you elaborate on what you mean by that?

Yeah, let me take an example. Let's say that you've got a wind turbine and you've got a prediction of what the winds are going to be in an hour from now, and let's say that there are configuration variables that one might want to set effectively in real time, essentially with the optimization goal of maximizing the energy output. So we think about it as rotating around to try to get the optimal wind direction. So now, based on some data that's coming in and predictions about the future based on what we know about the physics of the object that's going to be spinning and how long it takes for us to spin, you can imagine, given a set of inputs like what the weather is going to be and what the weather was an hour ago and how fast the turbine is spinning now, you could run it through effectively a physics simulation that says, if I turn at this amount I'm expecting to get this energy yield out, if I turn it by this amount I'm expecting to get this energy yield out. What I would posit is that one can take the results of those predictions, think about it very simply as an efficiency curve as a function of the azimuthal angle of the rotation of the blade, there's going to be a place where that's optimized. And clearly that number is going to be wrong because there will be other physics of that object, let's say that object's got a little kink in it or it's at a slight tilt, or the models of the winds are always systematically off by five degrees in the orientation. What I would then do is say, well, in the past we've had all these turbines choosing a next best

action for itself and those haven't been necessarily optimal. How could we take all of the data that we have coming in and build a model off of that to try to get a more optimal answer? And of course what you would do, in some sense because you have multiple turbines in the field, is try different potential outputs based on what the model actually predicts, and then as you get the results back you say, well, that looks pretty good. That's how you're effectively building up a continually learning model, you could call it a reinforced model, that you could then deploy and get better and better over time. So you use again the predictions from the non-machine-learning part of your model and use that as an input to the machine learning model.

That's a pretty fascinating take on reinforcement learning, right? We think about reinforcement learning as you've got these physical systems perhaps, that maybe super simplistic like an Atari game, or maybe a simulation of a robotic system that has some degree of fidelity to real life, and you're using the simulation environment as a way to give you feedback on what happens in real life. And so the model is kind of acting in this simulation environment. What you're describing is kind of flipping that on its head and making real life, your wind turbine farm, your simulation environment. In a sense — I guess not in the sense of simulation but in the sense of the environment in which your models take control, deliberate actions to try to minimize the error, maximize efficiency.

Yeah, and to be clear, I don't think that's a unique view that I have. One of, I'd say, the highest value results I've seen come out of Google's DeepMind is an optimization on HVAC usage in large compute environments. You can imagine you've got 17 different levers to push up and down, and there is no a priori understanding other than the physical thermodynamic physics of how a room responds to HVAC, other than to say that you've got a whole bunch of data coming in like, well, what's the server load on every single object, where is it located in my data center, and all I can do is move these 17 levers up and down. That's something where your simulated environment is actually the real environment, and getting to turn those levers up and down is something that you wind up learning how to do over time because you've got a very clear optimization metric, which is how do I decrease my energy costs of pumping AC into this room while still maintaining a level of reliability on the machines to not go over their expected heat loads. So I think that's a very clear example in an industrial context in some sense where that sort of notion of reinforcement learning winds up playing out.

To be clear though, I would say that reinforcement learning, as you've described and as I was describing in the industrial context, is really kind of one end of the spectrum of what we'll call continual learning. And the sort of other end of the spectrum is you build a model on static data, you deploy it into production, and you grab feedback of whether you're right or wrong, and then a year later you build another model based on that feedback and you deploy it. That's sort of a very gradual, punctuated continual learning. And then step farther forward into sort of what we were doing in Wise when we were doing customer support, where you wind up having a model that's rebuilt every day because the world of customer support is changing fairly rapidly, and those are deployed and it's taking all the feedback of what you've just learned over the last day. And then you can imagine another one where it's sort of a cybersecurity environment where you want to have a model which is updating itself based on different threats that are coming in, that could be an update on a minute timescale. The kind of continual, true continuous, real-time learning where you have these online models that are just getting better and better over time and adapting to changing

environment is a very natural place to be. And so I see that all as part of a continuum. Now, it has vast implications about the engineering behind it and even the data science and the certain techniques you would use, but conceptually I think it's all sort of very similar.

I think the distinction that I thought I heard there — and we can go back to the DeepMind HVAC example in the context of this continuous learning spectrum that you outlined — is clearly they're modeling a physical system, they're deploying models out to a physical system, they are continually optimizing this model and getting to a model that can control their seventeen levers in a way that produces optimal, or at least way better to use a technical term, output over a given period of time. What I thought I heard you describing in the wind turbine example was, if we kind of map that to this continual learning spectrum and say that their feedback loop is operating at such a scale that it's near continual, what I thought I heard you describing was almost like accelerated continual learning, meaning we take this model and then we push it out to the physical devices, again in this case the wind turbines, and direct them to act in specific ways, not to pursue the plan that is outlined in this model but to deviate from that in a way that we think will accelerate learning and thus produce a better model more quickly. Did I make all that up, or were you saying something like that?

No, I think you've got it. And the missing piece of why I see that connected to the HVAC example is that in principle — and I don't know this to be true or not — you also have a thermodynamic model of what would happen if you threw lever four up higher and increased the HVAC in that region of the room. You in principle could model that, and you can imagine that instead of just using the data coming off of the individual computers as the input, instead you could also use that data plus a thermodynamic prediction about what would happen if you made a change. And I agree this could actually be an accelerant, because in a world again where you knew all the physics, you wouldn't need a domain-driven model, you would just, a data-driven model, you would just take all the heat loads and you'd crunch some big supercomputer, which itself could add to heat load in the room, but then you'd wind up being able to say very precisely, if I change all the levers in every single possible combination, what is the optimal output. But that's obviously a very hard, intractable problem given the complexity of even a data server room. So I connect those two examples in part because they are physical systems, and in both cases you have the potential, whether you use it or not, of using the true thermodynamics and physical modeling of what the expected output should be, and instead of having to explore that space in a purely data-driven way, you then have the ability to explore it in a sort of simulated way and at least do an exploration in the real world around where you think the good answer is going to wind up being.

Is there anything that you've learned or any direction that you can point us in terms of

the very practical, tactical approaches to integrating the physics into the modeling process and the models themselves?

Yeah, in a time-series sense, if you've got a physical system that is behaving effectively like a sine wave and there's a, call it a linear oscillator, that's involved in producing the data, you can imagine fitting with your physical model — so this is if you have parametrized the data that you have to your physical model — and getting out internal parameters that more or less give you a good measurement of the past, and you can then use to make a prediction about the future. So again, if it's a sinusoidal model, you'd fit the spring constant and the mass of the object just to make it really simple, and then, if it's a perfect model and a particularly easy problem, you'd get basically just a set of residuals from your fit that's consistent with the noise properties of the data. But oftentimes in more complex systems you might have some residuals that are correlated in time and not zero and not consistent with the errors, which means that there are more things going on. So instead of building a machine learning model on this large sinusoidal wave, why not just build a machine learning model on the residuals of the data? And there you could then bring in other data points, you can bring in metadata. It becomes very powerful in some sense, we move away the signal that you know about and only model the signal that is unmodeled. If I've only got a certain number of data points and I've only got a finite-size model, if I don't imbue any physical understanding of this into what I've got, I basically now have to fit a sinusoid using my machine learning. Well, they can do that of course, but then you're using your power up in something that's knowable by other means. But imagine you measured that, and you said, well, I know it's a mass and a spring and I get these measurements, but boy, I can't predict the next time step, why is that? So you do what I just said, you subtract off essentially your physical model, and then what you wind up realizing is the residuals are growing in time. It's because you forgot to include friction. Well, now your domain-driven model is going to basically learn what the friction constant is, so that it winds up getting a better prediction when you combine both of those two together, and it may have had a harder time finding that if you just said I don't know anything about the system, let's just use pure data to figure it out. So I think the whole point here is that these are physical systems that have the potential to be modeled, and yet our modeling capability on the physics side is imperfect because we don't know all the physics yet, but that's clearly in some sense prior information that we should be using, and then removing that out of the original signal and then only trying to predict what those residuals are so that we get a better answer.

When I've talked to — to try to be more precise here — some of the folks from the deep learning perspective, they kind of say, to probably poorly paraphrase them in a way that they'd disagree with, forget about all the physical model stuff,

what's cool about this deep learning stuff is that it'll figure everything out, so why worry about trying to incorporate these models, let's just throw tons and tons and tons of data at this thing and the network will figure it out. And that's always been counterintuitive to me, and so I just wanted to kind of poke you at this a little bit to make sure it's really clear, and that us as a community are really clear, on why at least in this domain the physical models are important and can be very powerful.

Well, let's put ourselves in the minds of the people that made those sorts of statements. There's great evidence that they're correct. You have a whole history, decades long, in computer vision where people are trying to come up with essentially physical models of what it is that a machine is seeing, and building a very deep understanding based on our understanding of the physics of vision into being able to make predictions, being able to do segmentation, being able to make classifications. And then deep learning matures, the large large data sets, benchmark data sets, wound up coming out, and all of a sudden all the old models and all the old ways just fall by the wayside. So there's an example in the computer vision world, there's even examples in natural languages —

Yeah, I was going to say, in the NLP world, a famous NLP person I think from the 70s, “every time I fire a linguist my model improves,” right? This crazy notion now that, why do I need to have a complex understanding of how language works when in the end all I really need to be able to do is just throw massive amounts of data at a network that's capable of learning it? So there's certainly examples where physical modeling, or theoretically hierarchical models of how language works, just basically were inferior once you had enough data and you had sophisticated enough networks. But the operative word or phrase that you said is “tons of data.”

That's problem-relative.

Exactly, that's problem-relative. Again, let's come back to the jet engine example. We've got tons of data, and more data in jet engine world in principle than in any data lake of any computer vision researcher. So then you would say, well, we've got more data so you just throw it at it. Except in this case, as I was saying before, we have so few examples of things going wrong, because these engines are so good and so robust, that you have to appeal to physics in some sense, you have to look at a physical understanding of these objects and the different failure modes. Because in computer simulation land, or in just physical simulation land more broadly, you can test a whole bunch of different things that never get tested or seen in the real world, and so you can build off a whole bunch of failure mode environments that, if you start seeing something like that happen in real data, that becomes a trigger point and you say, look, we've got a problem that's upcoming, let's take care of it. Whereas that purely data-driven model, my hunch is at best it could say this is something we haven't seen before, but it can't tell you what's going to happen in the end because you don't have a predictive model, it's literally never happened before in any of the data you've ever collected, yet it is something that you could fit a physical model to and show, well, given all the data of what's happened the last 10 days, this outcome is now expected. And again, you won't be exactly right on those, but I would argue that, and those are great examples, your physics-based models are going to wind up trumping purely data-driven models. And in the end I think it's going to become clear that it's going to be the combination of both of those notions that will make the most powerful, most robust outcomes.

Great, well this has been an awesome follow-up discussion, and I'm super excited to have you as our first repeat guest back on the podcast. Is there anything else that you'd like to

leave our listeners with?

Well, first of all, I would love to be the first three-peat guest as well, so we can look forward to that in the future, and maybe somebody has a model for whether that may happen. But what I will say is it is a very interesting time to be crossing over from the consumer internet, working on valued problems for people and their interactions, to what we're working on at GE Digital, high-value problems for people living their lives. Ready to get on airplanes, your house is powered by a power plant somewhere, etc., you go to a doctor and generally GE machines are the things taking pictures of you and your insides. So your life as a person depends upon the industrial Internet of Things, and it's a great time to be part of that, and there's a massive amount of work to be done.

So I mean, you're hiring?

Oh yes, you're hiring, please. I would love to have any of your listeners contact me directly, email's easy, it's just josh.bloom at ge.com, or you can tweet at me, I'm just @profjsb, and I'd love to hear from you. I think the important other thing is it's not just, are you a machine learning expert in this one little realm of time-series, multispectral time series for blah blah blah, it's we're really looking for people that just know how to scale computation and work with data under very restrictive environments around security and governance. So for me it's exciting not just thinking about it from the ML perspective, but from the engineering perspective.

Fantastic. Well, thank you so much, Josh, it's great to catch up.

Great to catch up with you as well, thanks so much for having me on, and love the series and love what you've been doing.

Thank you. [Music] All right everyone, that's our show for today. Thanks so much for listening and for your continued feedback and support. For the notes for this episode, to ask any questions, or to let us know how you like the show, leave a comment on the show notes page at twimlai.com/talk/42. Thanks again to our sponsors Bonsai and Wise.io at GE Digital. For more information about Bonsai visit bonsai.ai/twiml, and for more on Wise visit wise.io. Don't forget to register for our upcoming online meetup at twimlai.com/meetup, and my newsletter at twimlai.com/newsletter. Thanks again for listening and catch you next time.

Industrial Machine Learning

Sat, 01 Jul 2017 00:00:00 +0000

Guest lecture at Berkeley Haas on industrial-scale machine learning — deploying ML against industrial data at GE Digital after the Wise.io acquisition, and the gaps between academic, startup, and enterprise ML.

Year approximate.

Systems of Intelligence

Mon, 15 May 2017 00:00:00 +0000

Keynote on building systems of intelligence - coupling machine learning with industrial data and workflows.

Emerging Needs and Opportunities in Data-Intensive Domains: Astronomy

Mon, 20 Mar 2017 00:00:00 +0000

Astronomy's data-science needs and opportunities, for the National Academies roundtable on data science education.

Machine Learning for the Stars & Productizing AI

Thu, 22 Sep 2016 00:00:00 +0000

An 84-minute interview on pioneering ML for robotic-telescope imagery at Berkeley, the founding and evolution of Wise.io, its data-science pipeline, and open challenges in machine learning.

Key Quotes

“The classic example I go back to is Galileo, who said, hey, there's this new thing that's been invented to look at the horizon for ships coming towards us — what if I just took it and pointed it to the stars? What could I do with that?” – Joshua Bloom

“In some sense, the way I view this now is that algorithms, and the accuracy that they can produce, and the ability to optimize them around a loss function, is really only table stakes for the utility of these in a real environment.” – Joshua Bloom

“We even got to the point where we were finding interesting objects in the sky without any humans in the loop… So by the time people woke up in the morning, we not only had the discovery, we not only had the initial inference, we then also had real follow-up — scientific follow-up.” – Joshua Bloom

“Your brain is a 30-watt supercomputer, unrivaled — at least for now — by anything else that's out there, and anything else that's out there is likely going to take megawatts or hundreds of megawatts to get anywhere close to that computational capability.” – Joshua Bloom

“Doing machine learning for machine learning's sake really doesn't make sense. It's probably the last thing you want to do if somebody hands you data — you do it because you have to do it.” – Joshua Bloom

Transcript

Sam: Hello everyone, and welcome to another episode of TWiML Talk, the podcast where I interview interesting people doing interesting things in machine learning and artificial intelligence. I'm your host, Sam Charrington. I think you all are really going to get a kick out of this show. My guest this time is Joshua Bloom, professor of astronomy at the University of California, Berkeley, and co-founder and chief technology officer of machine learning startup Wise.io. I was in California last week, and Josh graciously agreed to host me in his company's office for this interview. We had a wonderful discussion, and as you might have guessed if you happen to have noticed the length of this episode, we covered quite a lot of ground. But I promise you that you'll find this 84-minute interview to be jam-packed with great information, ideas, and war stories.

In this show you'll learn how Josh and his research group at Berkeley pioneered the use of machine learning for the analysis of images from robotic infrared telescopes. We talk extensively about the challenges they faced in doing this and some of the results they achieved. We also discussed the founding of his company Wise.io, which uses machine learning to help customers deliver better customer support — but that wasn't where the company started, and you'll hear why and how they evolved to serve that market. We talk about his company's technology stack and data science pipeline in fair detail and discuss some of the key technology decisions they've made in building their product. We also discuss some interesting open research challenges in machine learning and AI. Of course, I'll be linking to Josh and the various things we mention on the show in the show notes, which you'll be able to find at twimlai.com/talk/5. That's twimlai.com, talk number five. And now on to the interview.

Sam: Hey everyone, I am here at the Wise.io offices with the CTO, Joshua Bloom, and we've got a great conversation lined up for you. We'll start with Josh — why don't you introduce yourself to the audience here?

Josh: Great. This is Josh, and I am the CTO and one of the co-founders of Wise.io. I'm also a professor at UC Berkeley in the astronomy department. One of the important things that we'll touch on today is how does somebody go from astronomy and teaching to building an AI application company. A big part of the origin story of the company is, of course, my history, but I think it also has some interesting lessons for how we think about AI in production systems and why having diverse backgrounds is pretty important these days.

Sam: That's a lot of good stuff to talk about there. Why don't we start by learning a little bit more about you and your background and how you got to where you are?

Josh: I was trained as a physicist and an astronomer. I went to Harvard as an undergrad and caught a bit of the research bug over the summers working in Los Alamos, then went to Cambridge, England to do a master's, and back to Caltech, where I did my PhD, all in the context of astronomy, and then back to Harvard, where I was a postdoc — all the while working on what we could broadly term time-domain astrophysics: understanding the variable sky and why things do what they do, explosively, cataclysmically, or otherwise. And while there's a deep interest in understanding the origins of those events and how they're connected to other things that we study in the universe, I got more and more interested over time in the informatics of actually just doing the science — the statistics on variable sources and the presence of noise. Then as I became a faculty member at Berkeley, I started looking ahead to really a series of new surveys, particularly imaging surveys of large swaths of the night sky, and one of the great interests for myself and many others was in finding new events — essentially new explosions or new variable, eruptive stars that hadn't been known about before — and doing that as quickly as possible. Now, the traditional way in which that was done, and in some cases is still done today, is that as you acquire more data, you linearly hire more grad students. That scales with the total number of images that you're getting and need to sift through, and as I was becoming involved in some of those projects, I ended up realizing that as time went on, that really wouldn't scale anymore, and we needed to find effectively a replacement for domain experts who otherwise would have been looking at and opining on data, using alternate techniques. About seven, eight, nine years ago I stumbled upon machine learning as a real interesting potential avenue, and at the time machine learning really hadn't been applied to anything in astronomy in the variable sky, in a time-domain context. There had been a number of studies in using machine learning to do special types of inference on the static sky and understanding demographics of stars and galaxies and their distribution in space. So we really felt like there wasn't a lot of precedent in us applying some of these capabilities to this data, but we wound up realizing it was sort of an imperative. One of the things that people who know astronomers would probably say about them is that they tend to like to use tools that help them, and seek those tools out — be those new types of detectors (CCDs, for instance, were something that astronomers adopted almost as soon as they were invented) and obviously statistical techniques and computational techniques. Astronomers are willing to try things out to solve their problem. The classic example I go back to is Galileo, who said, hey, there's this new thing that's been invented to look at the horizon for ships coming towards us — what if I just took it and pointed it to the stars? What could I do with that? And so our use of the telescope was essentially a co-option of a technology that had been built for other purposes. That kind of proceeds apace throughout the history of astronomy, and so the idea of bringing essentially a fairly new technique into the fold is not at all unusual.

Sam: Do you remember how you stumbled across machine learning?

Josh: Part of it was just asking the question: if I've got a bunch of data and I need to decide, is this part of the sky interesting or not, is this event new or not, what type of event could this be — very quickly you wind up realizing this is a classification problem of some sort. And talking to people at UC Berkeley in the department, as I was starting to introduce some of these interesting challenges, it became very clear that machine learning, and particularly supervised learning, would be a fertile ground for us to start exploring. But one of the challenges that I saw is that even though we're in a very rich and fertile environment at UC Berkeley, and there's a lot of crosstalk between departments and individuals within departments, it was very hard to get even the language on both sides up and running, where both sides understood — the methodological folks who deeply understood what machine learning was and what it could be used for, and then people like myself from the physical side even learning to ask the right questions. So thankfully we wound up getting a group from the stats department and those folks from computer science together with me and my postdoc, and we were able to get a proposal together that the National Science Foundation funded, which allowed us to start building out new ways of doing inference on astronomy data. And that turned out to be a very fruitful place for us — for me in particular — to learn about the landscape of what other techniques were out there that we hadn't been taught in school.

Sam: Can you tell us a little bit about, from that initial discovery, what the research arc looked like? What were some of the first things you started exploring, and how did that evolve over time?

Josh: I really just started looking at toy amounts of data that we already had in the can, and we could start applying these different techniques and looking for tools that would be useful for us. Really the best thing out there at the time that we started was something called Weka, which was — and still is — a collection of machine learning algorithms that one can apply in a GUI, graphical way, all written in Java. Really, that was our original playground and benchmark, and we used that as a launching-off point to start understanding what are these different modeling techniques that are being exposed here, what is a support vector machine, what does it mean when people say random forest — and we used that as a way to educate myself and those in our group. And we started seeing some interesting results — we started seeing accuracies that were better than what you could get from random. Then as we poked farther and farther, we wound up seeing how far can we take these algorithms, how well does one of them work relative to the others to get the kinds of answers that we want, how do we build in a loss function — which turns out to be very important to get good answers, because in the case of what we did, when we're discovering something in the sky, it's not easy to articulate that loss function. And by that I mean: what is the cost of missing an interesting place in the sky? It means that you don't get to do new science. Versus what's the cost of saying everything in the sky is interesting, which means you burn all of your follow-up resources. And then we started to think about context-aware classification — now not just in the context of really just resources, but now time constraints: making an inference about something that could be of interest may be more important than waiting to get another couple of data points and saying something with even more confidence. So understanding how to calibrate confidences and probabilities, doing this in the presence of missing data and irregularly sampled data in time — all of these also wound up showing to us that there were parts of the machine learning sphere, at least in the academic world, that were not often exposed to the kinds of data that we were exposed to. And so noisy data, for instance — no one ever, when they talk about the iris dataset, says the petal of this is red plus or minus purple. So even just having uncertainties in your features, let alone your labels, became an interesting challenge, and we wound up realizing perhaps there were some new techniques that we needed to start innovating on to even do the kinds of inference we wanted to do with our data.

Sam: You mentioned the loss function and needing to wrap your arms around what that means. Can you succinctly describe how you grappled with that? How did you approach it, and what did you end up coming up with for the types of data that you were looking at?

Josh: One of the things we wound up realizing is that one person's loss function is not the same as another person's loss function, and so to get traction on your answers, one needs to at least be clear about what it is that you're optimizing for, and at least give people the ability to imbue their own loss functions. If, for instance, you're producing a catalog of different types of variable stars on the sky, we have a specific notion of what it means to get something wrong about, say, a very minority class versus a majority class. I wouldn't say that we solved that problem by any stretch, but at least we were trying to be clear about what our assumptions were of the loss function and articulate what it is that we're optimizing for. When people are doing AI or machine learning in a production environment, there is always going to be an optimization of some sort, and the typical one people will go to, without knowing exactly what the business value is, or scientific value is, of the answer, is to go for some notion of inaccuracy. And then when you get a level deeper in that, you say, well, what I really want to do is minimize false positives at a false negative rate of 0.1 — and that is an implicit statement of what your loss function is, and you hope that by defining it that way and by optimizing on it that way, you're actually getting very close to an optimization of the result of what you're emitting out of your modeling.

Sam: And so you're primarily looking at image-oriented data over time. Are there other fields where you've seen them adopt the same types of approaches to what you were working with?

Josh: Well, one of the nice things is you can work at the sensor-level data, which is effectively photoelectrons in a CCD, and counting those up as a function of position in x and y, and then trying to map that back to the sky — so that's what you might call noisy image sensor data. We worked at that level, but then we also worked at a metadata level, which was: now let's use traditional astronomy techniques to extract the brightness of a star as a function of time. And so we got ourselves out of the image plane and into the time domain, and then we're basically working with effectively tabular data. And again, there are lots of different models and feature engineering approaches that one can take to all of that. I wouldn't say that there was a common thread in our work across a bunch of these different sub-questions, other than to say that over time we wound up realizing that there were only really a couple of different machine learning models that did as well or better than everything else. And so even though, for instance, support vector machines are very popular because they have some great theoretical, provable properties, they tend to be kind of unwieldy, and for dealing with the kinds of data we were working with — which is heterogeneous, noisy, dirty, sparse, missing, and multiclass, where you needed to also get probabilities out that you could then calibrate — models like support vector machines really fall short for practical purposes. And so we wound up recognizing in our group — and I think that was validated in a conference that we ran at UC Berkeley on essentially streaming inference with machine learning; it was a week-long conference that involved folks from Netflix, folks from Google, and then domain experts, everything from biomedical to physics — a number of people would stand up, give their talk, and say, yeah, we wound up realizing that decision forests pretty much always won. Now, this was in 2012, before the resurgence of deep learning. I bet if we ran this conference again, half of the talks would be about how that's a better algorithm, as it were. But it was pretty eye-opening, and it was one of the things that we took to heart when we wound up starting the company: a recognition that to succeed, to produce value, very generally, the algorithm itself is not necessarily the key. In some sense, the way I view this now is that algorithms, and the accuracy that they can produce, and the ability to optimize them around a loss function, is really only table stakes for the utility of these in a real environment. So yes, you need to use a model that's very, very accurate and potentially can be retrained and get slightly better, but as most data scientists or most people that work in machine learning workflows will say, almost all of that work is in feature engineering — and if you're a deep learning person, you'll say almost all that work is in figuring out what the shape of the network should be and iterating over that.

Sam: On that note, before we jump into what you're doing at the company today, what were some of the results you saw out of your research on the astronomy side?

Josh: We looked at a couple of different realms. One was looking at large catalogs of variable stars and coming up with probabilistic classifications of what type of variable stars they were, what was the physics that drove them. And we did that in a bootstrap way, starting with effectively a few hundred known classes and a few hundred or few thousand known labels, and then extrapolated that to tens of thousands, hundreds of thousands of variable stars and produced probabilistic catalogs. One of the things I became adamant about as we were doing that was that producing a catalog where you say, hey, this object in the sky is of this type with this probability, is effectively useless unless it's then used for some new kind of science. And one of the things that I became — I won't say frustrated with, but I noticed often, is that people started using — not just in astronomy but in many other fields — machine learning as an end unto itself, saying, I'm going to apply machine learning to this data and I'm going to get a result. Until that result itself is novel, or until that becomes a stepping stone to another result which becomes novel, it's sort of an empty exercise. And so what we wound up saying is: what can we do with this probabilistic catalog that couldn't have been done with any other means? So one of the things we did is we looked for very strange types of stars that had certain properties and then followed those up with big telescopes and actually wrote science papers with those.

So we used that as a launching-off point in a real-time environment, where we actually were looking at images as they were streaming off of telescopes in Southern California, off of Palomar Mountain. Every 60 seconds or so it would get transferred up to Lawrence Berkeley National Lab, and we'd apply our machine learning to that to find new interesting objects in the sky and then populate databases of, for tonight, here are the interesting objects. And then we also had another machine learning code which would go into those databases and periodically make statements about what types of objects those might be, what they could be. And we wound up having, I think, of order 100, maybe 200, papers that came out of that — refereed papers — which, again, the machine learning part of that was really the stepping stone to discovery. The other parts of the machine learning were the stepping stones to initial inference, and obviously in the end you needed people to actually write the paper.

Sam: But it goes back to the grad students, right?

Josh: What's that?

Sam: It all goes back to grad students.

Josh: Exactly. But I really thought about removing people from the real-time inference loop and getting as far up the inference stack as we could. We even got to the point where we were finding interesting objects in the sky without any humans in the loop, identifying that not only is it a new object, but it's something we probably should be following up, and we were issuing alerts to robotic telescopes to go follow those up. So by the time people woke up in the morning, we not only had the discovery, we not only had the initial inference, we then also had real follow-up — scientific follow-up. One of the, I think, great achievements of the work that we and others did in one of our collaborations was to build this production system that had real consumers on the other side of it, and when it was broken or was wrong or didn't take feedback properly, we'd get nasty emails from our collaborators saying, your thing didn't work for an hour, you kind of screwed my science while I was at the telescope. So having an end user really keeps you heavily focused on making sure the things that it needs to do, it does right and robustly. But because we were discovering things even faster than a whole army of grad students would have been able to pore over all of these images, we were able to find, for instance, the nearest Type Ia supernova that had been found in 25 years, and get a whole bunch of people looking at that part of the sky and taking lots of data that led to a bunch of papers in Nature and Science.

Sam: Wow.

Josh: Not because that object wouldn't have been found by even amateurs — because it got so bright you could have seen it with binoculars eventually — but because the interesting science happened hours after the event blew up. And so it wound up also driving home for me the need for not only something that's working and is robust, et cetera, but where it's able to make statements quickly and do it in a way that's reliable.

Sam: Interesting. I'm sure that has led you to some interesting perspectives on the relationship between this technology and society and jobs and stuff like that. I'm hearing parallels to — a lot of people projecting that as AI is deployed, shifts in the job market will take place that put a lot of people out of work. But I'm also hearing in your example the counterargument you often hear, that really what it does is empower people and allow people to do different things. I don't necessarily want to go deep into the society stuff at this point, but —

Josh: Yeah, it's certainly a valid concern. What we do in our company at Wise.io is help customer support agents become more efficient at their work: by suggesting answers of how they can respond to an incoming inquiry, by automatically triaging incoming inquiries or tickets, emails, et cetera — that is, getting them to the right person or the right group who's going to be the best at answering that question — and then in some cases we will automatically respond to incoming inquiries. So when you write into an e-commerce site and say, my package didn't arrive, there's a growing chance that us or somebody else may be answering what looks like a bespoke question of yours with what looks like a bespoke answer — in the end it's just a templatized response that we ourselves are using. For us to be able to do that — obviously we can talk more deeply about how that works from an AI perspective — we have to get very confident in what our answers are. But what does this mean, on one side, to your question about labor displacement? Companies don't need to hire as many support agents — so where would they have gone? The other side of that is that the agents that they do have become better and more tuned at working on some of the harder parts of what their own products are about, what their customer complaints are about, in a way they wouldn't have been able to because they would have been distracted by the mundane. So if you say, how do I reset my password, and there's a person or sets of people that have to look at that email and decide how to respond, that's time that those people are not spending on really complex problems where empathy is required as well.

So we think of our product and what we do as a way of freeing people to work on the things that people are uniquely suited at, that machines really aren't going to be that good at until somebody solves the Turing test. Chatbots are not going to be able to understand people in the subtle ways that they need to, but we can take a lot of easy stuff off the table — we have. And so there was certainly a concern as we were starting to roll this out that we were part of this labor displacement movement, but we heard time and time again from our customers that their support agents became more and more happy the more involved we were. There was an entire team in Asia who had been tasked with basically reading an incoming inquiry or ticket and then deciding who else should be reading this to solve the problem, and because our triage capability came into play, one of our clients effectively deprecated a 20-person team off of triage, because we're effectively automatically triaging now. And we were worried — what's going to happen to these people? They have families to feed. And we got a whole bunch of really great quotes from them saying that because they had been reassigned to actually work these support tickets rather than push them along, they were much more happy in their job.

Sam: That's fantastic. So we jumped right into what you're doing at Wise.io, but the transition is a fascinating one as well. How do you get from astrophysics to a software company doing CRM stuff? And I know there was an intermediate step there as well.

Josh: Going back to the original part of the conversation, we had recognized in the team that I'd built and the people I'd worked with that, A, we had some great technical orthogonalities — some were good at software engineering, some good at ML, some at UI, et cetera — and, B, that what we had learned to do — recognizing the importance of putting AI into production and having real end users give real feedback in potentially a real-time loop — was something we at the time didn't see anyone else doing. We knew obviously that the Googles and the LinkedIns and the Netflixes of the world had this kind of baked into their overall data flow, but we certainly weren't seeing companies helping other companies do it. And one of my now co-founders had, more or less while he was between jobs, figured out how to make one of the algorithms that we liked a lot — these decision forests — scale very, very well, at least on a single machine, in a multi-core environment. And so we realized that we might have some interesting firepower. And given that there seemed at the time to be so much emphasis on massive-scale machine learning — it was certainly the pre-Spark era, but very much in the Hadoop heyday — it looked like most of the interesting ML that was starting to come out, and some of the other companies that were coming out, were really focused around helping the — I won't say exascale, but certainly petascale-level, Google-scale amount of data companies bring machine learning into their workflows. So we thought about skating to a place — using the analogy that's heavily overused — to the part of the ice where the puck was going to be, which was helping smaller companies and midsize-scale companies bring machine learning into production environments. And that was the impetus for starting the company. What the domain was going to be, we didn't know. Who the customer was going to be and who the buyer was going to be, we didn't know. We were, I'd say, blissfully ignorant about all the business challenges that we would wind up encountering over the next couple of years in bringing this to market.

And when we emerged out of our first accelerator — it was the Alchemist Accelerator — I gave a talk at our demo day where I said we're going to be GitHub for data scientists, and produce some interesting UIs of interactions to help data scientists like ourselves more easily build models that they could then push into a production environment. We wound up seeing over the next couple of months the challenges of selling products like this into data science teams. First, data science teams were few and far between, and the ones that existed were either too sophisticated, believing they could build it all themselves, or not sophisticated enough to get a large enough budget to pay for the things that we wanted to provide them from a tooling perspective. All the while, we were building out our underlying platform to be able to do exactly that: to be able to build templated machine learning models against certain types of data for certain types of use cases, and then, even though you built it at a laptop level, push it into the cloud and have, in Amazon or other compute frameworks, the scalability to be able to serve large numbers of customers around those same use cases — where what's emerged for us is that the difference between a customer is not new code, it's just a config file, if they're using that same use case.

So, all of that to say that we evolved — you could call it a pivot if you'd like, but I think of it as a series of pivots — to a place where we wound up seeing in customer support a lot of data, a lot of manual work, and some really nice CRM systems with open APIs and a fairly fixed schema. So the Salesforces and Zendesks and ServiceNows of the world really are the data lake and the transactional layer for doing customer support and related activities. And we thought if we could build now an intelligent system on top of that and do all the things that I mentioned — empowering these agents to become more efficient at their job and making the whole support desk more efficient at serving customers — we would solve a bunch of pain points. And as we wound up going into the market and started leading with products that could be more or less installed by a non-technical user and could be used by a non-technical user, we wound up getting a lot of feedback that indeed we were solving some pain points. There's obviously the efficiency question of needing less headcount, but there are also some really interesting customers of ours who were growing very quickly, and one of them said to us that if the CEO had given him an infinite budget, he wouldn't be able to hire good customer support agents quick enough. So helping them capture all the internal knowledge was something that, it turns out, machine learning actually does quite well at.

Sam: What were some of the biggest challenges in going from a product direction focused on generalized tools and platforms to one focused on a very specific application area?

Josh: Interestingly, it was all the things that we hadn't thought about, which was product management, and how do you get structured feedback from customers, what does it mean to build an MVP, roll that out, iterate on it, et cetera. A lot of Lean Startup 101 stuff was something that we hadn't really been thinking of when we started the company, and certainly didn't have, frankly, a lot of expertise in. And then as we started scaling, it was a recognition that there are large parts of a machine learning pipeline that don't naturally scale. So figuring out ways to containerize the parts that need special attention from PhD-level data science, and abstract that away from other parts of our engineering group that don't need to know about what's happening deeply, but need to be able to ask predictions of some other part of the stack RESTfully, in a services-oriented way — we just wound up realizing that what worked for us from a scaling perspective, a compute-scaling perspective, also wound up being what we needed to do from an organizational and HR perspective. When we hire front-end engineers and middleware engineers who are great at writing scripts against databases and managing Redis queues, et cetera, those folks don't need to know about machine learning. They need to know that there is a contract between their part of the stack and somewhere deeper in the stack: that if I ask you for a prediction for this client, for this model, I'm expecting to get it back in this format on this timescale, and if I don't, then our contract's broken. But likewise, I'm going to hand to that deeper part of the stack that's going to be providing those predictions effectively some data, in JSON or otherwise, that will have a fixed schema, so that the group that built that machine learning pipeline knows that this column is going to be of type datetime, this column is going to be an int, and it's going to join using these four indices on some other data. So once we wound up realizing that we could lock down the schema for a given use case, it meant that we could write data science pipelines against data we hadn't seen before. You need to see it once to make sure it's all working, and make sure it cross-validates in an offline sense and has the kinds of accuracy properties that you want, but then it means that we could start spinning up new customers where they get the base template that operates on their data, and when we need to make changes, those can happen from a more or less technical person rather than somebody with a PhD in statistics. So there were a bunch of challenges around that, and as we started solving those, it just sort of fell out that our stack really mimics what our organization looks like.

Sam: Okay. Can you talk a little bit about the data that you typically see in a customer environment? I'm imagining just loads of trouble tickets, but I imagine as well that there's ancillary data, supporting data as well. And you mentioned that there's lots of it — can you talk about the size you typically see, those kinds of things?

Josh: Our typical customer is doing of order five to 20,000 tickets a month, and we need to be working with companies that are achieving that level of volume — A, because the price points are reasonably high, and so it's typically the companies that have those large volumes that are willing to pay for what we do, and, B, because the machine learning models are built specifically for and on their data, and we don't use a common model, for instance, across our customers, so we need a lot of training data for a given customer. Now, again, this is not petascale amounts of data — we're talking tens to hundreds of gigabytes at the per-month level, per customer. The data is indeed a lot of human-to-human interaction, from emails, web forms, even chat. And there's also a lot of metadata: what is the value of this customer, what products are they using, how often have they been emailing — so there's a time-series component to this as well. And we've had to build these pipelines that are generic enough that we can then apply them to other use cases, but specific enough that they give good enough accuracies that wind up rivaling what humans can do. And so oftentimes our goal is to get to — we don't call it accuracy, we call it matching capability, because oftentimes when a human's labeling something and saying it belongs to this bucket, or this person should answer it, or it should have been answered with this template, they oftentimes can be wrong.

Sam: I think we've all had that experience.

Josh: We want to get ourselves to that kind of level of quality, let's say. So from a featurization perspective, we're doing lots of natural language processing, getting it to the point of rectangularization, a grouping in semantic space of outgoing tickets — that is, how agents are responding — that don't look like templates that are sanctioned by the company, which means that they're coming up with their own templated responses and potentially even sharing those with other agents. So we have a dashboard, for instance, that we show our customers — the ones running the support desk — of potentially new templates that they can use, because obviously if there's a new issue, for instance, with a product, then agents who are on the ground have to figure out a way to answer it, and if it's a recurring problem, within a couple of emails they will end up essentially having the right answer that they've already pre-formulated. So that's an unsupervised problem.

Sam: And do you see in that last example a future place for generative types of AI approaches, or is that more — do you think when you hear that, is that like the technology chasing the problem kind of thing?

Josh: Yeah, it's a good question. We've shied away from the generative component, and in fact we make that a big part of our sales pitch, to say: you, our potential client, really know the voice that you want to speak in and speak with your customers, and who is it for us to come in and say we're going to autogenerate — at the character level, CNNs or something — a bespoke answer, the way that Google Inbox does? Well, if it gives you five different words — sure, sounds good, I'll see you then, or how about Friday — those are fine. But because we're really focused on not just getting results into the hands of agents — where they can actually see, in a UI sense, within the dashboards they normally see, what our predictions are, and consume it in a way that they like to — we also want to take a lot of these tickets off the table in an automatic sense. The only way our customers get comfortable with that is if we're showing to them in an offline way: here's our accuracies for these types of templates. So every now and then somebody says, I'm very unhappy with what you've done, and we're going to send, thanks for your feedback, when it should have gone a different path — but we're only going to do that 1% of the time, at this level of false positives. And once we can do that, then our customers essentially can turn on a specific macro for us to auto-respond without any agents in the loop. To do something like that to potentially irate customers is pretty challenging, so — we certainly won't rule it out, but we certainly don't think about ourselves as producing generative answers in a bespoke way. We're just more or less turning all of our problems into multiclass classification problems of: which of the hundreds or potentially thousands of canned responses is the right one to answer with?

Sam: Okay, and just so I understand the comment that you made a second ago, in terms of sending out a given response a small percentage of the time — are you describing an error type of situation, or are you describing a feature, like an exploration type of feature?

Josh: Good point. There's an explore-exploit component to what we do, in a multi-armed bandit sense, that's typically not exposed or a knob that's tunable by our customers — so that will happen, and some of that will happen naturally. In the case of auto-response, we hold back 10% of the ones where we know what the answer is, or we believe we know with a certain threshold of confidence, and then compare after the fact with an agent who wound up now having to see it, because we didn't auto-respond, implicitly. No — the one I was pointing out is ones where we are essentially wrong. And that gets back to the question of the loss function, of what does it mean to be wrong. If one of the canned responses is, I'm so sorry for your loss, I will refund your entire vacation in the amount of $10,000 — the cost of being wrong on that is very, very high. But if somebody is mad and says, my vacation got ruined because of something you did, I want my 10K back, and we say, thanks for your feedback — being wrong on that side is not nearly as bad as being wrong on the other side of that. And so we give and empower our customers to make the decision about, let's do the easy stuff where the cost of being wrong is not a big deal. And that's for the automatic response, but for the recommended types of responses, if our first canned response is, here's your money back, and an agent looks at that and says, no, that's crazy, the right answer is farther down the list, they'll select that, and that becomes the feedback — our models wind up getting better as they wind up learning over time.

Sam: What are some of the most interesting challenges that you've run into in putting together this kind of hybrid ML-plus-human solution? One of the things that pops to my mind is just user experience, user interface — are there challenges there that are interesting, or what surprised you the most in trying to field these types of solutions?

Josh: Certainly. Because we're getting into the space and the face of agents who do this all the time, when we first started releasing our products, we didn't have a good training program for them. And so when they would see what we thought was an intuitive set of responses in the form of widgets that would show up on their desktop, they didn't know how to consume it, and they didn't know how to use it as effectively as we thought they should. There's all the mundane stuff around UIs, like responsiveness, and somebody saying, well, it doesn't look like your product's working because now there are no responses, and we'd say, well, that's because you've already responded — you're bringing up an old ticket that already has a whole conversation, and we're only getting involved, at least for now, in the first part of the conversation: what's the first response you should do. So then we weren't showing the results — and so how can we modify our widgets so that the agents understand we're not showing it for a purpose, it's not that our system is broken?

And then realizing also that many agents wanted parts of our UI — and UX more generally — that don't have anything to do with ML. So they wanted keyboard shortcuts, because we thought everyone would just click on stuff, but high-velocity support desks want to just use the keyboard. So we had to build that in for a set of customers, because essentially it was like the mouse had a disease on it — they didn't want to touch it. Getting feedback from the UI itself back into our system, making sure that we're getting the right metrics back, making sure that the KPIs that we're measuring are aligned with the KPIs that our customers wanted. I think one of the hardest things for us — and it frankly continues to be a challenge — is really just thinking about how fault-tolerant ML needs to happen. And again, going back to Google Inbox, for those of you that have used it: it makes a couple of suggestions about how you could respond to an email, and if you don't want to use those, you don't use it. So I would call that a great fault-tolerant ML experience. And the same thing in a spam filter within your mail system — it'll say, we think this is spam; if it's not, move it over, and then later on we'll figure out how not to call these things spam anymore that are like that. That sort of fault tolerance, where you're also getting feedback either implicitly or explicitly, is just something we've had to build up over time. But I think more broadly, that kind of approach needs to be built into any AI system in a production environment. Unless the AI outputs that you're building are going to be consumed entirely by machines, you need to have some level of understanding of who it is that's going to be consuming it, what are their concerns, and how can they give you feedback, so that your models will end up getting better over time.

Sam: Can you talk a little bit about the algorithms that you're employing and the toolchain, the pipelines? What does all that look like?

Josh: We stay out of what we call internally the algorithms arms race — we're not really selling the platform to other data scientists, so it gives us the freedom to focus on parts of the pipeline that we find most important. All of our algorithmic learning parts and then prediction parts are built in C++ and then surfaced back out into Python, which is where the data science team winds up working. We have our own notion of what a pipeline needs to be, and the data science team works entirely within the confines of what that pipeline ought to be. Which is: some sort of pre-filtering — so, for instance, if a ticket is from a voicemail, don't predict on it or don't use it for a build; so get rid of those that have, in this column, this value. Then there's the data transformation parts of that, and the joining across multiple data sets if that's needed, and then the featurization, which will often use open source tools in the Python ecosystem — pandas is one we use very regularly. And then once we wind up realizing that we've created a bottleneck — which typically will happen not so much in time but in RAM usage — we'll wind up rewriting other people's algorithms or code so that we create a RAM-efficient pipeline. And then once the featurization happens, the learning winds up happening in the C++ layer, and we've built a whole bunch of hyperparameter optimizations and feature selection capabilities, and then post-processing capabilities to get calibrated probabilities out of a multiclass problem. So we have a bunch of pieces that we've been building up that are not in the open world — it's something that we've decided not to open source for now — that allow us to work efficiently. So we think of it as high-velocity data science in building out a template for the first time.

But then, because the models have to rebuild every single day for every single customer on cloud infrastructure, which is not super cheap, we needed to make the cost of doing that as small as possible. And what we wound up realizing is that open source tools that many people use, like the scikit-learns and the TensorFlows of the world, even Spark ML, were vastly more costly to run — even if you could do it in the same amount of time, which we think we're much, much faster than most of those tools — because of the RAM requirements needed on multiple machines, or even a large single machine in Amazon, the cost of building a model just was X% higher — X being in the thousands. So having a RAM-efficient, speed-efficient, and — obviously, again, getting back to the original conversation about table stakes — highly accurate set of algorithms, which produce the kinds of answers we want, that we could then get into and modify if we needed to, was where we wound up settling as where we needed to spend our R&D engineering time.

Sam: Now, one of the areas that many of the machine learning platform companies have focused on is trying to close this gap between data science and production, and in essence eliminate the, hey, I've got this model that kind of works, throw it over the wall to developers and have it implemented. And it sounds like you guys have maybe embraced that, and you're using that as a way to build out the models in C++, presumably for performance. Are there ways that you've then compensated for that in terms of automation tooling, or do you just accept that, or even — we just have the best people on both sides of that fence that can deal with the existence of the gap? How do you maintain a level of efficiency and innovation in terms of the development pipeline — not the machine learning pipeline — so that it all works for you?

Josh: There's definitely this separation of concerns, which again is both an organizational one and also a computational one, to the level where we often talk about what we call the organizational API of who within this stack is the customer of whom. And so, for instance, the people who are the core ML and algorithms folks in the company are working in C++ and surfacing the great results back into a Python layer. Their customer is the data science team. The customer of the data science team is the people working on our architecture, who have to maintain this scalable, robust infrastructure, and their customers are the people working in the middleware, and their customers are the ones who are in the UI. And so each of them has a set of contracts of what it is that each part of that stack is looking for and how, in fact, they're supposed to engage with each other. And that's become very, very helpful for us, because what you find is that when you put somebody in a box, they figure out a way to innovate very highly within that box. So if there is a very strong contract of what data is expected to come in and what data is expected to come out, and everything in between there is really up to you to decide how to do well and efficiently — that's where, for instance, our data science team and implementation team will wind up working in building out a new template. They can work at their laptop — or glorified laptop — level on a toy data set, get some confidence that the pipeline is working, offline accuracies look good, and the whole thing is going to work. And once they're comfortable with that, they literally are just pushing a new version of a Docker image into our registry, which then is farther upstream from anything they ever have to think about from a production sense. Once a new build winds up getting kicked off for that customer, for that type of template, the new image will just get pulled and it will just get built with the config file for that customer. And so the data science team can wind up working within their confines, and of course we have a whole testing suite to make sure that if they build something new, they're not going to break something downstream from them, to gain confidence in that. And then they're literally just pushing the results of what they're doing on a semi-weekly basis into the Docker registry; that becomes the latest template for, let's say, triage, and then all the customers in production are automatically migrated to that. So having the data science team be able to push stuff into production without having to be on the ops side of things, nor have to think about the architecture, has really freed us up in great ways, I think, to innovate. And likewise, when they need a new bell and/or whistle from the core algorithm folks — because they say this part of our entire build chain is really inefficient — they can ask the people working on that to improve it, and they go through their own testing suite. And I think we're at 300,000 regression tests in our core ML, where we're also testing against every open source algorithm on customer data to make sure that we're staying as efficient or more on all these different axes before we cut a release. Then the data science team can just pull essentially a Python egg from our registry and use that in their system. So having those separations has been great. Obviously, if you're abstracting everybody from what the end use cases are, there can be a huge danger, but it's the job of people like myself to make sure that everyone is focused and innovating towards the right set of goals.

Sam: Great. I'm glad that Docker came up. You guys publish and maintain a set of Docker images for data science tools — I've come across that. My impression is that in general, Docker adoption within the data science, machine learning community is not particularly high. Is that yours as well?

Josh: We certainly haven't heard of many other companies using it in the ways that we are, but it seems like such a natural way to literally containerize and abstract the work of one part of an organization from the other — so long as that container will respond with a slash-build, predict endpoint, feedback endpoint, et cetera, in the way that everyone expects it to. I think that's a wonderful way to do abstraction. And then obviously it also helps you wind up achieving scale, because for us, scale is not, can we serve a billion of our customers with the same app? It's instead, well, we've got a new customer — we just spin up more containers to do the builds and the predicts for that customer, and if we need more compute capability, that's elastically scaling for us for free on top of Amazon. So I think of it as a very natural way to separate concerns from a stack perspective, and also a very natural way to do what is, for a company that's serving lots and lots of customers, a very embarrassingly parallel type of compute.

Sam: Interesting. I got into a conversation on Twitter or Reddit or someplace where someone was griping about just the dependency hell with Python and pandas, and trying to come to terms with managing different versions of different tooling versions and things like that. And I suggested — I might have even pointed to your Docker repository — and the response was, no, I want to make this simple, not more complex. And obviously you find it to be simpler. Can you give folks that aren't familiar with Docker and containers your 30-second Docker-for-data-science pitch, and where they can learn more about it?

Josh: Docker is a way of explicitly specifying not only what your, let's say, Python requirements are, which you can do with a simple file, but also what the entire OS shall be for running whatever scripts you're going to need. And once you build that, and you gain confidence that that image is doing what it ought to, you can essentially very rapidly turn a container on that is the almost instant instantiation of that entire OS, plus that script and all of the dependencies built inside of that. And you can hand somebody a link to the Docker Hub registry — or, if you maintain your own private registry, an explicit URI to that explicit version of that explicit image — and more or less guarantee that when they run that, with whatever data is contained inside of that, or whatever will be pulled over, so long as it's the same, you'll get the same answer out. So I tend to think, from a data science workflow — and then, getting back to just doing science more generally — Docker is a very nice framework for reproducibility. And so the idea is that now I don't have to share a machine with you, or an Amazon machine image with you — I'm just handing you effectively a Dockerfile that says, if you run this, you're going to wind up getting the same answer that I got. But again — doing the types of work that we do at Wise, and in some cases what we do on the science side of things, the final result is not what comes out of the Docker image or container. It's not, okay, here's a report of what my ROC curve is going to be, my false-positive-versus-false-negative curve, and then let me write a paper about that. It needs to be, for us at least in a production environment: now I've produced a prediction that now needs to get consumed by something that's farther downstream. So Docker is quite nice in that sense as well, because you can also now connect Docker containers explicitly, using something like a Docker Compose — there's many other tools out there as well — so that containers talk to other containers, and you allow each container to have, again, its own separated concern from the other ones, but still pull the results and push results to the other ones around it. In addition, some containers can just contain data, and you can build databases around that data. So now it allows you to build up a very lightweight version of what might be your entire stack, and do this in a way that's programmable. So we found that to be incredibly useful for testing purposes.

Sam: So is your GitHub the place that someone can go to learn more about what you're doing there, or —

Josh: Yeah, we've got a public Docker registry — you can go to the Docker registry and search for wiseio — or you can go to GitHub, wiseio, and see our other public projects that we've pushed out. So there's one around Docker and data science, which in that case — because we're not releasing any of our internal tools we've built — we're basically building up a container with open source tools that we find are really useful for doing lots of different types of data science. The other major project which we have up there in GitHub that's open is something we called ParaText, which started as just sort of a weekend hack from one of our engineers, Damian Eads, who wanted to see what it would be like to read data from disk in parallel, just to see what kind of speedups you could wind up getting. And it turns out pretty much every open source tool out there doesn't read in parallel, and the ones that do are explicitly parallelized, like over multiple machines — but if you just made use of the multi-core environment, how well would you do? And we wound up getting 100,000x speedups over some of the other tools that are out there, and importantly, also using vastly less memory. ParaText is not yet in our production environment, but we thought it would be a good example of showing off the philosophies that we try to adhere to within the company: of creating efficiencies that isn't just the one thing around accuracy, but around how fast can you read data, and how big is your model on disk — all these other aspects of what it means to do machine learning that have nothing to do with the algorithm. Once you're happy and you've reached some level of plateau with the algorithm accuracy, all that you're left to do is optimize all these other pieces of that pipeline. And so a lot of our engineering over the last year in particular has moved away from just optimizing accuracy to things like creating interpretability around the models that we build, making the model smaller on disk, making the other parts of the featurization pipeline be more RAM-efficient. And once you start playing whack-a-mole with, let's just say, RAM usage, you wind up finding really interesting parts of your entire pipeline that very few people wind up talking about — again, reading data, which should be the easiest part of your entire toolchain, is vastly inefficient, and whacking that mole, it turns out, you save a whole bunch of Amazon cost, because now you need a smaller-RAM machine.

Sam: That's great. You mentioned interpretability — have you spent a lot of time working on that, and what were the drivers for that?

Josh: We have spent a lot of time on that. It's one of what we think of as our trade secrets, around — getting back to the question of UI and UX for end users — we were asked often, at least in the early days, well, why are you getting the answer that you're getting? And you can't say, well, it's a thousand-dimensional feature space and there's covariance between all of these, and the model importances over the entire thing say that this is the most important feature — I don't know why we said for this one what the answer is. But that answer is what's called in the financial services world reason codes, and turns out to be really important — some places it's actually regulatorily required that you tell somebody why you got the answer that you got, even if it's a machine learning black box. And so some of our early R&D effort was around how to make, at the instance level — so an individual prediction level — how do we make these models interpretable, by saying these are the important features and these are what's driving this specific prediction. As an example, if you're working on customer churn and you want to predict somebody going to churn 90 days from now — it's a use case that we've also used on our platform, but not something we go to market with necessarily — two customers can have an identical probability of churning, but one of them may be churning because they haven't really used your product and they haven't done the training videos, and the other one may be churning because there's a high probability they're going to go bankrupt. In the first case, that's something you can do something about, and in the second case, you're kind of — and so even though they're identical in what their predictions are, and the probabilities of those predictions coming to pass, one is actionable and one isn't. And so it's not just people gaining a warm fuzzy about why did you get these predictions, and does it jive with my feeling about why that could be okay — which is critically important. It also then starts tying into next best action, because, I think again, an important part of machine learning in production is to drive value. If the value isn't the prediction in and of itself, then the prediction in and of itself is really just there to drive the next thing that happens. And so next best action is heavily coupled with the importances around which features are driving the prediction.

Sam: Okay. You mentioned value, and that's a great transition to one of the things that I really wanted to dig into with you, and that is this blog post that you wrote about cost-optimized AI, that I've incidentally mentioned on the podcast a couple of times. Do we have time to go into that?

Josh: Of course.

Sam: So I guess the first — it's actually come up several times in our conversation already, this notion of cost and value, but was there a specific thing that prompted you to, I really got to write this down now? What drove that?

Josh: That was a bit of an intellectual journey. I was wondering, to be really frank, why the hell are all these companies building these neuromorphic chips and all these specialized hardware to do deep learning — because I think much of the world's data and much of the world's value in data is tied up in — I'll use the word, quote-unquote, small data or medium data, not massive-scale, Google-scale data, Facebook-scale data. I was wondering why all these people are starting to build these very specialized pieces of hardware when deep learning — I think magnanimously one could say, or charitably — is incredibly good at a large number of inference problems, but not very good at a large, probably even larger, space of inference problems. That may be changing over time as people start applying it to these new realms, but the place where deep learning winds up shining is in really large amounts of data, because effectively what you're doing is turning millions or even billions of knobs to optimize a model, and to do that credibly without overfitting, one needs lots and lots of data.

So I wound up asking this question of myself: why are people doing this, and why isn't what we already have out there, even just in GPU land, good enough? And if you look at a plot, which I have in my blog post, of the efficiency — gigaflops per watt, which is something of, if I put this amount of energy in, which has this amount of cost, how many computations can I get out — that efficiency has been growing over time, but it's nowhere near what some of these other chips or these specialized pieces of hardware can do for these specific types of calculations. And those themselves are nowhere near what the human brain can do, which is of order, if I remember right, about 10 to the 5 gigaflops per watt. So your brain is a 30-watt supercomputer, unrivaled — at least for now — by anything else that's out there, and anything else that's out there is likely going to take megawatts or hundreds of megawatts to get anywhere close to that computational capability.

Sam: Incidentally, I don't know if you've come across it, but there's a parallel to using DNA for storage, and the storage density per unit energy is incredible in DNA.

Josh: Yeah, something like a drop in a teaspoon or something — it can take all the world's data. It's incredible. So getting back to this — that's an obvious one. And I started thinking about it when AlphaGo had its big set of results, the national or international championships, and you wind up looking at the computational capability that it took to win those competitions — it's just huge: thousands and thousands of computers, thousands and thousands of GPUs. The amount of power required there was several orders of magnitude larger than what was going on in the champion's head that it was playing against.

So I was thinking about that vast gulf, and wound up realizing that the companies that are pushing towards these specialized pieces of hardware — it's because they realize that for a given amount of time and a given amount of data, because these algorithms are all basically saturating on near-perfect answers, the only thing left to do is to get more energy-efficient machine learning for building. And the step after energy efficiency, when it comes down to it, is really cost efficiency. And so my takeaway on that part was that people are building these chips because that's sort of the last frontier of squeaking out and eking out the last amount of dollars coming out of the system for the number of dollars going into the system. And then taking a step back from that, I wound up realizing — or at least realizing for myself; it's probably obvious to most out there — that because machine learning is optimization, your good optimizer will find the optimal answer by definition, and if you're not writing down the function that you're trying to optimize — to get a minimum of or maximum of — in terms that actually matter, then you're creating, by definition, a suboptimal answer or system. And now that system doesn't just involve, is my algorithm more optimal at getting an accuracy better than yours, but now translating the accuracy into — well, let's go back to our loss function: what's the cost of being wrong in saying this thing is A and this thing is B? Translating that to a business term is something that's critical, and almost everybody knows that that's important.

But then you wind up realizing, well, if I'm going to build a model, what if it takes me 12 days to build one of these models, to get an accuracy which is only epsilon better than one that takes me 10 seconds? And what if I can build a model that may take 12 days, and the accuracy is much higher than one that took me less time, but the labor costs are very different? So I had to spend more data science time building one versus the other. And what about the opportunity costs of those data scientists not working on another problem in your business that may be more important? And when you wind up couching the problem that way, you get out of, again, just focusing on accuracy in the algorithm, to what is my cost of doing the entire pipeline. And now the entire pipeline isn't just running a machine learning model in production for this specific use case, but how does that couple to all the other things you're doing in your business? Are you hiring a data science team to do this, and then paying pensions? Or are you going to do a third party to do this and just write a check one time? And then, what are the societal benefits of all this? It becomes unwieldy at some point if you're actually being very honest about what's the cost of doing this. But at least I just wanted people to start thinking about, as we've started thinking about within our company, that accuracy is the table stakes. Let's assume that you all have your good algorithm that's going to do well — is it going to have strong scaling properties, so that if you needed to get the model built in X amount of time, you could just have N number of machines that get you X-divided-by-N amount of time on the clock, because maybe you need that model built very quickly, very often? And then questions around pipeline and RAM usage and AWS costs — in the end, as a small startup, when you start getting down to the brass tacks of what's our revenue and what's our cost of goods sold, what's our COGS — the COGS component is really, what does it cost to build a model and predict? And until we were able to boil down the fact that the cost per prediction for one of our customers is X, and we're going to be making X times some number — everything else is sort of moot. If you're losing, every time you make a prediction, effectively hand over fist, then you've got something wrong — that's unsustainable. So I started thinking about it as we were going through the exercise of, what's our cost of doing business? And the cost of doing business is running an AI system in a cloud with real customers. The labor part we can get, but all the other pieces — in the end, there's an Amazon bill, and because we put it all inside of Amazon, and we know how much money we're taking in, we can see how those two things relate to each other.

Sam: So you started with the question and ran through the thought exercise — what's next there? Whether it's you or someone else that does it, do you see this evolving, or co-evolving with someone else thinking about analytical frameworks for thinking about this, or tools — whether that's a spreadsheet, or — it almost lends itself to a machine learning algorithm trying to figure out how to deploy resources to do the machine learning.

Josh: Yeah, it's a great question. One of the nice things about a blog post is you emit it out to the world and you hope somebody runs with it. It's been helpful in focusing, for me, my own thoughts, and as we drive those sorts of efficiencies in our company — and then, again, more broadly, in doing science, doing astrophysics: choosing the right tools, choosing the right skill sets and people, choosing the right problems to work on or not work on. Those are very obvious outcomes from me having thought about it and framed it that way. One of the challenges — and I think people may wind up being able to pick pieces of this up and work with it — is coming up with articulations of, essentially, what is the conversion term between that item in the entire optimization equation and dollars. So one I'll throw out there that I don't know the answer to is: what's the dollar value of interpretability? And once somebody starts getting some handle on that, then optimization takes its wonderful toll or approach, or at least shall lead to a great outcome, which is: once you can really put a dollar cost to all these different pieces, then I think you can do a real honest-to-goodness optimization. So I know what the dollar cost is, for instance, of needing a RAM machine of this size versus that size on Amazon — okay, great. But what's the real dollar cost of — and can I know — how much time it's going to take for a data science team to build up this template from scratch and then push that into production? And how many people do I really need on that? Is it good to have one data scientist or multiple ones? And so all those things I think wind up becoming really interesting over time, once people potentially even wind up agreeing upon what's in bounds for this equation and what's not. Obviously out of scope is, what's the probability that my machine learning algorithm is going to start World War III — probably not worth talking about. But something smaller than that, smaller in scope at the company level, is probably worth starting to get some clear understanding around.

Sam: So now we've maybe come back full circle to graduate students. Sounds like there are a lot of interesting research questions in here for a PhD student or something.

Josh: Yeah, I think for those in computer science thinking about systems optimization who are also interested in machine learning, this is hopefully some fertile ground to start thinking. The other statement, which hopefully is clear from what we've been talking about, is that doing machine learning for machine learning's sake really doesn't make sense. It's probably the last thing you want to do if somebody hands you data — you do it because you have to do it. It's painful, and to run it in a production environment, given all the crazy bugaboos that many, many people have talked about — there's a great paper from folks at Google — D. Sculley is the first author — called Machine Learning: The High-Interest Credit Card of Technical Debt.

Sam: That came up on my last interview as well.

Josh: Yes, I'm not surprised. It's an important paper. It's got, I think, no equations in it, but it's a whole bunch of important lessons about how machine learning systems tend to be very different than typical engineering systems. So there's a lot in there to get right, a lot of bugaboos there that people who haven't done this before tend to get wrong. But what you wind up realizing is that once you realize machine learning, or broader AI, is the right set of tools to apply to the problem that you have, what you'll often wind up finding, I think — at the graduate student level, in terms of graduate student projects that could be worked on — is that it's still very much early days for the types of algorithms, pipelines, et cetera, in dealing with real-world data. I've often said to my colleagues on campus that real data is not doing sentiment analysis on Twitter — and yet many, many, many papers saying my scaling algorithm is better than your scaling algorithm will wind up using that as a toy data set. The real world is not toy data sets. Yes, we need to have benchmark data to have a lingua franca of who's doing better on these different axes, but when you wind up getting exposed to real questions, you wind up realizing that all the stuff that people know out there in the academic world, that people write about and do Kaggle blog posts about, are not what you really need if you're being truly honest about what needs to get optimized.

Sam: That's great. So how does one manage being CTO at a high-growth startup and being an astrophysics professor? It's becoming increasingly common to see folks, particularly in the machine learning community, have professorial posts and do academic work, or do work in these research labs and things like that, but — yeah.

Josh: I've been on what's called an industry leave for a number of years, and so it's allowed me to have also that separation of concern. So not getting paid by the university, not having health care, has made it easier for me to spend all my time as need be on the company, while still maintaining the kinds of links that I think are important. As I started thinking about coming back into the university setting — obviously there are a number of things I've picked up in management, ideas and capabilities, and then also thinking about how to evaluate new technologies: when is it appropriate to bring this into your toolkit, or when is it appropriate to wait? Those become really practical uses that I can take with me. But then also, again, recognizing, as I was saying before, that there's a whole interesting set of problems out there that are not being addressed by pure academic R&D research means that I can also start looking for those white spaces to actually do some pure academic research around those. I'm particularly interested in questions around interpretability and how you put metrics on interpretability, and that's something that I think I benefit from — having come from, having felt the pain of, customers asking about that — that I at least have a fresh lens on that. It doesn't mean I'll solve any of those problems, but at least I'll have a direction of potential interest. So it's certainly a challenge, but I think despite the challenges, the benefits to both myself, the company, and the university and my students at the university far outweigh all the gray hairs that I wind up getting. I'm teaching a data science class — essentially a Python-ecosystem data science class — right now. It's aimed at graduate students, and the things that I've seen in the business world have really helped me hone that class, and I'm directly giving back to the students from those learnings.

Sam: And is that a MOOC, or is that available only to —

Josh: It is not a MOOC. Other incarnations of that class that I've done in the past are probably online somewhere in the iTunes sphere or elsewhere. All the material can also be found on GitHub, and then we'll hopefully post some of the lectures online as well.

Sam: Okay, great. So if folks want to learn more about the company or get in touch with you, what are the best ways for them to find you guys?

Josh: Easiest is drop me an email, and you can find that by Googling around. So I'll add that as a little bit of a bar: if you really want to find me, you'll have to do a little bit of work. You can tweet at me — profjsb is my Twitter handle — and we can do a direct message. Maybe that's probably the best way to get at me.

Sam: Right, great. Well, I really appreciate you taking the time. It's great to finally meet you in person, and I really enjoyed the conversation. I think folks will enjoy it as well and get a lot out of it.

Josh: Great. Well, thanks so much. Thanks for your interest.

Sam: Great, thanks. All right everyone, that's it for today's interview. Thanks so much for listening. I haven't asked you all to do this in a while, but if you enjoyed this episode of the show, please, please, please do these two things. First, share it with your friends on Twitter, Facebook, good old email, or however you share cool things with your friends. Second, reach out and let me know how you like the show, who you'd like to hear on it, and how I can make it better for you. You can reach me on Twitter at twimlai and at samcharrington, and you can email me directly from the contact page on the twimlai.com site. Thank you so much for your support, and catch you next time.

Machine Learning (lecture)

Tue, 30 Aug 2016 00:00:00 +0000

Tutorial lecture on machine-learning methods for astronomers (classification, regression, feature engineering) at Astro Hack Week 2016, hosted at UC Berkeley and recorded by BIDS.

Exact day within Aug 29-Sep 2 approximate.

Key Quotes

“Start thinking about machine learning as just another tool in your toolkit for doing inference… There's a time and a place for applying different tools.” – Joshua Bloom

“There's been this almost Cambrian explosion of different algorithms and approaches… it can often be very noisy and very difficult to figure out what's hype and what's actually very useful.” – Joshua Bloom

“You have to look at your data before you start throwing it into these frameworks. Just because these frameworks are nice and easy to use doesn't mean that you're allowed to get away with not actually looking at the data.” – Joshua Bloom

“If you get a perfect classifier and you see a confusion matrix that has no power off the diagonal and all the power in the diagonal, you've overfit. I guarantee it.” – Joshua Bloom

“There are lots of ways to hammer your thumb with your new tool. And you may think you're building this amazing house, but in the end, you've built complete crap… Machine learning is fraught with places where you're introducing biases that you didn't know about.” – Joshua Bloom

Transcript

JOSH BLOOM: I'm Josh Bloom. Hi. Welcome to UC Berkeley. Welcome to Doe Library. Welcome to the Berkeley Institute for Data Science. And welcome to day two of Astro Hack Week. I hope you had a wonderful day yesterday. Today, we're going to talk about machine learning. I wanted to try to put machine learning into the context of an inference space that's certainly not rigorous, but one that hopefully helps you level set a bit relative to the types of things that you were hearing yesterday, and to give you a sense of how I want you to start thinking about bringing machine learning into your daily work.

What I should say from the outset is: start thinking about machine learning as just another tool in your toolkit for doing inference. This is not like, oh, now that I know machine learning, I'm not going to do Bayesian statistics, or I'm going to stop learning physics because, boy, machine learning is so awesomely data-driven that I don't need anything anymore. There's a time and a place for applying different tools. And what I'll teach you today is some of the at least Pythonic ways of approaching some astronomy types of questions around inference using machine learning.

But let's unpack this a little bit. I've got three axes. I've drawn them as orthogonal from each other, but for those that are reading ahead, you probably see that they're probably not that orthogonal. One is the statistical space, which is going in the left and right direction, Bayesian on one side and frequentist on the other. While I didn't see all the lectures yesterday, my prior knowledge of the two speakers suggests that you probably got beaten with the Bayesian stick pretty hard. As it turns out, most of the types of machine learning tools that we're going to wind up using are much more on the frequentist side of things. And those that, again, are practitioners of these various statistical frameworks for thinking about data and inference will often refer to the Bayesian-frequentist divide as really the wave-particle duality, that it's really both. And depending upon the situation, it's more helpful to think about data and inference in one place or another along that axis.

In the other axis, up and down, that's much more where you bring physics to the table or not. One is what you might call theory or hypothesis driven, where you've started from first principles, and you've derived a model that should lead to some set of observations. And that's the sort of hat that you wind up putting on. The other hat, which is the opposite end of the spectrum, is one where you say, I don't know anything about the system that I'm studying that I'd like to make inferences on, so I'm just going to throw the data into some framework and hope that I get the kinds of answers out to the types of questions that — because you've taken this hack day seminar — you've asked them correctly, in a well-posed way, although oftentimes, you'll find that people will be challenged in that respect.

And then in and out of the page here is the computer science view of this world, which is: are you doing the computation on a small enough amount of data that it can fit into RAM? Because if you can do that, then you get access to lots of really interesting algorithms that don't need to be that well parallelized, versus ones that are potentially out of core, where you're taking chunks of the data, and you're doing inference on chunks of the data at a time. And over time, you're building up a better and better inference about the questions that you're asking.

So this is the space that you should be thinking about. Many of you have been doing inference already, or you probably can localize the types of problems and approaches that you've taken in this space. And again, this is not going to necessarily be orthogonal from each other. I just wanted you to see the stats view, the computer science view, and the physics view of this. Yes?

STUDENT: Almost all astronomers have generally worked out of the board in the lower left, and Facebook is into the board on the upper right. But it's not clear to me that many of the occupants of this are actually filled.

JOSH BLOOM: No, that's true. It's like the fundamental plane of galaxies or the HR diagram. Just because you can draw temperature and luminosity doesn't mean that they're going to wind up populating everything equally. So there's definitely places of higher density versus lower density. That's a great exercise for us or for you to come up with some interesting problems that live in those different quadrants. I have got some thoughts on that, but I don't want to spend too much time. But you're absolutely right.

And that, in some sense, emphasizes one of the critical points, I think, of this whole morning, which is that you're going to choose your inference framework depending upon the types of questions you're asking and the type of data that you have access to. If you're at the Facebook level, you have access to, of order, several billion images. And so if you're trying to do insight into images and you can actually afford to get lots of labels, that is, answers for your previous images or videos, then you can bring a framework together that learns a lot from data, where you don't have to know much about, let's say, object detection. You can just throw it through some whole system that figures that out, like a deep learning network. Whereas in astronomy, oftentimes, if you're working right at the edge of signal to noise, you'll wind up not using machine learning because there, you would really benefit, essentially, pulling the signal out of the noise by bringing all your prior beliefs, both from a statistical perspective and also from a physical perspective.

Just a quick overview of what I'm hoping to cover today, and there may be a little bit more we can get into as well. What is machine learning? Obviously, we'll talk about that in all its gory details and cut as quickly as we can to the chase to try to do this in the most Pythonic way that we can. Go into two types of approaches and questions you'd ask of the data. One is regression, getting numerical values out of your data, essentially from a prediction perspective. We'll do a little bit of a runthrough on an existing notebook that I've created. And by the way, that notebook is up on the GitHub for Astro Hack Week, so if you want to follow along, feel free. And then classification, which is saying now I'm not trying to get some inference at the numerical level; I'm trying to understand which class an object winds up belonging to. And then we'll talk about how you actually improve your model. The first parts will be just introducing the topic, giving a little bit of theoretical motivation for those. But mostly, I want to get into what one needs to do as a practitioner. And then we'll start thinking about how to get these into production environments.

Most of what I'll be covering is what's called supervised learning. This is where you have a set of labels or answers on an existing set of data. And now you have a new corpus of data which is presumably taken and obtained and reduced in the same way as your original data, and then you want to ask the same sort of questions on that. So if you had a galaxy sample of, let's say, 1,000 galaxies that all had spectra, and you had your own classification scheme for what types of galaxies those were, and now you want to apply that to all the spectra from Sloan, this would be an example of a classification problem, but also a supervised one. So I'll make the distinction between unsupervised and supervised as I go along.

All right. So what is machine learning? The short answer is it's an offspring of statistics and computer science. The long answer is that it's a set of models which aim to learn something about a data set and apply that knowledge to new data. There's lots of different ways to unpack that, and there's lots of different ways to try to localize machine learning as something that has some boundaries to it within the broader context of artificial intelligence. What's started happening over the last couple of years is that there's been this almost Cambrian explosion of different algorithms and approaches, and commensurate with that are lots of books about each one of those and lots of tweets and people writing blogs about how awesome their algorithm is relative to other people's algorithms. One of the main difficulties, I think, not being a practitioner in machine learning and not having come through the ranks of having learned it from first principles — which I put most of us in that category — is that it can often be very noisy and very difficult to figure out what's hype and what's actually very useful.

The short answer on that point is don't believe the hype. But you probably should believe the hype at some level, because there's a lot of really powerful tools out there. And a big job that you're going to wind up having is trying to cut through that and trying to understand which are the tools that are useful and practical, and which are the ones that are highly specialized and really only need to be used in very specific cases. So it's the difference between a generic hammer that you would buy at Home Depot and some very specialized tweezer that can only open up a Nexus 6 or something. Very, very different things. One of them you absolutely need to have when you need it. The other one, though, is much more practicable and useful in your life.

Let me just step through these various components of what machine learning is, and then I'll break that out at an even higher level for you into different types. So it's using labels from training data to classify new objects. I've just given you a light curve. Is this a supernova or a nova? Domain experts can look at those and go, I know the answer. But if you want to do this at scale without any people in the loop, then that might be a good thing you'd want to use. I have a bunch of images. What's the galaxy type of these images, et cetera.

Learning the relationship between explanatory features and response variables to predict new data. What does that mean? There's a couple of terms in here that we need to get straight. Features are the variables. They're the x variable from what we saw yesterday. This is the input data. Oftentimes, we think of x in an abstract way. X could be an image. X could be a set of images taken over time. X can be metadata that's derived from those images. In the context of astronomy, it would be, let's say, photometry on a single object as a function of time. It can be photometry as a function of time on a given object, but it can also be the weather conditions at all the telescopes where that happened. This is all the data that could potentially get brought to bear on predicting this response variable, or what's sometimes called labels. And that's the y vector from yesterday.

Another utility of machine learning is its ability to discover natural clustering in data. Obviously, if you have lots of two-dimensional data and you plot it all up, your eye is going to wind up picking out clusters. You could draw circles around those and write papers about the objects in that circle. And that may actually get accepted to journals. And that's completely appropriate. But there's a more robust way to come up with the notion of clustering, even a two-dimensional space. There's obviously some parameters that define what it means to be a cluster or not that you'd have to decide based on your problem. But where machine learning tools wind up becoming really useful in this context is its ability to find clusters in very large dimensional spaces. If you have not two dimensions or three or eight, but 10,000 dimensions that you're looking at, and you're trying to find clusters that actually have meaning, you can use a bunch of different tools for that.

And similar to that idea is the idea of taking a very large dimensional set of data and reducing its dimensions down to just a few that are actually informative. A classic example of that in astronomy would be taking a bunch of galaxy spectra. While every single flux value at every single wavelength has its own meaning, and you can interpret that in a physical way, there's only a small set of representative spectra of galaxies. And every galaxy, in some sense, can be represented by that. If you've done principal component analysis on spectra, you've seen this, where if you take lots and lots of Sloan spectra at the same redshift, you can wind up picking out just a few representative samples. And then you can get an admixture, even in a linear sense, to build up most of the signal that you see in new galaxies. So that would be an example of taking a very large dimensional space — in this case, flux as a function of wavelength, potentially thousands of measurements — and condensing it down to just essentially a few weights on eigenspectra.

I should say, that particular example of PCA is where you start blurring the line of what is a machine learning technique and what's not — PCA is in the gray area there. Oftentimes, PCA — and this is a bit of an aside — will be used for doing dimensionality reduction on very large data sets with a large number of dimensions. And then the output of that dimensionality reduction would then be thrown into a machine learning framework. So in the context of, let's say, galaxy spectra, you might take the weights on the top 50 principal components to reproduce the spectrum that you saw and throw that into a more traditional machine learning classifier.

And last, because you can do clustering and you can find things that are like other things, the flip side of that is also true: you can find things that you haven't seen before. And anomaly detection is a very important thing in astronomy.

What I was going to try to unpack a little bit here — these are all the different components of what makes up machine learning. There's another way that I'd like you to also start thinking about what machine learning is and what it isn't, and that's in the types of questions that you wind up asking on the data. I'll call one type type 1, for lack of a better term, and that might be called exploratory. And the other one, which I'll call type 2, might be predictive and prescriptive. So why am I making that distinction? Exploratory in the context of, let's say, anomaly detection — that's a great example of what I'd call a type 1 activity with machine learning. You have a large amount of data. You want to find something you've never seen before. And then you want to look at that and say, wow, that's interesting, I should get more spectra of that object, let's keep going. And eventually, that could wind up leading to a paper.

It could also be that it's exploratory just visually. So this is a way of, let's say, taking very large numbers of dimensions and bringing it down to a smaller number of dimensions so that you could even visualize and play with, to get a better sense of the data and the low-dimensional structure that is effectively embedded in larger dimensions. I don't think of exploratory, in the context of lots of types of machine learning that's out there in the wild, as all that useful, because it doesn't oftentimes give you a tremendous amount of insight about what you need to do next. In the context of Facebook, the reason why they're building new frameworks for doing inference on videos and images is not because they want to learn about when somebody is happy, are they wearing this color or that color, and they're going to write a paper which is going to get accepted in some journal. They're doing it because they want to make predictions about what they can actually get out of that image. And then ultimately, they want to monetize the results of that.

And it's prescriptive in the sense that the best types, I'd say the more leading types of machine learning, are ones that say, not only do I understand what happened here in this data that you've just acquired, I'm now going to make suggestions about what new data needs to get obtained in the future to get to some objective function. And so while all of machine learning is an optimization — and as long as you can start couching your frameworks around optimization, at some level, you're doing some form of machine learning when you include data — I think the type 2 are very, very interesting.

In the company that I started, it's called Wise.io, we're working on more of the prescriptive kind, where we're looking at interactions between people, and we're making predictions about what they're saying and then making suggestions about what could happen next. What's nice about that type is that it winds up creating nice feedback loops, where if somebody takes an action that you didn't suggest, you wind up getting counterfactuals to your models. And then the models themselves can wind up improving over time.

I actually think while this is very exciting and there's lots of classes of machine learning frameworks that are being developed here, a lot of what we do in astronomy is much more on the exploratory side. Going back to this example of classifying large numbers of galaxies, that is saying, I'm going to build up a catalog of galaxies of this type. I've got a couple of exemplars from that space. I want to find more of those things. So you go and you find more of those things. And then it's really up to you to decide what to do with that. Not baked into the optimization technique traditionally is what you need to do next. So you're using it as a starting point to say, aha, I just found some interesting data that I didn't know existed there. And now I'm going to use that, because I understand the physics of what I'm looking at, to go off and do something interesting with it.

Is that clear on the distinction? By the way, it's obviously not a very hard and fast, black and white set of rules. I just wanted you to start seeing the two differences there. Any other questions about overall statements that I've made about what machine learning is? Yeah.

STUDENT: Maybe you said this and I missed it, but couldn't we say that detecting low-dimensional structure or lower-dimensional structure in high-dimensional data is all of these things?

JOSH BLOOM: The question was, couldn't you say that detecting low-dimensional structure in high-dimensional data is all of these things? I don't quite understand the last piece of that.

STUDENT: Well, all of the other stuff up there you could rephrase in a way that makes it look like detecting low or lower-dimensional structure of high-dimensional data.

JOSH BLOOM: Yeah, it's a good point. At some level, you've got a function on your input data, and you want to predict something else. This is a generic statement of making models on data to predict outputs. It could be that I take a very large dimensional data set, and my output is three numbers. And that's my way of reducing it down to something that I could potentially visualize and play with. It's not the same as saying, what class do I belong to? Just because I can take a 100-dimensional data set and then show that it's got some actual structure in a two-dimensional projection doesn't mean that I know what class of galaxy that is along those two dimensions. So you're right in that machine learning is a summarization of the data, and it is a compression, in some sense, of the data to get out an answer that you care about. But they're not identical. Any other questions? OK.

We're going to now start looking at the landscape of machine learning capabilities within Python. For those that haven't already, you can do a conda update scikit-learn, as you see up there. If you're using a different package manager distribution system, you can do a pip install scikit-learn. I believe the most current version is 17.1-1. It's probably fine if you have 17 and beyond. Some of the things in the notebook might wind up breaking if you're at 15 or 16. So I'll give you a second to do that.

While you're getting your environment up to speed here, what I will say is that there's a large number of packages that exist, even in the Python sphere, that do some parts of machine learning. There's mlpy. Orange, which has been around for quite a long time, is much more visual. Keras is a wrapper around a couple other machine learning frameworks, and particularly deep learning frameworks, and that gives you a very high-level view of deep learning. Nolearn is a competitor, at some level, with Keras. And then there's some of the lower level things, like TensorFlow, et cetera, that give you a little bit more direct access to some deep learning frameworks. And then astroML. There's a whole bunch of these things that are out there.

By far, I think it's fair to say scikit-learn is the de facto starting point for doing any machine learning in Python. And that's because of a number of things. One, it's very well-maintained. It's got something like 20,000 commits at this point, hundreds of committers. And they have a very well-documented and really sharp-elbowed API system. So you can wind up writing an entire pipeline against one type of machine learning model framework and then essentially rip and replace that out and stick a new one in once you wind up realizing you want to try something else, and not break a whole bunch of the rest of the code. Any questions about Python, machine learning frameworks in Python? Anyone else beg to differ? OK. This is a fairly complicated slide from the scikit-learn folks of what they call the cheat sheet for doing machine learning. I'm asking you questions about what type of data you have and what kind of approaches you're going to take, and then what's available within the scikit-learn world. The first one is, do you have more than 50 samples? If not, go get more data. So stop. That's not completely fair, because of course you can do inference on just 50 data points or 50 instances. Are you predicting a category? And do you know what the label is? If you're predicting a category, you're going to do some sort of classification. If you're predicting a response variable quantity, you're probably going to wind up doing some type of regression. Do you have large-dimensional data? Et cetera, et cetera.

I'm not going to flow through all of this, but what I will say is that once you've arrived in each one of these different boxes, what scikit-learn gives for you is a whole bunch of choices of the different algorithms that you can use. And one of my fairly strong statements that I'd be happy to be challenged on is that there's only a few algorithms out there now that are so battle-tested and so useful, that you probably don't need to learn nor try all these different ones once you've arrived in one of these different boxes here. That is to say, if you've got low-dimensional data and not a lot of it, some form of a linear model is probably fine and appropriate for your data. If you've got a medium amount of data, let's say at the gigabyte level or even the terabyte level, something like a group of decision trees, something called random forest, will almost always get you to the best answer possible. And then if you've got much more data than that, typically you'll wind up using some type of deep learning to get access to the best results.

It should also be clear, because all of you do do computation in one way or another, that the best answer, the one that essentially maximizes the objective function — that is, one that gives you, let's say, the best accuracy in a galaxy-star-quasar separator — isn't always the best one to use. So anyone want to come up with some arguments why the best accuracy algorithm might not be the one you'd want to use? David?

STUDENT: It might be outrageously computationally expensive.

JOSH BLOOM: It might be outrageously computationally expensive. And Dave talked about yesterday the trade-off that you have to make as scientists between what's the right thing to do and what's the tractable thing to do, because you need to publish. If it takes the age of the universe to get 2% better accuracy on a star-galaxy separator, and you have to burn all of your campus's resources to do that, you're probably not going to do that. So computational efficiency, efficacy, is an absolutely important one. What else? I'm going to save these. So why not use the most accurate? We said computational efficiency. Yeah.

STUDENT: You overfit to your input data.

JOSH BLOOM: Good — overfitting. Given a sufficiently large number of tuning parameters, I can fit a sufficiently arbitrarily large amount of data. But then that doesn't mean it has predictive power on the data that I haven't seen yet. So that's a great thing of what we want to protect against. Yeah.

STUDENT: Just building on that, accuracy might not be your goal.

JOSH BLOOM: Yep. So accuracy might not be the right metric. Indeed, your goal could be to minimize the number of false positives. A great example of that: if you've got a drone that's flying around, shooting down other drones, or airplanes shooting down other planes autonomously, you probably don't want to have any false positives, and you start shooting down your own people's drones. You want to shoot down somebody else's. I don't know, I'm coming up with examples. That's an example where accuracy is really important, but it's not the most important thing. There, you'd want to create a model that drives false positives down to zero. In the context of false negatives, if you're building a classifier on images as it streams off of telescopes, and you want to find things in the sky that are new and novel, you want to minimize your false negatives. You don't want to miss anything that's new and novel. But in that case, you also want to not find and say everything is interesting. So the two extremes of that, in the context of, let's say, a transient classifier that's happening in real time: one would be everything in the sky is interesting right now, and the other side of that would be nothing is interesting in the sky right now. And both of those are horrible systems in production.

What other reasons might you not want to use the most accurate? Is the most accurate the most informative? That is, if I have a classifier which just beats all the other classifiers, and then I look inside of it, and I say, I have no idea how it's combining the data — and yeah, I can see the math, but I don't get any insights out of that — for astronomers, that could be a pretty big deal. Unless you're working in a regime where the proof is in the pudding — that I don't care how you got the answer, just show me that, in a calibrated sense, you get essentially the best, most accurate curve or the perfect metric — very often, you're never going to be in that space, because you have to write a paper eventually about the things that you've built. And somebody's going to ask, well, why did you get to where you got? So they may not be all that informative.

All of these are reasons that you'd want to take a bit of a pause when you meet somebody on an airplane and they say, I work at startup x or a large corporation in the valley y, and we have the most accurate classifier on something. And you'd say, well, that's great, but it's like when you talk to an astronomer in a field that you don't know, and you say, what about magnetic fields? And they go, aha, magnetic fields. These are the things that you want to keep in mind when you're actually working with real frameworks, recognizing that overfitting isn't just what could potentially happen when you work with one model. You could wind up doing some sort of meta overfitting, where you choose a model that happens to look best across multiple different types of models. And that's because you've gone through effectively a multi-trials problem, and you've essentially glommed on to something that's best, but within the noise. Did you have a question?

STUDENT: Yes. Going back to that you wrote on the board that as predicted was your (INAUDIBLE). So out there in the tech industry, how much work goes into understanding the systems they work with, compared to just making models that can predict new data and enable them to place ads and (INAUDIBLE)?

JOSH BLOOM: So the question is, out there in the non-academic world in industry, for the type 2 kind of predictive sort, how much do they care how much they can unpack an understanding of what their models are producing? I think the answer varies. My guess is that in the ad world, in the ad placement world, it doesn't matter, because there again, the proof is in the pudding. If I get epsilon better than somebody else, that means millions or billions of dollars in accuracy. And boy, I can throw the whole kitchen sink at that because I can afford the computation to do it. Again, in the context of understanding whether you're happy or not, to suggest a different ad to you inside of Facebook, I suspect there are people within that organization that do want to understand what it is that their models are doing, and what type of data do they need to acquire, what type of data is informative, what's not, for the purposes of making a better prediction — but the end goal is not to actually understand why.

I think in the context of, let's say, financial models — let's say a risk assessment model — there are plenty of reasons, the most important one being regulatory, that when you deny a loan to somebody algorithmically, you have to say why you denied the loan. Just like with your FICO score, which is a terrible model of your ability to repay a $10 loan from your friend in two days, you can actually figure out exactly why you got the score you got. There's a regulatory requirement around that. There are regulatory requirements around making some types of inference models essentially completely transparent of how they got to that answer. I believe the EU just started passing a law, or they've already passed a law, that's starting to make some of those actually required to be transparent and then actually known by the public, of why they got the answers that they got.

I think the informative component of machine learning models is vastly understudied in the academic circles. Most people are focused entirely on, is it more accurate than somebody else's on the same data? Does it scale better by a couple of different metrics and something else? And very little of it has to do with interpretability. And I think that's a fantastically interesting open subject.

OK, so supervised learning. We're going to use a set of training pairs — so this is an x vector and a y outcome — to predict new y outcomes when we see new x's. In the context of regression, that's predicting a continuous variable, which will be our y, from an input set of features. Lots and lots of things and lots of different approaches exist within Python and scikit-learn, from linear regression to very nonlinear model capability.

All right. Why don't we jump into the notebook now. I'll mirror displays and see if we can get this to work. If you go into Astro Hack Week, Astro Hack Week 2016, and then you click on day two-machine-learning, you should see the latest version of the notebook. I'm going to do this and make this a little bit bigger. And we're going to do some astronomy examples. I adapted a previous notebook from Python 2 to Python 3, but I try to make it backward compatible.

Just show of hands, who's using Python 3? OK, so most of the people in the back. And then who's using Python 2? Most of the people in the front. That's interesting. I did some clustering in my head — which could be wrong, might not be right. That's actually, by the way, one of the problems that you generally will wind up having with unsupervised problems, is that how good you did tends to be very subjective. And with supervised problems, you can actually define a metric that you can test yourself against.

STUDENT: (INAUDIBLE) that result. That result was pretty good. You should take a photo of that and see if it's (INAUDIBLE).

JOSH BLOOM: OK. And now you can't change your result. And you can raise your hand twice, because you might be using both kernels.

STUDENT: Yeah, so who is a Python 2 user? OK, who is a Python 3 user?

JOSH BLOOM: Oh, it's more right than it is–

STUDENT: (INAUDIBLE) OK, great.

JOSH BLOOM: OK, good.

STUDENT: That's ridiculous. [LAUGHTER]

JOSH BLOOM: But how good is my pattern matching? Boy.

STUDENT: Going to find (INAUDIBLE).

JOSH BLOOM: Yeah. All right. We're going to work some examples using the Sloan Digital Sky Survey data. Again, this should work in both Python 2 and 3. I wanted to grab photometry that's corrected for extinction for 1,000 quasars that have known redshifts. As you can all probably do SQL in your head, you can see where this is all going and what this gets. But we're going to pull over 1,000 quasars from Sloan. And you should be able to do that locally if you just run and execute this — oh shit. I'm running it on demohub just because I don't want to use resources on my laptop. If you run this, you should get the data.

What does this data actually look like? Probably appropriate to look at it. It's not getting bigger. So here, we have the object ID, RA, dec, dereddened u-band photometry, dereddened g-band, et cetera, et cetera. Somewhere, we've got a — we don't have a redshift here, or we do have a redshift? Thought I pulled over redshift. One of these is redshift. We've got RA, dec, et cetera. OK, so that's the data. We just now pull over some other stuff you need — pandas, seaborn, matplotlib stuff to actually do plotting — and then pull over scikit-learn. I don't think I need to run this, but I'll just try it.

Does everyone know what demohub.jupyter.org is? It's a site that is built and maintained by the Jupyter collaboration, where one can have not ephemeral notebooks, which you can get from try.jupyter.org, but actually dedicated computation space. So you can do a git clone of a repository there and then pull in all of your stuff and then just have it there. And those computers, in principle, will run forever, and your servers will run forever. That's obviously not true, because it's maintained by a third party. I think it lives on top of Rackspace. But the whole machinery for building these things is actually now out and fairly robust, obviously open source. So for those of you that are teaching, it's a nice thing to be able to build a space for you and your students to all work together that's in the cloud. This notebook, obviously, I could be running from my own machine if I'd like, but I'm just running it from the cloud now. And that'll probably come back to haunt me because I just extolled its virtues.

So what are we going to do? We're going to import numpy, matplotlib, pandas, seaborn, et cetera. And we'll take a look at that data. Just doing a head on the first 1,000 quasars. And you see all the stuff that I got out of that. Oh, here's the z. Spectroscopic redshift is spec_z. And here's the class. Now, we're not going to just do machine learning on it. One of the things that you have to know is that you have to look at your data before you start throwing it into these frameworks. Just because these frameworks are nice and easy to use doesn't mean that you're allowed to get away with not actually looking at the data. So let's look at the data.

What are we going to do? We're going to pull only some of those columns out, because we're going to wind up doing a regression problem or, later on, a classification problem. I'm going to drop something like right ascension and declination, because I want to not care where you are in the sky if I want to decide if you're a quasar or a galaxy or a star, for instance. This is a great example of bringing prior knowledge to this problem. It should be the case that if you build a nice classifier or regressor that included right ascension and declination in the data set, that you would find that those features, those values of x in that vector, have no informative value and no meaning. But because they're in there, and because they're going to wind up not being distinguished between something else that may be much more informative, putting on our domain-driven hat, we have to allow ourselves to use domain knowledge just to make our models more understandable, more tractable, and actually have some meaning.

So we're going to pull out the object ID. We're going to pull out the spectroscopic redshift. We're going to pull out the different colors, et cetera. And then we're going to make features, which is just a copy of that whole data frame. And then we're going to get redshift. We're going to try to build a predictor of redshifts using just photometry data. And we're going to delete, out of our feature set, the answer, because that would be pretty bad if we had the label in the thing that we're trying to use to then predict our y values. And then let's see what the result is of that.

So this is our features. And now we've got a smaller number: one, two, three, four, five, six, seven, eight, nine. And some of these features, by the way, are the difference between aperture photometry and Petrosian photometry, which could be informative for the size and the shape of the galaxy at some level. I'm not taking all the other metrics that are obviously also available. I'm just trying to do something simple here. So we've reduced this down to a nine-dimensional input y vector. And for those that are used to pandas, I've created an index around the object ID.

And now, let's plot the histogram of the output variables. And what you see right away is that there's very little numbers of quasars beyond redshift of 2 in the Sloan catalog, or at least of the ones that I wind up picking out. So there's a pretty big cut around 2. So you can imagine that when I create a regressor, if I don't give it any other information, it's going to try to do a very good job around the places where there's lots of data. And it might not know that actually, what I really care about is finding large redshift quasars. So again, this would be how you would wind up constructing the model. You'd have to think about that.

So let's plot the data against itself, pairwise, every single feature against every other feature. Let's just see what looks informative. And we'll colorize that by redshift. This takes a little while to produce here, so give it a second. Sorry to Stefan, who's sitting over there, because I'm using Jet and not your color map. There we go. OK. So there it is. And it's going to be very hard to see here. We've got dereddened r, and it's just a histogram of that down the diagonal. And you can see indeed that there is correlation, heavy correlation, between the difference in the u-band between the two different photometry methods and the g-bands and the i-bands and the z-bands. Those ought to be correlated with each other, and indeed, they are.

You also see something kind of crazy here. What is the dereddened r-band magnitude? It looks like it's mostly around zero, but some of them are around minus 10,000. This is like somebody setting off a supernova in your eye or something. So that, obviously, isn't valid data. And we just realize that there's something crazy with the dereddened r-band magnitude. Some of these actually are not detected in r-band, and we can decide what we want to do with those. Again, here, we're not even doing any machine learning. We're doing what's called feature engineering. We're going through a process of understanding our data. We've articulated the question that we're going to wind up asking: can we predict redshift from just photometry on quasars?

And what we'll now do is we'll say, look, I'm just not going to care about all those ones that don't have r-band magnitudes. Maybe those ones would be really easy to find what their redshifts are. But just for the sake of this demo, if the color is sensible and it's actually a sensible magnitude — that is, it's not this minus 999 — we're going to keep it. Otherwise, we're going to get rid of it. So now we can look at that matrix again and make sure that we get what looks like pretty sensible answers. And then once we're done with that, we'll wind up saving that data to a CSV file so we can use it later on and bring it back in. This also takes a little while. I guess these computational clusters we're running on are not all that beefy. Maybe I should have used my laptop, but we'll give it another second here. There we go.

So it may be very hard to see from the back of the room, but you can start to see, in some of this data here — let me see if I can zoom in even further. No, that doesn't help. You can start to see, in some of this data here, that there's different colors. There's some green and there's some blue. And it starts to look like there may be some separability in that. So this should give you a little bit of hope that you may actually wind up being able to figure out the different redshifts in this large dimensional space. But it's very clear that it's not like in two axes, we get perfect separability of high redshift versus low redshift, for instance. So this is actually a pretty hard problem to do. And those that have worked on things like this know that it's very hard to do it.

All right. But we'll say and declare, for the purposes of this demo, that we're done with the pre-processing steps. I'd still call this part of the machine learning pipeline that we'd have to build if we're going to do this for real. But now we're ready to do some basic model fitting. To do basic model fitting, we need to create a training set, and then we're going to create what's called a testing set. We're going to hold back some data on the side, and we're going to take our model and predict against data whose answers we already know. And then we're going to try to see how well we did. So we're going to create our x vector and our y vector. The x vector is going to be features, and y vector is our answers. We've got 9,988 examples of data that meets our filter criterion in nine dimensions, and we've got an output of the same size. So we're off to the races. Just for the purposes of this demo, we're going to choose half of the data for training and half of the data for testing. What did I just do, other than just create a train set and a test set? What did I just potentially introduce in terms of biases into the thing that we're going to wind up building? Yeah.

STUDENT: (INAUDIBLE)

JOSH BLOOM: That's right. So it's not a random selection of the data points. I assume that what came back from Sloan was not something ordered by magnitude or ordered by location in the sky, which could have some very subtle impact on whether you're in the supergalactic plane or not in the supergalactic plane. We just introduced a whole bunch of biases in the way I even created this train and test set. A better way to do it, or at least another way to do it, would be to create a random test set. We'll see that Python has a whole set of methods that allow us to, without having to do what we're doing right now, implicitly create train and test sets as you wind up building up the model. But good. So really important to bear in mind the assumptions that you're making about the data as you wind up building up these models.

So we can build a linear regressor on that. Typically, what you'll wind up doing is instantiating one of these different models, and you call it clf. And you can wind up doing a tab completion to see all the different things that can be done on this model fitter. The first one obviously is to run fit. Once it's been fit, we can get the parameters of the fit, et cetera, et cetera. And we can apply that, if we want to, to new data.

So let's fit the data. For all the learners, they will all have a dot fit method associated with them. And so it should be where you can swap in and out these different learners and still apply it to the same format of the data. So we'll train on the x data, given the y outputs. And we got a result here. We're going to wind up predicting on our test data, and then we're going to see how well we did. We're going to get the mean squared error relative to our test data. Visually, we can see what that actually looks like. And you see that this is pretty crappy. Yet at some level, it's actually pretty good, because it did the best job it could for a linear regressor on nine dimensions in fitting most of the data. But you see that it was a massive underfit of the high redshift results.

Here's what we would do if we just took the average of the training data. So mean squared error of 0.65. And if we'd just chosen the average, we probably would have gotten about the same result. So this is not a very good regressor. And you can see some things here from scikit-learn, where we just don't have to build up the notion of what a mean squared error is ourself. They have all these different scoring functions.

Before I jump into other classifiers, what I'll do is jump back into the lecture notes, and I'll introduce some of the other — I won't call it theoretical motivations, but at least visual motivations for these other types of classifiers. But anyway, I wanted you to just see at the very beginning here that we've got reasonable end-to-end of starting off with raw data, and then we're building a model, and then we're able to use that model on new data. It just turns out to be a really crappy model.

STUDENT: I have a quick question.

JOSH BLOOM: Yes, question.

STUDENT: When you do this mapping, what does it actually do? Can you tell us?

JOSH BLOOM: What does the fit actually do when we do the matching? Give me one second. Just put something up here. Sorry. The question was, what are we actually doing when we're doing the modeling?

STUDENT: The fitting — because you use the first data set, and you fit a regression model to this data set and then use that model to the–

JOSH BLOOM: So the question is what's happening under the hood. When you run dot fit, the machinery of scikit-learn is going through, looking at your input data, looking at your outputs, and coming up with a linear model that winds up predicting this. So it's coming up with — however the linear model is created and run within that; there's logistic regression, there's all these other ways of doing modeling — it's just coming up with essentially weights on the data. What are the nine numbers I need to multiply to get out another number? It's just coming up with that. And it's storing those results so that when you then apply that to new data, you wind up getting out essentially what the result should be. And because you know the results on the test data, you can compare them directly.

What I wanted to show you is this thing called k-nearest neighbors. K-nearest neighbors is represented right here, which is to say I have a new data point. This is showing you a data point in two-dimensional space, so I'm not showing you what the axes are here. But I have a new data point, which is green, and I want to know what its value is, its result is. In a classification sense, you might just say, well, if I have a parameter which is called a hyperparameter, k equals 3, I'll just take the 3 around me. And if I know the two classes are of class blue and class red, then I can wind up saying, well, probably with a 66% probability, this class of green dot should actually belong to the red triangle class. If k is equal to 5, you may wind up saying that it's equal to the blue class. So this is a tuning parameter of a model like this.

In the context of regression, what this is doing is saying, let me take the y value variable outputs of those three things inside of that first box. And I'm going to, let's say, average those together. Or I could take the median of those. And you can define exactly how it winds up doing the combination. And then that should be your answer. And this works in any large number of dimensions. But as you can imagine, you get into very weird places, where you're in 1,000-dimensional space, and really, nothing is near you. Almost by definition, nothing will be exactly near you unless you have a repeat of the instance from your training set. And so the notion of what that distance is becomes really strange.

One of the things that's really weird about k-nearest neighbors and linear regressors and lots of the types of machine learning models that are out there is that it assumes something implicit about the data. Does anyone want to guess what that is, based on some of the words I just used? I've got a nine-dimensional space in the problem I just used. What are the units of all those nine dimensions? Those happen to be magnitudes, right? If we go back to the notebook here, I've got magnitudes. But look at this. Most of these are color differences, so their values are pretty close to zero. But I've got one, which is the apparent magnitude of the object, and these numbers are nowhere near zero. When you build a linear regressor, there's no difference between this value and this value here. And when you're doing k-nearest neighbors and you're trying to find out the distance between you and the nearest object, the probability of having a distance of 0.4 in u minus g is much different than the probability of having 0.4 of the difference in the dereddened r-band magnitude. All this is saying is that many of these different classifiers and regressors that we wind up using have this implicit assumption that the data across all the dimensions are of equal importance and have an equal notion of distance, in a Euclidean sense. And so when we build a distance metric implicitly in some of these machine learning classifiers and regressors, we're implicitly assuming some metric in that space.

What would happen if I added another column here that was something that was potentially qualitative, like near a galaxy or not near a galaxy, or near the galactic plane or not near the galactic plane? How would a linear regressor deal with that? One possibility, of course, because now you're dealing with categorical features, is you could turn those into zeros and ones and say if you have a galaxy near you, we're going to call that zero, and if you don't, we're going to call that one. So then you have binary variables as some of your input, and then you have continuous variables as some of your input as well. Many of the original classifiers used in machine learning don't do all that well in that context.

Let me now do a k-nearest neighbors and see what happens, and see if we actually get an improvement in our model here. We already saw the result of linear regression, and we saw that the mean squared error was not much better at all than what we got from just essentially choosing the mean. So here, we're going to wind up now dealing with this metric space and doing something that tries to put all the different nine-dimensional parameters on the same footing. And that's where we're going to wind up scaling them. We're going to wind up scaling them all to approximately the same distribution, so they'll have, I believe, a mean of 0, and they'll have standard deviation of 1. And this scaler winds up remembering what it did to the input data before it wound up running a k-nearest neighbors regressor.

So I'm going to use k-nearest neighbors of 10. I'm going to take the nearest 10 objects in my nine-dimensional space, and then we're going to use whatever the defaults are from scikit-learn. And we're going to wind up fitting. You see that fit was pretty quick. It assumed a Minkowski metric, which is fine because I already did the pre-scaling. And we're going to see what our mean squared error is, which is much, much improved. So now our mean squared error is 0.23, as opposed to 0.6-something. And you see that I still have sort of the same structure I had before, so that I'm missing some of the high redshift stuff. So effectively, if you think about it, these high redshift things at redshift of 6 were assumed to be off by 4. It assumed all the high redshift stuff was at redshifts of around 2, which is near the mean. And you see, I got a result here which is much better than what I had before. But I have a hyperparameter. So is that result better because I chose k equals 10 or k equals 5? Let's see the results out. That wound up improving things a little bit. Let's try k equals 1, so we'll just take literally the nearest neighbor. You see the result is much worse. And you can start to see some structure in the data here, which is a little bit scary. I don't know, let's try 20. The results are all about consistent with each other.

So here's an example now where we had to do some pre-processing. If we didn't do the pre-processing — I think we can try this. What would happen if we didn't pre-process that? We already pre-processed x. I don't want to go back up and reprocess all that. But it'd be worth just trying it for yourself, what happens if you don't do this pre-processing step here. OK. So here's a pretty good example. I think I played around with this for a while, and I found that k-nearest neighbors of 5 gave me approximately the highest accuracy answer. Any questions about that, k-nearest neighbors? I'm going to jump back to another type of classifier and regressor called random forests in just a second. But if there are no questions, then I'll jump over.

By the way, when you split up the space with k equals 1 — this is, again, in two dimensions, so I've got three classes, red, green, and blue — this is the space that, if you now sample densely across your learned space, you get a very different classifier. That is, for the same point that shows up — just randomly choose a place in your head there — you often wind up getting a different answer depending upon what k value you wind up using. There's a nice thing on the web, by the way, I wanted to show you, where you can see — I don't know, these are just support vector — all right, I'll come back to this. It's support vector machines and random forests. I'll be able to show you visually and interact with that. Let me show you now a little bit on decision trees. Yes.

STUDENT: (INAUDIBLE) comment earlier, but could you actually talk about your favorite clustering algorithm (INAUDIBLE)?

JOSH BLOOM: What's my favorite clustering algorithm? I often don't do clustering. I feel like the most interesting man in the universe. I don't do clustering, but when I do, I drink Dos Equis, and I do k-nearest neighbors. But those tend not to do great in very large dimensions. So there's something called DBSCAN, which tends to be quite good. And oftentimes, when I'm doing clustering, it's because I really just want to see low-dimensional embeddings. So I'll often use some manifold tricks that I'll show you towards the end of the lecture to do that. And then most of the time, again, because I'm generally in the business of doing lots of unsupervised problems, I'll wind up trying to couch the problem in a supervised way. And then I won't use k-nearest neighbors. Questions?

STUDENT: What if you have, for example, periodic features? So right ascension of 0 being very close to right ascension of minus 1, which is the same as et cetera, et cetera, et cetera.

JOSH BLOOM: So the question's about periodic features — right ascension being very, very close because you loop over on the celestial sphere, for instance. That's a great example of where domain knowledge needs to come in. If you don't care about location on the sky, throw that out of your feature set. But if you do, because that's going to wind up being informative, put it in terms that you believe will be most useful for the output that you care about. So if you want to know, for instance, that this thing that you just found is an asteroid, rather than give it RA and dec, you probably want to give it ecliptic latitude and longitude, or ecliptic latitude only. So that would be an example of taking that data, changing it to get closer to what you think is going to wind up being informative. The probability of building a — I shouldn't use the word probability so easily — the chances of one building an amazing classifier which winds up learning how to transform RA and dec into galactic latitude is more or less zero. You're not going to build something that's going to wind up inferring the math behind that. If you already know the math, by all means, you should be entitled to actually apply that in what you would generally call the featurization step. As most people who do machine learning, both in industry and academia, know, when you're applying it to real data and asking real questions of it, you're spending almost all your time in featurization. That munging that we just did, which I breezed through really quickly and was just really simple filtering and stuff — that's the kind of thing you're going to wind up spending a whole bunch of time on, because it's the place you're not only introducing more informative features, it's also the place you're starting to protect yourself against bias. There was another question over there.

STUDENT: So you need to standardize the data in the second round (INAUDIBLE) and I was wondering how (INAUDIBLE) and how much that affects (INAUDIBLE).

JOSH BLOOM: So the question was, we just scaled the data as a pre-processing step before we did the k-nearest neighbors — how much do the parameters of that scaling wind up impacting the results? The short answer is, I don't know. We just have to try it. And if we were going to build a real paper around this or a real result around it, we would spend a lot of time trying to understand how sensitive our results are to the different scaling techniques. And it may be that we want to get our results very sensitive to the scaling techniques, because we want to highly optimize on the data we have. The flip side of that could be maybe I want to build a star-galaxy separator using k-nearest neighbors that don't just work on Sloan data, but work on other types of data, in which case we'd try other types of data after we finish that whole exercise and convince ourselves that we got the types of accuracies that we were expecting.

So we really don't know what the impacts are of the choices that we're making until we actually explore that and try that. And this is the crux of why you all would be better at building a star-galaxy-quasar separator than if I handed this to the most advanced data scientist who worked at pick your favorite company, Facebook. They might know a lot more machine learning than you. They might know a whole lot about all the intricacies of pre-processing of data. But because they don't understand the domain, they wouldn't do as well. And they'd potentially overfit without even knowing it. Any other questions?

All right. Let me jump now to decision trees and random forests. I'm going to present this section mostly in the context of classification, but it also has implications for regression as well. Before I do that, let me just show you a very classic data set that people wind up using in machine learning. This is called the iris data set. It comes pre-packaged with scikit-learn. And here, you've got essentially three dimensions — sorry, four dimensions — measurements on these various different flowers, and three different classes of what type of iris these flowers are. Has anyone not seen this before? OK, so I'll breeze through it because only a few of you haven't seen it. This is the thing that you will typically apply your classifier or regressor against to see how well it performs. Here, you can visually see that I can make cuts on the data that allow me to separate out different classes very easily.

For instance, in sepal length and petal width, if I made a cut in the top right quadrant over there, up and down vertically, around 0.5, I would be able to perfectly remove all the classes of red. And then I'd be left with needing to make some other cuts in another dimension that wind up giving me the separation between the green and the blue. So this is in a classification sense. If I now wanted to say, for a regression sense, given the values of, let's say, the first three parameters, what's the fourth parameter? Can you actually predict that? That's another question you might ask of the data.

But let me step back to decision trees and how it views that. This is a decision tree that's built on the iris data set. All it's doing when you build decision trees is you're building essentially these branching points, where you wind up, in the case of prediction, when you wind up having a new input y variable, at the very top level, we're saying, is the petal length greater than 2.4? And if it is, go to the right. If it's not, go to the left. And if it's in the left, you see what's called the terminal node up there. Do I have a pointer? Yeah, so this is what's called a terminal node right up here. But if not, I'm going to go down here, and I'm going to ask another question. Well, is the petal width greater than 1.65? And if it is, go down to the right. And if it's less than 5.5 in the petal length, go down here, et cetera, et cetera. And eventually, I get to another terminal node for the given instance of what I was just given that tells me that this was of type Iris virginica. So that's classification. In the context of regression, the idea here would be to go all the way down, and you would wind up just getting the value of one of the examples from the training set.

How you actually build up these different trees is kind of interesting, and that's where the real interesting math lies. But the way to think about it is you wind up trying to figure out how to create the most amount of separability between the classes that you know about, or if you're doing a regression problem, you try to get the most information out by making one of these splits. There are always splits at every single node that happen for one of the variables in your feature set. And how those variables are chosen as you progress down the node is where all the interesting math winds up coming in.

But you can see here that I can build a perfect classifier. In the x2 dimension, if it's greater than 1, then go down this path. If not, I've just gotten the green class. That's a perfect classifier on that. And then if x1 is greater than 1, then I will wind up splitting here, and I can classify between blue and red. How you actually do that is beyond the scope of this tutorial, but there's a couple of different ways in which one does it. Most of it — the way to think about it is you're trying to gain the most information by making a split. What you will typically do, when you're building up one of these trees, is you'll randomly choose a feature to try out of your large-dimensional feature set. And then you'll move effectively a pointer around for all the different values that you could split on that. And you'll stop the pointer where you wind up, on the other side of it, getting less information by cutting. And the information gain, or an entropy notion from a Shannon information theoretic perspective, are just simply defined these ways over different numbers of classes. Or you can define a similar type of thing when you're doing regression.

But depending upon exactly which feature gives you the best gain, you will just say, OK, for this node, I'm just going to cut on that one. And then as you go farther down into this tree, you wind up taking all the examples from the training set that made that cut, and you'll keep on going. And then you'll have a stopping criterion of when you decide, I'm actually done, and get to a terminal node. Oftentimes, you'll stop when you get to one single instance of your training data. But many times, what that means is that you've got lots and lots and lots of cuts to get all the way down to isolate every single training example. So one of the parameters of these decision trees will be, what's the minimum number of objects that I need to have in my node before I wind up stopping, or the maximum number before I stop? You can also imagine, as you wind up building up these trees, that you would actually want to stop when you get to a fixed length.

Now, it turns out that trees are really good for classification and regression. And what I'll show you here is that given a classifier, I can make a decision, when I put down some new data, exactly where I'm going to wind up showing up in this space. But what's interesting is that if I show up in this space here for, let's say, new data — that is, given a new x2 and x1 — not only do I get a good classifier (so if I'm in this space, the answer is green), I also get some notion of probability of which class you belong to. So here, I had one example of blue, and I had a couple of examples of red. If I'm somewhere in here — and that's the spaces I partitioned using my decision tree — then I wind up actually being able to get some notion of probability, which can turn out to be very, very helpful and useful.

Now I can show you a demo. Here is a demo of decision trees. This is essentially the result of having built not just one tree, but multiple trees. And when you build multiple trees, those are called large numbers of trees, or random forest. And it's got some other names associated with them. Now, if we've got a class of red in our training data and a class of green in our training data, this is the space as it's partitioned by not just one tree: as I put in essentially a new value here, which space do I belong to? You can also see that there's some notion of a probability. So out in here, I'm not sure whether I belong to green or red. And in fact, in this case, you're going to get an answer of what the probability is. But because you're so far away from your original data, this might be an example where you give pause and say, I've got a probability out, I got an answer, but I'm not sure if it's right, because I'm far away from my original data. That's another thing.

If I now plop down another data point — so I plop down red and build a new set of trees, and now took an average over a bunch of these different trees to get what the result is — you see that I wind up changing the space. If I put one right inside of here, I wind up changing the space; put here, et cetera. So if I want to do a green data point, shift-mouse click. You notice here, I'm not changing anything except I'm making this quadrant very, very strong for green. If I put a green right in here, things get pretty ugly. So I just drew a little red box around this one red point.

And you see that it's not just a fixed number of parameters. If I change the number of parameters of my model, I wind up getting a different space out of that. So if I have a few number of trees — in this case, I've got one tree — this is what one tree would wind up splitting up the space to be. If I have lots of trees, I split up the space — so this is 189 trees — I split up the space like this. The number of trees you have may actually wind up being part of your protection that you have against overfitting. If you have lots and lots of trees, in principle, you could wind up overfitting on the data because you've just seen it too many times. Random forest does a pretty good job of protecting yourself against that, not introducing too many biases with it. I can change the depth of how deep the trees can go, et cetera, et cetera. Any questions about what I'm showing you here? Yeah.

STUDENT: Is there a notion of error in the parameter? So the red and green points are next to each other. What if one of those has an error that (INAUDIBLE)?

JOSH BLOOM: Good point. So is there a notion of errors in the data? Yes, because there are astronomers in the room, and we know that our measurements aren't perfect. And there's noise in the data, and we have biases and errors in our measurement. But no, in the context that random forest doesn't take those into account explicitly. One could take it into account by making the error on, let's say, the r-band magnitude another feature. But what you're not doing is telling random forest — or any other of these classifiers, by the way — feature two is the error on feature one. It would just potentially wind up learning that if there's a large number in feature two, then I should discount the values in feature one. One of my big knocks against a lot of the academic work in machine learning is that very little of it contemplates uncertainty in the input data. Likewise, very little of it contemplates the fact that you might have mislabeled data. So in this example here, our original data-taking may tell us that this value is red, but the class is actually green. And so we just messed up our classifier locally around that point. So in a Bayesian sense of having parameters of some probability that every data point is incorrect, you might want to actually take that into account in your classifier — we don't do that. The only way to really do it is with some notion of a brute force, where you would simulate what your data sets would look like under different types of assumptions, so you could jiggle around each one of these different data points. It's mildly appropriate to create lots and lots of instantiations of your data by randomly sampling what you observed from what you believe to be your noise properties of what you observed, and then using that to build up what is effectively a much larger data set, to build up a potentially more robust classifier. But that's a bit of a cop out. Again, most of the machine learning that you're seeing here and as done in the real world is very frequentist. And we don't typically have a good way of incorporating errors from the data into it. Any other questions?

STUDENT: (INAUDIBLE)

JOSH BLOOM: So the question is, given that we know that we're changing the whole model space as I change the hyperparameters of the model, how do we go about choosing the right hyperparameters, the most appropriate ones? That's what's called model selection, a hyperparameter optimization. That's what we're going to do a little bit later on. It's somewhat of a black art. There's machinery within scikit-learn to help you do that and run all of it. But in the back of your head, you should always be thinking, as you're doing these model selections and hyperparameter optimizations, am I just overfitting on all of this because I just happen to choose the right one that gives me the right answer or the best, most accurate answer? But we'll talk a little bit more about that. It's a good thing that you're thinking about it.

I want to be cognizant of the time. When should we be giving some thought to stop?

STUDENT: About halfway.

JOSH BLOOM: All right. Why don't I go through, and I'll just finish up on regression, and then we'll come back and do a classification example. And then we'll do a little bit of anomaly detection after that. All right. Let's do some random forest. You instantiate a random forest regressor. Here, we're going to build not one tree, but 100 trees. We're going to create a criterion by which we decide on the different splits at every single one of the nodes — this is just going to be mean squared error. Minimum number of samples in a leaf will be one. These are different parameters that you're going to have to get to know if you're using random forest. So I'm running that now. You see it takes a little bit longer than the other ones did. And it just finished. So now, we can plot up the results. And you see that my mean squared error is now lower than all the other ones that I had before.

One of the major problems of random forest and these nonlinear learners — because it should be obvious that this is a highly nonlinear learner. I'm not partitioning the space with just a bunch of different hyperplanes. I'm actually making very, very nonlinear cuts when it comes to variable interdependency. And it's the job of a good learner to figure out the relationships implicitly between these various different features. But what should also be clear about random forest is that it allows you to make cuts on the data where you don't have to do this pre-processing step. And for that reason alone, it's why I tend to stay away from the ones where you do the linear regressors or the k-nearest neighbors, where you have to effectively define the metric space ad hoc.

In this case, because every time you wind up making a cut on the data, you're deciding to go left or right, you're deciding it based on the units of just that column of that feature. So if I'm cutting on dereddened r-band magnitude, I'm making a decision based on, what's your dereddened r-band magnitude at that split point? I have no reference at that split point to what the other values are in the other features. Whereas when I make a decision around how many neighbors do I go out to, you have to make a decision in a large-dimensional space, which means that all the features have to be effectively scaled correctly with respect to each other. One of the beautiful things about a random forest is that it doesn't require you to have to do that. And so it takes away one of those really ugly places where one, as a data scientist or a physical scientist, would have to imbue a lot of knowledge to get it right with some meaning. So now we have some questions about model selection — choosing the best model. We have this notion of what's called cross-validation. What I just did is I built a very poor man's cross-validator, where I took half the data, and I essentially took it out of my model building. And I built a model, and then I applied it to the other half just to see how well I did. But what you can imagine one can do is instead build a model on, let's say, 80% of the data, and then predict on the other 20%. And then move your window to another 20% of the data, and build a model on the first 20% and the last 60%, and apply it to your second batch of 20%. And now what you have is a prediction on all of your data, where you've now not used half the data, but now you've used 80% of the data. So in principle, your model should be better. And as you wind up tuning your model, you can imagine making those windows smaller and smaller. So this is called cross-validation.

Just visually, I'll show you what that looks like. This is what's called five-fold cross-validation, as depicted here. This is just a two-class problem, but it works as well on regression problems. You have a test set and a train set. And so you take the first 20%. You hold that out. You build a classifier on this portion here, this 80%. And then you apply it here. And then even though you're applying it on a small amount of data, you store the results of that application. And then when you're done, you basically have built up a whole bunch of predictions on data that weren't part of the training set. And as long as these models are trained on the same hyperparameters, as you go to each of these different so-called folds, you wind up getting a very accurate measure of what it would be like to have a model that's built on all of the data and applied to another set, which is out here.

By the way, oftentimes what people will do is not just create a test set and a train set. They'll also build what's called a validation set. In that case, let's say you have 120 examples. You take 20%, and you put it fully aside. And you say, I'm not going to do anything with that, other than to say, when I'm ready to write the paper, I'm going to apply my best final model to that and say, that's what I expect my results to be. And then you can do these clever examples with train and test to help you get a better and better model. But it's considered very good practice to hold out even more data in this validation set and not use it at all to help you train your model, because as soon as you start building models and changing hyperparameters based on the results of what you got on here, even though these data are, in principle, completely held out from here, there is some implicit crosstalk between the train data and the test data. So there's lots of different ways to do these different cross-validation holdouts, as it's called. The one that's often useful in the context of classification is something called stratified k-fold, where you're saying, for a large number of classes — let's say I've got 12 different Hubble types of galaxies, and I want to classify something in — if I have a class of galaxies that is a minority number, let's say there's only 10 S0's in my galaxy classifier in my training set, but there's 1,000 of some other type, then you can imagine that a classifier that's built correctly on the data, a little bit like we saw before with the redshift regressor, would try to fit most of the data. And so there's a nice thing about this k-fold, where it winds up holding out the appropriate amount of data for the given distribution of the classes in your training set. Don't worry if that's a little bit beyond what you've thought about for now. I just wanted you to hear it and think about it a little bit for now.

All right. So we're just going to build a little helper function, which is going to tell us what our cross-validation score is. We're going to do our cross-validation using k-fold cross-validation. We're going to do essentially 20% holdout, 80% prediction. Oh, wait. What? Oh, I forgot to run this. I get an answer there. We'll do cross-validation k-fold with 10 using random shuffling. This gets to one of the questions earlier about the potential bias that we introduced by holding off the first half of the data versus the second half. This will actually shuffle it for us.

And then we can do it on other classifiers. So this is clf2 — this was the random forest classifier. You see this takes a lot longer. This is building up a more intricate model. Let's see how well that does. Any questions while this is finishing up? Is this running on a Raspberry Pi? Is Rackspace all Raspberry Pis? I guess not. Well, we'll see. So what is this doing? It's actually doing a three-fold cross-validation, which means it's building three random forest models. And you see that our mean score winds up getting better. This, by the way, is different than the mean squared error, so this number should be larger.

All right. I'm going to stop here. We'll take about a 10-minute break or so. We'll come back and do some classification. We'll do some hyperparameter optimization. We'll do some parallel computing. And then we'll try to do a little bit of anomaly detection as well. I'm happy to take questions during the break if people want to come up.

Let's start up again here. I've been making reference to classification problems. Before we jump into that, I will just formally describe what I mean. Predicting the discrete class — that is, are you in class A, B, C, or D through n — of an object from an input set of features. So instead of it being a continuous variable that we're going to wind up predicting, we're going to try to decide which class you want it belonging to. And of course, you could turn a regression problem into a classification problem, where if you're above the value of 1, then you belong to this class; if you're less than this, you belong to that class. And vice versa. You could take a classification data set and say, if you're in this class A, that means that, from a regression perspective, I'm going to put you somewhere in the bucket between 0 and 1. But the closer you can get to defining your problem that's consistent with the question that you're asking and the data that you have, obviously, the better. Anyway, if I've got a bunch of instances — these are examples of x vectors and pairs of y, which are the labels, say 150 of them — and I want to predict these three different classes, that's actually what defines the iris data set.

So lots of different ways to do classification within scikit-learn. You can turn a logistic regressor into a classifier. K-nearest neighbors classifier — we saw that a little bit already. LDA, Latent Dirichlet Allocation, Naive Bayes, support vector machines, classification trees, random forests, blah, blah, blah. So there's a whole kitchen sink, like there was in the regression side, for you to make use of. And then there's also something which is also quite useful called feature selection, which is an automated way of deciding which features are informative and then rebuilding models based on what's informative and what's not informative.

Because I haven't shown it before, why don't I just quickly introduce yet another way of doing regression and classification: support vector machines. Typically, you think about these in the context of 0, 1 classification. So do you belong to the red box class or the blue circle class? And support vector machines is a mathematical model for building up what are essentially called hyperplanes — because we're showing this in two dimensions, it looks like a line. But in multiple dimensions, you want to create these nice places to separate.

So in the iris example, there's obviously some nice places that you would wind up separating between the red and the green. The question is where, and how would you build those lines? Obviously, I could build a line that cuts right close to the red and say everything to the left of this line in data we haven't seen yet is going to be of red class, and everything to the right will be of the other classes. But is that the optimal place to put it? The whole idea of support vector machines is that you try to get these hyperplanes in between the data, where you create what's basically the largest space right here. So this is the largest support, as it were. So this A is the optimal line to separate this data set and this data set. Even though B also does a perfect separation, it gets very, very close to this value here and gets very, very close to that value here. So it has less support. The larger the support you can get, the better. This works really well, in many cases, because if the data is nicely separable in these hyperplanes, then if you were going to use something like a random forest, you have to make cuts which are effectively always perpendicular or parallel to one of your axes. And so there, a random forest or even just a decision tree would have to make a cut that's here, and then on a subsequent run, it might have to take another cut in another dimension, just to build up this — what is a very easy math to figure out what these hyperplanes are. So this, again, is an optimization where you're trying to find all the hyperplanes that give you the maximal separation here, subject to some constraints: that if you had, let's say, a red value right here, the constraint would be, how many values of one class are allowed to be on essentially the wrong side of this line? So is that clear conceptually what's happening? There's some very nice properties of these things for these linear separator models, where you wind up getting some very, very good classification results.

Here's an example, though, where you've got data that might be hard to see, where it's blue here inside, and then it's red here in these two dimensions. And so you basically can't build a hyperplane that winds up separating these two things out. But instead, what you can do — if you click on this link here — oh, this link. Go to page. What you can do is actually project your data into a higher dimension than you had before, and then you can create hyperplanes that actually do give you perfect separability. So here, we're taking two dimensions. We're going to multiply those and push those into a third dimension. And now you can create a hyperplane that optimally splits between the blue and the red. And then you can project that back down into the dimensions that you had before. So it turns out that in a very, very large — in fact, infinite — dimensional space, you can build a perfect classifier, even for highly nonlinear relationships between the data.

The problem with that is you wind up having very, very large models. And again, you come back to this problem of the metric space that you're doing all of this work in. If this direction here is magnitudes and this direction here is which telescope you used, what is the metric in that space that you would actually wind up using? Again, I come back to things like random forest, because when you make your cuts, you're making it in the metric space of the thing you're cutting inside of. So it tends to be much more powerful.

All right. Let's go back to classification. Any questions about support vector machines? What I will say — it's not even anecdotally, I guess you could look it up — is that for lots and lots of different types of problems, both in astronomy and outside of astronomy, support vector machines rarely do better than things like random forest on real-world competition. They have other issues as well, like their model sizes can be very big. So if you try to move them around in memory and you have to use them often, you have to hand them to people, the typical rule of thumb is that they're about a third the size of the original data set, which can be quite large. Anyway, lot of people use support vector machines, but that tends to be mostly on the academic side. The other major issue with support vector machines is that there's no natural way to get probabilities out. Whereas with k-nearest neighbors or with random forests, you can just look at what all the different, effectively, votes are and say, well, I voted mostly for green, so that's the answer, but I had a couple other red classes for this new instance — then you can create a realistic notion of probability. With support vector machines, there is no way of getting a natural probability out of that. And many times, we want to get some notion of, I belong to this class with this probability. Yeah.

STUDENT: I think one of the main reasons that academics love support vector machines is that the optimization it does inside is closed form, in the same way that PCA is closed form, or the linear fitting, I think yesterday, is a closed form. And so it's truly a convex problem, closed form solution. So you know you're getting the best possible answer, even in the infinite dimensional reduction version. So it is a very beautiful algorithm, and it deserves its props there in that respect, the same way that PCA and linear fitting are just wonderful.

JOSH BLOOM: Yeah, again, I completely agree, obviously. And it's beautiful at the mathematical level. It just isn't practical. For all the reasons that we have up here, they actually can be very computationally expensive — not to build it necessarily, but to move it around. If you're doing a nonlinear kernel, it can be pretty expensive. But it's not all that informative either. You're making all these massive hyperparameter splits — it's not really clear what's going on. And then again, because we care about probabilities, support vector machines don't give you probabilities. And for me, the biggest knock is our data is heterogeneous in terms of what the different units are in the features, and we also have missing data. And so you can try to coerce what is a real-world data set or astronomy data set into something that you can build a support vector machine against. But in the end, you're basically doing a square hole in a round peg, whereas there are other models that allow you to natively take care of things like missing data.

And the last one is that in context of classification, support vector machines works in a 0 to 1 way. So if you're trying to build a multi-class classifier, you're effectively building up, do you belong to this class or another class? Do you belong to class B or the other classes? And then you wind up choosing the one that gave you the best answer, and that's your classifier. So it's not very natural for multi-class problems either.

STUDENT: I just want to add one more thing, which is that — this connection — all of the reasons that you don't like support vector machines are all of the reasons that I don't like PCA. And the two algorithms are very closely related, because they both are these closed form linear algebra solutions. And the reason that they work is because they depend on there being a well-defined metric and no missing data and homoskedastic and all that stuff. And so basically, whenever an algorithm is absolutely trivial, it's probably not appropriate for us.

JOSH BLOOM: For those that didn't hear that at home, this is a tweetable moment from Dave. If it's tractable — can I rephrase it a little bit? If it's tractable and it's beautiful mathematically, it's probably not that practical. OK. Now we're going to build a star-galaxy-quasar separator. Star-galaxy should be pretty easy if we had shape parameters. But again, we're going to use the original data we had, which is just photometry parameters. So this is a somewhat non-trivial problem. We're going to pull over 1,000 quasars, 1,000 stars, 1,000 galaxies. We're going to remove all the bad data. And we're going to take a look at our data. So here, we've got our object IDs and our different colors here. And we'll take a look at our different classes. So we've now got 3,000. This is what's called a balanced label problem — we've got one third, one third, one third. So this means the classifier really is going to have effectively an equal shot at making predictions on all of them. This obviously doesn't take into account that there are far more stars that you're going to be able to see in Sloan than you are going to be able to see quasars.

So again, we're not using our intrinsic knowledge about this. This is saying, given a source and given no other priors on what that is — if I told you I found a source in Sloan, the first thing you'd say is that's probably a star, because there are more stars in Sloan than other types of objects. If I said, well, it's not a star, you'd say, well, it's probably a galaxy. And say, OK, well, it's not a galaxy — well, then it's a quasar. So without giving you any other data other than saying it's not a star, you can pretty much figure out what something is. But given a new object in a purely frequentist way, without any priors, I want to know, given its data, what type of object is it? And here, we'll make our y value. So now, instead of it being redshift — a continuous variable — we're going to wind up having a three-class problem. And we'll run random forests on that with 200 trees.

And interestingly, before we see the results, you can actually count up the number of times that a feature is split upon during this process. And the number of times a feature is split upon is actually indicative of how important it is for the answer that you care about. Because I didn't go into all the details of exactly how you choose which feature to split on at every one of these different nodes — but effectively, there's a hyperparameter which says, choose a certain number of features to try out every time you get to a node. Figure out the one that is best at doing the split — this is what's called a greedy algorithm — and then split that way on that one. And the next time, just randomly choose another number. Typically, the rule of thumb is that you'll choose a number of features which is the square root of the number of features that you have. So if you have nine features, like we do, it will choose, at any split point, three random features to try. It'll figure out the optimal split across all three of them, and it'll choose the one that gives the best information gain. And then it'll split there on that feature. And then it keeps on going. So if you keep splitting on the same feature to improve your information at each of the nodes, it stands to reason that that would be more important. And there's a more rigorous way of keeping track of importance in your model with random forest. And then you can plot that up. Was there a question?

STUDENT: Yeah, how do you pick the number of estimators? What's the best–

JOSH BLOOM: So the number of estimators, this thing of how many features do I try — that's called mtry. How deep is the forest? How many instances should be in each node, in a terminal node? All of those things are called hyperparameters of random forest. And support vector machines is a different set of hyperparameters, et cetera, et cetera. We're going to show you how you do that sort of selection. But effectively, you have a bunch of rules of thumb of how many you should have, and then you wind up looking around that. Effectively, what you do is you either do a grid search or a random search over those hyperparameters. Again, with the little bird on your shoulder or devil on your shoulder saying, if you do this, you may be overfitting, you may be overfitting — so protect yourself. So there are ways to try to do that. But the answer is, logistically, you do a search over the hyperparameters.

OK, so u minus g color is pretty indicative of whether you're a star, a galaxy, or a quasar. The difference between your g-band magnitude and your aperture or your Petrosian magnitude is the next most important. These other two are important. And it turns out your i minus z color is not important at all, or not very important at all.

We can get what's called an out-of-bag error when we do our score, when we do random forest. Let's try to figure out what out-of-bag actually means. The way to think about out-of-bag is that whenever you are deciding to build up a tree, the first thing you do is you say, OK, I'm going to try a bunch of different features. But I also have to know all the different instances that are going to be part of my selection. You randomly choose your instances from the data, but you do it with replacement. So that means if you happen to randomly choose instance two, which could be, let's say, a star, you put it back into the bag. And then you choose again, and you might actually get number two. But you choose the total number of instances that you have. I'll do it in the column way. So if this is instances — it's the number of rows, and columns are the number of features — you're choosing, let's say, n instances. But you can do the math and figure out that you typically will randomly leave behind about 30% of your data at every single one of the top splits that you wind up doing when you build a new tree. And that's what's called the out-of-bag data. So the out-of-bag data, when you're building up a tree, is actually left behind. It is not part of building up the tree. But you can take out the things that you hadn't used, apply it, get what your answer is, and if you store up all the results, you get, effectively, the train-test split for you very nicely. So the out-of-bag score is, for all the data that randomly wasn't used, what were my predictions, and how well did I do? So this means that I got 95% of them correct with this classifier.

We can do the same thing in support vector machines, and there's lots of different kernels you can use to build up your SVM. And we'll print out what those look like and what the results are, hopefully. Any questions while this is running about the statements I just made about out-of-bag error? OK. So what am I doing here? Let's take a look. I'm just creating a bunch of different support vector machine models. And I'm creating different types of kernels, which is whether you're in a linear kernel or a non-linear kernel, which actually allows you to morph a little bit what the metric space is that you're using. And then there's some other hyperparameters of that model. Again, we're not going to go into the details of what all these different hyperparameters are, but it's useful for you to look at that if you want to play with it.

I don't know why this is taking so long. If this takes too long, I'm just going to jump back to my laptop. Just curious, what do we have here? Does anyone know how to figure out what kind of machine we're running on inside of a Unix? No? Is it cat etc proc-something? No. Cat /cpuinfo. Oh, it looks like a lot of cores. All right, let's see if this finished. What the hell? It's still going. All right, why don't I talk about hyperparameter optimization while this is running? If this takes another minute or two, I'll just jump back to my local version on my machine.

So this is where you would actually do this hyperparameter optimization that you were asking about. It really shouldn't take this long. Here, we grabbed this thing called grid search. And we can now create a dictionary that has a set of parameters, where we're now not individually creating all of those support vector machines by hand. We're going to create a set of parameters which will be fed in when we're creating our different models that we want to loop over. Or in this case, we're going to create a large-dimensional grid over all these three different parameters: what kernel it is, whatever gamma means, and whatever C means. I'm going to just kill this. Oh, we got some of the images. All right. So here's two of the support vector machines in the space that they wind up creating. Here, you notice that the linear kernel is splitting up the space as best as it can across the three different classes. Here, with radial basis function kernel, you wind up having some nice behavior here. Again, we don't know whether we're overfitting on this one blue dot or not. But that's just to show you what the different spaces are that get built out.

All right. So let's take a look at what we're doing here. GridSearchCV will run the fitter on this type of model, where it's passing different parameters and different combinations of those parameters. So we can run this here and now run it in a parallel way. Hopefully, this will be faster. And we can figure out what the best score is. So we're doing 168 different fits. That finished pretty quickly. So it looks like I have a lot of cores, but terrible CPU or something. And I got my best answer out here. And so you see, this actually got comparable to random forest. And here's the best model. The best model, in this case, was gamma of 0.1. It chose a radial basis function kernel or the other parameters we searched over. I think we looked at degree. So that's actually kind of nice, because we can run it over multiple cores. And there's something called joblib, which is what scikit-learn uses to actually push this out to as many cores as it can get access to. And it's running all of these embarrassingly parallel jobs for you. So we did 168 of those in seven seconds over however many cores there were here.

All right. So we can look at what's called the confusion matrix, which is a nice way of showing you how we wind up labeling correctly or mislabeling between classes 0, 1, and 2. I forgot what we called those things — 0, 1, and 2. So 0 is quasar, 1 is star, 2 is galaxy. And we can get an actual numpy array showing us what those values are. So this means that for all the things in the training set, we built a classifier, and then we applied it to the testing set. We got 231 of those right of quasar, whatever, 254 of star, and 255 of galaxy. And you see, these off-diagonals is where we had some confusion between those different classes.

If you get a perfect classifier and you see a confusion matrix that has no power off the diagonal and all the power in the diagonal, you've overfit. I guarantee it. Or you're using a trivial data set, where you shouldn't even be playing with it. But if you don't see some off-diagonal power, it means that something in your training set is leaking into your testing set, or something in your features knows about the answer in a way that you don't want it to know. If I had a label that said the label q, s, and whatever, g, probably we'd have a perfect classifier, because it would wind up figuring out that whatever column I wound up using in my feature data was actually exactly predictive of the thing that I wanted. Another way to figure out whether your classifier is any good is put the actual answers into your feature set. Run your classifier. And the most important feature better be the thing that's the answer. If it's not, you have a bad classifier. You've done something else horribly wrong. These are the types of things that you have to start thinking about as you're building these up.

OK, so let's do the same thing, where the question was how many trees do we use? We'll actually build one where we're looping over a whole bunch of different questions that we have for the hyperparameters of this random forest. So we're going to loop over 108 different types of models with different hyperparameters. Hopefully this will also finish in a finite amount of time. OK, that was pretty good. There we go. All right, so we got an answer, which is a little bit worse than what we had had before with the models that I chose before. We don't know whether this is an inferior one. It just happens randomly that we wind up getting slightly worse results, because again, there's a random component to the construction of this model. I thought I saw a hand somewhere. OK, no.

So here's our answer. Max_features 3, min_samples in the leaf node is 2. Number of estimators, 200. We can get those parameters, and we could save those for later. These are the best parameters of the random forest we built. Here's the best parameters of the support vector machines that we built. And I won't go into the details of what you can do in here, but you can actually set what your different scoring functions are. So if you want to do your correct choosing of which hyperparameters are best, you don't have to use whatever the default is, which is probably mean squared error for regression, and it's probably just accuracy or score for classification. Any questions about that?

It should be obvious that when you're doing grid search, it's not RAM-optimized and not compute-optimized. If you actually look at the code that's doing the grid search, it's doing a whole bunch of nested for loops. And oftentimes, it will wind up not saving pre-processing that you might do in the step. So if you had a pipeline that was part of your grid search, you might actually want to save the data and then reuse it for something that's inside of a for loop. These are not always optimal. When you have a lot of compute power and a lot of time to sit around, that's fine. What it turns out is that somebody wrote a paper fairly recently showing that if you just did a random search over your space, you get an answer which is pretty good, and it's pretty close to the right answer, because over multiple hyperparameters, it turns out most models are only sensitive to a few of those hyperparameters. And so searching over an entire grid in lots of the hyperparameters that you think you care about, you're spending and wasting a lot of computational time in a space that's not actually all that useful. There's also a whole Bayesian formalism for deciding what next hyperparameter to wind up using, given the results of all the other runs that you've had. That works irrespective of the model that you're actually building.

I want to jump back into the notebook and go into a little bit more details here, and some questions that you would be asking when you're doing hyperparameter optimization. How do I choose a model is really the critical one once you've done all the featurization and pre-processing steps. K-nearest neighbors is, what's the number of neighbors? Support vector machines is, what's the kernel that I'm going to use? What's the bandwidth? Random forest is, how many trees? What's mtry? With Gaussian processes, it's different sets of questions. I showed you this already. And then there's lots and lots of different metrics that you would potentially wind up using to create your optimization, and lots of different ways for you to show plots of how well you did on the models that you wind up choosing. Obviously, I won't go into all the details here. I will post this notebook in the Astro Hack Week GitHub repo, so you can see that for later on if you want to go into that. So all these things are available to you. And you may decide none of these metrics are the ones that I actually care about, because as we talked about later, we might not want to be optimizing on something like accuracy. We might want to be optimizing on something which is much more closely related to the problem that I have at hand.

What I will say — and this is from a paper we wrote in an astronomy context, where we looked at a bunch of image differences on the Palomar Transient Factory, and we had a bunch of labels of whether this thing that looked like it was a new source in the image difference was a real source or whether it was not a real source, as in a transient or not a transient. So this is what we call the real-bogus problem. And given the same input featurized data — across, I think it was 42 dimensions in this case — what you wind up seeing is that in all metrics for false positive and false negative, random forests wind up beating out everything else. And this is with the best tuning of each one of those different models. I won't claim that this is a universal type of curve that you'll see. But if you're asking the question, which model do I try, I often would try random forest first. And then maybe if that doesn't give you the results you want, you might do support vector machines. But the other ones tend not to do as well. And again, we already talked about this question about what we can pull probabilities out. So in this case, we want to minimize the false positive rate and minimize the false negative rate, the type 1 and type 2 errors. So you want to push that curve all the way back down to the bottom left. And at all places, we found that random forest actually did better.

We're not going to do the breakout session. I will continue to extol the virtues of random forest and say that random forest is built into Kinect. All this is doing inside of Kinect in hardware is running a random forest that's been pre-trained on lots of probably postdocs at Microsoft — maybe grad students, probably postdocs — where they walked around, and they then labeled which part of your body each pixel was. And then random forest is trying to figure out and colorize, in this case here, what body part you are in a given pixel. And so all it's doing is, at pixel level, it's saying what body part are you? What body part are you, given the input images? And random forest, because there's a bunch of if-then statements effectively, can run incredibly fast on the prediction side. So this is what they did. They took all this input data — left hand, right hand, shoulder, neck. They built a straw man model of what that would look like to create some notion of tracking. And then they just built a classifier at the pixel level. So here are the postdocs doing crazy things that you do in Kinect. They had a million training examples.

All right. So let me just finish on the hyperparameter optimization stuff. And then we'll jump back, and I'll show you some other types of models, show you some other examples of using machine learning in astronomy. All right. We're going to run this without doing a full grid search. And we got an answer which is pretty comparable to what I had before. We've been using this thing called joblib. Has anyone heard of Dask? OK, so a few people. Again, it's the people on the right. So it's the Python 3 people. Interesting. People at Continuum have built essentially a distributed computation and scheduler that allows you to make use of multiple threads and multiple cores and even multiple computers, where it keeps track of what the computations are that are going to be needed and are then going to be needed farther down in the computation tree. And it keeps in memory the things that it believes it's going to need later on. And then once it's done with that, it will wind up excising it out of memory. So it's a computationally and RAM-efficient way of doing computation. If you run install Dask and distributed, we can try to run this grid search using Dask, which will do the distributed thing in a completely different way. So here, I'm just going to pull over a very, very not safe for work — well, not safe for work in the traditional NSFW; not safe for work in the sense that it's not ready for production — way of this thing called dask-learn, which is trying to use Dask and rewrite all the scikit-learn fitters and optimizers. So we're just going to do that and pull that into our namespace. Hopefully that will work. Oh, nothing called Dask. I forgot to run this. Oh, right, because I'm running it in the cloud. I'm not running it on my machine. Installing Dask. Extracting packages, complete. OK, so this should now work. Yep. That worked, right? Yeah.

So now, we'll do the same exact thing. But instead of doing the grid search that comes with scikit-learn, we'll use a different grid search. And this one actually may take a little bit longer. We'll see how long it takes, just because it sets up a lot more infrastructure. But if we did many, many more parameters, or we did this on a very large cluster — let's say it took five minutes to build one of these models — this would totally win. So we'll let that run. I won't execute this thing here, which allows you to build up a cluster and then run it over a cluster. I'll come back to a different kind of clustering. So here, this took 31 seconds. We got about the same answer as we had before. Let's see how it did timing-wise relative to the other one. That one took nine seconds. So still much faster on the scikit-learn side. But there are workloads that you can find on the web using dask-learn that actually do much faster and much more computationally efficient. What I didn't do is keep track of the RAM usage. I think one of the conceits of Dask is that it's just going to be more efficient at using RAM. All right. I'm going to jump back now to a couple other things I wanted to show you. Who's heard of deep learning? OK, now almost everyone, not just Python 3 people. Effectively, the idea around deep learning, or what people are very excited about, is not only does it give very, very good and probably the most accurate answers on a class of problems — typically around language understanding and around voice-to-text, around image processing, around video processing — the thing that people really like about it is that you don't have to do a lot of featurization. In fact, the idea is you can throw raw data at a deep learning network, which, in the end, is just taking your raw data, and it's deciding how to combine different pixels from different places on, let's say, an image, with weights that it winds up learning through a bunch of very clever techniques that allow it to learn in a finite amount of time. And then it takes the results of whatever it wound up taking from your raw data. That's going into what's called the next layer. And then you have another set of weights which are multiplied against that data to get to another layer.

And the idea for image processing is that you don't have to know a lot about what it means to be an edge to detect something that is an edge. You don't have to know what it means to be a cat to find cats in images. As long as you have an objective function on the other side that you're trying to, let's say, classify am I a person or a cat — if you're trying to get a two-class problem as an output — if you have enough data, you let the machine figure that out. So what's happening — unlike what we just showed you throughout this whole tutorial, where we took our raw data, we did some feature engineering, we threw some stuff out, we built some new features, and then we did our classifier — in this case, with deep learning, the idea would be that you can really do both the feature engineering and the classification in the same sort of process.

This is very computationally expensive, although again, there have been a large number of advances in the field that make it at least tractable. But oftentimes, because a lot of this is a lot of matrix multiplication, these are things that you can actually wind up doing on GPUs very effectively, or even FPGAs. It's worth taking a look at this result here, where people built up a network to do some inference on galaxy images, essentially to classify galaxy images, using raw Sloan data. And here, you can see that their input was RGB data. And they wind up building up a network to get to some sort of answer that they wind up caring about. I'm not going to go into the details of what all these different types of layers are and how you wind up combining all of the data, other than to say that yes, you give up on feature engineering. That's a big plus one for deep learning. But in the end, there's this whole black art of how you build up your network. And how did this person get to the point of building a network that looks like this? Because everyone's network is going to wind up being different. And you wind up getting different results depending upon what your network is and all the different hyperparameters of the training of that network. All of that is stuff you still have to do.

So we've pushed featurization, which, as domain experts, I tend to like a lot, because it allows you to commune with your data. It allows you to ask questions that you know are physically sound and relevant for the question that you're asking of your data. And it allows you to build features that are going to be informative and then throw it into a classifier. Whereas in this case, you wind up potentially just throwing everything in, the whole kitchen sink, and hope that you get a good answer out. Now, there's nothing forbidding you from using deep learning networks to work on featurized data. But because these tend to work well in the context of two-dimensional data, most of the time, you're not doing featurization. And then there's some sort of notion of some local connection between pixels. In that case, you certainly could build a deep learning network on an input vector, which is what we've been operating on now. But it typically will wind up wanting to work on two-dimensional images. Or at least that's how it's been used to most effect recently. Any questions about deep learning? Yes.

STUDENT: I would just say that that blog post by (INAUDIBLE) here is excellent, and it's really worth reading. And if you're interested in thinking about deep learning and the surrounding context, it covers a lot of the ground. (INAUDIBLE) choices in there, but it's a very good, straightforward introduction to how you (INAUDIBLE).

JOSH BLOOM: Just like machine learning is not the answer to all your inference problems in astronomy, deep learning is not your answer to all machine learning problems in astronomy. And in fact, in the image problem, where we looked at the real-bogus, we did a traditional featurization step on that, and then we did random forest. We also tried using deep learning, and we got inferior answers. And the reason being is that while you can take networks that have been pre-trained on other images and apply it to your image, what's actually happening in many of these cases is it's learning a lot about the details of the questions that you're asking of the original image set. And we didn't have enough data to train what is effectively millions and millions of nodes in this network. And so we just got inferior answers. So here's an example where we needed to put it in production to work on real-world telescope data effectively in real time, and deep learning just didn't work. You could argue we're not deep learning experts and we didn't know how to build the networks correctly, but we did spend enough time on it. I think Danny kind of banged his head on that enough. But it's not going to solve all of your problems.

Another approach of using deep learning, which I think is gaining more and more traction, is to use this notion of autoencoding, for those that have heard of that, where you don't try to predict an outcome with labels. You try to build up the data that you had before. So you have a deep learning network that looks like this. And what you're doing is you're compactifying your data and getting it more and more summarized over time, so that by the time you get to these deep layers, you, in principle, have learned concepts about your data. Whereas at the very high layers, especially for image processing, you're, in principle, learning sort of low-level features that you would apply to data if you were going to do it from scratch. So this could be like histogram of gradients. It could be other types of low-level featurization that you would ordinarily do if you didn't know about deep learning. It's been shown that some of these higher-level layers closer to the raw data actually do some traditional filter bank techniques and are learning those intrinsically, which is pretty amazing. But farther down, you've got concepts and deeper concepts of what it means to be the data that you've given it.

If you then take this compact notion of what it means to be your data, and then you actually start building and start expanding that data back out, you can try to build something that doesn't get smaller and smaller, but now gets bigger and bigger. And then you wind up trying to predict the data that you started off with. That's what's called autoencoding. What's nice about autoencoding is that you don't have to come at this problem with the classification set in mind. You can just say, I've got a whole bunch of data — a bunch of images, let's say. I want to build a deep learning network that gives me back the data I started off with. But now, I can use this stuff in the middle, which are the deep concepts, and I can, in principle, use those as features for another learner. So what we've been doing in my company, what we've started thinking about in the astronomy context, is marrying both deep learning and something like random forest, where the deep learner is not the classifier itself, but is just the thing that's building the features for us. That's definitely something worth exploring. If somebody wanted to build a hack around that this afternoon, that'd be pretty exciting. Happy to talk with them about it.

The other thing I should say, in the context of deep learning and more generally about machine learning, is that when you have more data, you wind up traditionally doing better. So even if you have the same learner, you have the same featurization process. And this comes — something Peter Norvig at Google said a while back: more data beats clever algorithms, but better data beats more data. So get more data, but make sure it's pretty good. And oftentimes, if you want to improve your model — given that you've just spent a bunch of time featurizing it and extracting what you think is the most information out of this and then throwing it into a good classifier — the only way to get a much better result is typically getting more data. There was a question.

STUDENT: Yeah, the question was related to this. How good example size is good enough for deep learning? How many tuples is good enough to move to a deep learning model? Especially, is there a rule of thumb between the number of features that you have and the number of examples?

JOSH BLOOM: Oh, good. So the question is, what's the rule of thumb around whether you do deep learning or something else? I don't have a good one. What I will say is a couple things. One is, if they're images or they're sonograms of frequencies or something like that, that may be a pretty interesting example of where you'd start thinking about using deep learning, because those are the obvious places where they're doing really well. If you have lots of metadata and heterogeneous data, in the sense that you have categories of your features, and you also have numerical values and maybe even strings, that's a pretty terrible place to think about deep learning. And the only time you'd have to do it is after you did a whole bunch of featurization. But once you've done all the featurization, you might as well just use a traditional learner. So if your data looks like that, that's a little scary for deep learning. If you're looking at pixel data, and every pixel goes from 0 to 256, that's a good place to think about it. Now, in terms of the volume of the data, millions of images. You can do well with, say, tens of thousands of images. There's a whole notion of what's called transfer learning, which I alluded to, where you could take a network that's been built on images taken off the web, and then take that whole network and just use it to apply it to your images. And in principle, there are features that it's effectively learned and concepts it's learned that it could wind up using and reusing, that could be informative and allow you to use smaller amounts of data. I think once you're into the millions of instance level, then you can credibly start thinking about deep learning. People on the web may be flaming me right now, because they say, no, there's an example where we can do 10. If you're at the 10-instance level or 100 or 1,000, you have to be using some of these other techniques. Any other questions?

STUDENT: You mentioned the concept of rights. (INAUDIBLE) Is this something that you have to set up (INAUDIBLE) yourself and then validate over there?

JOSH BLOOM: I missed the beginning of your question. I mentioned the concept of what? Rights?

STUDENT: Weights. Weights. (INAUDIBLE) the raw data and then the weights. You have the layers that you want to choose. (INAUDIBLE) I just wonder, when you said the machine or the algorithm learns later on what (INAUDIBLE) to choose, I wonder whether that's something that you said initially and then–

JOSH BLOOM: Right, so the process by building up these networks. What should be clear, just to step back, is that a big part of the work that people do in machine learning at the academic level — and a lot of that's moved into R&D centers inside of companies — is to figure out optimal ways to do the optimization problem of, how do I build up the network? Or in random forests, what's the optimal way to make the splits to get the answer out that I want, where optimal now can be not just accuracy, but RAM efficiency and CPU cost, et cetera, et cetera? So a lot of the work is on building up the networks and learning how to learn. Once the network's been built, then it's just a matter of just cranking data through. And one of the nice things about things like random forests and deep learning networks is that it's not that computationally expensive to take the data, throw it through, and you get your answer out really, really quickly with not a whole lot of extra math. But so the question about how the weights are created — that comes down to how you decide to build the network. Oftentimes, the weights will be randomly assigned at the beginning. And then you have this notion of what's called backpropagation, where you start from the answer. And there's a way, in a mathematically tractable way, to start and update the values of the weights going backwards, forward. There's also forward ways of doing it as well. So that's all of the techniques that people are working on to try to make that better.

All right. We already talked about improving models. I guess what I'll say — I'll just show you a couple examples of — how are we doing on time?

STUDENT: It's 11:36.

JOSH BLOOM: OK. Couple examples of machine learning approach to classification of what we've done. I told you about the 0, 1 classifier of real-bogus. We've also done this on variable stars. So now we've worked in the time domain, and we've taken photometry and other things, and we wind up building a classifier off of that. So here, we took what is very heterogeneous data. Sometimes we've got photometry in r-band magnitude. Sometimes it's regularly sampled data. Oftentimes, it's not regularly sampled data in time. And instead of throwing the raw data in, in RA and dec and all that stuff, we build a whole bunch of features off of that. So you can build features in the frequency domain. You can build features with unordered statistics, et cetera — so variability metrics, periodic metrics, shape metrics on the light curves, and then context metrics of where this thing is in the sky and what's near it. And we were able to get some pretty interesting results out of that.

We wound up building up — and this is now still a work in progress with some people that are in the room — essentially a framework that allows any of you all, and hopefully a lot of your colleagues, to start getting not just the pipeline that I showed you at the IPython notebook level, but even data handling and project handling and reproducibility notions out of your data. This is a project we call Cesium, and we just released the initial version of this. This provides a lot of the featurization capabilities around time series data, and then also a whole bunch of access to scikit-learn modeling and feature selection, et cetera, and then even plotting that allows you to build up entire frameworks even around some inference problems you might have in the context of time domain. This is different than scikit-learn in the sense that scikit-learn is a set of models, and it focuses very much on the modeling part, not so much on the featurization part. We've spent a lot of time on the engineering components around the featurization. And our idea is that we're going to be able to take this to other domains as well, like in seismology and neuroscience, and take what is fundamentally a similar type of signal — just measurements as a function of time to predict outcomes — and use essentially our feature bank to actually get good answers. So I'm happy to talk with people about that if they want to start using it and playing with it. There's a way to install it and get it working on your laptop with Docker. So we applied our classifier that we built on variable stars to something like 50,000 stars from ASAS, where we only had a training set of 810 over 25 different classes. So this was a hard problem — our training set size was really small. And we try to get, out of light curves like this, what the probability of it being an RR Lyrae is. And we try to do that as far back to the raw data as we could. And we got something like a 15% error rate, which is pretty good across these multiple different classes.

And we built a website called bigmacc.info, if people want to check it out, where it allows you to peruse through the hierarchy of variable stars and then click into those. So here's pulsating, and we'll go into different types of RR Lyrae. We'll get a fundamental overtone RR Lyrae out of this. And these are the ones that were predicted, 405 of them out of the ASAS catalog. And we show the probability of it belonging to that class. And then you see the probability vector of what comes out of that, of belonging to the class. You see the raw data and the folded data. And then we made it social so that Facebook could buy us one day.

But anyway, for those of you that are working in variable stars, the whole idea is that you've got to start getting used to probabilistic catalogs. So when we say this belongs to this class or that class or that class, the real answer is it belongs to this class with this probability, and it belongs to that class with that probability. And oftentimes, as astronomers, we want to take spectra of things that are of that type. But intrinsically, we know that there's some chance that they may not be part of that type. We're trying to formalize that in studies like this that allow us to put these different objects into different buckets and allow us to do science across those different buckets. I can go into the details of that if people want to ask me about it offline.

The last thing I'll say — and then I think I'll probably end, and leave for you in the notebook, you'll see there's some stuff on doing anomaly detection — is that we did a study with a student of mine named Adam Miller, where we looked at Stripe 82 in Sloan, which was effectively a five-color study in time of all the stars that had essentially shown some level of variability over several years. And then we built a classifier on the variability metrics, using all the parameters that I showed you before, to predict stellar quantities that you could only get out of spectra. So that's temperature, gravity — log g — and metallicity. And we got about 5,000 spectra or so ourselves, or they existed in the catalog already. And we wound up showing that using time domain parameters plus colors, we're able to get root mean squared errors which are comparable to what you would get out of low-resolution spectroscopy itself. So here, we used random forests. We did lots and lots of featurization, tried to protect ourselves as much as we could against overfitting. But this is just another example of using machine learning in real-world contexts. What we wouldn't claim is that you could take our model and then apply it to another time domain survey and get as good of an answer out of it. That's a whole separate problem. But in principle, if we got more Sloan data in other parts of the sky without taking spectra, we could, in principle, learn what the fundamental parameters are of that star or stellar system.

All right. I'm going to end there. We'll have time for a couple of questions. Let me just say, again, just to reiterate: think of machine learning as another set of tools in your toolbox. And if you haven't been trained on it — and by the way, I've just given you an opening to all of this if you haven't seen this before, so don't consider yourself well-trained — there are lots of ways to hammer your thumb with your new tool. And you may think you're building this amazing house, but in the end, you've built complete crap. That happens a lot. Machine learning is fraught with places where you're introducing biases that you didn't know about. But it can be very powerful. And as long as you're asking a question that's appropriate of the data that you either have or you plan to get, where machine learning is the right type of tool, you're off to a good start.

And when in doubt, it's worth asking other people in your field. And in particular, for those with home institutions that have stats departments and people working on machine learning, it's not a bad idea to go to them and say, here's the type of problem I'm trying to do. Give them a little tutorial for five minutes on the kind of science you're trying to accomplish. And let them tell you whether they think this is appropriate for machine learning or whether it's not. Because oftentimes, it's not appropriate. So just bear that in mind.

I often come back to this nice quote that Jim Gray — who was at Berkeley and then at Microsoft, sort of prototype of a modern data scientist — whose quote was, “I love working with astronomers because their data is useless.” And he meant it, because if you're in Microsoft and you're trying to build new algorithms against data, the data that you typically have access to has personally identifiable information. If it leaks out that you're using it, it's really bad. But he loved working with astronomers because our data — who cares? Like, oh wow, you showed me a galaxy that I wasn't supposed to see. It's not the end of the world. You're not going to start a war. So statisticians and computer scientists actually really like working with astronomers, because our data is pretty big. It has some interesting properties — noise aside; most of them don't like to think about that. But there's lots of questions to ask of that data, where machine learning may actually be useful. And they can hone models, try new scaling curves, get new benchmark data sets around that. But the flip side isn't always true. Just because you've got data and you're an astronomer and you've been at this Hack Day doesn't mean that you should be trying all of these out against all these cool, fancy new approaches that you've been reading about at blog post level. So just be careful. And if you're careful, I think you'll go a long way. So I'll end there, and happy to take a few questions. [APPLAUSE] Yes.

STUDENT: On that note, have you seen machine learning being used in contexts where you could tell straight away, that's completely inappropriate?

JOSH BLOOM: So the question is, have I seen machine learning being used in places where it's completely inappropriate? Yes. Oftentimes, if it's truly inappropriate, it won't make it all the way out to the public sphere. But there are lots of examples from people who reside in computer science and stats who have applied machine learning to questions that aren't all that interesting to astronomers, using astronomy data. I won't name names. But they have the other problem of not knowing the right questions to ask, and not being able to evaluate independently, other than at the accuracy level, how well they're doing. So if you build a photometric redshift estimator on photometry, and you only look at the r-band magnitude or something — even if you get a classifier which is pretty good, that's OK, but we know that for all intents and purposes, when you're doing photo-z's, you're doing it so that you can do something in cosmology. And there, you need to also have your uncertainties, but you probably also bring in lots of other data to bear. And so looking at this one benchmark data set that has just that data available just is cool because they use astronomy, but it's not all that useful for any of us. Any other questions? Dave.

STUDENT: Just wanted to make a comment about probabilistic catalogs. Maybe you thought about it a little bit. We're obviously very interested, in my group, in probabilistic catalogs. And one issue is that if you want to combine data from different sources, you really want likelihood information to combine data, not posterior information. Implicitly, random forest generates posterior probabilities that things are in different classes, and so there's an implicit prior that's been applied. And — just first of all, just as a general comment — it's very important that when we release probabilistic catalogs, we also release the prior information, the implicit prior information. But can you say a few words about what the implicit priors are for the random forest classifier?

JOSH BLOOM: So one obvious prior is that the distribution of your labels in the data you haven't applied your model to yet is the same distribution as in your model. If you're building a star-galaxy classifier, again, there's vastly more stars in the Sloan catalog than there are going to be quasars or galaxies — I think that's true. And so if you now apply it to just a pure, let's say, subset of galaxies, you're going to wind up getting answers that don't make sense, because I already knew that they were all galaxies. Why am I sometimes getting the answer of star? So that's there. And the imbalance in the data set, in principle, is being learned by all these different classifiers. You can figure out ways to, post-hoc, pull that assumption out without breaking the classifier itself. But that's an obvious one.

The other one, which is obvious, is that you're assuming that the classifier that you then apply on new data isn't just looking at the same universe. It's acquiring the data with the same noise properties, done in the same part of the sky. So again, if I built a star-galaxy classifier and I just happen to use it around places where there were a bunch of stars, like in the galactic plane, and then I applied that blindly to a place somewhere else in the sky where there's just more galaxies around, you're going to get wrong answers. Or if I did it not on the Sloan catalog, but now I did it on PTF, I'd get wrong answers. So there's the obvious mismatches of what you're actually looking at and how you obtain the data. There's probably other, more subtle priors that I'm not coming up with at the top of my head, but those are the obvious ones. OK, thanks.

STUDENT: Thanks again, Josh. [APPLAUSE]

Machine Learning for Businesses

Tue, 19 Jan 2016 00:00:00 +0000

On deploying ML in enterprise software: why ML is not a drop-in module, data-scientist stratification, abstraction layers for ML applications, and technical debt in ML systems.

Machine Learning in Production: Trials, Tribulations & Triumphs

Wed, 23 Sep 2015 00:00:00 +0000

Lessons from putting machine learning into production at wise.io, for the SF Machine Learning meetup hosted at Instacart.

A Systems View of Machine Learning

Sun, 26 Jul 2015 00:00:00 +0000

Keynote at the Microsoft campus arguing that real-world ML must be understood as an end-to-end system — data ingestion, featurization, modeling, deployment, and feedback — not just algorithms.

Key Quotes

“What are machine learning systems? In my view they're living systems, both influencing and reacting to their environment. At best they're valuable, resilient, functioning systems composed of many imperfect parts, with many weak contracts between them, built by fallible individuals with broken communication channels amongst them.” – Joshua Bloom

“If you can't define your loss function you can't optimize, and if you can't optimize then basically you're fishing.” – Joshua Bloom

“Real data is not platonic, it's plutonic… It's ugly, it's a dusty snowball with mountains and warts and geysers, and it's got NaNs and missing quotes and all that stuff. It is incredibly ugly, but at the same time it's incredibly rich.” – Joshua Bloom

“When it comes to ML systems, it takes a village. Or another way to say this is, data science as a team sport. You have to have interdisciplinary teams.” – Joshua Bloom

Transcript

Hello everyone. So when I talk about machine learning systems, I'm thinking about those that are in production rather than those that are one-off data science projects. Not to say anything bad about those, but this is really the focus of this talk. So what are machine learning systems? In my view they're living systems, both influencing and reacting to their environment. At best they're valuable, resilient, functioning systems composed of many imperfect parts, with many weak contracts between them, built by fallible individuals with broken communication channels amongst them, living in a resource constrained world that's constantly changing, the results of which are consumed by capricious and exacting individuals. In short, this is very hard stuff.

I'm very thankful to be here today. It's a great honor to speak in front of you. I also had the honor of being one of the keynotes at the last PyData, so the organizers have presented a major challenge to me: to essentially give a completely different talk to the same audience in the space of one year. So I'll do my best. You've heard a bit about me already from Fernando. I thought I'd leave you, or at least start off, with a graphic that would maybe shock you out of your lunch stupor. Does anyone have a guess what this is? It's Pluto, yeah. One of the new Pluto images from the New Horizons project run through Google's deep dream. If you stare really carefully, you will freak out.

So the reason why I'm showing this is because in some sense this is the confluence of who I am. I've been doing machine learning in production in the context of academia for well over a decade, and as you heard from Fernando, started a company that's doing machine learning, not quite as a service, in a slightly different way. So this is the marriage of those two worlds visualized for you, again just to freak you out. Of course, for those that actually know me, they'll know that I'm not a solar system expert, nor am I a deep learner expert, but this is the closest I could do in some sense.

That work that I've done in the past, both in academia and in industry, informs some of the things I'll be talking to you today about. But the point of departure for this talk really is this paper that came out by folks at Google called machine learning, the high-interest credit card of technical debt. One of the things that I think is very shocking about that paper, which by the way is probably even more shocking than the image that I just showed you for those that are practitioners, and it's certainly worth a read, is this quote that you see up there: it may be surprising to the academic community to know that only a fraction of the code is actually doing machine learning. A mature system might end up being at most 5% machine learning code and at least 95% glue code. So when we think about machine learning and we think about the innards of machine learning and the algorithms and the theory behind it, when you start thinking about machine learning in practice in real living systems, it tends to be a very small fraction of that. By the way, that word glue code should be a bit of a trigger point for you, because that's where Python starts to come in. Python is a wonderful glue code to all the different projects and implementations at perhaps a lower level programming language, and allows us to expose that to a broader audience.

So I'm not going to go in and summarize and give a talk around that paper, but just some of the highlights for me are the following. The idea that there are complex models that effectively erode the abstraction boundaries between the different components of that model, and that's the data dependencies, which are more expensive than actually the code dependencies, which is somewhat unusual if not completely novel for software engineers. It's a system of spaghetti code that has to interconnect, and again it's interoperating and interacting with the external world. So again, this paper is well worth a read. It's good summer reading. It's going to be on the test, so please do read it.

All right, so what's the agenda for today's talk? It's to discuss in some sense from the inside out, from the algorithms, the software, to the hardware that this software has to live inside of. And then of course, as we start thinking about hardware in the modern era, modern hardware is basically composable and programmable with software, so we've got another layer around that. And then we've got the people that actually build this stuff and care about this stuff and maintain it. They of course are only doing this because it'd be crazy to do it by any other means or for any other means, with a consumer in mind. And so the result of a machine learning project or machine learning system in production must have the end user in mind. And then of course there is societal impact to everything that you wind up doing in machine learning. The society could be at the organizational level of the company that's surfacing it, and it could actually be something that's even larger than that. In some sense we heard from this morning's keynote about some of the systemic issues of dealing with and understanding machine learning and some of the trade-offs that we have to wind up making software in this world. And this worldview is connective tissue between the hardware, and it is the thing that's instantiating all the theory and surfacing it, or at least as part of the original layer surfacing it back out, for all of us to be able to use it.

So I'll go into a few aspects of these different components. I'll present some of the new facilitating tools that I and my group and others have been building to help with these components, and then talk about the impact of how you do problem definition in the context of this and also in the context of teams. So of course many of you will see your favorite algorithm or sets of algorithms on the left hand side. Many of you won't see that and you'll get mad at me, and that's completely fine. This is just examples of the kinds of things that we all know are being used in practice in machine learning systems. And the wonderful thing about where we've arrived as a community in Python is that there are some really great packages that are exposing these amazing algorithms that have these wonderful properties. And I think it should be clear to anybody here that despite the provability of some algorithm being convex on paper, until it gets exposed at the software layer and until it actually gets run in real world hardware, it really is vaporware. And so in some sense software is where the rubber in theory hits the road in practice.

What I wanted to point out though is that as we start thinking about growing out of these layers, from just getting out of theory land, there's a really wonderful blog post that John Langford, who's now in Microsoft research, wrote I think it was in 2007, entitled all models of learning have flaws. Lest we forget that our favorite model or classes of models that we like to use to solve our problem actually have some fundamental flaws with them. And so this is a couple of images that are basically subversive to a well-trained deep learning model, that basically trick it into saying something. The other thing that's obviously too much detail to go into for this talk, I just wanted to point out, is that oftentimes we can convince ourselves that these algorithms seem to work really well on our own data and we've got some beautiful held out testing data, when they actually are then applied, almost, I don't even say transfer learning, you just take an entire model and you apply it to a completely different data set that should work because you've told people that I have bounds on my errors, in the context let's say of images, don't work, or at least they don't work nearly as well as people think they are, because there is no guarantee when you build a model in machine learning, even that you've got testing errors, there's no guarantee that's actually going to work when the distribution of the data actually winds up changing.

So this is a couple of different projects and training sets on which you can build these very large classifiers to predict the word car basically from an image. And what you wind up seeing in that diagonal is what was the reported testing error for those projects. And then if you actually apply that same model to the other sets of images and you look for the word car, you wind up noticing that the testing error is actually much worse, or, that is, these are actually accuracies, the accuracy is much worse. So compare 28.2 to 10.6 and you wind up realizing that something's really wrong for a paper that has basically said

these are our errors and this is how well we do relative to everyone else. By the way, does anyone know what the original adversarial image is for deep learning? I found it actually in the literature. It's from McCulloch in 1929, he wrote this up in ICML. Only a few people got that, that's okay. So really the question is, as you're building out these systems, what are you optimizing for? And if we start again from this inner part of the onion and move ourselves outwards, obviously at the algorithmic level and the model level you want to think about learning rate, you want to think about convexity, you want to think about error bounds, you want to think about scaling. And all those things are really important, and people write papers in the academic literature that say my scaling curve is better than your scaling curve, give me a PhD. And that's okay, but when you're thinking about that, typically those are the things that you wind up caring about and you forget about things in terms of the trade-offs between the accuracy that you're going to wind up obtaining and the memory usage and the disk usage and the CPU needs and the time it takes to learn and the time it takes to predict. These things don't show up very often when you're just thinking about theory, and that's again okay, but it's a smaller view of the things that you have to care about.

And then at the project level, and the people that are minding over not only the construction of these systems but the maintenance of these systems, it's what is the time to actually implement one of these projects? Think about it this way: if I could give you a perfect spam filter, would you take it? And you can't answer that question truthfully because I haven't told you how long it's going to take me. Could take me the age of the universe and I guarantee it's going to be perfect, or maybe something less time, but you get my drift. You can't talk about building something that has perfect properties and not think about all the impact on resources. What is the impact on people costs? What is the impact on their time? What is the reliability of these algorithms? How robust are they to the changing world? Are they maintainable? Can you do experimentations with them easily? And we're not done. Then start thinking about something even broader: what are the things that consumers actually care about from these machine learning systems? They want direct value, they don't care about really what's happening under the hood. They want usability. Many times you actually need explainability in why you got to a decision that you got. You'll hear time and time again from some of these large corporations that have been using machine learning in production for decades that they're using linear algorithms. And the reason why they're using linear algorithms is because the CEO can understand it, and when it doesn't work they can explain to the stockholders, well, this is a plus b times x, and that's it. Try to get that into production when you have an incredibly complex model with thousands of dimensions, you have some nonlinear mapping over that, and that just gets harder and harder and harder. Explainability for real consumers in the real world turns out to be one of the most important things for machine learning in production. And then of course at the societal level, what are the direct impacts of the existence of this machine learning system in production?

I think the big view here is that if you have a narrow view of just one part of this component, it can be very very costly when you actually start putting it into practice farther up the stack. So I'm going to pick apart a couple of things around, say, accuracy, still staying close to the algorithmic level, and then also start moving more and more in towards the hardware. When it comes to optimization, we have to ask this question: what is the metric that I really care about? So this is something that we did within my company. It's a 100 dimensional space that we've done a regression problem, we're basically predicting a certain number. And it looks pretty good if you look at the R-squared, it's 0.91. You can come up with some other metrics, root mean square error, etc., and all this stuff looks really good, right? So I'm done, right? In fact, this is the best that I could do. But what I haven't told you is what I actually really care about. Maybe I want to minimize the scatter about this line. Maybe I care a lot about the bias in the numbers that are predicted to be below zero, and maybe that's where all the value is for me. And maybe I don't want to ever be wrong about outliers. So until I tell you what it is that matters, you can't just throw your data into an algorithm, get an answer, and move on.

I think throughout this conference you've been exposed to these so-called ROC curves, receiver operator characteristic curves, where you're plotting mis-detection rate versus false positive rate. It's basically type one versus type two error. And typically what people want to do is minimize all the stuff to the bottom left to get a very good classifier, right? And for some reason, well, I know why, but it's kind of weird, this is called area under the curve and you want to maximize that. So the larger the number, the closer to one, the better, you go all the way down to the bottom left with these curves. So who can do the integral in their head and figure out what's got the smallest AUC? Who thinks red? Who thinks blue? Green? Okay, everyone's sleeping. It's red. So red is the best one, you want to basically minimize your false positives while you also minimize your false negatives. So which one do you choose? Which one do you put in production? This isn't quite fair, it's a bit cheeky, I haven't told you what the question is yet, so you can't answer that my AUC is better than your AUC. But here's the reason why I would choose a blue curve over a red curve, or a green curve over a blue curve.

Let's take two extreme examples. So if I have a product that works in the intensive care unit and I want to make sure that the people that are connected to this thing, basically when they have some problem with something in their cranium and you get a lot of expansion, you want to make sure the nurses are alerted to some extra pressure in the brain. This is actually a real world problem. And so you can build these systems that basically come up with alerts, and of course you want to minimize the number of false alerts, but you don't want to miss too many of these actual alerts, right? So in that case where do I want to be on the curve? Well, I don't want to have any mis-detections because then the patient dies, and for that I'm willing to give up on a whole lot of that AUC and I have to choose the blue curve. So with the blue curve my false positive rate can be very very high but I have zero mis-detection rates. Likewise, if I'm setting up some, let's say, drone system, and I want to shoot down enemy planes and not my own drones, I probably want to be on the other curve. And the other curve is one where I'm willing to take a large mis-detection rate just so I don't blow up my own stuff. My false positive has to be zero and my mis-detection rate has to be whatever it has to be to get there. So if you can't define your loss function you can't optimize, and if you can't optimize then basically you're fishing.

Accuracy winds up coupling in real world systems not to other properties of how this thing scales. Of course it does that at some level, but the thing that you have to really care about is some notion of implementability. If I can't put this stuff into practice, it doesn't matter how good it is on paper. So I think one way to illustrate this is by looking at both the Netflix prize, which I think is now about 10 years old, is a million dollar prize to try to get people to improve their recommendation engine by 10%, and the team that won, won basically with a few decimal places better in the metric that Netflix cared about over the other teams. What I've done here is I've taken both the Netflix prize and all the public leaderboards from that and all the public leaderboards from all the Kaggle prizes that have ever been done, which are running Netflix-like prizes, and you can split this up between very valuable prizes and not very valuable prizes. And I've normalized it by whatever the winning metric is, so if it's a number they're trying to minimize, I've basically reversed the normalization, so all of this stuff should be largely on the same scale, because again, the way that Netflix was evaluated and the way that Kaggle is evaluated is with typically a single number. And what you wind up seeing is that there are lots and lots of teams for these expensive prizes that get within some normalized metric of the final answer, and they do really really well. For the ones that aren't very valuable, you wind up seeing fewer teams actually participate and fewer high quality teams participate.

So which one do you wind up implementing? If you get within Epsilon of the best answer you could possibly get, maybe somebody who came in second place has essentially a linear classifier and you could write this in 12 lines of code, and maybe the one that wound up winning has some incredibly convoluted classifier that would take a long time to do. So when you think about the trade-off between accuracy and putting things into production, you have to be mindful of this, and Netflix themselves saw this in spades. Those of you that know the history of what happened after the Netflix prize was given out, is I think they figured that it would take about a 100 man years to put the winning algorithm into production. And I encourage you to read this blog post by Xavier and Justin on just what they learned out of that. They looked at all the different pieces of that code, that was one of the requirements of that competition, and they realized there are a couple cool things they could take from that, it just didn't justify putting it in production. Now you could say this was a complete failure in some sense, to have launched the data science competition world, not to have had a bigger impact on data science more broadly, but you could also say, well, a million bucks was a pretty small price to pay for Netflix to not have to throw tens of person years, data science years, at trying out a bunch of stuff. They just got everybody else to try it out for them. So it's not clear that this was obviously a failure.

So putting algorithms into production is absolutely critical. In my company at wise.io, we wound up realizing that the kind of things that we wanted to do for our learning algorithms we weren't able to do at the time on existing public code bases and existing implementations of algorithms. And what you wind up realizing is that while these algorithms work really well on paper, until you start putting them into computers and start dealing with issues of RAM, you wind up realizing there's a lot of inefficiencies out there. So this is a very big picture view of the core ML stack within the company. This of course is orchestrated in a much larger environment with containerized data science workflows, etc., that interconnect with the rest of the world. But we needed fast, memory efficient, scalable, composable data science that we could actually do on a laptop and then move it into the cloud very easily. And really the end goal of the company from a data science perspective and engineering more broadly is to have our implementation folks manage that entire setup. So it's not the data scientists or the engineers that are working on and setting up new clients, where we basically wind up seeing the difference between customers from the back end is the difference between config files.

So what our API was meant to do, from the very low level, was reduce the friction of common tasks in data science, like subsetting, cross validation, computing ROC curves, etc. And we really focused a lot on computational efficiency, so most of the work that we've done is at the C++ level, in the form of something we call Wise data set, which I'll spend a little time on, and Wise transfer, and then I'll also talk about this other component called Wise wind tunnel, both of which we're hoping to get out into the open at some point. Some of these things are pretty new, especially Wise wind tunnel, so we're going to work on it a bit, but I've already started talking to many of you in the audience about it. So let me just talk very briefly about Wise data sets and how they're laid out. Again, we're trying to be able to do most of this work very efficiently in C++. The idea here is that once I've laid out the data very well and intelligently in memory, then I can build fast algorithms against them, again in C++. And really I think one of the highlights of what we've done is we take a given data set, say in CSV, and we pre-parse it into what we call variable groups. So all the things that are ints go into one variable group, the things that are strings go into other variable groups, and you can keep track of those at a metadata level, so that you wind up having mappings between, and actually mappings between the strings and the ints for instance, so that when you're doing comparisons you're doing them with ints and not with strings. And the nice thing about it, of course, is that this is an abstraction layer that the Python folks don't have to really think about, it just gets exposed to them, and I'll show you what that looks like. The cool thing also is that, because we've abstracted the notion of what these different data types are, we can simultaneously store dense versions of our columns and sparse versions of our columns.

The way this surfaces back up into the Python land is using something we call Wise transfer. See if I can get that to work, so you can see how that actually looks. And the goal here is to move as little data as possible from C++ land back into Python land. We found that we had to do something like that, and so we also had to play around with transferring data across from C++ into Python, and we wound up realizing that all the various implementations of this, like protobufs, etc., really didn't work. There were enough limitations on

those that we had to build our own transfer protocol, so we can basically transfer arbitrary data, can arbitrarily serialize C++ stuff back up the stack if we need to. And so we've tried to make this as easy for the data scientists to be able to use without having to think about all the stuff that's happening under the hood. So you see here we basically instantiated a data set, and the important thing that isn't clear from what you just saw is that train and test, which were part of basically a stratified sample, aren't copies of the data, they're just copies of pointers into C++ land. So we're effectively just creating views on the data, and it allows us to operate again very efficiently in C++.

So where does this live? It's funny, because in the 80s everybody wanted to be a DJ, I feel like now everybody wants their own data set abstraction. We did that as well. This is somewhat opinionated, backed up by our own internal tests against these various different implementations. Starting from the far left to the far right, it's the sorts of things that data scientists care about down to the sorts of things that implementation engineers, C++ folks, care about. So slicing induces copy is a really important one. If you've got a big part of the data and you want to build, let's say, a train test for a holdout set just by slicing your original data, if you have to make a copy, if that data is very large, that becomes problematic. Immutable columns, things like queries, transfer between Python and the lower levels, that's actually very important as well, that's something that Spark data frames is doing quite poorly right now. C++ SDK, it means that I can write against these data set objects at the low level. Is it distributed? Is there a memory efficiency with the way that you've laid data out in memory? Have you optimized the notion of what categories are of those data sets? And again, can you use sparse and dense at the same time? So this isn't to say that what we've built internally is a panacea, but for us, building very fast machine learning algorithms that are memory efficient without giving up accuracy and having to make trade-offs is, I think, really really important.

Okay, so let me move on to a somewhat larger view of this system, and that's about enforcing weak contracts between the lower levels of the system and the higher level ones. And in some sense this is really how we monitor deployment. So here's a very generic example that those of you that are working with ML systems in production probably know about pretty well. So I build a data science workflow on a test set, I like the offline testing accuracy, so I'm like, great, I'm going to deploy it, and I'm going to start monitoring the results in practice. And of course, because this is how I've set up the problem, online, when we actually start looking at the results, it's worse than we expected from our offline test. So what do we do? Well, you can bang your head and say, well, I overfit on the model, I'll go back to the drawing board, I'll do some more data science, and that's probably fine. I can just retrain because more data has better answers. You've probably heard that a lot of times, that's Peter Norvig's thing. Or maybe it got worse because there's this thing called concept drift, that the underlying data distribution changed and the model that approximates the predictions you wanted to make should indeed get worse. So if you've built up a data science workflow that, let's say, hardcodes which features you wind up using and you're not doing automatic feature selection or automatic hyperparameter optimization, you can wind up basically hurting yourself quite a lot. If it turns out that upon retraining, where you figure out those other parameters, you still get a pretty bad answer in production, you can go back and change that data science workflow, maybe you realize that new features that you hadn't thought about become important. Or maybe, just maybe, that's okay, because your prediction influenced the outcome, right? That was the goal in the end of what these ML systems are.

If I'm making predictions, as we heard earlier today in the context of Stripe and in the context of payments from Mike, if you're basically making predictions about whether this transaction is going to be fraudulent, and you don't let what you think are going to be fraudulent transactions through, the next time you go and retrain, unless you're being very very careful, you've basically influenced the world. And so there's something about the way the data looks today that's different than the way the data looked yesterday, and so when you try to glom all that stuff together you get a really strange view of the world. So there's a lot of people obviously thinking hard about these problems, and real companies that are putting real ML systems in production are thinking about this probably as the forefront of the things that they have to care about to get good answers. There's also a very good talk by Chris yesterday on A/A testing and A/B testing that's relevant to some of this discussion. But there's more, of course. There's obviously the testing that you normally do, because all of you are awesome, across your entire software stack: you've got unit tests, regression tests, you're looking at integration tests, etc. But then there's also what I'll call a semi-new concept, at least in the Python world. We heard yesterday from Trey about a couple of cool tools that are helping along these lines. Basically, is the data changing, and are my predictions different than they should be given that changing data? Is the contract that I set up with the larger software infrastructure that I've set up affected by that changing data? And then there's another component of this, I'm still kind of trying to get to the words myself, of model deployment testing. I need to know when things are too different than they were before, I need to alert a human, and then I need to use automated tools to try to isolate the cause of that change. Was it data that changed, and that's what the ETL stuff gets to, or was it the code that changed? Because again, you're not deploying a machine learning system in production and then walking away and looking at it every year, you're constantly deploying new things into the system, so you need to make sure that you're not actually introducing new bugs and new errors that other parts of the stack didn't know about.

So we built this thing in pure Python called Wise wind tunnel, and the idea here is that if you start from a known good implementation of a known algorithm and then you start monkeying with it and start adding bells and whistles and things, in some ways you can basically start understanding whether, with the data that you're throwing through, you're getting the answers out that you expected, given the fact that I've made changes. So instead of hard coding, the AUC should be this value or the accuracy should be this value, say within some constraint, instead the way that we allow ourselves to do it at the decorator level is to say the AUC shouldn't change by more than a few sigma, where we've defined what a few sigma is from the previous run using different random seeds given the same data set. That has to be true. But it's not just testing on things like accuracy, we're also testing on things like memory usage and total time it takes to train, total time it takes to load data into memory. And so when you start doing that you get a probabilistic inference about how well you're doing relative to what you had before. Now you still have to have something at the very beginning that you wind up trusting, but to the extent that we're thinking about this as almost like a git project, everything that we do when we do a new deployment becomes a new git hash and allows me to go back and reproduce the code that got us those exact answers along with the same seeds, so we can go back and make sure that we're not messing anything up. Unexpected surges in memory cause crashes, for instance, and you can protect against that, or at least test against that, and that can be as critical as protecting against loss of accuracy.

So when we think about these weak contracts between the different layers here, one way to think about it is that the abstractions of how you think about the implementation of your algorithm, in the context of the world that you have heads-down focus on, those abstractions wind up leaking into the other parts of your system. So one thing that Leon Bottou basically said at ICML along those lines, he came up with the very clear example I think that illustrates this leakiness very well. You've got a smart programmer which makes an inventive use of a trained object detector, recognizer. The object detector recognizes data that it hasn't seen before and it produces an answer, and then this smart programmer basically gets an answer that's complete garbage. And there's nothing in the way that we think about, say, an algorithm in deep learning that protects us naturally against this, we have to build larger systems around this. And why is that? It's not just because the theory is really bad for this algorithm or that algorithm. The way that the Google paper put it is machine learning packages are inherently a tool that mixes data sources together, and because they're doing that you create entanglement, and that entanglement is very hard to take out. We have not software to blame for this, or theory to blame for this, we have data to blame for this. We like to think, when we're building out our algorithms, of this platonic form, this crystalline sphere of data, but real data is not platonic, it's plutonic. Sorry, I had to do it. It's ugly, it's a dusty snowball with mountains and warts and geysers, and it's got NaNs and missing quotes and all that stuff. It is incredibly ugly, but at the same time it's incredibly rich.

If we wind up looking at, say, one document or a series of documents, you wind up realizing that the different pieces of that data require different sorts of attention in the featurization of it. So you've got natural language processing, which could produce very sparse output, you've got computer vision, which may be a nested thing, you've got metadata, which can produce dense output, time series, so there's a concept of streaming, and of course if you're pulling over third-party data about this, many times that turns out to be missing and noisy. So the richness is wonderful, and what you wind up realizing is that to do this right, to do this featurization correctly, you need to have very deep domain knowledge. And so doing machine learning without that knowledge is very painful, and in fact may even be too hard. And when you think about domain knowledge and you think about your requirements of having that, in some sense what you're really asking is, are you asking the right question? How many of you know a botanist? Wow, that's amazing, are we near a big botany school or something? I wasn't expecting that at all, I was expecting crickets. So the reason why I'm asking is because there's this famous machine learning data set, it's the hello world of machine learning, called the iris data set, where you have to predict the different types of irises. Who's asked a botanist, of those that know a botanist, do you even give a crap about that? I've got this amazing classifier, it's got 100% accuracy and I can take an image, and they'll say, no, I don't care about that, right? Who's asked themselves or maybe somebody else about the provenance of this data? How come there are no errors on the length of the petals? Were all these data points taken of the same area with the same type of soil at exactly the same time after they were planted? No, or we don't know, right? We need to start asking ourselves about questions that are actually important. Not to say that this isn't an interesting data set as a hello world, but this doesn't actually have any real value, and because it doesn't have any real value it's hard to define what the real loss function is over that data.

One of the things that we found very hard in our academic work was the recognition, which should be clear to all of you, as me being essentially a trained physicist, that I'm not a machine learning expert, I'm not a statistician by training. And so for me to be able to answer these questions in a production environment that mattered, I needed to find people that were really good at the things that I wasn't, in the computational sciences specifically. I could ask the domain questions, but then getting those people to essentially build these large-scale machine learning systems so that we could answer really important scientific questions was hard. What I started realizing after we started building some projects around this in astronomy is that when you look at time series data in the sciences, there's a lot of commonality, from neuroscience to seismology. And the way that we started thinking about building out a project in astronomy, we started realizing some of the same tools that we were going to need to do that were going to be broadly applicable to those in other sciences. And so what we did is we got a grant from the National Science Foundation to basically build a platform for domain scientists to very quickly go through and get essentially featurization off of time series data, to be able to build out these scientific projects on time series data and answer questions that they want to ask themselves without having to spin up a whole team of domain experts in machine learning and statistics. So I won't go into the details of this, this project is on GitHub and we're certainly very happy to have people contribute to that. One of the things that we're really excited about is the way in which we do continuous integration, basically leveraging Docker, and we've been very thankful and fortunate to have Microsoft Azure devote some time on their cluster for us to be able to work.

What was the output of this? We wanted to produce basically some of the scientific papers that we put out several years ago with the same raw data sets with essentially just a couple clicks of a button, and our MVP, which we're now arriving at, is our ability to produce results that are comparable to the ones that we had before. So one of the things to recognize is that making predictions, let's say, about the classes of variable stars is one thing, but you really want to use that as a launching-off point to something else of some sort of deep interest. So based on all of the work that we did in the past where we classified tens of thousands, if not hundreds of thousands, of variable stars, we were able to find some really interesting needles in a haystack that were essentially impossible to find by any other means. And I won't go into the details of all these things, but the point of this is that we basically had an end goal in mind, we had something that we really cared about, which is pushing the envelope of knowledge, and in doing so we had to do machine learning. I didn't do this because machine learning is fun. I mean, it is fun, but it is really hard.

And one of the things that we all have to ask ourselves is, you see this data set, you say, I've got this pretty awesome hammer, I'm just going to start hitting it because it looks like a cool nail. There is a real danger in starting off and trying to solve problems with machine learning systems before you actually think about the problem deeply and can you solve it in other ways. In the company at wise.io, we're basically selling products into the CRM world with a non-technical buyer and a non-technical user, and we're basically focused on routing, say, support tickets. When you send an email to a company, there's a lot of people that have to look at it, and optimizing and automating that is the domain problem that we decided we're going to solve. What you wind up realizing whenever you're solving some sort of large machine learning problem, even if you're doing it well, is that you're still going to make mistakes. And the way that you make mistakes and don't shoot yourself in the foot is you build what's called fault tolerant machine learning. Gmail is famous for doing this really well: I've categorized this message as spam, if it's not spam, that's okay, tell me why, and next time I go back I'll basically build a slightly better spam filter for you. Microsoft Kinect uses random forest, and if I get the pixel wrong of, is this the left thigh or the right thigh, the next frame you get it right, it's not a big deal. At wise.io, this is a view of what an agent sees within Zendesk as they're basically going through the tickets. We're giving them recommendations about how to respond, and if they don't like those recommendations, they can search and figure it out. And it seems very simple, it's abstracted at the UI level, but you have to think about your audience, right? You have to think about, as you build these and deploy these new ways of doing machine learning in production, what's the impact?

I think it was about a week ago, maybe two weeks ago, that the Gmail group said we're using deep learning now and look how great the ROC curves are getting for all of our people using our spam filter, and it was pretty impressive. And then a couple days later Linus Torvalds posted this, he said, something you did recently has been an unmitigated disaster. Of the roughly thousand spam threads I've gone through so far, now 228 threads were incorrectly marked as spam. Google pissed off Linus, he didn't like his ROC curve. I mean, let's face it, Google is the best machine learning company in the world, and they sometimes struggle at what is one of their core end-user facing products in machine learning. And it isn't just Google, Netflix also makes this mistake sometimes. This was on Reddit a couple days ago. Is this fault tolerant ML? Yeah, I mean, this guy isn't going to cancel his subscription, Linus isn't going to stop using Gmail. This is very very hard stuff. And one of the things that Leon said in ICML was that machine learning disrupts software engineering. So what should be the machine learning engineering process? The my-scaling-curve-is-better-than-your-scaling-curve is not the answer, or my version of this algorithm has a slightly better accuracy than this. You have to have, and this is the scary part of it all, you have to have a 50,000 foot view of the entire system as you're building out and working on these various different components. It's not an intractable problem, it's a really hard problem, and I guarantee you it's not going to be solved by one type of person.

And really the last thought here, the parting thought, is that when it comes to ML systems, it takes a village. Or another way to say this is, data science as a team sport. You have to have interdisciplinary teams. And this should echo, for those that were there at the keynote this morning, some of the things that you heard there, of the cry for that. We often talk about in academia the need to build these pi-shaped people who are really good at domain questions and really good computer scientists and PhDs and everything. And my view of this world is that there isn't this notion of a pi-shaped person. It's dangerous from an educational perspective to be thinking about how we create pi-shaped people. We need to create people that are very deep in something and maybe somewhat deep in some other parts, and we need to get a bunch of gamma people together, gamma-shaped people, to solve some real world problems. So, unless you haven't been paying attention, I think it's reasonable to say that just because you've got a good software package that implements a whole bunch of machine learning algorithms, it doesn't mean that that's a functioning ML system. There are many other considerations when you build these ML systems. It better be because it's the last resort, because you have to. What I like to say is, if it's not worth doing, it's not worth doing well. Machine learning systems require optimizations across components, so we better understand the true loss function, not just in our restricted domain but at the largest level. And hopefully this isn't a completely lost problem, and the novel testing frameworks can strengthen those abstractions within components and the contracts between them. And regardless of what you do, the last line of defense is some notion of a fault tolerant machine learning. So with that, I'll say thank you. [Applause]

Data Science Infrastructure & Time-Domain Inference

Wed, 15 Jul 2015 00:00:00 +0000

Data-science infrastructure and time-domain inference work, at the Moore Foundation DDD investigators meeting.

Data Science Education: Needs & Opportunities in Astronomy

Wed, 01 Jul 2015 00:00:00 +0000

On the growing need for data-science training in astronomy curricula and the opportunities for cross-disciplinary education, from Berkeley bootcamps to institute-scale efforts.

25-slide deck on SlideShare; exact venue/date uncertain.

Astrophysical Insights in the Presence of Uncertainty & Under Duress

Fri, 15 May 2015 00:00:00 +0000

Astrophysical inference under noise, uncertainty, and real-time pressure, at the Simons Foundation.

Machine Learning in Production

Fri, 15 May 2015 00:00:00 +0000

As Wise.io co-founder/CTO: what it takes to deploy machine learning in production enterprise settings, and how industrial ML practice differs from academic ML.

Month per FirstMark slide upload (May 2015); one listing says Nov 2015.

Key Quotes

“Astronomy in some sense had a big data problem 120 years ago. When we had this big data problem, what we did is we hired people.” – Joshua Bloom

“When you're deep in the weeds in a data science organization that isn't highly connected to the product… you can lose sight of the fact that what you build and what you put in production has to be useful.” – Joshua Bloom

“Netflix and Google, some of the best machine learning companies in the world, and these are their core products, and they still make mistakes. Yet these are not fatal mistakes. They've built fault tolerance into the machine learning.” – Joshua Bloom

“We don't really see ourselves as a big data AI company… we're solving interesting problems that don't show up in the academic literature, more around featurization than it is around the actual model building itself.” – Joshua Bloom

Transcript

Hello everyone. My name is Josh Bloom. I'm co-founder and CTO of wise.io. I'm also a professor at UC Berkeley. Tonight I'd like to talk about machine learning in production. And when I talk about machine learning in production, what I'm actually trying to talk about is the data products and the interactions that all of you are applying to data that's unique to your own business and transactions that are unique to your own business. But many of you are also thinking about building outside of the core of your business into some of the aspects that actually are very similar across multiple industries. So the key question that I want to pose tonight is: where is the AI build-buy decision boundary? And many of you as data scientists are going to wind up having influence on that decision. So I hope I can give you some thoughts that help you frame that question for yourself and your organizations.

My own point of departure is as an astronomer, and astronomy in some sense had a big data problem 120 years ago. When we had this big data problem, what we did is we hired people. So we had what were pejoratively called computers. These are mostly women who are looking at data, and they were making high-level decisions at some times, when many times they were basically just pulling data out of images. There were too many photographic plates that were coming from the southern hemisphere, so the director of the Harvard College Observatory had to invent grid computing or parallel computing. And what's amazing is that if you look at this slide it seems so quaint, but this is actually how a lot of science and actually a lot of industry is still done today. When we look at data and when we touch data, we tend to think of needing people in the real-time loop. And one of the things that's so amazing is that this sort of repetitive knowledge work is still rampant throughout organizations.

After we wound up building essentially a machine-learning-based workflow that helped us to do astronomy better on some more modern problems, we started to turn our attention to other aspects where repetitive knowledge work was being applied in an industry. The place where we wound up landing is in customer success. So for those that don't know about wise, we basically build an artificial intelligence layer on top of SaaS-based data sources. And so we touch the conversations that you are having with your clients, and we try to make them more efficient and better. And we see customer support as a pretty interesting launching-off point to other value propositions within organizations. So rather than ask people to build new workflows around AI, instead what we do is we add essentially an augmentative or automated layer on top of the workflows that people are already using. I won't go into the details, and I'm happy to take questions on what the company actually does, but I just wanted to give you a presentation of our stack.

And this is the first time in this talk we start thinking about the build-buy decision. In this case not build-buy around AI, but build-buy around engineering practices. So it would be very easy of course, if you've got a good group of people, to stand up your own Kafka cluster or manage some multi-zone Postgres server. But instead, why not just have Amazon do it, or your other favorite compute cloud? So we made, as a young startup, the decision that if there was a managed service around what we needed to do to get something into production — and we're at the millions of predictions level a month over dozens of customers now — we were just going to buy it, or in this case we're going to wind up leasing it. So obviously I won't go into the details of where our stack is and all the things that we've tried to build, but suffice it to say we try to build a composable set of services that wound up talking to each other, and we try to make the most vanilla plain architecture as possible, so that we could focus on our core differentiators, which was building very fast and very memory-efficient AI and solving some real problems.

I also won't have time to go into the details of what our stack looks like, but what we wanted to be able to do is let data scientists in our own organization build, almost at a config-file level, the differences between different clients that we have, and put those into production without the need to talk to engineers. So we try to abstract away all that other stuff that you saw on the previous slide and allow a data scientist to essentially work in the kind of environment that he or she wants, typically in Python, and then be able to put it directly into production after testing it on their own laptop. And so that's what we've been engineering towards, and that's what we've been able to accomplish. And in part it's getting into some of the nitty-gritty of what the core IP of the company is.

When we're making these decisions — when I hope you're making these decisions — now in the context of machine learning, there is this question of what it is that you're optimizing for. And obviously if you're a machine learning expert and you come from academia, you might really be thinking about how your scaling curve is better than somebody else's scaling curve on some large amount of data on some toy dataset. And that's fine, right? So getting better scaling is important, getting better accuracy is important. But oftentimes when we're building these systems, we often will wind up neglecting other parts of the stack, and more importantly we'll start neglecting who our actual customers are of the types of products that we wind up building. There's this multi-axis optimization that many people don't think about. So deep inside you work on the algorithm, but you have to put that into hardware, right? And so is this algorithm that you've just built actually memory efficient? Are you trading off accuracy, memory usage and speed, and are you actually explicitly making those trade-offs in your head as you're starting to build these systems into production? And then of course there's a time to implement, which is often highly neglected when organizations are starting on a new project.

How long is it going to wind up taking you to take this cool algorithm that just got published in ICML and put it into production? How many person-years? How many person-years after that is it going to take to actually run the system? And then of course, is this actually valuable to customers? And that's something that we obviously always have to have in mind. But typically when you're deep in the weeds in a data science organization that isn't highly connected to the product, that can be a real challenge, and you can lose sight of the fact that what you build and what you put in production has to be useful. So I'll talk about some of these different trade-offs.

There's several hundred people in the room, so I'm probably pissing off most of you right now. In this trade-off between accuracy and interpretability, you'll hear more from some of the other speakers about their thoughts on this, but oftentimes in organizations the data scientist is trying to optimize towards higher accuracy, which makes perfect sense, but they may be biasing themselves towards algorithms that are harder to understand why you got that answer. There's a great example from a friend of mine who gave a talk from basically a real estate analytics company, saying that he showed us all the different accuracies for all these different algorithms, and he had everything from a linear algorithm all the way to random forests and even some deep learning. And he said, which one do you think we wanted to put in production? And it was very obvious that as you go far to the right there was a better and better accuracy on this one problem he presented. And in the end he said, we just chose the linear one. And the reason why is because their clients needed to be able to explain it to the CEO, so that CEO could go to the board and explain, well, there's this A times B times X plus C, and there you go, that's why we chose what we chose. And so explainability or interpretability turned out to be a very important optimization that very few data science teams will be cognizant of unless their investor is really thinking about it.

There's obviously this other optimization metric of the trade-off between accuracy and implementability. And so here I've taken old Kaggle prizes, including the Netflix Prize as well, and normalized by essentially the winning metric. So if there was a low number I normalized it, if it's a high number I normalize it, so the winning team was all the way on the right on this graph. And then what you see on the y-axis is the percentage of teams that did within some normalized distance of the final answer. And you wind up seeing, for low-value Kaggle problems, you wind up having not a lot of teams doing really well, and for high-value ones a lot of teams come out of the woodwork and do quite well with those things. And so when you wind up winning one of these competitions, you typically win in the fourth decimal place. But there's an obvious question of, what about the second team, and the team before that — was their algorithm or their approach easier to put into production when you actually want to start doing this at scale? And that is the question that you have to ask yourself. Accuracy is not the only thing that's most important.

And what I love is this quote from Xavier Amatriain, who talked about what the Netflix Prize meant to Netflix. And essentially what they said is — for those that don't remember, it's now a 10-year-old prize, where they gave out a million dollars for getting an improvement in the recommendation engine from Netflix — they basically say, we couldn't put it in production. They paid a million-dollar bounty, and they wound up looking at the code, and there were hundreds of separate models that were then boosted together, and they said there's no way we can do it. They took a couple of interesting pieces of that, they put it in production, but accuracy — the team that won in the fourth decimal place — wasn't the most important thing for them. It was about implementability and usability. I think some of those lessons of data science are echoed in a very scary way in the new paper by Google, who titled this “Machine Learning: The High-Interest Credit Card of Technical Debt.” And they talked about machine learning in production as a very different type of beast than typical engineering efforts, because of the lack of composability and because of the abstraction boundaries that wind up bleeding over from say algorithm into software into hardware, etc.

And by the way, if you are a data scientist, this should be in your canon, right? This is like your required reading. If you're a CIO or a CTO, I think you have to read this as you're making your build-buy decision. If you're an investor, this is the kind of thing that you want to be asking companies that are coming to pitch you, where they're talking about how they've got this next great machine learning algorithm, next great machine learning platform — asking these questions, because these are the things that people are going to want to know as they start putting it in production. And so there's this quote up here where they basically tell all the academics of the world, yeah, by the way, algorithms are important, but ninety-five percent of all machine learning code in production is actually just glue code, connecting all of these different pieces together. Again, happy to talk about this in the question-and-answer session. There's another wonderful talk by Léon Bottou, who is now at Facebook, in ICML, talking about some of these concepts as well.

So again, I'm going to piss off a lot of people who don't see their favorite platform up here. When you are in a data science team and you are making that build-buy decision, are you trying to optimize for something that's very quick and dirty to get an answer easily but may be very hard to put in production? Or are you doing something that needs to be on-prem because perhaps the data is very private or very secure? Do you need to do something at a very fast speed with low latency, or are you willing to do something as a service? And there's nuances in all of this, right? Are you willing to give up interpretability to get a very good answer, in which case you might go farther to the left? But again there's multiple axes here. And the other ones that I think are critical when making those decisions about which platforms to use and implement within your own organizations is: what does that cost in time to actually put this into production, and then once you put it in production, what is that cost to actually maintain it?

So when we think about putting machine learning in production, we're often thinking about giving our end users the ability to go everything from what they're currently doing now, which is manual processes or manual business rules, to some sort of augmentation — we're giving the end users of our products the ability to weigh in on our decisions — to a full automation, where we're basically taking data off the table and removing conversations that agents and others actually have to see just to solve those problems. So on the left-hand side is just a schematic of what you might see as the output of the confidence of a prediction from a machine learning project on this piece of data — purposely being abstract here — and then as a function of basically the frequency over say a day's worth of data. So when you first start out you may not be very confident in your answers, and so once you've built this into the organization, into the workflow, most of the time they're still going to just keep on manually going through these normal processes, but some of the times they might like to get your suggestions and act upon those suggestions. And over time, if you build the appropriate feedback loops into your systems, the system itself will wind up learning from those processes and get better and better, so you can actually start automating those processes.

So really on the right-hand side what you see is perhaps that risk and cost, because this is now another thing that you have to optimize over. If I make a mistake, what's the cost to the business, right? If I send a “here's how you do your password reset” back to somebody after they said “hey, I like your product,” it's not a terrible thing that you've done, right? But if you say “here's your password reset” and somebody said “I'm going to be suing you for X, Y and Z” and you've missed that interaction, that's a serious problem. So you have to understand the cost of being wrong in really both directions. So we like to think of what we do, and many others, as building fault-tolerant machine learning. So when you make a mistake you have the ability to give feedback into the system. So Google does this obviously with Gmail — oh, this was in a personal message, this was spam — allows you to move that, and then behind the scenes there are models that are getting rebuilt specifically for you and your feedback. And what we do in wise is we give essentially the agents the ability to take our suggestions for how to answer a support ticket, and if they don't, then that becomes feedback for us, and if they do, that also becomes feedback.

I'll just end with a couple of slides showing you how hard machine learning in production actually is, lest you think that this is something that you can wrap your heads around. You might have remembered Gmail getting very excited about the fact they're using deep learning now to get better spam filters, and they talked about their accuracy rates. And then a couple of days later Linus Torvalds wound up putting this on his Google+, saying “something you recently did has been an unmitigated disaster. Of the roughly thousand spam threads I've gone through so far, 228 were incorrectly marked as spam.” So Google pissed off Linus Torvalds because their machine learning wasn't perfect. Netflix does the same thing, right? So here's this showed up on Twitter a couple of weeks ago. Netflix and Google, some of the best machine learning companies in the world, and these are their core products, and they still make mistakes. Yet these are not fatal mistakes. They've built fault tolerance into the machine learning. So as you think about building your own machine learning products, please give some thought to that build-buy decision of whether this is your core competency or whether this is something that you perhaps should bring in-house. Okay, thank you. Thank you.

[Q&A with Matt Turck]

TURCK: This was awesome. So many very interesting things you said. So you were talking about the ability to explain to the ultimate buyer in an organization, the CIO, what actually is going on, as being a very important aspect of this. And so I understand this is not a completely black box, you can provide some feedback to the system, but are you finding that it's one of the big issues — that living inside the technology, but the reality of selling to the enterprise, that the issue is that people need to understand what the hell's going on behind the scenes?

BLOOM: Yes and no. I think the reality is that there are multiple players in that enterprise software buy decision. And fundamentally there has to be an ROI discussion. If there isn't that discussion up front, then you're going to wind up having a big challenge later on the road, because even if they can understand how it works, if they don't like the numbers it doesn't make sense. But once you get over that hurdle, then there's the data scientists themselves, who are typically part of, in our case, the types of decisions that are being made of whether they're going to actually purchase this and bring this in-house, and they're really the influencers on this decision. And for them, typically they wind up asking — not to denigrate all data scientists — but what algorithm you're using, doesn't it scale, why is this not this one that I just read about or I just wrote about? And the answer you say to them is, look, we would love your help on understanding the data and the exact problems that you're having, but we just want to provide — the proof is in the pudding, right? This is a great answer, and your boss's boss needs to understand why we're getting these answers and how much the accuracy, or what other ROI metrics they care about, are actually improving over time.

TURCK: I'm going to open up to people in a second, but you very kindly didn't turn this into a product pitch, which I appreciate, but can you tell us actually a bit more about what you guys do — what is the experience of a customer service agent when using the product?

BLOOM: Yeah, so the interesting thing is we have a bunch of different users throughout the organization. At the base level it's the agent, and for them it's really where the augmentation comes in. We're recommending how they should respond to an incoming message. Some of our clients have hundreds or even thousands of macros of how to respond to somebody's inquiry, and instead of having them go through all of those or memorize those, we're basically giving them the top three suggestions and allowing them to search deeply into all the other macros when they don't get that. We're also doing triage, so as a ticket winds up coming in, we're routing it to the correct queue. And so because you asked, I'll say a couple of numbers that we find pretty exciting. Typical time will be, especially when the triage team is in another country in another time zone for a US-based company, it could be six to ten hours before a ticket that is your email where you say “I like your product” or “I don't like your product” even gets looked at by a human. And when they do, the first thing that happens is they wind up routing it to another team, and that team's job is then to respond, because this is actually a fairly straightforward process. And we're baked into systems like Zendesk and Salesforce, it allows us to route those almost instantly, so within a minute or two you're taking an eight-hour process to a few-minute process. And so even if we do nothing else after that farther downstream, the amount of time savings is crucial, let alone the labor savings.

I mean, I think what's really exciting is that obviously there's this buzzword around customer success now, and there's this understanding that customer success is the core of what a company has to do, obviously after they get a customer. Customer satisfaction is only one of the metrics that really drives whether you're doing well in customer success, and we think all of that really starts with support. The other thing that's quite interesting is that these companies are growing very rapidly, and we've been told multiple times by people running support desks worldwide that if they had an infinite budget they wouldn't be able to hire fast enough for how much their support needs are growing, to keep customer satisfaction at a fixed level. And so what's really interesting, I think, is that you could have a software company that's figured out how to do all the scalings correctly, so you have one person and you get a hundred customers, you add another person and you get two hundred customers on your own team — support is still that last place where you wind up having to scale the total number of people in your support system and support operations by the number of incoming tickets. And since tickets scale with revenue, companies that are growing quick start feeling this pain very much. And that's where we wind up coming in.

TURCK: One or two questions. Okay, that's this one here.

BLOOM: So the question is, are we going to try to solve the problem or create the best response? That's really where that automation-augmentation layer lives. At first we're just going to basically do augmentation, so we're giving help to the end-user agents. And I didn't answer part of your other question — their bosses can wind up understanding overall macro usage and things like that, and understand their efficiencies and how those are changing. But over time the system becomes so confident on some fraction of the answers — five to ten to twenty percent — it can basically just answer them without any humans on our client side actually looking at it, and those tickets get solved and people are satisfied. So that, we hope, shifts over time. But what we start off with is that proposition to the agents, so that they can feel like we're essentially their assistant.

TURCK: Quick one more.

BLOOM: So the question is about low-hanging fruit. I think we actually are solving a pretty hard problem, in the sense that we have to build data science workflows against data we haven't seen yet. And the only way that we were able to do that, I think with some success, is by bounding the problem from just that general statement to: it's going to be around natural language conversations that people are having with their clients. And there's a small vocabulary, as it turns out, of the ways that individuals are interacting with companies. And so what we wound up doing is essentially build models off of those specific interactions. So for us actually I think it's a pretty hard problem, and by bounding it to natural language for multiple companies in a real-time environment — not just in English but in other languages — we see a whole bunch of really interesting problems. And one of the things I'll just add is that we don't really see ourselves as a big data AI company. Most of the problems we're solving are at the hundreds, or sort of maybe thousands, of gigabyte levels at biggest for the types of models we're building. And so for us, we're solving interesting problems that don't show up in the academic literature, more around featurization than it is around the actual model building itself.

TURCK: Great, well thank you very much. We're going to keep moving.

Computational and Data Literacy for Domain Scientists

Sun, 15 Feb 2015 00:00:00 +0000

How universities can train domain scientists in computation and data science, in the AAAS symposium on interdisciplinary data-intensive career paths.

Supervised Machine Learning

Thu, 18 Sep 2014 00:00:00 +0000

Morning lecture on supervised ML for astronomers — classification and regression, random forests, and a scikit-learn demo on SDSS star/galaxy/quasar data — at the first Astro Hack Week.

Large-Scale Inference in Time Domain Astrophysics

Thu, 19 Jun 2014 00:00:00 +0000

Statistical inference on massive time-domain astronomy datasets, for the algorithms-for-massive-data community.

Astrophysical Applications of Machine Learning at Scale and Under Duress

Thu, 22 May 2014 00:00:00 +0000

IPDPS keynote on machine learning for astronomical discovery under real-time and scale constraints.

UC Berkeley Data Science Startups

Thu, 08 May 2014 00:00:00 +0000

Panelist (as co-founder of Wise.io) discussing the wave of data-science startups emerging from UC Berkeley research.

Machine Learning in Astronomy

Wed, 07 May 2014 00:00:00 +0000

Colloquium on applying statistical machine learning to large-scale astronomy datasets in batch and streaming modes, as co-chair of the LSST Transients and Variable Stars collaboration.

Data Science at Berkeley

Fri, 02 May 2014 00:00:00 +0000

Keynote at Facebook HQ on the emergence of data science at UC Berkeley: training, the newly founded Berkeley Institute for Data Science, and the interplay of theory-driven and data-driven inference in scientific practice.

Computational Training and Data Literacy for Domain Scientists

Fri, 11 Apr 2014 00:00:00 +0000

Invited lecture at the NAS workshop on big-data education, drawing on the Berkeley ‘Python for Data Science’ bootcamps and course, and what domain scientists need in computational training; proceedings published by National Academies Press.

Slides title matched to this event (probable).

Data-Driven Astronomical Inference with Machine Learning

Fri, 21 Mar 2014 00:00:00 +0000

Machine-learned inference for astronomical surveys, at the UC-HiPACC exascale computational astrophysics workshop at LBNL.

Overcoming the Barriers to Production-Ready Machine-Learning Workflows

Tue, 11 Feb 2014 00:00:00 +0000

With Wise.io co-founder Henrik Brink: practical obstacles to deploying ML in production, including data preprocessing, scaling models to growing data, and deployment into production pipelines.

Large-Scale Inference in Time Domain Astrophysics

Sun, 15 Sep 2013 00:00:00 +0000

Boot-camp lecture on large-scale statistical inference for time-domain astronomy, during the Simons Institute's Theoretical Foundations of Big Data Analysis program.

Key Quotes

“Make no mistake, computation is not our goal. Instead, the novel computation and algorithmic techniques are enabling what we do, which is fundamentally to conduct physical science.” – Joshua Bloom

“We'll find more supernovae, which are basically big explosions from dying stars in various different ways; we'll get more of those types of supernovae in the first two weeks or so once the survey turns on than mankind has found throughout all of history.” – Joshua Bloom

“We haven't quite got to the point of having machines write the papers for us, but that's maybe one of the ambitions.” – Joshua Bloom

“Very few people actually talk to botanists and ask whether they care about classifying irises; if they go out in the fields, is that the thing that they really want to know?” – Joshua Bloom

Transcript

[Music] So I love this quote from Jim Gray. He meant this in, I think, the nicest way you could possibly take it, in the sense that when people get to work with astronomy data, and in particular people working on large-scale algorithms and sort of novel computation, they get to do so in a sandbox that is a lot easier to work with and work in than some of the other types of data that are large and streaming and noisy and dirty, like astronomy data is. So he saw this as this wonderful playground for Microsoft, and more generally for people like yourselves, to be asking questions of scientific importance,

but not so much on the physical side, more on the computational, algorithmic side. And of course if you make mistakes with astronomy data, you don't start wars, you don't blow up your company, you don't have privacy leaks, etc. So there is something very special about that. And of course from the physical science side we think our pursuits are reasonably orthogonal to commercial ones; I won't try to claim more consequential. So I'm very excited to be here and want to thank the organizers for the invitation to speak. So while our data is maybe what you could consider a fertile ground for interdisciplinary work between

the methodologists and the physical scientists, make no mistake, computation is not our goal. Instead, the novel computation and algorithmic techniques are enabling what we do, which is fundamentally to conduct physical science. That is, we're working with novel computation and algorithmic techniques because we have to, to solve a physical science question. For those that don't know, astronomers pride themselves in actually using new tools, and in the context of this talk we'll talk about algorithmics and computation as some of the tools that we get to play with, and we really pride ourselves in using those to achieve our end. So this is a picture now essentially 120 years old, from

the Harvard College Observatory, and you notice a lot of people looking down and meticulously taking notes and essentially looking at data. There was a big data problem back then, 120 years ago. The Harvard College Observatory had just opened up a new observatory in the Southern Hemisphere, and they were getting more images of the Southern Hemisphere than they knew what to do with. And they had a specific science problem in mind: they were trying to basically look at binary stars. You can see what's called a light curve in the back here, which is basically the change of brightness of a star as a function of time, and these are actually binary stars

that they were interested in looking at, because if you measure the periods of these oscillations you can actually wind up inferring, with some other data, what the fundamental properties of those stars are, like the mass, the temperature, etc. So that was of great interest to people and still remains of great interest. But in some sense you could view this as a prototype of grid computing, or crowdsourcing. And we used to call these people computers, in a pejorative sense, because they were almost all women at the time. But in a positive light, what we wind up learning is that many of the people

that were in this room were making fundamental contributions to astronomy, in particular Henrietta Swan Leavitt, who's pictured here. In a modern sense we have our own data deluge to deal with, and the 800-pound gorilla in the room for us is the so-called Large Synoptic Survey Telescope, which has really just kicked off construction essentially this month, and is a sort of billion-dollar-level project funded in part by your taxpayer dollars, that will be taking essentially more data on the sky at visible wavelengths than has ever been taken before. And just to give you a sense of the sort of raw numbers that we care about, the light curves that you saw

before on the back of that room, we'll be getting something of order a billion of those updated every 3 days. And so obviously getting people to look at that data doesn't really make any sense. We'll find more supernovae, which are basically big explosions from dying stars in various different ways; we'll get more of those types of supernovae in the first two weeks or so once the survey turns on than mankind has found throughout all of history, and it goes on and on and on. It's of order 20 terabytes a night of streaming image data. There's a satellite which recently launched called Gaia, which is in some senses a pared-

down version of the Large Synoptic Survey Telescope, getting sort of hundreds or so updates on the light curves of essentially a billion stars in the sky over the course of several years. And one of the things that you'll see throughout the talk today is that in advance of these surveys, a number of us are trying to cut our teeth on real data with sort of precursor surveys that have largely the same ambitions but perhaps pared down in the size and the velocity of the data. It's not just visible wavelengths that we're interested in; basically across the electromagnetic spectrum there are endeavors already ongoing that have very specific scientific ends.

In particular, one that I wanted to highlight here is one that's headed by my colleague Aaron Parsons here at Berkeley, and the goal here is to detect the signatures of the emergence of the first stars and galaxies after the Big Bang. And these are enshrouded in essentially lots of hydrogen, and what happens is you wind up sort of burning away, that is ionizing the hydrogen, at these very large distances, and the signature of that would wind up being bubbles that we might be able to detect on the sky. But it's many, many orders of magnitude below the foreground emission and just general noise, and so this is an incredibly hard measurement. And just to

give you a sense of the data rate that comes out of these radio interferometers that's already being achieved essentially today, it's something like 210 terabits a second. And over the course of an observing season, several months, after it's been correlated — that is, basically you take all the interferometric data and multiply it by itself and do a little bit of math after that — you wind up getting something like 200 terabytes. And this is a pretty significant endeavor for astronomers, in particular because these telescopes are in very remote sites. So one of the innovations here, while the algorithmics are actually pretty straightforward, the actual work that had to get done

was to build very specialized hardware, going from custom FPGA boards down to GPU and then ultimately doing the final analysis in clusters. Astronomers really do consider ourselves quite lucky in being able to play around in this large data inference space, and so depending upon the problem that we have and the tools that are available to us, we get to put on different hats. And so there are certainly cases where we need to be incredibly theory-driven — we've got very deep physical reasons why something needs to happen, and so we might apply physical theory to the data that we get — and in other cases we don't have much theory at all and we need to be

much more data-driven. In some cases a Bayesian analysis makes sense, sometimes a frequentist analysis makes more sense, and we have those tools available to us, and really it is just deciding — and one of the hardest parts is in making that decision about which toolkits we apply when asking questions of the data. One of the things that I've been interested in over the last several years is understanding this taxonomy of so-called variable stars. I already presented really two of them to you: one, I said, supernovae, exploding massive stars; another are these so-called eclipsing binary stars. But the taxonomy of these stars that we know about that

go bump in the night, that is, what we think has separate physical explanations, there are about 150 different classes and it's growing every day. Some of these classes have essentially one example in them, others have thousands or tens of thousands of known examples. And so what I wanted to do, and the question that I wanted to ask, is, given existing data sources — essentially synoptic images of large parts of the sky taken repeatedly in similar wavelengths — could we actually infer just from the change of the light curve itself, the change of the brightness as a function of time, could we infer which class a star would wind up belonging to? Now you might imagine we just take

all of the stars that we know that are the prototypes of each of these different classes and just apply templates to them. But that's actually very difficult, because almost all of these classes are sort of fuzzy in some sense, and people have made judgments about whether something belongs in one class or another, so there is no actual real prototype that everything has to look exactly like. The other important thing of course is that our data is noisy, irregularly sampled, and so that adds complications to the analysis. And then of course we also have spurious data; that is, we're pulling essentially values of the brightness as a function of time out of

raw imaging data, and sometimes we make mistakes, we basically make mis-estimations in how much noise we think is involved in each one of these observations. So these are actually wrong data, or incorrect data, or incorrectly inferred uncertainties. And then if we're going to try to actually extract more information out, it could be that the interesting things of what's going to happen in this event — and this is actually a theoretical curve from a gravitational lensing event — it could be that those actually haven't happened yet, and so we have to have some sort of notion of where things are going so that we can start marshalling resources to be able to do even more science. So

taking that sort of heterogeneous set of light curves and turning it into something that we can actually do inference on is something that I've been doing for a while, and we've taken a machine learning approach to this, and basically we try to imbue some of our domain knowledge into what distinguishes different sources from each other. And we build basically featurization codes that take this heterogeneous data and turn it into a homogeneous, if not sparse, large dimensional space — a rectangularization of the data. But the location of where something is on the sky — and so we have of order a hundred different features that we apply, really just looking at a whole bunch of different things you might ask

of time-variable data. And that's gotten us pretty far along in being able to make strong classification statements about stars that we've never seen before, without any people actually looking at the data. So on a data source of about 50,000 stars where we had some notions of labels, we're able to get sort of a gross misclassification rate across three different classes of only about 5%, which hadn't been achieved previously. And making use of the taxonomy to build a loss function was one of the things that we worked on for a number of years, and that actually improved the classifier. Another thing that improved the classifier, as you can imagine, because

in some sense this was a semi-supervised expert expedition, is to use active learning, where we would build a classifier on those stars, about a thousand of them, that had really strong labels, and then the classifier would ask questions of experts and said, if you told me the label of this other star, that would improve the classifier dramatically, or at least the most. And we'd go through several rounds of active learning where we're essentially getting experts to essentially buy more labels, which are expensive to do, and we improve the classifier a whole lot from that. So really what we wanted to be able to do in the end is

to take a subset of the really well-studied and well-known classes of variable stars, down from 150 to about 25 classes, and take essentially a light curve that looks like this and then spit out probabilities that it belongs to some class of variable stars — in this case 94% probability it belongs to a so-called RR Lyrae class. And we tuned down the total number of features using modern feature selection techniques, and we got a fairly good classifier, 15% error. And what you'd like to see of course is that you have basically all your power in this confusion matrix on the diagonal, and you can see that there is a bit of confusion

across some of these different classes, but you notice in some cases we only have sort of one or two examples in our training data, so we're not too worried about those. So the output of this endeavor was to take a survey that had been around for about 10 years and produce a probabilistic catalog of all these different stars. And we made a website that, we tried to help people basically go through and be able to search through this taxonomy and find the objects that were highly likely to be part of that class. So here you can see we're parsing through the taxonomy, and then we bring up basically a list

here of all the different classes of stars that are part of that, and then we're going into some of the subclasses of the RR Lyrae, and each individual star has its own page made for it: here's the raw data, here's the folded data, here's the probability vector of what we get. And in here we have a little social thing; we're hoping maybe this gets bought by Facebook. Okay, good, you're still awake. But what's kind of interesting about this, of course, is we're making these statements, but they're fuzzy statements, and this is quite unusual for astronomers. Astronomers like the idea of being able to go out and do science on a catalog of stars that are part of

Class A or Class B, but doing science with probabilistic catalogs is not something that people are generally used to in the variable star field. So we really have sort of two different ways that you can imagine doing your science with this catalog. And to emphasize again, we're not doing this machine learning endeavor because it's fun — it is indeed fun — but we're doing it in service of novel science. And so what are the different kinds of things one can do with this? One is to do sort of demographic surveys where you don't have a lot of capability of following up any of these individual objects, and what you're willing to do there

is trade high purity at the cost of lower efficiency. And so if I want to use a subset of stars that I know are very good for probing the 3D structure of the Milky Way, I want to make sure that that survey that I have, and that bucket of stars that I have, really only includes those types of stars, and even if it means losing a large amount of them out of my catalog, I'm really happy to have a highly pure sample. On the very opposite side of that spectrum is what you call novelty discovery, the notion that I don't mind dedicating lots and lots of

follow-up resources if it means I can find something that's quite novel that's actually really interesting. And so I'm willing to follow up with many telescopes and burn lots and lots of time and people resources and dollars to be able to find that sort of needle in the haystack, and for that I'm willing to have a very high efficiency sample at the cost of low purity. And so that's sort of what we did; we said, well, now that we've built this survey and this catalog, what is it that we can do with this to actually do something novel that wouldn't be able to be done by other means? And we found, in the novelty discovery

part of the spectrum, some very strange stars. And there had only been about 10 known at the time; we found basically about seven more of these really weird stars that change over courses of decades in their brightness. They basically will fade by factors of 100 to a thousand, and people still don't even know what causes this fading. But the fact that we're able to find it in a survey and data that had been open and public for the last decade, I think, is a testament to the fact that if you're doing these fuzzy catalogs you can get a lot of traction. One of the other things that I'm interested in is

not just variable stars, things that are changing in the sky, in our own Galaxy generally, but things that are blowing up. And there we have another sort of interesting challenge in that we're sometimes looking for things that we know about — these are so-called type Ia and type IIP supernovae, these are pretty normal explosions in the universe — but then there are other things that have been theorized, really bright events that take a really long time to evolve, and really short events, and faint events, that will evolve over the course of just a couple of days. And in the case of these events we really benefit from being able to recognize

these events more or less in real time, so that as new data streams in, if we can make inferences about the fact that something like this is happening, we can train our telescopes and optimize our resources to do the follow-up. It also means that if we're wrong about things, and we basically only figure out that there was an interesting object happening over here and we figure that out a year later, well, that's basically useless to us. As scientists we want to be able to optimize our scientific follow-up now. The other thing that I can't put on here, by definition, are sort of the unknown unknowns. We want to be able to build classifiers, streaming classifiers, that are going

to be able to identify objects essentially in real time that we've never even envisioned before. And having people listening or looking at data clearly doesn't make sense in this sort of streaming environment, especially as the volumes of the data are increasing. So a big part of what I've been doing is really trying to automate that whole inference stack, from the strategies of how we observe in the sky, to how we actually schedule our telescopes, how we do the observations and the preliminary analysis, how we do the finding and the discovery, and all the way up to actually getting robotic telescopes to follow up on discoveries made by other robotic telescopes

before any people actually wake up and know that something's going on. We haven't quite got to the point of having machines write the papers for us, but that's maybe one of the ambitions. So I'll just talk for the next couple of minutes about the discovery engine that we built, which we put into a real-time system, basically asking the question, are these objects on the sky real, or are they bogus artifacts of having observed the sky with a detector that has noise properties associated with it, and our inability to actually very cleanly subtract out new images of the sky from reference images of the sky? And what we did is, again,

featurize these images and then used what amounted to a number of different classifiers, and figured out that for our purpose here random forest was quite good at producing a classifier that bested any of the human labels that we had even made. We had noisy labels and we were able to figure out that the humans made mistakes with that. One of the new endeavors that we're working on along these lines is to be able to use deep learning, so we don't have to imbue our classifiers with domain knowledge to build the features. And some of the preliminary work on that, of being able to use essentially these different image postage

stamps, has already been ongoing, and we've been doing that up at LBL and in conjunction with some people here in the EECS Department. But we were able to get this classifier using random forest into basically a survey that was ongoing over the last four years, and found a number of interesting events, as you might hope. One of the ones we're most proud of is having identified a supernova in a nearby Galaxy. And one of the important things — and you can see basically there's an arrow, which doesn't appear in the sky, by the way, you actually have to draw that afterwards, to tell you where to look — a supernova that wasn't

there, and then it appeared, it got brighter and brighter, and it was the earliest supernova after explosion ever observed; basically about 11 to 12 hours after explosion we think we got our first glimpse of it. The important point here is that this object was in a very famous Galaxy that's observed by amateurs every night, and probably about 3 or 4 days later, had we not found it with our machine learning framework, it would have been found by amateurs, but it would have been too late. We got a lot of great data in just the first day after this event that would have never been able to be obtained by

any other means, if we had observed it and detected it 4 days later or so. And so it shows you, I hope, the sort of onus on getting your inference not just robust but getting it very quickly. Just in the interest of time I'll skip over one of the results that came from that, but just show that we have in the survey basically a robotic inference machine called the PTF robot, which goes in, looks at every new image that's been streaming off this telescope, and not only does discovery but then starts to say, well, I think this is a supernova, and it actually logs in before any

people are awake. And then people over the next couple of days wind up saying, oh yeah, this actually looks like a supernova, and then people say, actually this looks like a weird supernova. And this thing then turned into a Nature paper that came out last year, where people wound up realizing that there had been sort of a pre-explosion in that part of the sky, which was quite interesting for understanding the progenitors of what makes those types of events. And then last month another sort of early discovery of a supernova by our team came out in Nature, where we were able to make observations very, very early on, which

helped us make some inferences about the nature of the thing that wound up exploding. So this is maybe a bit too pedantic here, but what I'll end with is a statement about using novel computation techniques, or machine learning techniques, or algorithmics, for the sake of physical science. Those of you that are in the machine learning world know the iris data set, where you're trying to classify three different types of flowers based on four different sets of observations of the length of the petals, etc. It is a really interesting sort of toy data set for people to test new algorithms, and

if you're building a new one you want to make sure that it would scale to a larger data set. But very few people actually talk to botanists and ask whether they care about classifying irises; if they go out in the fields, is that the thing that they really want to know? So a big emphasis when I talk to groups that are not astronomers is to point out that when you're working with physical data or social science data, there has to be a very specific question that obviously taxes your new algorithmic techniques but ones that you hope are of great interest to the actual physical

scientist or a social scientist. What we've done here is not just build sort of real-time streaming frameworks to be able to do inference; we did it so that we could basically understand the nature of exploding stars, etc. And so that's an important thing to keep in mind, is how you wind up working with people and how you wind up finding problems that are of interest on the physical and social science side as well. That is really one of the main thrusts, if you haven't heard about it already in this conference, of a new institute that's just been started here at Berkeley called the Berkeley Institute for Data Science. And the idea here is

to be kind of an incubator for this type of activity, where people working in algorithmics and computation can come together with people on the physical and social science side and find sort of novel problems together that tax both sides of that house. So I'll leave you with some parting thoughts. I hope it's clear that the astronomy data deluge demands an abstraction of the traditional role of people actually looking at data in the scientific process, and what's emerged of course is that the only way to deal with these data streams is with the types of techniques that many of you are working on. And I'm happy to talk with people offline

about what we're working on. BIDS will hopefully be one of these places where more work like this winds up emerging. So I'll end with that. Thank you.

Big Machine Learning

Thu, 05 Sep 2013 00:00:00 +0000

Machine learning at astronomical scale, in the Big Data session of KIPAC's tenth-anniversary symposium.

Astrophysics of Streaming Time-Series Data: Discovery and Inference

Mon, 01 Jul 2013 00:00:00 +0000

CITRIS Research Exchange seminar on ML pipelines for discovery and inference on streaming astronomical time-series data (variable stars, transients, synoptic surveys).

Exact date within 2013 unconfirmed.

Key Quotes

“My own field is in astrophysics, and understanding in particular transient phenomena, that is, understanding objects which change and go bump in the night.” – Joshua Bloom

“Basically Kepler is able to see incredibly small changes. It'd be the equivalent of flying over New York City and looking at the Chrysler Building and noticing that one light was out on somebody's desk through looking at all the windows.” – Joshua Bloom

“The greatest insights in astronomy… come only when we do great follow-up, but follow-up is incredibly expensive.” – Joshua Bloom

“The manifest destiny of where astronomy is heading is this full abstraction of the discovery steps and even the scientific inference steps, allowing us to sit in our armchair during the daytime and make strong statements, and perhaps transformative statements, about how the universe works.” – Joshua Bloom

Transcript

Great. Well, thanks very much for having me. Lauren ended his talk very nicely hinting at, well, not even hinting, essentially saying that some of the problems that he's encountering in one domain are similar to the challenges that he's finding in other domains in genomics. And I think this really speaks to a commonality of challenges that we all have on the application side of big computation, in trying to make the best out of a deluge of data and come to important scientific inferences that push our own fields. My own field is in astrophysics, and understanding in particular transient phenomena, that is, understanding objects which change and go bump in the night. Some of these are explosive events. Some of the changes are actually much more subtle. But what we're seeing, like Lauren, is that there are tools that are out there and some algorithms that actually need to be refined and perhaps in some cases be created, to make use of the data and to maximize the science that comes from that data.

What's interesting, I think, in the context of astrophysics, and this adds an important wrinkle from a computational perspective, is that when we're talking about time-domain data, this is watching things change as a function of time in the sky, there are a number of events, a whole series of classes of events, that need immediate attention. So it's not just going back a decade later and looking at a data stream and saying, oh, there was something really cool there, I wish I had known about that at the time. It's actually identifying what's worth spending resources on before you get swamped and another new event comes up. So it's a maximization problem in the context of massive amounts of data. As Masoud mentioned, this work is being done in the context of what we're calling the Center for Time Domain Informatics, which is housed in Evans Hall here on the Berkeley campus. We've been funded by the National Science Foundation for some of this work, and some of our computation time has come from Yahoo and Google. The important people here — well, everyone's important in their own right, but the important people here are the fact that we have a number of professors from the statistics and computer science departments collaborating with us. Again, we have a bunch of work happening not just on the astrophysics but also on the algorithms that will drive the frameworks for us to maximize our understanding.

So it's perhaps appropriate to start off, for those that aren't astronomers, with a traditional view of astronomy. This is Vermeer's painting, a famous painting called The Astronomer. It hangs in the Louvre, painted 1668, and this shows a fairly quaint view of an astronomer looking at a globe of the heavens and some maps of the heavens in this book here, and making sense of it all. This is not so much different from a Greek philosopher just thinking about things and making grand conclusions about how the universe works. A more modern view is one that's actually not so different than what was going on in 1949, where you had an astronomer, in this case Edwin Hubble, smoking his pipe, looking through a very large telescope, in this case just the finder, and making grand inferences about the universe. Obviously he would do this with the lights off, although legend is that he never stopped smoking a pipe. This is not so different from the way that astronomers work today. You know, replace the human eye with a charge-coupled device, replace this control here, a hand paddle, with some robotics control, and you more or less have a prescription for how we take data today in astronomy.

What's important is that Hubble discovered with small telescopes, what we now consider small, some of the grand insights that have driven 20th-century and now 21st-century astrophysics, in particular the expansion of the universe. And he did that by discovering a time-domain phenomenon called Cepheid variables in a nearby galaxy, recognizing that the Andromeda galaxy is not actually within our own, but is actually a dissimilar object that's quite far away. And he found a number of other Cepheid variables that allowed him to infer a relationship between distance and velocity moving away from us, and hence the expansion of the universe. So time-domain astronomy has been critical in a number of different ways, and Edwin Hubble is a good epitome of an early time-domain astronomer. What we're talking about here, when we think about how we're going to deal with this data deluge, is in taking the traditional way of doing science, of observing, finding objects on, say, images, making discovery on those objects, conducting follow-up, and then getting to a scientific set of inferences. It's how do we deal with this in the context of this data deluge? And from the beginning there's been humans in the loop in every one of these various steps.

I should say, by the way, there's a finer distinction between finding and discovery. One, you might consider taking metadata out of an image and sticking it into a database; that might be called the finding process. But the discovery that what you're looking at is something interesting or something new takes a bit of a leap. There's a famous example of Galileo who, while studying the moons around Jupiter, actually observed Uranus but didn't realize that he was looking at another planet, so he's not credited with the discovery, although he actually found it and it's in his journals. So there is a big leap that has to happen from finding to discovery. So let me talk to you about the dynamic universe, some of the astrophysics that we're interested in, and then we'll talk about some of our approaches to this problem. The first thing to recognize is that every single star in the sky changes its brightness, in some cases very dramatically and in most cases actually quite subtly. These are what are called light curves. Six of them from the new Kepler telescope, which is flying in space, has a very precision photometer that's capable of measuring the brightness of bright stars as a function of time. And you see a scope here of about a week. So this is something of order a month, and you see these very clear undulations in the brightness of this star.

That might not be too surprising until you start looking at the scale here. These undulations are of a size of sort of 12 parts per million. Basically Kepler is able to see incredibly small changes. It'd be the equivalent of flying over New York City and looking at the Chrysler Building and noticing that one light was out on somebody's desk through looking at all the windows. It is an incredibly small change in brightness. And what they've noticed is that there is no such thing as a quiescent star. Every star is variable, and the origins for this variability are quite varied and diverse. Stars pulsate, they rotate, that is, they spin. Some stars move in front of other stars, or planets move in front of other stars, and you get what are called eclipsing systems. There are some episodic systems where you get unsteady mass flow and you can get some outflow, that could lead to a brightening event, and there are explosive systems, basically when stars wind up blowing up. There are hundreds of classes of variable sources and events, and making use of a generic data stream means trying to put things into their respective buckets, and that's obviously a tremendous challenge.

The subtleties are one thing, but the truly exciting events, at least for me, are the ones that are quite rare and also herald the end state of massive stars, that is, supernovae. And here's a picture, essentially a before picture, looking at some of the Magellanic Clouds, satellites of the Milky Way, and you see the after picture, and I'll leave it to your eye to figure out where the new star is. It's right there in the middle. It's a really bright thing. This was found in Chile in February of 1987. It's the famous supernova 1987A. It was found by the human eye, and somebody just happened to go outside, probably for a smoke on a pipe, and happened to look up at the heavens and notice that his favorite part of the sky was a little bit different than it had been the night before. One of the most famous discoveries in astronomy in the last 100 years, and it was done by somebody who just happened to be outside at the right time. There's not just supernovae of types that we know. It doesn't matter that you know these various different nomenclatures here, but Type 1a supernovae are the things that tell us about the expansion of the universe. Type 2p supernovae are the most common types of supernovae in the universe. And then there's a whole bunch of classes of sources that could go bump in the night, that only theorists have dreamed of, that we've yet to see.

So we have sort of the known knowns, and we have some known unknowns — sorry for those that aren't fans of Donald Rumsfeld — and of course there's the unknown unknowns that we haven't even thought of yet and been able to put on this plot. This is another light curve, days since explosion. And this is just a measurement of brightness. Some things could be incredibly bright, some could be very faint. Some take time scales of only a day to evolve. And to truly extract the great science out of these sorts of events — which, by the way, neutron-star neutron-star mergers are the objects that are thought to produce most of the gravitational radiation that we should see with the next detectors that are coming online in something like 2015. This would be the first detection of gravitational wave events, and these are probably the objects that are going to do them. They will create an electromagnetic signature that traditional astronomers might be able to see if we can react quick enough, if we can actually find these things in the sky. So there are multiple thrusts in understanding and exploiting transients, and the introspection one is a clear one. We want to understand what gives rise to the changes that we see. What are these events? There's the exploitation of them, which is a statement to say that we don't really care about the physics of an individual event, we only know that they're great for studying something else — in-situ laboratories, or perhaps lighthouses or beacons to other places in the universe.

A classic example of this might be pulsars, which are rotating neutron stars putting out pulsed radio light. We don't really understand the physics of pulsars, but we know that they're tremendously good clocks. So you can use those tremendously good clocks in understanding, and maybe even finding, primordial gravity-wave radiation. They've also been very good for understanding gravity waves in general, in the context of strong gravity. Nobel Prizes have been given for the study of gravitational radiation in the context of pulsars. So exploitation is one of the categories. And then this exploration, the idea that we want to search the time domain and find new things that we haven't even conceived of yet. Okay, so let's start getting a little bit closer to some actual data. But before I get there, I want to make clear that when you make discovery on a massive data stream, that is an incredibly difficult event. Extracting metadata out of images is one thing, but saying that of the million things I just looked at, these five might actually be fairly interesting to follow up on, is quite important. But then understanding which of those five you want to spend your precious telescope resources on is an important one, and discovery is really only the start.

The greatest insights in astronomy — and this is where the real-time needs for classification and understanding come in — come only when we do great follow-up, but follow-up is incredibly expensive. We have people involved in that. This is using large telescopes to, say, follow up discovery of a number of different transients on a small telescope. People, telescope time, resources, and ultimately it's about money. So we can couch some of these classification needs in the context of some need of maximization of resources. Talking about data scales over the next several years, in the context of surveys that I'm interested in, mostly optical waveband surveys. There's a couple different ways to capture this, given that every pixel of a CCD on the sky captures about the same amount of data. When you think about this metric called étendue, which is essentially a statement about how powerful the survey is, you can translate that roughly into total sizes of data volumes, and you find out that we're living in an era now where traditional surveys are going to be producing orders of petabytes of data. But we're getting to a point, say in 2019, with the Large Synoptic Survey Telescope, where we're looking at something which is approaching 200 petabytes of raw imaging data, and making sense of all that in real time is obviously a significant challenge.

Just to give you a sense of scale, don't be scared with this silhouette man standing over here. This is the scale of the Large Synoptic Survey Telescope. 3.2 gigapixels of imaging, getting data more or less every 15 seconds. So it's a tremendous data rate. So this is really the 800-pound elephant, or gorilla, or whatever, in the room. And we're looking at having to get scientific inference out of 800 million sources every 3 days, and that amounts to something like a million supernovae a year, and something like 20 terabytes a night of data. Gaia, which is an interesting space-based mission which is going to precede LSST by a number of years, will be looking at something like a billion stars, but only sort of 70 times over 5 years, and we'll also find an appreciable number of supernovae. Now the good thing is we have the ability to now cut our teeth with real data from real-time data streams. So what our group has been doing is looking at the Palomar Transient Factory data, which is a survey using the same telescope that you saw Edwin Hubble looking through, but now completely roboticized, and now not with a bunch of Edwin Hubbles’ eyes sticking at the end of the focus, but instead essentially paving the focal plane with silicon.

And you see that we get a tremendous number, and it's actually quite a wide field of view. Here's the full moon for scale. We get a quite large number of candidate objects a night. This is more or less just detections of these individual objects. But only about a hundred, or say a thousand, of them are bona fide, actually varying sources. And of those maybe 300 of them are variable stars, and maybe about 10 of those are new transients. And our group is most interested scientifically in getting at these 10 transients. So we have a huge compression ratio, factor of ten-to-the-five, essentially, in going from things which we're not at all interested in, in fact are artifacts of the way that the data were taken and processed, down to the things that we really want to spend our resources on. So what we've introduced, because obviously we can't get a million graduate-student eyes on this data in real time, we've introduced the ability to have expert training on a small subset of the data, looking at these subtractions to find these new objects. And we present to the experts essentially an image of the sky as it is in quiescence, the new image, and a subtraction of the new minus the reference. And we ask them, is this real or bogus? Essentially, is this a discovery or not? And this, to a trained expert, no, this is probably no. And to a trained expert who looks at this, they say, oh yeah, that's a real supernova. And in fact, that is actually a real supernova.

So it allows us to get a tremendously good rejection, because we can now do some machine learning on the input, the labeling, that comes from these experts, on data on a massive scale. And just in the first month of data last year, you can see that most of the stuff that we were looking at was tagged as being essentially bogus, and a good fraction of it, although much smaller, is tagged as being something which is potentially very real. So that's allowed us to take humans out of the discovery loop within the Palomar Transient Factory. And by requiring a number of epochs of detections for a true discovery, we're able to plot on the sky where all our different transient discoveries are. And it's a very large number. You know, the Palomar Transient Factory found more supernovae last year than any other project, even those dedicated to supernovae. And that was just one small facet of the entire endeavor of that project. So we're already making great progress there. But what's interesting is not to say, well, we've got something of interest there, but we now want to know what that is. Because it may be that we don't even have enough resources to follow up every one of these objects. And that's actually true; it's quite daunting to believe that.

And it should be pretty easy. It should be that here's the taxonomy of all these variable and explosive sources I was telling you about. It should be that we just get a lot of data. This is again a light curve, flux as a function of time, or brightness as a function of time. And you see there's some time scales of different types of events. There's a Mira variable, which has a very large amplitude of variability over very long time scales. Supernovae, which is a huge amplitude of variability and then it winds up declining over also really long time scales, and then other types of sources all belonging to different classes of variable sources. And we are trying to make sense of that and try to understand that from the PTF data stream. The problem is, and this is what's kind of interesting, is that it's not like we're sampling the data in regular time and we can just get a period out of that data just doing an FFT. We are getting the data in an irregular way. The data have noise on them, and we have to try to understand what this event is. In this case it's a microlensing event, but here are all these different data points in time. So trying to get a sense of what a source is in the context of noise is a tremendous challenge for us. And perhaps even worse is that in many cases the metadata that we extract out of those images are actually spurious, that is, incorrect, or we haven't correctly characterized the systematic error, and that's a significant problem for us. And then if we're trying to actually do some follow-up, maybe the telltale signature of the event hasn't even happened yet, but we want to send out some alerts to help us get going on it.

So we've identified machine learning as a reasonable approach to dealing with this data deluge. And here we create something called features, which are a homogenization of the data from this noisy and irregularly sampled data set into one that's much more regularly gridded, in this feature space. And we have a number of different metrics. These ones in blue are related to the time domain. And then context, that is, where this object is, near a galaxy for instance, where it is in space, is also quite useful. And what's pretty remarkable is, just taking the immediately available data from the Palomar Transient Factory and trying to identify transients from that, that is, just any object which is truly essentially explosive and not just a variable star doing its thing, is that what we're finding is we can do very well in a cross-validated machine-learned way. We're getting to something like 95% efficiency at 98% purity. That is, we would lose 5% of the true transients if we were willing to take follow-up data and only have 2% of the follow-up data not be of true transients but of variable stars. So this is, I think, an impressive step for us, that we're now able to go not just from extraction of the data out of the images to discovery, but now we can make some of these rudimentary statements about what those objects actually are.

But we want to go even further. We now want to follow as a function of time how these events are changing, and try to get an even stronger statement without the need for even more data. So what we've been doing within the group — and this is a paper by Joey Richards, who's a postdoc in our group — is look at existing well-labeled data sets from, well, one space and one ground base, at Hipparcos and OGLE. These are ones that have very nice labels on them, and we can try to learn off of those labels. And what's clear is that in feature space — this is an example of two different features over all the different classes of objects we're concerning ourselves with, in amplitude of the variability and in some sort of frequency sense, if there is enough data to be able to get a dominant frequency — you can see that the very long frequencies, or, well, sorry, the long frequencies, as in long periods, are out here. This is the Mira variables. And then we have a whole bunch of short-period objects over here. And then you can see that there's an anti-correlation, for instance, of this feature with this feature. So if we learn off of this, we can then, when we get a new instance which may have a value in this feature here and another value in this feature here, we can pretty rapidly throw out whole classes of sources here and another whole set of classes here, and we can hone in on the ones that are most interesting.

And what was shown in Richards’ paper is that we can do quite well in a cross-validated sense, in showing now the confusion matrix of how well we do essentially classifying a source relative to itself. And what you'd want to see is a total amount of power of one along this diagonal. And what you could see is that there's some spillover of some classes into other classes. What's remarkable about this is that when you highlight what types of physical sources these classes are, you wind up seeing that a large amount of that spillover happens within some of these larger subclasses. And so, in some sense, what the machine is doing is figuring out that there's an intimate relationship between some of these classes here, because it has a very hard time distinguishing them. Their distance in some abstract space is very small. I'm running short of time, so I'll just say we're trying to make use now of the hierarchy and of this taxonomy so that we get even better classifications. And what we find is that if we take our variable-star classes and we just separate them into three very broad classes, our misclassification rate is starting to go down pretty dramatically, to the point where we're only misclassifying about 5% of the variables.

The idea now is that we can start abstracting this large amount of data to a human who might start asking questions about a given source or a given position on the sky, and present to them a machine-learned set of statements about what it thinks that object is, before resources are used, and in some cases abused, to follow up sources that really don't deserve it. It would allow the human operator, who's at the very end of this long tail of scientific inference, to be able to make great use of this. And what's quite remarkable is that the LSST Corporation has now put out an iPhone app where they're able to push out events using a protocol that I invented called VOEvents, that allows an amateur astronomer to essentially get real-time alerts of these things. Now these alerts are currently vetted by humans who go through and say, yeah, I think it's that. But the next step, obviously, from the Palomar Transient Factory and other new surveys that are just now coming online, is that we basically have the machines making not just that discovery step, and not just that initial statement step about whether it's a variable or whether it's a transient, but going deeper into the classification and saying this is a supernova that may be important for this type of science. So it's enabling some great citizen-science follow-up.

So I think we can come back now to this original picture from Vermeer, and it doesn't look so quaint anymore. It kind of looks fairly modern to me, in the sense that we replace this globe here with a screen that's visualizing the results of an SQL query that was asked of a machine that's probably not even local to this guy's office. And what you wind up seeing is that the manifest destiny of where astronomy is heading is this full abstraction of the discovery steps and even the scientific inference steps, allowing us to sit in our armchair during the daytime and make strong statements, and perhaps transformative statements, about how the universe works. So I'll end with that, and just remind you that discovery engines in the context of astronomy are already swamping our available resources. We literally cannot follow up every object that we find. So knowing what to follow up is key, and this autonomous classification is going to play, and is becoming, an important tool for us, and is really abstracting the traditional role of astronomer in the entire scientific process. What's also very clear is that we have this rich interface between machine learning, computer science, statistics and astronomy. And there is so much commonality with the other data-driven fields, as you already heard today and you'll hear more of now. I'll end with that. Thank you so much. [Applause]

[Q&A] We have time for one quick question before I get to the next speaker.

So you mentioned the fact of getting the people into the loop, but again, how do you actually trust when actually they represent something in a sense, because you don't know what type of rule each of these astronomers are using. So how do you actually make sure the quality of the results?

Well, so what you can benefit from is having multiple experts all labeling the same data, and then you can take some aggregate sense of what the true label actually is, and then see how individuals were, in a distance sense, from getting that label correct. So here you just say that the answer is what a large number of experts say it is. And if that's the case, then you can learn how to weight people according to their expertise across all of this different labeling process. So we have a very rudimentary algorithm for how we go about that in the context of discovery on the two-dimensional images. But we're now thinking about how you get experts weighing in on individual sources which, when identified through some techniques to tell us that it might be an important source for us to understand, because it lives in a feature space that isn't already well labeled by other sources around it. So getting expert learning as part of the process is critical. The important thing here, though, is that all of that happens in a relaxed sense, that we haven't yet identified a place where we think there's tremendous use for experts in the real-time loop. And in fact, that's kind of what we're trying to get rid of, is the need for that. So getting experts in the learning process means that we can get them out of the application process. Thank you so much.

Extracting Actionable Insight from Dirty Time-Series Data

Fri, 21 Jun 2013 00:00:00 +0000

How to extract real-time, actionable insight from noisy, incomplete streaming sensor data (earthquakes, supernovae, traffic), and the interplay between domain scientists, statisticians, and computer scientists.

Webcast archived on the BIDS video resources page.

The Modern Astrophysics Stack: Automated Action and Insight

Thu, 30 May 2013 00:00:00 +0000

Talk on the automated, ML-driven software stack of modern time-domain astrophysics at the Simons Institute's inaugural public symposium; reportedly the earliest video on the Simons Institute YouTube channel.

Verify YouTube ID at build time.

Key Quotes

“It's easy to think about astronomers as portrait artists, but we like to think of ourselves as celestial cinematographers, because for us time is a very important axis in understanding the physics of what's happening.” – Joshua Bloom

“It's of course preposterous to think about scaling up, say, 10 to the 6 Jodie Fosters to listen for what it is that we think we may hear in the sky… observing and listening to the universe doesn't guarantee discovery, and discovery of course doesn't guarantee insight.” – Joshua Bloom

“We have to learn as astronomers to say goodbye to black-and-white catalogs, because these catalogs, if we're getting lots and lots of data, have to be made in this probabilistic way.” – Joshua Bloom

“Even though we're using machine learning to really automate the whole astrophysics stack and remove people from the loop, can machines be taught to ask the questions that we haven't, or we can't? Will machine intelligence ever replace the eureka moments by people?” – Joshua Bloom

Transcript

Maybe a little bit darker — oh good, that's good. Those of you that knew Jim Gray know that this was an endearing quote; he meant this in the best of all possible terms. Astronomy's data is big, there's no privacy concerns per se, and unlike satellite imaging or with medical data, if you make an incorrect statement about it because you're trying out some new algorithm, no one dies and we don't go to war unnecessarily. Jim was a computer scientist at Microsoft and an alumni at Berkeley, and he saw astronomy as a sandbox to develop new formalisms of fundamental computer science research. In some sense we're trying to reach back the other way: astronomers are starting to play around with some of the new concepts in computer science and view all of that as our own sandbox, and we're trying to do something novel with it.

I want to thank the organizers and thank all of the other speakers — it's been an absolutely amazing couple of days. In some sense, as a way to end this workshop, I wanted to try to make this talk fundamentally about a conversation between computer scientists and astronomers, or physical scientists in general. Our stack really ends with insights about the universe, but it starts with the things that you all do, and I hope, as you'll see throughout the talk and especially towards the end, that there's something astronomy is bringing to the kinds of thoughts that you're working on every day.

One of the fairly well-known astronomers is this guy named Galileo. He was pretty opportunistic, and in some sense he's, I think, a pretty good example of the way that astronomers look at toolkits that they themselves didn't necessarily invent. He knew about this thing called the telescope, which most people have been pointing out at the horizon looking for enemy ships. He said, what if I just do that and look up? And the rest in some senses is history. But repurposing telescopes that were invented for military purposes to point towards that horizon is in some sense what astronomers have been doing ever since then. And obviously with those new eyes came some amazing insights. In some ways he was the guy that really put the nail in the coffin of the Ptolemaic order of the universe — the planets were supposed to be fixed spherical orbs, and they weren't supposed to have moons — and that picture that he made about 400 years ago of this discovery of the moons around Jupiter was obviously an amazing landmark. I also wanted to show this because this is really our first glimpse of so-called time domain astronomy. Time is moving in the axis going down and angular position on the sky in the axis on the right, and synoptic observations — taking observations repeatedly over time — plays a very big role in the work that I do. I also wanted to show this to you because it emphasizes the crucial role that humans have in data collection, or at least had in data collection — and not just data collection but data analysis and inference. You could think of this piece of paper as not just a log book but a database. What's so amazing is that this one man with his toy and intellect really changed the world.

What happens — and it's not a lot of data — what happens when astronomers get more data? Well, we just add more people to the loop. Harvard College Observatory in the late 1800s was dealing with a vast inflow of photographic data that was coming from remote sensors, in this case photographic plates from the southern observatories, and there was a set of questions in the time domain around finding and understanding variable stars. But there weren't the same types of computers we have today. So these computers — these people who are poring over actual data — were in some sense a prototype of crowdsourcing, or, you could think of it another way, a prototype of grid and parallel computing. It's also sort of a personnel innovation. In some sense it also sets the groundwork for modern day data mining. Viewed from a modern lens, this is obviously fairly pejorative, to call these people — and they were all women at the time — computers, in a modern way.

However, when we look at it, we can also see great things in the fact that they were close to data, and they made some very important contributions, as you'll hear about later. At the time they were really interested in understanding the fundamentals of stars: what are their masses, what are their radii, what are their temperatures? And in making measurements of binary stars in particular, it was thought that one could, in concert with other types of data, make some fundamental measurements of stars of a whole different type. So this is fundamentally what they were looking for — binary stars. We'll come back to that later on.

Today, of course, we have graduate students and post-docs who are actually using the types of computers that we all know very well, and the concept behind what we try to do in our group, which is called the Berkeley Center for Time Domain Informatics, is figure out ways to take people out of the real-time loop. As you'll see throughout the talk, there's a great need, as we get more and more data, to be able to automate the sort of tasks that people way back when — and even just a few years ago — were doing in a manual sense, even ones that seem fairly cognitive. These are ones that we're actually trying to automate away.

We're concerned with the dynamic universe, this notion that everything in the universe changes if you have enough sensitivity, and we can see those changes unfold before us. It's easy to think about astronomers as portrait artists, but we like to think of ourselves as celestial cinematographers, because for us time is a very important axis in understanding the physics of what's happening. How fast or by how much tells us a great deal about the underlying physical processes.

With that, I wanted to show you a very important — probably the most important — three-frame movie you've ever seen. I'm sure you've seen longer-length movies before. Why is this important? Well, first of all, this is a snapshot of the same place in the sky taken about 45 minutes apart between these images, taken on a fairly small old telescope, about 60, 70 years old. And why is this so unbelievably important? First of all, you can see the kinds of raw data that astronomers are dealing with. There's some noise in here, there's this effect called fringing from the detectors, and there's an object moving here. Can anyone find that object? Yes, you're all pointing right there — very good, there it is. This is the discovery movie of this thing we now call a dwarf planet, called Eris. It turned out to be larger than Pluto, and this happened 10 years ago. For those of you that have children that are upset about Pluto being demoted from its planet status, this is the movie that essentially did it. This was a massive data mining effort, and it was done not entirely in manual mode, but obviously had a huge impact upon our understanding of our own place in the solar system.

Here's another two-frame movie, actually just superimposed on top of each other, from the Hubble Space Telescope. There's a dot here, a dot here. A very, very bright star, through various different techniques, was occulted out, so we could see, essentially behind the glare of that star, this very, very faint object, which moved from here to there in two years. It turns out from there to there is a Keplerian orbit, and this is now believed to be the first image of an extrasolar planet by direct imaging. And this is just happening essentially in our own lifetimes. Watching things move in space is obviously having a transformative effect upon our understanding of the universe.

The thing that I spend most of my time on is how things change in brightness or change in color. Everything, as I said, if you have enough sensitivity, appears to change. Stars, even though they might seem fairly quiescent — if you have the ability to collect enough photons quickly enough with enough signal to noise, you will actually see it burble around. That's not what I work on either. What I spend a lot of my time on is working on more dramatic things, where I'm not looking for changes in one part in a million to the light; I'm looking at something that wasn't there and now it's there, and then it's going to go away. Typically this means that something catastrophic happened: stars die and they blow up. This is the realm of supernovae, gamma-ray bursts, and potentially even new phenomena. One of the important themes that you'll see throughout this talk is the notion that just because we can make measurements of these changes doesn't mean we actually understand anything about it. In particular, even if we've made an interesting discovery of something new that's happened in the sky, the greatest insights in astronomy happen when we're able to do follow-up imaging or follow-up spectroscopy or look back at archives.

The important thing about all of this follow-up after discovery, which is in some sense where the real fun comes along, is that it's expensive. There are real resources involved in all of this: there's people, there's time, there's precious telescope time, and it just goes on and on and on, fundamentally, which I think makes this whole problem quite interesting. We could probably turn all of these limited resources into a notion of a dollar term, and what we learned is that when we start couching the problems that we have in astronomy in raw terms like that, the metrics that we wind up applying to decisions actually become even easier to follow.

It's not just the objects in the sky that we know about that are interesting. And I should point out — this is one of the terms that I'll introduce here — this is what we call light curves. This is basically time on the x-axis here, and this is brightness in crazy astronomer terms, where this is faint and this is bright. And there are a bunch of different behaviors of brightness as a function of time of these very important types of exploding stars. These are the two most common ones: Type Ia supernovae are the exploding stars that are used to measure the very precise expansion rate of the universe and have a great importance for cosmology, and Type II-Ps are more ordinary stars blowing up. These are the things that we know about and we find all over the place. Although they're incredibly rare on the sky, people have gotten pretty good at finding them. And then there's things that have been hypothesized to exist — the so-called pair-production supernovae, which is brighter, and all these other kinds of exotic events. It doesn't matter what these terms actually mean; the point is that I'm showing you things that we know about and things that we think we know about. In a Donald Rumsfeldian view of the universe, there are the known knowns, and we can go after those pretty easily. There are the known unknowns — maybe you might say that the theoretical work might guide us in the types of pictures and the time scales that we'd be applying our observations to, to find these things. But then of course, by definition, I can't plot for you the unknown unknowns.

One of the major difficulties in doing needle-in-the-haystack work — discovery — in astronomy is making sure that you keep your eyes open for all these different classes of possibilities, and making sure that if you're going after one of these events — which, by the way, would be a major result if we could actually find an electromagnetic signature of two neutron stars merging — we have to make sure we don't completely exhaust our resources and throw everything we have at every single possible event. So we need to be very careful.

It's of course preposterous to think about scaling up, say, 10 to the 6 Jodie Fosters to listen for what it is that we think we may hear in the sky. Yes, we may be able to hear subtleties that might be hard to program a machine to uncover by itself, but observing and listening to the universe doesn't guarantee discovery, and discovery of course doesn't guarantee insight. This is especially true today as we just get more and more and more data. For optical astronomers, those of us working in the visible regime, this is really the golden era of time domain astronomy — and Big Data time domain astronomy. This is a depiction of the Large Synoptic Survey Telescope, which is now on track to come on sky in about 2020. This will be surveying the sky every couple of days, essentially imaging every available part of the sky to unprecedented depths, and the amount of data and information we're getting just about the changing parts of the sky is truly mind-blowing. For us, we're going to get light curves of almost a billion sources, which will be updated every three days. We're expecting to discover about a million supernovae a year, and it goes on and on — pick your favorite time domain object, and LSST is going to find these things in spades. Twenty terabytes a night is more than most astronomers at most wavelengths are used to collecting, although there's some other wavelengths where that's actually not that much.

But in the context of scaling up those computers that you saw at the very beginning, if you think about that as actually happening today, with saying, well, I've got more data, I'll just hire more graduate students — this is obviously not going to work anymore. This is easily a petascale problem, and this is happening at a number of different wavelengths: LOFAR and SKA is an even higher data throughput problem. What's exciting, I think, is that we know this is coming, which is good, so we can start preparing for it. But there are a number of precursor surveys that we're starting to cut our teeth on, and this is for me my real sandbox, saying, well, we can try out new ideas of how to do this sort of automated astrophysics so that we're ready for LSST, the Large Synoptic Survey Telescope.

So for us the Big Data challenge, and this question that we ask, is: how do we do discovery, follow-up — which remember I said has got resource implications — and inference when the data rates and the requisite time scales preclude human involvement? It's an interesting question, because going back in retrospect and looking at a database and saying, oh wow, there was something really interesting a year ago, I sure wish we had used all of our telescope resources to do some real great science with it — that obviously doesn't cut it anymore.

What's exciting about the time domain is it adds this extra element of urgency: I need to take action on the things that are happening in the sky right now for me to maximize the return. I think I've mentioned this a couple of times already, where we're trying to automate the scientific workflow, and in a cheeky way I'm plotting the barrier to entry as a function of intelligence required to do this. So there's the observing stack, there's the finding stack, the discovery stack, the classification, follow-up, scientific inference — writing papers, getting awards, standing at podiums, et cetera, that's not on the plot. And some of these things we're starting to really get a good understanding of how to do well under the types of conditions that we all operate in. I'll talk more about all that later.

Observing is getting to be fairly routine: robotization, queue scheduling — it's a constraint satisfaction problem. Finding — that is, actually identifying those objects in the sky that potentially could be new; maybe they're just noise, who knows — but really getting down massive amounts of data into, say, medium amounts of data. And then discovery, and this is where we're going to draw from computer vision and image recognition. And I make a distinction, by the way, between discovery — where we might say, aha, this is something of interest, I don't know what it is yet, but I recognize that this is something that I may be interested in, I want to do more work on — I make that distinction between discovery and classification for the following, or at least drawing on the following story from history.

Again it goes back to Galileo. In all of his observations of Jupiter, it turns out that Neptune, at the time when he was making these observations, was pretty close to Jupiter in the sky, and Galileo made note of it in his database — in his notebook — and he said, here's a fixed star. Galileo is not credited with the discovery of Neptune, even though we can go back and with essentially no doubt believe that he actually recorded it. He found Neptune; he's not the discoverer of it. It took another 220 years or so for the actual discoverer to find Neptune and realize that it was something interesting. I think that's remarkable — he would have been a very famous person if he had actually discovered Neptune. Once you've said, aha — I've not only found it, I've not only said this is something that I care about — now you want to ask, well, what is it? What is the source of this change on the sky? What's the physical origin of this? And this is where we're starting to apply some machine learning techniques. And then you want to do follow-up, et cetera, et cetera.

Let me just talk a little bit about automating observing. One of the great leaps, I say, for astronomy — and optical astronomy in particular — was when we went from photographic plates to digital imaging in the form of charge-coupled devices, CCDs, because for the same size aperture telescope we could collect essentially light at 50 times higher quantum efficiency, which means that we were able to build up a much better signal-to-noise image much quicker. It also has some great properties, like they're linear, and so the data that you get out of it tends to be more accurate. But after we went from photographic plates to digital imaging, we also had to take humans out of the loop of operating telescopes, and so we've gone from human operations to robotic telescopes. My colleague Alex Filippenko has been operating the so-called KAIT telescope on Mount Hamilton, not so far from here, for almost two decades now, and that's been incredibly important, because getting these people sitting on top of a mountain night after night taking data is a lot harder than just getting a robot pre-programmed to do things.

This is something that I did early in my career: I actually roboticized an old telescope that had been mothballed, and then I decided to go beyond just the robotization of it, so that I could put in a whole list of objects that I cared about, where I actually asked the telescope to figure out what it thought was most interesting to observe at any moment, given all the constraints that were put in by all the observers that had gotten time on the telescope. So beyond robotization, we started creating what I guess we could deem intelligent data collection agents. This is me standing in front of my telescope. I highly recommend people getting pictures of themselves in a smug pose in front of stuff they built, and if you haven't done that already, that's okay — for those of you that code a lot, you can just print out code and have somebody take a picture, so you can show it in talks like this.

What it does is autonomously schedule itself based on complex prioritizations — again, that's a constraint satisfaction problem that we worked on for a while. It does detailed weather sensing, so it senses when it's in danger and will try to shield itself and close itself up when there's a problem. And it will react to new things in the sky without any people in the loop — again, remember, we're trying to take people out of this loop, because the idea here is then we can be more efficient with the way that we do science. And so if there's an event, say from a satellite, it will be able to slew if it believes that it's important to look at that object in that place on the sky, before anybody even wakes up. It tweets where it is on the sky, so that other people can follow this up and actually try to draft off the locations of where the telescope is. And under the hood it's some interesting stuff that we worked on: daemonizing all the different subsystems and making this into a formal state machine, so that we could wrap our heads around all the different problems that wind up cropping up when you're doing a software-slash-hardware problem.

The results have been pretty great — we've been very excited about it. This is a movie up here going from one minute to 30 minutes after one of these gamma-ray bursts exploded in the sky, essentially without warning. A minute after an event was recorded by a NASA satellite — and it more or less beamed down a text message to the ground — this telescope had slewed over and started taking data. And what you see is a negative image here, where it's very bright at the very beginning, and you see after about 30 minutes it's completely lost into the noise. It wasn't just a burst of gamma rays; it was also a burst of visible and infrared light, and you can see the light curve here just fading from very bright — your eye can see down to about here. This was incredibly bright, and then it was tens of thousands of times fainter than what your eye can see within just a half an hour. If we had had a grad student in the loop, then maybe they would have had to have their cell phone go off, and they rub their eyes and they fumble for their computer and they forget their password and they log in, and all of a sudden we're way down here and we get to miss all the juicy science in there.

We're not the only people who are trying to push the envelope a bit on robotic telescopes. This is now becoming a more and more interconnected system: instead of well-connected graduate students talking to each other, we're trying to figure out ways to have telescopes talk to each other and actually federate themselves to actually do observing campaigns without any people in the loop. And so what we see here is an initial network of these telescopes which are starting to talk to each other, and we obviously have different roles for the notions of publishers and subscribers. What's pretty neat is that this topology of this network has been growing organically, and I think it probably looks a lot like some of the early days of the internet. And for those that are interested, I created essentially the language that telescopes use to not actually just talk about where they're going to look in the sky but describe different events in the sky, and this has been tremendously exciting, and there's been some actual great results that are coming out of just this network.

I want to keep going, though, and talk about what happens after we've now scheduled the telescope to look at something and we've now gotten the data — we then automate this reduction process — but now how do we wind up doing discovery? I've been involved in one of these precursor programs to the Large Synoptic Survey Telescope called the Palomar Transient Factory, and what you have here is the kinds of things that, for the first several weeks of this project, which started in 2009, we were having ourselves — and not just graduate students and postdocs but also faculty — looking at images. These are two little places on the sky. What you see here is a new image — it's a little postage stamp around a new place in the sky, and all that nebulosity, by the way, is from a nearby galaxy — and then here's this new image right here of a different place in the sky. And what we have is a deeper reference image of the sky, built up over a number of nights before this night, and we more or less just do a subtraction to look for something that's new.

Nature is not usually so kind as to draw green arrows and cross hatches around where there's actually a new object, but I wanted to point your attention to this object right up here. It got this name, 11kly — that's not all that important. The important thing to note here is that this is a pretty easy find for an automated algorithm that's just passing over these images: it's just noise, and then there's something that's obviously not noise. But then down here you see something which is a little bit fainter — which, by the way, to your eye it's pretty hard to see. This is a new supernova that happened on the outskirts of a galaxy which is quite far away — halfway to the edge of the observable universe. And what you see are these red boxes around here, where the subtractions were actually imperfect. It's a very hard problem to do a subtraction in the presence of noise, because there's subtleties in how the atmosphere winds up distorting the image; the telescope is not exactly in the same state today as it was in the past. So this is a really hard problem. We get about a thousand images a night from this project, and from that we're able to extract — including all these red boxes — all the possible candidates: about 1.5 million of those objects a night. Of those, about a thousand of them are actually bona fide new objects in the sky, about 300 of them are variable stars, and about 10 of them are new transients — new things that weren't there and are now essentially there. These are new things that are exploding somewhere in the universe, and we want to now follow these things up.

So we've been working on trying to automate this process of discovery, and one of the things you wind up really realizing right away is that discovery is imperfect. Whether something is new or not, especially when I'm getting close to the noise floor — you don't really know if it's actually a new object. You might learn later on, after it's gotten brighter or fainter than it actually was, but if I want to take action on that right now, this is an important thing to understand. So we started more or less taking those images and trying to do even further dimensionality reduction, more or less creating of order about 50 different so-called features that really describe each one of the candidates: fits to the Gaussian, what's the signal to noise, what's the distance to the nearest object, et cetera, et cetera. And what you can see in some of these pictures right here, in the two different colors, are examples from our training data of real candidates — so these are real astrophysical candidates — and bogus ones. I didn't point out before, but going from 1.5 million down to about a thousand is a pretty massive needle-in-the-haystack problem, so there's a huge imbalance in these two classes of good versus bad or, as we call it, real versus bogus. And so we have all of these different features, and it's really just a two-class problem: we're asking, is this candidate real or not? And so we've couched this in the way that people who do machine learning can understand, and we threw as many algorithms at this as we could, and we wound up finding that some of them did much better than others. This is the so-called ROC curve — receiver operating characteristic curve — of false positive and missed detection rate. It's an important place to be able to understand where you want to be on this curve. If I really didn't mind more or less missing all the interesting objects in the sky, and I was really afraid of going after and using resources on all my false positives, I might put myself right over here, where I have a very low false positive rate — that is, calling things that are actually bogus real — but I would miss a whole lot of interesting stuff. If I put myself here, I'd miss half of the interesting stuff, so that's a problem. If I went all the way down here, and I didn't want to miss anything, well then I would really start blowing up my false positive rate. So you have to decide, if you're now trying to build a framework around this in terms of follow-up, where do you want to be on this curve.

What we wind up doing is bootstrapping the survey, so that we started off with just a couple of training examples, and then as the survey went on we wound up getting real ground truth about what actually was real and what wasn't. And we wound up finding, as you might expect, that as we added more data to the training set, we're able to get better and better classifiers. Here's where we were in the original paper that I wrote in 2011, and just by adding a larger training set we went all the way down here. For me this brings back one of the well-known statements from Peter Norvig, who said that more data beats clever algorithms, but better data beats more data. I have a little bit to say about that on the next slide, but it's very clear that more data helped us tremendously in getting a better classifier, even just using the same algorithms. But what's interesting is that what we wound up doing is we started flipping the training data. So we took a real object and we said that we actually now believe that it was bogus; we took a bogus object and we flipped it around and said that it was real. And we kept re-running things as a function of how much we messed around with the ground truth, and what we wound up finding is that only a few algorithms did very, very well in the presence of what's called label noise. We got up to about seven or ten percent label noise — that is, being wrong in the training set — and still produced a very, very good classifier. So that was pretty interesting for us.

Now, those that were paying attention to that thousand number going down to 10 — it turns out that most of the real astrophysical objects in the sky are actually asteroids. Asteroids are very interesting to a large number of people, but they're not interesting to me at all, so we needed to figure out a way to get rid of all of those so we weren't wasting our resources on following up known objects. So we built essentially a parallelized version of a resource that was being used and is still out there called the Minor Planet Checker. We just did this in a couple of months with one graduate student; it's now being used worldwide. But for us this was really the last step to be able to say, aha, we have actually made a discovery — and moreover, because we have training data, we can go back and figure out what the probability that we're wrong about that discovery actually is. So we can ascribe probabilities to that discovery.

This has been a great project — and in fact it's already ended, and now there's a new incarnation just starting up called the intermediate Palomar Transient Factory — but one of the things that we were very excited about was illustrated in this three-panel image here. On August 23rd in 2011 there was a nice image of this galaxy — there was, again, no green arrow that nature was telling us to look there. On the 24th of August there was this little new object that popped up, and it didn't move, so it wasn't an asteroid. And then on the 25th it was getting brighter and brighter and brighter. This turned out to be a discovery of a supernova which, after we did a lot of analysis, happened just 11 hours after the explosion. It's the first time we've been within 24 hours — or maybe even three or four days — after the explosion of what turned out to be an ordinary Type Ia supernova. It's the nearest Type Ia supernova in three decades, and it was discovered by our machine learning framework that essentially promoted it to the top of the stack for people to actually do more inspection on and follow up.

And because we got on so early, and because we were able to throw as many resources at it as we could and we recognized the importance of it, we were able to make some pretty interesting and useful discoveries. I'll highlight two of them here. This is a bit of a busy plot, but I want you to just look at this green area here and note that these are all different regions of exclusion in effective temperature and average density of the actual object that blew up. There's a lot of physics that goes into making this plot, but fundamentally all these different lines rely upon a single observation of a non-detection — what turned out to be four hours after the explosion — that an amateur astronomer had essentially taken in Spain. The non-detection of that turned out to be very, very useful for limiting the actual radius and the properties of the thing that would eventually wind up blowing up. And what we were able to do is rule out all types of normal stars and only allow, in this space right here, very dense stars — compact objects, either a white dwarf or a neutron star. And people have been talking about white dwarfs as the objects that blow up for Type Ia supernovae — and as you know, Type Ia supernovae are incredibly important for understanding of the universe as great probes of the acceleration of the universe — and yet we haven't really known for sure what it is that's blowing up. Astronomers are pretty great at this: they'll say, I don't really understand my probe over there, but it does its purpose, so that's good enough for me. Here we're actually able to rule out all other reasonable possibilities for the first time.

Type Ia supernovae are also thought to be part of a binary system, where the larger star is transferring mass onto the thing that winds up blowing up, causing it to explode after it reaches an important limit, and there's lots of different possibilities of the configuration of that binary system — what is the size and what is the age of the donor star, et cetera. And we were also able to exclude, in this yellow region up here, many of the well-described theories about what makes Type Ia supernovae, because there was pre-existing imaging of this place on the sky with the Hubble Space Telescope, and a non-detection of objects there — in the place where a supernova was going to explode just six years later — turned out to be incredibly important for ruling out all but just a few models. I'm trying to point this out to show that by putting machine learning into practice, we were able to do this massive needle-and-haystack problem incredibly quickly, and we're able to do this very important science that informs really all aspects of astrophysics, just as quickly as one could hope for.

But I still haven't told you what happens after we do discovery. After discovery you want to ask that question — what is it? And here again we're bringing in some machine learning concepts, where we're using machine learning as a surrogate to ask that question: what is the nature and the origin of the variability that I see? And in this sense this is what I would call classification. The problem is, in the context of variable stars, it is a zoo — it is a variable zoo of variable stars. And if I gave you an object which is changing in the sky and asked you to tell me, even if you're an expert, what it is on the sky, you might have a hard time putting it into one of these broad classes, and you might have a very hard time, until you got more data, putting it into one of these subclasses. If you've got great data, you might think that you can just do some template matching. This is four examples of variable objects on the sky that are varying on a variety of different time scales, from three different large classes of variable stars. There's the so-called eclipsing systems — these are binary stars; there's pulsating stars that are changing their radius as a function of time; and cataclysmic variables — these are things that are undergoing catastrophic mass loss or some type of eruptive explosion. Here again are some light curves. This is a pretty nice-looking so-called W UMa class star, and it's changing on time scales of hours. Here we've got a so-called Beta Lyrae, which is changing on time scales of order days. Here's again a Type Ia supernova light curve, and here's a long-period so-called Mira variable.

So if you had great data and you had great examples from all of these different classes of variable stars, the easiest thing would just be do template matching. The problem is, even though we have these broad classes, this classification system is really kind of an admixture of phenomenology and physical thought and physical belief. Some of these classes are called X-ray because they emit X-rays in addition to optical light, and some of them are called eclipsing because we actually have a physical belief that one star is moving in front of another. But we don't typically have perfect templates of all the things that are changing in the sky, and even Type Ia supernovae, which are incredibly regular in their light curves — again, which allows us to make these great measurements to large distances and measure the expansion of the universe — they're all different in their own right. And when you get down to the subtlety of it, each Type Ia supernova and every other type of star is actually different in its own way. It's sort of like Tolstoy and Anna Karenina, I guess — every unhappy family is unhappy in their own way. So there's really a lot of interesting problems that start coming up when you're looking at the actual type of data that astronomers get. We don't get to have those perfect light curves, and one of the things we have is noisy and irregularly sampled data. When we actually extract information out of the data that we've now reduced from the raw data down to the nice images, sometimes we could get the answer vastly wrong and underestimate the uncertainties in our flux measurements — might call that spurious data. And if we're thinking about follow-up, the telltale signature of what it is that we might want to be actually observing might not have happened yet, so you also have to have some predictive modeling of what might be happening in the sky.

So the machine learning approach to all of this is to take all of the data that we can possibly get our hands on and coerce it into a large-dimensional feature space, and then with training data hope that we can build a classifier — now not just a zero-one classifier but a multi-class classifier — that would actually allow us to do this well. And so we do this with about a hundred different features on a very large data set of variable stars. All the things in blue are time-domain-related features — again, doing this in the presence of noise and irregularly sampled data, so some of this stuff is non-trivial — but then also context features, this notion of where did this thing go off in the sky.

What's interesting is, if you came up to me afterwards and you said, Josh, I saw this interesting object in the sky last night, it got brighter and it went away, I might ask you first, well, was it in the ecliptic plane — where a lot of the asteroids actually live — and you'd say no. And I'd say, well, was it in the galactic plane, because then maybe it's associated with a star, and you could say no. And then I'd say, well, but was it near a galaxy? You'd say, oh yeah. And was it at the center of the galaxy? And you would say, no, it's on the outskirts of the galaxy. And I'd say, was the galaxy red or was it blue? And you'd say red. I'd say, oh, that's a Type Ia supernova — and I'd be right 99% of the time. So without telling me anything about the time domain, just the context of where this thing occurs in the sky — if we're able to go after all the different databases of that place in the sky and extract all this information, some of it rich, some of it in places of the sky we've never even looked at, and build a classifier on that, we can do great things.

And so we're able to build a pretty good classifier over — I think this is about 25 classes of variable stars — where this is the true class and this is the predicted class, and this is a whole cross-validated error matrix. And what you see is that you want to have all your power along the diagonal here, meaning that your classifier is always right — but unfortunately it's not. What's really interesting for us, though, is that when you draw boxes around these larger classes and don't get into all the different subclasses, what's pretty interesting is that a lot of the off-diagonal power still lives within these larger classes. And what's pretty neat about this, I think, is that even though we didn't tell this machine learning classifier anything about the physics or anything about the actual connection of all these different stars with respect to each other, there's something about the fact that they all kind of look the same as why the classifier maybe got some of this wrong. So the classifier in some ways is discovering some of the physical behavior and some of the physical connections between these larger classes. And we're getting down to about 15 percent classification error, which is pretty good, but when we actually start using a structured classifier, where we actually use the taxonomy of this classification tree as it exists, as people created it, we get down to five percent gross misclassification rate, which is pretty good.

One of the things that has been touched upon throughout the last three days, which I found pretty interesting, is this notion of how we can learn on, say, one survey. And I should say that when we take data with one survey, we're doing it with a certain set of filters, we're doing it with a certain cadence — so we go back to the same place in the sky at some level of frequency — we're doing it with a certain telescope, et cetera, et cetera. Every survey is different. So now if I handed you that classifier and I gave you all of my codes, and I said, go ahead, now you've got your new survey, go ahead and classify — you would do a very terrible job with my classifier. And that's because the way that you observe the sky actually winds up changing inherently the underlying feature distribution.

This is illustrated pretty well in James Long's thesis — he's finishing up right now; he's been doing a PhD in the group — and what you have here is a depiction of a three-class problem where you have feature one and feature two, and you can build a perfect classifier — a decision tree of, if you're on this side here, then you belong to the yellow triangles, et cetera, et cetera. But if I take that exact same classifier from this survey called Hipparcos, and I apply it to data — now where we actually know the answers for all these three different classes — you see I've produced a very lousy classifier. And so this question of how you use decision boundaries and classifiers from one survey and transfer that knowledge onto another survey that has different characteristics is a very interesting question. And what's pretty neat is, of course, if you're observing with a different telescope in a different part of the sky, you're going to have different reasons for doing so; you're generally going to be probing a different population of stars. And so there may be whole places in this large-dimensional feature space that are not even populated by one survey and are incredibly well populated by another survey. That's illustrated here with a plot of the period of known periodic stars and the amplitude of how much they're changing as a function of time over the course of their period. These are very large amplitudes — changing by an order of magnitude — all the way down to very, very subtle changes. And what you see here are data from two different surveys. One is a survey called ASAS — this is where we had all the ground truth. And then you have this other survey that we'd like to know the answers in — sorry, this is where we didn't have the ground truth; we have all the ground truth from the training data — and you see that down here we have no examples. We have no idea what classes of objects those things are.

And so what we've been doing is building up a framework for doing active learning, to figure out how we can imbue experts into this problem to allow us to identify and manually label the testing set of data, so that when we use the finite amount of expert resources, we'd be able to, in future iterations, do a better job on the classifier that I took from survey A and then applied to survey B. I won't go into the formalism here of how we did this, but I just wanted to show that, with a few iterations — if we have our off-the-shelf error rate on this essentially transfer learning problem — very quickly, with just a few iterations of active learning, we're able to get to a very, very nice classification error that's essentially acceptable for this problem.

I'm sure all of you know this, but astronomers have a very hard time with the slide I'm about to show. The classification statements that we make on this noisy data is surprisingly fuzzy. I can't tell you it definitely belongs to that class; I can only give you probabilities that that object belongs to that class. And what you really hope for is that I do a great job in calibrating those probabilities. I can get something that comes out of my classifier and say there's a 20% chance of this type of supernova, but what you really want — in the same way that weather people will say there's a 50% chance of rain — you actually want it to rain 50% of the time. If I'm wrong in either direction, you might get angry with them for different reasons. So the catalogs of transients and variable stars that we're building up are necessarily probabilistic, and this is incredibly subversive to astronomers, because they like to go to a paper of known class of this type of star and say, ah, I'll observe that. But those were all populated by people and experts over the last 100 years, and a lot of those things actually turn out to be wrong — that's another sociological question. But we have to learn as astronomers to say goodbye to black-and-white catalogs, because these catalogs, if we're getting lots and lots of data, have to be made in this probabilistic way.

And so we tried to make it easy for astronomers to digest this. How am I doing on time? Yeah, okay — okay, good. What we did is we built a website that allows people to probe through the probability distribution for any one of these stars and search around for the different types of objects that they care about — this is using the Google Fusion Tables under the hood, if people are interested. So I can go to this type of RR Lyrae, I can actually click on one that I might be interested in, and there's the probability vector of the top 10 most important ones, there's the actual light curves — and you see how crappy that data is. And then we even made it social, just in case Marissa Mayer wants to potentially buy us — maybe Yahoo will buy this thing, I don't know. But we've gotten a lot of traction with this, in the sense that astronomers who care about these different subclasses are actually pretty interested, and they're learning how to actually use these catalogs. We've been trying to go beyond this and now actually do our own science. Again, just because you make a catalog doesn't mean that it's actually all that useful. You're doing this not just for the exercise of, cool, I can do machine learning and make probabilistic catalogs; you're doing it because you actually want to do novel science. So building a catalog so that you can do novel science, I think, is only justified after the fact if you actually do that. And what you really want to do is figure out what kind of science do you want to do. Do you want to do demographic science, where you want the purity of your sample to be very high — and there you might really tamp down and put yourself on a part of the ROC curve where you have very few false positives — or do you want to work in novelty discovery, to go after the rarities, where maybe only one out of 100 objects that you actually spend telescope resources on turn out to be actually all that interesting?

And so we were able to do this with one of my other graduate students, Adam Miller, where we wind up discovering some very, very bright stars that were essentially misclassified for the last five or six decades, where the probabilistic classifier said it was of this type — so-called R Coronae Borealis stars or DY Per stars — and with just a couple of spectra we're able to confirm these very, very weird objects. I don't think I'll go into the details of how well we do relative to just blind searches and blind cuts.

I want to just end with a couple of statements not about looking at catalogs of data that have already been taken, but about doing this classification on the fly. I already talked about how we do discovery on the fly; now we actually want to say, is this a supernova, is this a variable star? And one of the great places we got some training data from was this group called the Galaxy Zoo, where we presented to them — citizen scientists — a bunch of images just like the ones that I showed you at the beginning of the talk, and asked them, is this a supernova or not? And this is not a supernova, and that is a supernova. And the citizen scientists got very good, and they wound up marking up a few hundred of these images a night and built up a huge training set for us. We then of course went back and we looked at those places on the sky and applied our resources of follow-up, and we got some ground truth. And what's pretty amazing is that we were able to build a classifier using their training data, and the classifier outperformed the people at any single false positive rate. So our ROC curve — you actually want it down here as much as you can — was actually below the citizen scientists themselves. So that was pretty exciting.

And we're now putting this into practice, where we're not just doing discovery on the fly; now we're actually able to make classification statements on the fly about what type of object it is. And we don't have a lot of data — when we just have one new data point, we have to more or less rely on context, and so we have a simplified version of the taxonomy. What's pretty cool, I think, is that we now have the robot that's doing this classification log into the system that everyone in this group actually uses, and it actually will log in now and say, I think this is a transient, and I think this is a supernova. And so then you see a whole bunch of people who logged in and said, oh okay, it's at this distance, it's this type of supernova, after they got more and more data. But the robot now is responsible for more discoveries and more classifications than the people are in this project. What's exciting about this is this actually turned out to be a fairly interesting supernova — it was a paper that appeared in Nature a few months ago, because we were able to again get great data and maximize our resource usage.

I've been thinking a lot about machine learning, because we've been drawing from all the great literature on machine learning to do the kinds of science that I'm interested in doing. But I've been thinking about how that may be a bit of a misnomer. That notion of learning is a difficult one. In a cheeky way, I would say if you went to the Library of Congress and you learned everything there was to know in the universe, and you had some amazing insight, and then you got hit by a bus, it wouldn't be all that useful that you learned all that. So for me, trying to find another term that has meaning, I've come to this term of machine intelligence — of putting machine learning into practice.

I think it's easy on the theoretical side, and maybe with a computer science lens, as you're thinking about a new machine learning algorithm, to think about classifying irises with your new great algorithm. But the data that we actually get in the real world is noisy and it's subversive and it's fuzzy, and we don't really have a great answer, and the color of a petal of an iris is not purple plus or minus red. The kinds of data we actually get in the real world has noise, and sometimes it's missing, and it's dirty. And so putting the results of machine learning into practice — I haven't seen much of that in academia. We see it obviously a lot in industry, where they've been able to recognize the importance of doing great machine learning that works in practice, because if they got it wrong, there are real resource implications. Does a botanist really care whether we've got a great classifier of different types of irises? I don't know — has anyone actually asked the botanist, do you care about that? What's important is that we need to figure out ways to get machine learning working for the types of science that are collecting lots and lots of data, asking these very complex questions on that data. And I think it's only really just beginning.

I wanted to come back at the end here to this picture that I showed you at the beginning — it's very, very dark — of the computers in the room at Harvard, and not point out the light curve but actually point out this person in the foreground here. Her name is Henrietta Swan Leavitt, and she was the discoverer of the period-luminosity relation of Cepheid stars. And that discovery that she made in this room, around the same time when this picture was taken, was because she decided she wasn't all that interested in binary stars, and she started thinking about pulsating stars. The work that she did became the basis for Edwin Hubble's first measurement of the expansion of the universe. It's the genesis of modern cosmology.

And so what I end with is a question that I don't have the answer to: even though we're using machine learning to really automate the whole astrophysics stack and remove people from the loop, can machines be taught to ask the questions that we haven't, or we can't? Will machine intelligence ever replace the eureka moments by people? And will the entire astrophysics stack — not just from the data collection, but to the initial classification, but to the real breakthroughs — will that ever be able to be truly automated?

I'll end with just a couple of concluding statements. The modern astrophysics stack is obviously drawing very deeply from contemporary computer science theory. It's great because these frameworks exist; it's not so great because it means it's a massive educational challenge now. It means I have to teach my students not just about the mundane things, like exploding stars, but now I need to teach them about toolkits that perhaps I don't even know about. But it's pretty clear, in the context of the data deluge for astronomers, that there is a huge demand for the abstraction of the traditional roles of people from the ordinary scientific process that we've been used to. And yet despite all that, I think it's pretty clear that there remains a very important role for people, and I don't want to give that up. So with that, I'll end the talk and say thank you.

Well, thanks for a beautiful talk. Let's have a few questions.

Slide? Yes — good, okay, I can do that. Yep. So the question is, what are people actually doing with this sort of emergent network? I think they actually are looking at asteroids, and using these networks to follow up interesting near-Earth asteroids that could be pretty dangerous — that is actually happening. What's great, though, is that if you have this network, they don't all have to be doing the same thing at the same time. Maybe they spend 10 minutes of their night doing this; maybe one telescope is more interested in doing it than another telescope. Again, because this is all about resource maximization, we've tried to think about how you actually have telescopes from different groups that have different needs sharing data with each other, when perhaps they've never even collaborated with each other. So we've invented a currency for these telescopes called Starbucks — hopefully we won't get sued — but that again makes it very easy, once there is that currency and people believe in that currency, to actually do the best with the resources that you have.

The classification scheme you showed us — has it been frozen for ages, or it started developing? And do you expect during your lifetime to add another branch?

Yeah, so the question was about the classification scheme, that rat's nest of taxonomy. It has to change, because first of all we're making new discoveries, and we usually try to slot a new object into one of the existing classes, and eventually it becomes pretty clear that it just doesn't belong there anymore, and it will wind up branching out. There are new classes of variable stars found every, say, couple of years, and usually it happens now with the very subtle changes of light curves, because the really explosive events have been announcing themselves for 300 years and people have been following them up and trying to understand them. So yes, I think that classification taxonomy will change, and one of the things that I think might be interesting is to take machine learning concepts and figure out actual distances between all of these different objects, and see whether there's any phylogenetic tree that winds up emerging out of it, or concepts of species.

You folks are trying very hard to use computer science, but don't you also want to recruit from computer science?

This is an excellent question, and I'm glad to be able to address that a bit. The question was about whether we actually want to recruit from computer science and bring people in to essentially help us out. That would be great, but I think what's more exciting about this conversation that astronomers are having with computer scientists — and I hope you all see this — is that we're presenting to you data sets and types of questions that perhaps there is not yet a well-established theory to answer and to address. And so I see this much more as a collaborative kind of communication, rather than one where we're just drawing on people in a more rote way. I think what we've done in the Center for Time Domain Informatics, with other faculty in the computer science department and in stats, is produce for them a set of really important science questions that we would love to be able to answer, if only they could develop the theory and the formal mechanisms for us to do that with enough grounding that we believe we're on to something.

Okay, yeah — so the question is, if you're gaining confidence that you've got this awesome pipeline that doesn't require graduate students anymore in the real-time loop, how do you gain confidence that you're not missing something really important, something you hadn't thought about? I don't know the answer to that. One could try to throw dummy events into the data stream and hope that you actually wind up catching those. But then again, if you're throwing a dummy event in, it's still within the realm of the known unknowns. If something is truly different than anything we've ever thought about, it's going to be pretty hard to guarantee that you're never going to miss things like that. And that in some sense maybe is where the frontier gets pushed to: the next generation of people doing machine learning and automating workflows have to figure out ways to uncover the unknown unknowns in a more systematic way. It's a good question.

Automated Astrophysics in the Big Data Era

Wed, 27 Feb 2013 00:00:00 +0000

Invited plenary (IP6) on automating astrophysical discovery, classification, and follow-up at survey scale.

Extracting Novel Insight from Probabilistic Machine-Learned Classification Catalogs

Wed, 27 Feb 2013 00:00:00 +0000

Doing science with probabilistic machine-learned classification catalogs, in the Big Data minisymposium (MS158) - his second CSE13 talk besides the plenary.

Relativistic Tidal Disruption Events

Mon, 03 Dec 2012 00:00:00 +0000

Yale Physics Club colloquium on relativistic tidal disruption events.

Automating Science in the Time-Domain Survey Era: Machine-Learning Challenges

Fri, 21 Sep 2012 00:00:00 +0000

Machine-learning challenges in automating discovery and classification for time-domain surveys, at the SAMSI massive-datasets program.

Classification Challenges

Wed, 15 Aug 2012 00:00:00 +0000

ML approaches to astronomical classification for LSST: real/bogus candidate discrimination, variable-star classification, and probabilistic cataloging, with examples from the Palomar Transient Factory including SN 2011fe.

Deck on Speaker Deck (LSST AHM, Aug 13-17 2012); he was LSST Transients & Variable Stars co-chair.

Telescope Targets Black Holes' Binges and Burps

Tue, 31 Jul 2012 00:00:00 +0000

Featured expert in Lauren Sommer's piece on NASA's newly launched NuSTAR X-ray telescope, explaining black-hole ‘binges and burps’ and debunking the vacuum-cleaner misconception.

Python as Super Glue for the Modern Scientific Workflow

Thu, 19 Jul 2012 00:00:00 +0000

Invited talk on Python as the glue of the modern scientific workflow, from robotic-telescope automation and real-time transient-discovery pipelines to machine learning in astronomy.

Listed on PyVideo; verify YouTube ID at build time.

Key Quotes

“We are not, in some sense, portrait artists of the sky; we are celestial cinematographers, and time is a very important component of us understanding what's happening out there in this vast and dynamic universe.” – Joshua Bloom

“We can't be listening to all of our data, we can't be observing all of our data as a person, we can't scale Jodie Foster to ten-to-the-six people; that just doesn't make sense at all.” – Joshua Bloom

“My view of doing probabilistic classification with machine learning is that the way you find needles is that you get very good at identifying hay.” – Joshua Bloom

“Python is this incredibly wonderful super glue. And I think, although I haven't heard it said here yet, I think it's poised to become the de facto engine for modern science, and I think we should all be very excited and very proud about that.” – Joshua Bloom

Transcript

Thanks for the introduction, Fernando, and I want to thank the organizers and the sponsors, but mostly I want to thank all of you for your efforts in making Python really the engine of modern science. This talk is really meant to highlight the top of the stack of Scientific Python and to show you that many of the results that make it to the popular press are actually results that are based on Python. So a lot of the science that you're seeing, that your parents are asking you about, that you're reading about in the papers and seeing on the news, so much of that has Python inside, and I thought it'd be useful for everyone here to hear about that. My name is Josh and I'm an astronomer.

I'm certainly not as badass as the most famous and outspoken astronomer, so I apologize for that, Neil deGrasse Tyson. And I'm definitely nothing like Vermeer's astronomer. He's unlike any scientist I know. Actually, first, he's clean-shaven. Second, optical astronomers really don't do a lot of their best work during the daytime; you'll notice this is the middle of the day. But most important, this Eureka moment is something that he's having because it's been handed to him on a silver platter, in this case the celestial globe that's been constructed by monks, probably. I think the important thing to recognize with modern science is that we're really working in the muck. We're knee-deep in data and we're throwing every tool that we have at it. This quaint view doesn't really cut it anymore.

Python, I think it's fair to say, is in every component of my scientific life, and rather than tell this story and form this talk from the perspective of toolkits, from what's under the hood, I think it makes sense to present this in some sense anchored by the scientific challenges and results that have been coming out because of Python. Because fundamentally it's the scientific interests that drive our need for solutions. Science is the reason why this community does what it does. Why build and optimize an application if there is no application at the end of it? So what is it that I use Python for? Well, I'm not going to talk about everything, of course, but I will talk about the automation of data collection and inference in my group's work, and in particular talk about robotic telescopes and data reduction, talk about communication, logistics and visualization, talk about machine learning with an eye towards Big Data and astronomy, and then touch on a couple of different aspects of dissemination and education, and in particular mention some of the teaching that we've been doing at Berkeley.

The idea here, of course, is that Python is making science happen through this whole stack, as I said. So I think the thing to do, of course, is show you how we do that in a Python file, and you can get this on GitHub and you can fork it. I guess, are you allowed to say fork here? Is this public? Okay, anyway. So first we have to do some exploration and discovery. So we try to get a new idea; that doesn't work out, take somebody else's idea, try to get some funding. You have to make a proposal, you've got to wait six months, probably won't get funded, you keep on looping until you get that funding. And when you get the funding you instantiate a bunch of grad students, postdocs, undergrads, etc., throw them into a pool, and then you just have them write that up. You have to get the data, find the results, make a paper, submit it to Science, and of course you don't have to go through any iterations there because it gets accepted without revision. And then of course you have to get some credit, stick it on your CV, and that's it. Right? That's all we should be doing here.

This is not quite Import Flying from XKCD, but it's close. Obviously this is not at all like what we do in astronomy and science in particular, but Python is there at the nitty-gritty, and I hope to show you different aspects of that. My interest as an astronomer is in the changing universe, the time-domain universe, the so-called dynamic universe, the idea that everything in the universe is moving or changing its color or changing its brightness, and with enough sensitivity, even on human time scales, that kind of change is now accessible to us and becoming more so. We are not, in some sense, portrait artists of the sky; we are celestial cinematographers, and time is a very important component of us understanding what's happening out there in this vast and dynamic universe.

So we'll go through some examples of all those different types of changes. One is motion in space, and this is perhaps the most important three-frame movie you've ever seen in your life. You didn't realize that, but here it is. Why is this so important? Well, you see, first of all, the noise; you see a bunch of different stars, but there's an object that's moving in here. Who can find that? Yes, I see people pointing in the general direction. It's somewhere over there. Yeah, right there. There, you see that thing right there. That is the discovery of Eris, the object which turned out to be larger than Pluto and killed Pluto as a planet. It was taken with a half-century-year-old telescope, 20-second exposures, and was a massive needle-in-the-haystack effort, but obviously paid off in spades. This is a composite of two Hubble Space Telescope images showing the motion of what is now believed to be the images of the first extrasolar planet.

So motion in space obviously is teaching us a great deal about our universe around us. What's amazing, of course, is that if you have that precision to be able to see and notice things changing and moving on the sky, you can actually not only measure their velocity with respect to Earth, but you can also start to make measurements of distance. Finding the position of an object on the sky is pretty easy in two dimensions, but getting its distance is incredibly hard, and so much of what astronomers do at some level can be boiled down to finding distance. That back-and-forth wobbling that you see on this one object right here, and this is a model fit to a bunch of data, is called parallax, and the motion across from top to bottom is what's called proper motion. And you see a bunch of different data points here. Just to give you a sense of scale, that motion is Guido at about 16 times the distance to the International Space Station. So it's an incredibly difficult set of measurements to make; it's very fine motion. Yes, I just did a star wipe, Guido away.

So how do we make those measurements? Well, we take images of the sky, but of course the images that we take are imperfect, and so when we try to align those images and find how these objects are changing in the sky with respect to the other stars, you have to make maps of how all these different stars line up on top of other stars in other frames. And this distortion map is something that people have been doing for years, if not decades or centuries, but what's remarkable is that people have been doing this in a fairly frequentist way. Measuring the rotation and scale is something that should obviously be a Bayesian framework for doing this, but this hasn't been done yet. So my group has started working on this problem, trying to do essentially very good astrometry, as it's called, measurements of positions of stars with respect to other stars, and we've been using SciPy for that. Because we have to invert essentially a large matrix, a covariance matrix of the position of every star with respect to every other position of the star. And so here's an image of two different covariance matrices, and your job here is to figure out which one of these is going to wind up being invertible. The one on the left. The point here is that when you want to do an inversion of a matrix, if your matrix is not positive definite or positive semi-definite, then you've got a big problem.

What we do here is we essentially change the matrix a little bit and find all the very small and negative eigenvalues and set them to a very small positive number. And there's some good theoretical constructs for why this makes sense to do, but the nice thing is we can then just go ahead and invert that matrix and get the solution that we need, and then we run this through a big MCMC chain to be able to get posterior distributions of the locations of stars with respect to other stars. And this code is now up on GitHub as well; the paper on this we hope to hit the archives in the next week or so. So that's motion. Stars are also changing their brightness. Here's six stars from the Kepler Mission, launched several years ago, and this just looks like stars burbling along, and then when you start looking at what the y-axis is and the scale of the y-axis, you realize how unbelievable the photometric precision we have now is. This is one part, or a few parts, in a million, and this burbling is obviously of great interest to a number of groups. And the Kepler mission is poised to find the first extrasolar planets that live in the habitable zone that have a mass and radius similar to Earth, and they'll be doing that by looking at these extreme subtleties in this data.

My interest is really in things that go boom, and so not one part in a million, but things that get brighter by a factor of a million or a hundred thousand or a hundred million. In some sense those are easier to work with. One of the things that I've been working on recently is understanding how black holes gobble up gas and grow, and we have a number of different light curves, as they're called, brightness as a function of time, from a number of different surveys. And one of the questions we asked is, if we have a representative light curve over several years in a couple of different bandpasses, it turns out that every black hole grows and burps and belches in different ways, but they all do it in a way that's statistically similar. So there's this interesting question: in irregularly time-sampled and noisy data, how do you make a statement about whether some change or sets of changes are similar to another set of changes? And so we have a sample of something like a hundred thousand light curves of sources, and we want to find out which one of these things are like quasars or massive black holes that are putting on lots of light. And so we do this also with SciPy, some of the linear algebra programs, maximizing a posterior probability of a damped random walk, which is the model that people have found best describes the statistical behavior of these light curves. And then when we have new light curves, we essentially ask, is this data consistent with being drawn from this model? And this is a nice code; it works very fast, and it's all basically due to some of the great modules that we can find within SciPy.

And how good is it? Well, this is a bit of a busy plot, but if you focus on the red and you look at the x-axis, you wind up noticing that we have very good separation between the red, which are known quasars, and the blue, which are known stars in the same field. And this is the sort of separation that people have been dreaming about for a long time, because once you find these quasars, you then need to go follow them up with precious spectroscopic resources. So we were able to produce what we think is a 99% complete catalog of this large data set, and something which was very pure, so very little contamination from non-quasars. And instead of looking in the traditional color space, we're able to look in the time-domain space and pull these things out. So that was, I think, a very exciting result. The point about the time domain, in some sense, is that we have lots of things changing in the sky. It's not just quasars, it's not just twinkling stars. The extragalactic explosive systems are of great interest. Many of you probably know about Type 1a supernovae and Type 2p supernovae; those are the most common types of supernovae in the universe. But there's also theoretical constructs of what other types of stars might do when they blow up. There's a so-called pair-production supernova, which are very bright, and then there's the neutron-star neutron-star merger models, which may be related to gravitational waves. And you can see these things are different brightnesses, they happen on different time scales, and the goal here in time-domain astronomy is to be able to find these things and then follow them up very rapidly.

So we have, in some sense, a Rumsfeldian view of the universe. We've got the known knowns in the Type 1a supernovae and the Type 2p supernovae, the unknown unknowns, which I obviously can't put on this plot, and the known unknowns, and we're trying to look for all of these things at the same time. What's obviously incredibly preposterous is the idea that we would actually be communing with our data in the way that you read about in the popular press or watch on movies. We can't be listening to all of our data, we can't be observing all of our data as a person, we can't scale Jodie Foster to ten-to-the-six people; that just doesn't make sense at all. And the important thing here is that even if you're able to find objects that are going bump in the night, how do you know what they are, and how do you decide what to do with the resources that you have at your disposal for following those up and getting the real science out?

The exciting thing for us is that we are entering an era of great exploration in the time domain. There are a number of surveys that have come before, that are happening now, but the thing that's very exciting for many of us is the so-called Large Synoptic Survey Telescope, which, it was announced yesterday by the National Science Foundation that they've been given the green light to essentially get funding, starting with construction in fiscal year 2014. So this is really this massive discovery engine waiting to happen. Just to give you some indication of the types of numbers we're looking at, we're talking about light curves for order 800 million sources that will be updated every three days, something like a million supernovae a year, and something like 20 terabytes per night of raw imaging data that we're going to have to go through. And then there are other surveys at other wavebands happening: LOFAR and SKA, and the Gaia space astrometry mission, which will be the mission that finds those little astrometric wobbles to get distances. And so while we're very excited about this changing and dynamic universe, one of the things we're obviously worried about is how we're going to deal with all of this data.

So this is the data deluge challenge that we have in front of us, and let me pose it in this way, this question: how do we do discovery, follow-up and inference when the data rates and requisite time scales preclude human involvement? The obvious answer to this, of course, is that we have to automate what used to be that workflow that had scientists in that loop, but we need to get those scientists out of that loop and abstract them from their traditional roles. So, automating the scientific workflow. This is a bit of a cheeky slide, but I thought it'd be useful to show. We've got this barrier to automation on the y-axis, and how difficult is this to do on the x-axis, and we've got a chain here of observing and finding and discovery and classification and follow-up and then the scientific inference. And each one of these aspects require and demand their own specialized sets of codes, their own specialized sets of algorithms. There's, on the observing side, robotization and queue scheduling; finding, that is essentially taking the data and actually sticking new objects from raw data into databases; doing discovery, perhaps using computer vision and image recognition; classification, there we're using machine learning techniques; follow-up, there you have again robotic telescopes that are going to help you, and making sure that you're not spending all of your resources on things that aren't going to have scientific payout.

This should be, I think, fairly intuitive for all of you. One of the things I wanted to point out is this distinction between finding and discovery, the idea that you have data in your database and you have discovered that object or that data is preposterous, right, especially when your databases are large. Just because you know that that object exists doesn't mean that you've made that recognition that that object is interesting. And the classic example from astronomy is Galileo and Neptune. While Galileo was looking at his more nearby object, it was Jupiter's moons, he happened to jot down in his notebook a star that happened to be moving a little bit, and it turned out about 200 years later was discovered to be Neptune. So Galileo had in his notebook Neptune, but he's not credited as being the discoverer of Neptune, and I argue he would have been very famous if he had been.

So let me talk about automating discovery and the robotization of telescopes, and this really brings me to my personal story with Python. In 2002 I started my postdoc at Harvard and recognized that there weren't a lot of telescopes, in fact no telescopes out there, that were going to be able to follow up on new gamma-ray burst positions, essentially very large exploding stars, at infrared wavelengths. And there was a telescope that had been used by the so-called 2MASS survey for almost a decade that had been essentially mothballed in Arizona, and I asked for some funding to basically roboticize it, bring it back from the dead, modernize it, and get into all the different subsystems so that we could respond quickly to new alerts, but also so we could basically get the observer out of that loop and have a fully functioning system that could do this by itself. So in 2002 I asked my hacker friend at Los Alamos, Mark Galassi, which language should I use, and I was thinking he was going to say Tcl/Tk, which I'd used before, and he said use Python. And I said, what the hell is that? And he said, just try it out. So I decided this was going to be my first Python project and started ramping up from there. And we got first light in 2003, and we were up and running at the time that the new satellite from NASA called Swift was launched in 2004.

So what we did is we basically created a fully autonomous telescope out of what had been a 10-, 15-year-old project that had a lot of people in that loop. We had autonomous scheduling based on complex prioritization, detailed weather sensing, quick reactions to new events without any of these humans in the loop. So essentially it was listening to some TCP/IP socket waiting for new alerts to come through, and then it would just slew over and start taking data. We also had the telescope tweeting where it was looking so people could follow this, and we tried to bring robotic transient astronomy into the 21st century with this. So I'll do a little demo of the facility, which is still working on a nightly basis. I was going to try to open up for you and show you the sky, but it turns out it's pretty cloudy and rainy in Arizona right now, so I'm a little bit worried about opening up on a several-million-dollar facility just for shits and giggles. Sorry, we can bleep that out later. What you're looking at here is basically a video feed from inside the dome, and you can see this little crack here of light because it's daytime, obviously, in Arizona. And what I can do, if things work out — who, don't do this, exit on me — oh no, we're good — is try to turn the light on. Okay, so we can see inside, you can see inside the dome now, and there's the telescope, and there's the camera right there.

And what I can also do is I can start moving this telescope. First of all, let me show you the Python code that we're using. Might be a little bit hard to see here; I'll try to do a which version of Python. Am I using 2.3.3? Basically we got this whole thing working, and it was working, and then we said, hell no, we are not upgrading at all. So we're stuck back in the Dark Ages, but the point here is that Python has been working almost now for eight years and running this robotic telescope. So we can try to see if we can move this thing. It's hard for me to see. Okay, so we're going to send some low-level commands through a serial port. I'm going to tell it which elevation and azimuth to go to. Let's see if that actually works. Question mark there. Yeah, watch the screen. There's no sound, sorry; you can think about whatever music you have in your head. The thing I wanted to point out here is that we run this thing as a state machine, and so every few seconds this little GUI that we have shows us what the telescope is doing. And each one of these different cells here is run by a different daemonized Python code which is essentially writing little states, and we have a master daemon which is overlooking all of these and knows how to make changes. When it says, oh, it just went from nighttime to dusk or nighttime to dawn, let me now close down the telescope and go through these various sets of operations. And then we can see where that telescope is pointing at any time, we can get a last on what the telescope was doing in any one of these cells, and this thing has basically been unchanged for eight years or so. I better put the telescope back to where it should be instead of 70. What should I write, elevation of 90? Point back to zenith. Okay, so that didn't fail horribly.

So I guess the point here again is that essentially everything here has got Python under the hood, and it's not just the automation of the telescope and all of its different subsystems, it's the data reduction of all that data that's coming off. There's obviously a number of different results to highlight from here, but I wanted to show you one of my favorite movies of a gamma-ray burst afterglow which was observed one minute after it was detected by a satellite, which then sent down essentially an alert to the world, which got broadcast out from Goddard Space Flight Center, made its way up to the mountain, telescope slewed over, started taking data. And so we have a number of sub-one-minute responses, and you can figure out which object it is that's changing very rapidly. You can also see that if you get on with a large telescope later on, this source is fading out very quickly, you might be able to catch it, but even with a small telescope — and this is a 1.3-meter telescope, I think it is the largest robotic telescope still in the world — you can actually see these things when they're fantastically bright and do some very interesting science at early times. I have a grad student working on that right now for his PhD thesis.

Robotic telescopes are really all the rage, and one of the things here is that it's not just telescopes which know how to open and close themselves and respond to weather, but can then also start talking to each other and start making federated decisions about what it is to observe. It's an interconnected and very exciting ecosystem, and it's only getting bigger and bigger. Okay, I want to talk a bit now about automated discovery, and again this idea that just because you've got something in your database doesn't mean that you've actually found it or actually identified it as being interesting. And the way that we do discovery on transient or changing objects is essentially something called difference imaging, where you have a deep reference image and you have a new image, and you basically subtract the two. And doing the alignment is hard, and obviously with all the defects in the detectors and in the optics, this is an imperfect process. You see a pretty good subtraction on the top panel and a not-so-good subtraction on the bottom panel. These are actually two very well-known now supernovae found in the Palomar Transient Factory, both found by our codes. The top one I'll talk about in just a bit, but the thing I want to point out are these red boxes in this bottom panel on the far right with 11kx. Those are detections in a sense that they show up in the database, but they're obviously spurious because it was an imperfect subtraction.

So we have again a needle-in-the-haystack problem that we've been thinking about how to handle, and what we did is in 2008 we built a website to allow a bunch of experts to basically say whether they thought the subtraction was real or bogus, and then weigh in on a couple hundred or a couple thousand of these initial images from this survey. And we built this on the Google App Engine, and it made it very easy to get up and running and to essentially get lots of feedback from people on these various subtractions. So people would essentially slide this bar from bogus to real and things that are definitely real. So, to the trained eye, the object on the bottom is pretty real, the stuff in the middle is probably bogus, and the stuff at the top is almost certainly a bogus subtraction. But we have a thousand objects that we don't really care about, or spurious detections, to every one that we do. And when we start talking about the massive amount of data that we're getting just with this precursor survey called the Palomar Transient Factory to LSST, we obviously have a major bottleneck where we can't be throwing real human eyes at this problem all the time. So what the machine learning codes basically do is act as surrogates, and they try to say what they think this subtraction is, to mimic what experts would say.

And to put some numbers to this, in the Palomar Transient Factory we have about 1.5 million candidates a night. Only about a thousand of those are bona fide sources, and maybe only 300 of those are variable stars, and 10 of those are perhaps real interesting transients that we want to follow up. Most of the real things that we find are actually asteroids, and what you see is a distribution of pretty much what the machine said about 20 million candidates from the first month of data. And we're now reaching, I think we're basically at a billion candidates in that survey, and machines have essentially weighed in on all this to make sure that we don't have an asteroid and start throwing our resources at things that none of us in our survey care about. We built a parallelized minor planet checker called PIMP Checker, and it runs on an eight-core machine, and it's about 10 times faster than the Minor Planet Center's MPChecker, which is essentially what everyone has been using. This is a web service that we've exposed, again it's Django, and it's using an astronomy package called PyEphem, and it was written by a grad student about two months in the first summer when he arrived to start his PhD at Berkeley. And we're getting a lot of traction on this; there's a number of other groups around the world who are starting to ask questions of whether their source is actually an asteroid or not. So that allows us to weed out the bad sources and then get down to the really nitty-gritty.

And here's the visualization of all the thousand new variable stars and transients that were found by the Palomar Transient Factory in its first two years, and this was done with matplotlib. The red circle which is coming around is Andromeda, and we obviously spent a lot of time on top of Andromeda. We try to look at lots of nearby galaxies in the hope that there is something really interesting happening, and that really interesting thing happened, actually, wonderfully, at the end of last summer. An object called PTF 11kly was detected within the Pinwheel Galaxy, which is one of the most nearby beautiful spirals, and it's well observed by amateurs. And we think we caught this thing 11 hours after explosion, and we wound up getting a spectrum within 24 hours after the explosion occurred, which is unprecedented for any type of supernova but especially for a Type 1a supernova, and it turned out to be the nearest Type 1a supernova in more than three decades. And the important thing here is that it was promoted to the top of our candidate list by our machine-learned codes, which again have basically a lot of Python inside. This got a lot of press, of course; the Huffington Post quoted me as saying this was the supernova of a generation, which I like to show to my family and now I'm showing to you, because you're my family in some way.

But more importantly, obviously the press stuff is very interesting and very important for outreach, but more importantly, because it was the most nearby Type 1a supernova, we were able to get the best constraints by a factor of 100 on what makes these things. Type 1a supernovae are incredibly important for cosmology, they really touch all aspects of astrophysics, so having an object that's this close by allowed us to really rule out a number of so-called progenitor possibilities, and this is really the first time we've been able to do something like that. This is a plot that I made in matplotlib and used PySynphot from HST, and it took me a long time to make this plot, so I hope you like it. This showed up in Nature at the very end of last year. Okay, so just because you've discovered something doesn't mean that you actually know what to do with it or you understand what it is. And so this is the end of the line, at some level, of how do you do classification, how do you do human levels of cognition, where you used to show to somebody, hey, what is this object, and they say, ah, that looks like a Type 1a supernova to me. Now you need machines to be able to do that.

But there's considerable complications with time-series data that we deal with. First of all, it's noisy and irregularly sampled, we often have spurious data in our databases, so we have to be able to handle that, and sometimes, if we're thinking about follow-up, the main event, the thing, let's say in this case the thing that's getting brighter, hasn't even happened yet, so you might need to have theoretical constructs to help you decide where to spend your energies. And so, without going into details about feature-based machine learning, what we do on all of these light curves that we spend time with is we homogenize this very noisy data into a large space of essentially real number lines, and we derive metrics off of these light curves. And everything in blue are the typical metrics that you would think about using if you were given a time series of data and you wanted to pull out the essence of what was in that time series. And in red are these so-called context metrics, the idea that even in the absence of lots of time-series data, if you just tell me the location on the sky and I can then go and query the web and find out what's around there, I may be able to make a very interesting case for what that type of object is. Obviously if it's in the ecliptic plane, it has a high probability of being an asteroid or a minor planet; if it's in the galactic plane, it's some type of stellar activity; if it's near a galaxy, it's probably a supernova; if it's in the center of a galaxy, it's probably some quasar. So without telling you the nature of the time variability, just from context alone, I can tell you something about this.

And so we've been pushing with historical catalogs on trying to get very good automated machine learning classification. This is one of our more recent error matrices that shows us how well we're doing relative from the training set — sorry, the true class — to what we would say in the training set. And you can see we have a lot of power in the diagonal, and that's good, but you can see we also have a lot of off-diagonal power here. And I'm not going to go into obviously all the different of these whatever 25 classes of variable objects, but the thing that's interesting here is that if you draw a box around these big-picture classes — the pulsational variables, eruptive variables and multi-star variables — what you wind up noticing is that a lot of the off-axis power remains in these boxes. And so at some level, even though we haven't told the machine anything about the physics of these objects, it started recognizing that it can't really distinguish between these large classes. But what's exciting is that we're now getting the global classification errors that are at the 15% level over order two dozen classes of variable stars, and if you just look at those large classes, basically if you think about the classification as a hierarchy, we're getting down to five percent error rates, which is very exciting.

We've found a random forest is outperforming pretty much any of the other machine learning techniques that we've been able to spend time with. We started this project before scikit-learn was as mature as it is today, and so we've basically been using RPy2 for all of our work, but we're starting to think about scikit-learn. And what we did is we went back into an old catalog that had been published a number of years ago, and we started to try to classify all that old data, and we built up the first probabilistic classification catalog and made this basically available on the web. We built this also with Google App Engine, and we put these very large tables into Google Fusion Tables, so Google is basically doing all the hard work for us. And you can inspect each one of these different sources, look at the classification probability vector; we've got a social component, so maybe we'll get bought by Facebook. But that's it, you can check it out, it's called bigmacc.info, and this is again using Python for all the web work; it's got Python under the hood.

What's exciting is that when you start trying to build science out of these probabilistic catalogs, you have to do things in a different way. You can't just say, give me all the objects that are classified as this and go off and write a paper about that. You have to ask the question, how pure do you need your sample, and how efficient do you need that? So if you're doing demographic surveys, perhaps you want a very high-purity sample, but you don't really care about missing a whole bunch of other objects if you're getting that high purity. But if you're doing novelty discovery, so you're actually going to try to find some anomalous source that doesn't really fit the mold, you might want to actually go off and take spectra and spend a lot of your resources following up objects that turn out to be more mundane, but you're willing to trade that high efficiency for low purity. And so you get to dial in what your science is when you have these probabilistic catalogs. And we're also very excited about the fact that we were able to find some very nearby, very bright, very rare objects that have great interest for understanding Type 1a supernovae, in this catalog that had been published essentially 10 years ago. So it's a massive needle-in-the-haystack problem, but in some sense my view of doing probabilistic classification with machine learning is that the way you find needles is that you get very good at identifying hay.

So we're not just looking back in retrospect at old catalogs, we're also now running our machine learning codes essentially in real time on real data through the Palomar Transient Factory. And you can see that the robot essentially logged in and saved this candidate and discovered it, and then said it thought it was a supernova, and then you can see all the chatter that happened after that, and this indeed turned out to be a very interesting supernova. So it's not a toy idea that machine learning is interesting and useful in astronomy; it's actually happening, and we're very excited that it's happening on real functioning systems. I want to change gears here just for the last couple of slides and talk a bit about education. This came up in one of the mini-symposia questions last evening, about what's happening on various campuses and how are we getting people trained to be able to do science with Python. And we started essentially a seminar class called Python Computing for Physical Science, and we were basically teaching all the different aspects of how Python can be used in a scientist's daily workflow. We had guest lecturers speak about these various different topics, and it's obviously dynamic, as we wind up realizing that different code bases are potentially more and more interesting or more and more useful, more and more core for people's work, we wind up introducing them into these various sessions. And at the very end we asked people to basically write a large Python code base and check it in and deal with all the software carpentry that they need that is going to help them with their own research. And we wind up getting grad students and even upper-division undergrads to go through this full, very intensive course, and we've obviously gotten some good feedback from that.

Leading up to that, we also teach a three-day boot camp at Berkeley. In 2010 when we taught it for the first time we had 85 campers, and then just in January of this past year we had 135, and we're planning now for next month to get something like 200 just on the Berkeley campus. And this is free and open to anyone who's at Berkeley, and we make most of our materials also available on the web. The other thing — and I don't want to steal too much of Fernando's thunder, he'll be talking about the notebook in just a bit — is that we're also seeing the IPython notebook as incredibly important for teaching. It's a great didactic tool, and I'm sure all of you who have been using it know that already, so I'm preaching to the choir. But we're also starting to check our notebooks into git repos, and it's becoming part of our scientific workflow, and even in the paper-writing process as well. But one of the things that I don't think is mentioned all that often — I'm sorry I missed the high-performance computing session yesterday — is that IPython is in some sense a gateway to doing parallelism with Python. And obviously a large part of what we're talking about with parallelism here is embarrassingly parallel, but I think this is a very important aspect of what Python can do in a way that some of the competitors of Python are having a hard time dealing with.

This leads me to some of my parting thoughts. First, I think the drive to get native machine learning and statistical packages competitive with R is very crucial, and I think once we have the benchmarks that show that we're doing as well or outperforming some of the same exact modern algorithms as can be found in R, then there's going to be this phase transition where people who have been resistant are going to start coming over to that. And in the era of Big Data, as evidenced by the Large Synoptic Survey Telescope coming online in several years from now, I think open-source parallelism is a key selling point for Python. People may be very comfortable with MATLAB or IDL, and it might work very well for them on a single core or single machine, but then when they have to start doing lots of jobs, and then they start having to worry about large licenses to be able to do even embarrassingly parallel projects, I think that's going to be a major stumbling point, especially for those of us in academia where money is not as easy to come by these days. So I think we need to tout that as a community, that open-source parallelism is very important. And when we're starting to teach our students, teaching them in another language that doesn't have the free and open-source view of the world really doesn't make sense anymore.

And of course, Python scientific computing isn't just SciPy, it isn't just NumPy, it isn't just dealing with arrays and doing things efficiently. I hope you've seen here that Python is really part of the entire ecosystem, the entire workflow of a modern scientist. We're using PySerial, PyParallel, we're doing web interfaces, we're interfacing with NoSQL, we're interfacing with everything out there, and Python is this incredibly wonderful super glue. And I think, although I haven't heard it said here yet, I think it's poised to become the de facto engine for modern science, and I think we should all be very excited and very proud about that. So I come back to this slide with Vermeer, and maybe we can see this in a slightly different way than we had before. Maybe we want to look at Vermeer's astronomer as a very modern scientist who is now sitting, of course, during the day, even though he's an optical astronomer, but because he's looking not at a globe made by monks but at a view of the universe that was created by this automated workflow, and with Python inside. So with that, I'll say thank you, and happy to take a few questions.

Co-ordinating Facilities

Wed, 25 Apr 2012 00:00:00 +0000

Discussion lead on coordinating follow-up facilities across transient experiments, at the Kavli satellite meeting following the Royal Society transients discussion meeting.

Data-Mining and Machine Learning in the LSST Era

Tue, 24 Apr 2012 00:00:00 +0000

Invited discussion-meeting talk on machine learning as a surrogate for rapid human analysis at LSST data volumes: discovery algorithms, classification methodologies, and real-time transient identification.

Deck on Speaker Deck confirms event and date.

GRB Phenomenology & Neutrinos

Tue, 20 Mar 2012 00:00:00 +0000

Gamma-ray burst phenomenology and the prospects for coincident neutrino detection, for the IceCube collaboration.

Constraints on the Progenitor System of the Type Ia Supernova SN 2011fe/PTF11kly

Tue, 10 Jan 2012 00:00:00 +0000

The SN 2011fe progenitor constraints presented to the AAS winter meeting.

Relativistic Black Hole Tidal Disruption Events

Wed, 09 Nov 2011 00:00:00 +0000

Colloquium on relativistic tidal disruption events at STScI.

Relativistic Tidal Disruption Events

Tue, 04 Oct 2011 00:00:00 +0000

Colloquium on Swift J1644+57 and the discovery of relativistic tidal disruption events, months after the Science paper.

Constraints on the Progenitor System of the Type Ia Supernova SN 2011fe/PTF11kly

Tue, 20 Sep 2011 00:00:00 +0000

Ruling out red-giant and most main-sequence companions for the nearest modern Type Ia supernova, weeks after its discovery (Li, Bloom et al., Nature 2011).

Machine Learning and Classification in the Synoptic Survey Era

Fri, 01 Jul 2011 00:00:00 +0000

ML-driven discovery and classification of transients and variable stars in the era of synoptic surveys (PTF to LSST): real/bogus discrimination, probabilistic variable-star catalogs, and automating the discovery loop.

51-slide deck on SlideShare; exact venue/date uncertain (estimated from related 2011-2012 work).

In a Flash of Gamma-Rays, a Star Is Gone

Fri, 17 Jun 2011 00:00:00 +0000

The day after publication of the Science paper on Swift J1644+57, Ira Flatow interviews Bloom (with Lawrence Krauss) on the gamma-ray outburst interpreted as a supermassive black hole shredding and swallowing a star 3.8 billion light-years away.

12-minute segment; NPR transcript available.

Transcript

Bloom's segments (condensed): Bloom explains how observations of Swift J1644+57 — its energy, duration, spectrum, and location near the center of a distant galaxy — led to the interpretation of a black hole swallowing a wayward star and beaming a relativistic jet at Earth. He discusses black holes as efficient converters of mass to light, radio follow-up to watch the jet emerge, the ~once-per-million-years-per-galaxy rate of tidal disruptions, the special jet-aligned viewing geometry, and plans to search historical high-energy archives for similar unrecognized events.

Black Hole Swallows a Star (Swift J1644+57)

Wed, 15 Jun 2011 00:00:00 +0000

BBC Newshour interview on the discovery of a possible relativistic jetted outburst from a massive black hole fed by a tidally disrupted star.

Linked as MP3 from his old Berkeley homepage; original audio link now offline.

Real-Time Classification of Astronomical Events

Wed, 02 Mar 2011 00:00:00 +0000

Machine-learning pipelines for classifying astronomical events in real time, for the computational science community.

SASIR: A Wide-Field Synoptic Survey for this Decade

Mon, 29 Nov 2010 00:00:00 +0000

The Synoptic All-Sky Infrared Survey concept, presented at a post-decadal-survey community town hall (alongside an LSST status report from Tony Tyson).

High-Redshift GRBs with Lobster

Tue, 12 Oct 2010 00:00:00 +0000

Using the proposed Lobster wide-field X-ray transient monitor to find the highest-redshift gamma-ray bursts.

The Transient Universe

Fri, 01 Oct 2010 00:00:00 +0000

The dynamic sky - explosions, flares, and ephemeral events - for the SACNAS community of Chicano/Hispanic and Native American scientists in training.

EXIST: Probing the Epoch of Reionization with Gamma-Ray Bursts

Wed, 21 Jul 2010 00:00:00 +0000

The proposed EXIST hard X-ray survey mission as a probe of the reionization epoch through high-redshift gamma-ray bursts, as chair of the GRB-EXIST working group.

Finding Utility in the Diverse Origins of Gamma-Ray Bursts

Mon, 04 Jan 2010 00:00:00 +0000

The Pierce Prize lecture: what the diverse progenitors of gamma-ray bursts - collapsars, mergers, and oddballs - teach us, and how bursts became tools for cosmology.

A Redress of Short Bursts

Wed, 18 Nov 2009 00:00:00 +0000

The revised observational picture of short gamma-ray bursts five years into the Swift mission, with the Berkeley GRB group.

Real-Time Classification in the Petascale Epoch: Ramping Up with PTF

Thu, 25 Jun 2009 00:00:00 +0000

Machine-learned real-time classification of Palomar Transient Factory events as the ramp toward petascale synoptic surveys.

The Synoptic Infrared Imaging Survey

Mon, 27 Apr 2009 00:00:00 +0000

The SASIR synoptic infrared survey concept, presented at the second Hot-wiring the Transient Universe workshop.

The Future of Gamma-Ray Bursts

Sat, 18 Apr 2009 00:00:00 +0000

Where gamma-ray burst science is headed, at the symposium honoring Vahe Petrosian.

GRBs in a Cosmological Context

Thu, 25 Sep 2008 00:00:00 +0000

Colloquium on using gamma-ray bursts as probes of the distant universe.

SASIR: The Synoptic All-Sky Infrared Imaging Survey Concept

Tue, 23 Sep 2008 00:00:00 +0000

The concept for a synoptic all-sky infrared imaging survey (SASIR), a proposed 6.5-m IR survey telescope.

Real-time Astronomical Time-series Classification and Broadcast Pipeline

Fri, 15 Aug 2008 00:00:00 +0000

A Python-based pipeline connecting synoptic-survey telescope data streams to real-time machine classification and event broadcast, from the Berkeley Transients Classification Pipeline project (with Dan Starr and John Brewer).

Proceedings paper; talk co-presented with first author Dan Starr.

GRBs as Cosmological Probes

Thu, 15 May 2008 00:00:00 +0000

Gamma-ray bursts as probes of reionization and the high-redshift universe, at the Harvard Sackler 21-cm cosmology conference.

Building a Classification Engine for the Palomar Transient Factory

Tue, 05 Jun 2007 00:00:00 +0000

Session talk on the machine-learning classification engine behind Palomar Transient Factory discoveries, in the event-classification session alongside LSST and Caltech colleagues.

Agenda archived on the IVOA TWiki.

Gamma-Ray Bursts: The Long and the Short of Them

Tue, 27 Mar 2007 00:00:00 +0000

Colloquium on the two burst populations - collapsars and mergers - and the evidence separating them.

The Host Galaxies of Short Gamma-Ray Bursts

Fri, 06 Oct 2006 00:00:00 +0000

Invited review of short-burst host galaxies at the HEAD meeting, cementing the demographic case for compact-object mergers.

The Hosts of (Short) Gamma-Ray Bursts

Fri, 09 Jun 2006 00:00:00 +0000

Solicited talk on the galaxy hosts and environments of short gamma-ray bursts and the case for an old-population progenitor.

Gamma-Ray Bursts and Their Hosts

Thu, 01 Dec 2005 00:00:00 +0000

Invited review of gamma-ray burst host galaxies and large-scale environments as progenitor diagnostics, in the first year of Swift.

Shedding Light on Short Gamma-Ray Burst Progenitors

Fri, 22 Jul 2005 00:00:00 +0000

The first afterglow localizations of short gamma-ray bursts and what they reveal about compact-merger progenitors, weeks after the breakthrough events.

Gamma Ray Bursters

Fri, 03 Jun 2005 00:00:00 +0000

Ira Flatow interviews Bloom about the origin of short gamma-ray bursts, following the first localization of a short GRB (GRB 050509b): the evidence that short bursts come from merging neutron stars rather than the deaths of massive stars.

NPR transcript available.

PAIRITEL Science Overview and Commissioning Update

Tue, 19 Apr 2005 00:00:00 +0000

Science goals and commissioning status of PAIRITEL, the robotic 1.3-m infrared telescope for transient follow-up.

Implementing a Real-Time VOEvent Messaging Network

Wed, 13 Apr 2005 00:00:00 +0000

Design and implementation of a real-time astronomical event messaging network, at the workshop that launched the VOEvent standard.

New Perspectives on Gamma-Ray Bursts

Wed, 06 Apr 2005 00:00:00 +0000

Colloquium on gamma-ray bursts in the first months of the Swift era.

GRB-Supernova Connection: Evidence and Implications

Wed, 15 Dec 2004 00:00:00 +0000

The observational case connecting long gamma-ray bursts to supernovae and its implications for progenitors.

Optical/IR Observations of GRB & XRF Transients

Tue, 25 Nov 2003 00:00:00 +0000

Colloquium on optical and infrared observations of gamma-ray burst and X-ray flash transients.

Cosmological Gamma-Ray Bursts: Perspectives and Prospects

Thu, 06 Nov 2003 00:00:00 +0000

Colloquium on the state of cosmological gamma-ray burst studies, delivered shortly before joining the Berkeley faculty.

Optical/Infrared Observations of Gamma-Ray Bursts and X-Ray Flash Transients

Tue, 28 Oct 2003 00:00:00 +0000

Colloquium on optical/IR follow-up of gamma-ray bursts and the newly identified X-ray flashes.

Energetics and the GRB Hubble Diagram

Thu, 11 Sep 2003 00:00:00 +0000

Standardizing gamma-ray burst energies and the promises and limitations of a GRB Hubble diagram for cosmology.

The Case for Late Optical Bumps as Associated Supernovae

Wed, 18 Sep 2002 00:00:00 +0000

Solicited case that late-time optical bumps in GRB afterglows are underlying supernovae - the photometric argument for the massive-star origin of long bursts.

Optical/IR Afterglow Observations

Mon, 20 May 2002 00:00:00 +0000

Review of optical and infrared afterglow observations at the Harvard Sackler conference on gamma-ray bursts.

The Progenitors of Gamma-Ray Bursts

Thu, 28 Jun 2001 00:00:00 +0000

Invited review of gamma-ray burst progenitor evidence: afterglow offsets, host galaxies, and supernova bumps.

Toward the Progenitors of Gamma-Ray Bursts

Fri, 18 May 2001 00:00:00 +0000

Progenitor evidence for gamma-ray bursts - offsets, hosts, and the emerging supernova connection - presented at the IAS.

Toward the Progenitors of Gamma-Ray Bursts

Thu, 03 May 2001 00:00:00 +0000

Progenitor evidence for long and short gamma-ray bursts from afterglow positions, host demographics, and supernova signatures.

The Observed Offset Distribution of GRBs About Their Hosts

Sun, 15 Oct 2000 00:00:00 +0000

The measured offsets of gamma-ray bursts from their host galaxies as a direct test of progenitor models - massive stars versus compact-object mergers.

GRB Host Galaxies: Hints for the Models

Tue, 04 Jul 2000 00:00:00 +0000

What the properties of GRB host galaxies imply for progenitor models, at the relativistic-astrophysics meeting.

Emerging Trends in Gamma-Ray Bursts

Wed, 23 Feb 2000 00:00:00 +0000

Survey of the rapidly moving observational GRB field for the IPAC astronomy community.

GRBs & SNe: Investigation of GRB 980519 and GRB 980329

Wed, 20 Oct 1999 00:00:00 +0000

Searching for supernova signatures underlying the afterglows of GRB 980519 and GRB 980329, early evidence bearing on the burst-supernova connection.

The Discovery and Broad-Band Follow-Up of the Transient Afterglow of GRB 980703

Mon, 02 Nov 1998 00:00:00 +0000

The discovery and multi-wavelength follow-up of GRB 980703's afterglow and its dusty star-forming host galaxy.

The Spatial Distribution of Coalescing Neutron Star Binaries: Implications for Gamma-Ray Bursts

Sat, 10 Jan 1998 00:00:00 +0000

Population-synthesis predictions for where merging neutron-star binaries occur relative to their host galaxies, and what burst offsets imply for the merger progenitor model.

The Host to GRB 970508: A High-Redshift Dwarf Galaxy?

Mon, 15 Sep 1997 00:00:00 +0000

Evidence that the host of GRB 970508 - the first burst with a measured redshift - is a distant dwarf galaxy, tying bursts to faint star-forming hosts.

Extragalactic Hosts

Tue, 27 May 1997 00:00:00 +0000

On the extragalactic host galaxies of gamma-ray bursts, in the first post-BeppoSAX year as afterglows and hosts were first being identified.

The Corrected Log N–Log Fluence Distribution of Cosmological Gamma-Ray Bursts

Sun, 15 Oct 1995 00:00:00 +0000

With Fenimore and in ‘t Zand: correcting the observed log N-log fluence distribution of BATSE bursts for detection biases, an early constraint on the cosmological burst population.