Panel on foundation models and AI agents for physics, with Mariel Pettee and Ioana Ciuca.
“The only real test set is data that hasn't been observed yet. The only real test of the efficacy of foundation models is when they're applied in production to data that wasn't in the can at the time those models were created.” – Joshua Bloom
“The idea of using a very large foundation model that's good at looking at images from telescopes and then somehow plugging that into some little downstream task and then actuating a billion-dollar facility is kind of preposterous. It's even more preposterous when you realize we only have access to one CPU and a tiny amount of RAM at runtime.” – Joshua Bloom
“Now we're in this mode where it's not just you and your grad students and your collaborators putting your names on papers; it's you and your collaborators and your students and your agents putting names on papers. And so the trust still ultimately comes back to us.” – Joshua Bloom
“It is hard to see how any of us really go back to the way that we worked before LLMs came into our lives… It's changing and upending how we live and how we work professionally.” – Joshua Bloom
MODERATOR: So our first panelist will be Josh Bloom. Josh is a professor of astronomy at UC Berkeley and he's a highly decorated pioneer at the intersection of AI and the physical sciences, and I heard that with his talk he's going to give us a very unique perspective on foundational models. Please welcome. [applause]
BLOOM: Hi all. So hopefully this will be both unique and somewhat spicy. That's what I was told to do: inject some spice into the room. All right. So we've heard a lot about foundation models today and the construction of them, the utility of them. And I had a bunch of rants. I have rants about other things as well; you can ask me later. But this is mostly framed in the form of questions for people in this room. We know about scaling is all you need. But when we're building foundation models, adding more modes — so you have spectra and images and other metadata, more training data, more diverse data sets — we have to ask ourselves, are we guaranteeing that we're getting better and better results on all the downstream tasks? And I think oftentimes we don't ask ourselves what actually makes a better foundation model. There was a question that was asked earlier in the day today about how do you know, when you're building a foundation model, that the downstream tasks that you're building them for have actually improved. And part of that comes deep inside from how we actually do representation learning. And if we're doing this in a self-supervised way, which you pretty much have to if you're using lots of data that doesn't have a huge amount of labels associated with it. This is an example from a multimodal foundation model that I built called Astro M3, where we were trying to build a foundation model around time series data, so photometry and spectra and also metadata. And we wound up realizing that if we're doing this in a CLIP way, contrastive way, where you're trying to align embedding spaces across these different modes, oftentimes you can get modal degeneracy where the same light curve, as you see here, actually gives rise to two very different spectra. And that's because you could get a light curve from two different sets of stars that could give you the same light curve but actually are very different types of stars. So as we're building these models, the thing that we're baking in are assumptions about the underlying physics and the connections across these different modes. Oftentimes when we're building these models, we're building on very high quality, high signal-to-noise training data, but we're applying it downstream in application to lower quality, different types of data from different instrumental setups. And inherently, at least in astronomy, we're probing different populations, right? If I build something on nearby galaxies and I try to apply it to galaxies that are farther away, there's different distributions of those galaxies in all of their properties, in their numbers. And those sorts of biases are not generally contemplated inside of foundation models. And so, the thing that people have for a while heard me say, the only real test set is data that hasn't been observed yet. The only real test of the efficacy of foundation models is when they're applied in production to data that wasn't in the can at the time those models were created, and preferably by groups that didn't themselves create those models, which is kind of the state-of-the-art of where we are right now.
And so when groups are building foundation models applying it for themselves, they're hoping other people will use it. And the question is, can we just throw that over the fence and hope those are used appropriately? If those models are getting bigger and bigger, there's questions about is it big enough, and do we stop caring about accuracy and do we start caring about other things like inference speed and actual cost and who gets to host these models and who's going to pay for these models in production. These are questions that are not often asked by many of us who are working with foundation models. And importantly, how do we convey those costs in a rigorous way, or at least in a non-implicit way? And the last thing I was just going to say is I have a deep concern about the role of foundation models in doing what I guess I would call citation capture. We rely in astronomy, and through all of academia, on citations, and if our work is getting subsumed into foundation models and then used for downstream tasks, oftentimes those downstream tasks are not citing the original data from whence those models came. And so we have a serious problem there. And we're also not really working with raw data. We're working with corrupted data and trying to fix that data. And that corruption is exogenous to the physics of the things that we care about. So that's it for now. You'll hear more of my spice later on. [applause] MODERATOR: Thank you very much, Josh. The next panelist is going to be Mariel Pettee. And Mariel is an assistant professor of physics, and she just started and leads a new group. They're designing new AI methods to enable discoveries across physics and astrophysics. [applause]
PETTEE: Hi, great to be here. Dennis, I was counting on 10 minutes. I hope that's okay. I will move efficiently.
MODERATOR: Sure.
PETTEE: So it's great to speak with you all today. I'll talk briefly about foundation models and then I'll transition over to agents, since this panel is covering both topics. So first I'll talk very briefly about foundation models in physics, which is a field I've worked in for the past few years. So I think it's a fair question: will foundation models revolutionize physics? And I think what I can say today is that it's clear that foundation models, especially in industry, are catalyzing incredible progress. But I'd say that their potential to yield really surprising discoveries in physics, we have not fully understood today. However, I still think that the pursuit of foundation models, as Josh just alluded to, encourages us to explore these questions of multimodal representations and self-supervised learning strategies for physics. And I do think that what we learn in that pursuit is going to be critical for the future of physics analyses. So I've just flashed a handful of examples of these types of projects that, at least with AstroCLIP and AION, have integrated multiple modalities of astrophysics data — either two modalities or three modalities or, on the order of tens or 30 modalities — and we're starting to see more efforts of this in other fields like high energy particle physics too, where I just think that there's ripe potential to think of our detectors as highly multimodal sources of data. So in the process of this, we're essentially distilling a lot of our data into a different form. So just like maybe ChatGPT is sometimes analogized as a blurry JPEG of the web, answering this question will boil down to understanding what makes our data special. How is fundamental physics data distinct from, say, industry standard machine learning data or even internet data? And how do we wrangle that data? How do we turn it into a meaningful representation? And what does meaningful even mean?
So in the process of this, I think there are some interesting questions around methodology. Which self-supervised learning method do we choose? And then also, what goes into a good pre-training data corpus? There are all kinds of different properties that we might imagine emphasizing in curating a data set for a foundation model. And maybe that includes geometric properties, statistical properties, semantics, shared information, complexity, or also some sense of data set diversity where there's an interplay between the size of the model as well as the heterogeneity of the data sets that you're including. So there are a lot of open questions here, and I don't think scale alone will erase the potential of some of these questions. And I'll also say that I continue to think that one of the values of foundation models in physics, and one of the impacts of it, could be how it forces us to maybe rethink the structures of our social alignments in the sciences. We tend to organize socially around these traditional academic disciplines that were decided decades ago. And I think it's certainly possible that we can recall that particle physicists helped bring us into this era of big science, from a time when science was done quite differently with much smaller resources and collaborations. And I think we can, as fundamental physicists, be potentially part of a rethinking of our paradigms of how we do physics. And this is not a new problem. And I think it's been argued pretty well that AI for science can be seen as a social problem. It's a question of data access and fragmentation. It's a question of resources and allocation of resources, and it's also just a question of how do we communicate across disciplinary boundaries.
Okay, so switching over to agents and physics. This has been such a remarkable year for changing the way that I think about my own workflow in physics for sure. So I think agents are incredible, and I also think that there's potential to think about agents with some more imaginative power as we deploy them in our work. So first off, I love to think about not only, once we have this interesting tool, how are we using it, but rather can we take a step back and just remind ourselves, what are agents even for? Why incorporate new technologies into our workflows? I like this quote from an essay about agency, but I think it applies just as well to the concept of agents in physics. Without emphasizing its purpose, maybe it allows us to ignore some harder questions about what does a good life look like? What does good research look like? What does impactful research look like? So I really encourage us to think about using agents and some of these new tools to really raise the bar on what scientific work can look like in ourselves and in others, to bolster a sense of trust and rigor in our field. And I think that historically we've enjoyed a lot of respect and trust from the general public, but this is changing. And moreover, forget the public — just among each other, I think it's hard enough to find trust in other people's work, especially from people who we don't know personally, and this is being exacerbated by LLMs. But the paper mill and the concept of an onslaught of new work was not caused by LLMs. It's just being exacerbated by LLMs. So I worry sometimes that this is going to encourage us to really just fall back on personal connections and make access to doing research and science even harder if you're not already well connected.
So how do we push back against this? We've seen some evidence of this today thinking about reinforcing signals of high-quality work. And I tend to believe that these things should be opt-in as opposed to enforced. These are things that authors can choose to do to add extra validation onto their work. So for instance, agentic reproducibility of results and assessments of whether the code reflects what's actually stated in the method sections of the papers, for instance. And I think we've been having some super interesting conversations about this on the editorial board of PRX Intelligence, which was mentioned yesterday by Anatol, thinking about how do we talk about LLMs in the review process, right? And I think that LLM use is not just a checkbox to indicate. I think we should encourage people to really describe in depth how LLM use is a part of their process, just like we would expect for other parts of the scientific method. Finally, LLMs can suggest improvements clearly, but I don't believe that they can be held responsible for decision-making. And there's a quote that sort of summarizes this: LLMs can make decisions, but only people can choose. So I really think that expert, insightful peer review and curation is still going to be a critical part of our scientific process here.
Now I want to say a couple things on the idea of craftsmanship, skill atrophy and joy. So cognitive offloading leads to skill atrophy. I think this is clear, and yes, this is still true even if you have a PhD. And I also emphasize that this is not necessarily a bad thing. I don't think anyone really misses the days before having maps on your phone or memorizing phone numbers. So cognitive offloading is something that can really benefit us in our work. But personally — I don't know about you — my happiness is also tied in part to a sense of craftsmanship and mastery in my work. So very quickly, as we saw yesterday, we get into this territory of why do we do what we do? What is your craft? And I think that as scientists we should have some agency to choose what parts of the process we really take pride in, what our sense of craftsmanship is. But I also think that there's some external metrics that can guide this around the sense of high complexity versus low complexity tasks. It's evident to me that coding has more structure than free form writing, for instance. And so I see free form writing as a higher complexity task than coding in some senses. And so that informs a little bit of my choice of where I put LLMs into my workflow.
And I just encountered this idea recently, just this past week, of adding an egg. So it comes from this maybe apocryphal anecdote of Betty Crocker in the 1950s, where we were first seeing evidence of pre-processed foods in the supermarket, and you could buy a cake mix that had everything included. Just add water and you make a cake. And they found that sales were really not doing so well when this was first introduced. And a lot of this was attributed to our psychology, a sense that just mixing water with the mix and the cake is ready — it kind of depressed people, or it made them think, what is my role in this process? And so, as the story goes, marketers on the team decided we should actually encourage consumers to add an egg. And that's supposedly still why that's part of cake mixes, because it makes you feel like you've really contributed to the process as you've baked a cake. So I've been thinking about this recently in terms of how do I — what egg am I adding, what is that in my own workflow? And I've seen some — I'll just flash some quick quotes that summarize this. This is from someone who I don't know on social media who was talking really enthusiastically about agents in their workflow. And then they followed up with this statement saying, actually, my goal with agents is to keep cognitive traction, and if I lose that, my internal sense of aliveness with the project fades too. So paradoxically, agents materially forwarding the work and removing myself as the bottleneck chokes the source of project emotional investment. So I resonated with this, and I imagine some of you might too. And from another perspective, choreographer Martha Graham talked about how learning is achieved through practice. It is the performance of a dedicated, precise set of acts, physical or intellectual, from which comes shape of achievement, a sense of one's being, a satisfaction of spirit. So this is elusive. It's hard to find in any practice. But it's something I'm still working through in this field, of how agents intersect with this desire to feel mastery of my craft. Okay, final thoughts. I think we should make agents weirder. I think agents and agentic ecosystems are often — and I understand the temptation — we anthropomorphize them and we think of them as maybe one-on-one replacements or analogues to individual scientists or even to scientific labs, and they are sometimes pitched as products. But I think that we could instead think about embracing their wildness and weirdness and see them as more reflections or distortions of our own thought process. So sometimes LLMs are described as mirrors, right? They give you what you give them. If you give them higher quality material, they'll give that back. I think really agents are kind of more like disco balls that reflect some parts of you and then also lots of other parts that maybe you didn't say explicitly, as well as lots of other sources. So I think potentially, imagine this is a disco ball collabial manifold. I think if we really thought of these as quite different from our own intelligence and not trying to directly analogize them, there are some cool possibilities here. And this is a slide with some advice on how to do this in practice. This is a slide I made, I think, eight years ago. And this was in my other life, thinking of integrating AI into my art practice as a creative, and in the course of this early development many years ago I thought of interacting with these models that I had developed for my own creativity along these different axes, thinking about who has the creative power, who is making the creative decisions in my practice, versus do I think of the model as my friend or as either my enemy or a critic, or at least someone distant from me. And there's a lot of possibilities here in reimagining the relationship between you as the scientist and an agent or a set of agents.
And I'll conclude with just a couple quick cautionary tales. As I mentioned, I've been involved in the AI and creative space in the early days before ChatGPT. And so I was interviewed recently as part of this kind of review paper reflecting on, how do people who were starting out in this field think about the state of AI and art now? And this might be familiar to a lot of us, right? The introduction sort of describes how a small group of people began to experiment with artificial neural networks. They created conferences and exhibitions, and they talked about machine learning and how it was shaping their field. But it really concludes with this current state of AI and art where a lot of these early practitioners don't really recognize the field anymore. They feel that their legitimacy has been affected, and there's a big mismatch between their creative process and the way that the technology has gone. And I think that potentially the way that we think about agents in our work might not only look like this, it might also look, for instance, like the music industry. So there's this quote that I've heard in the context of tech: whatever happens to musicians happens to everybody. And I worry that this might be increasingly true, where a lot of musicians, with some dark humor, describe their practice as t-shirt salesmen and less of practicing musicians. They go on tour and they sell t-shirts and that's how they support themselves. So this is kind of a sad story, but I see it as a real state of the music industry now, that we kind of have titans at the top, and then we have a number of people who make a living from their music practice — and I would say that that's on the order of a thousand t-shirt salespeople, as they might jokingly call themselves — and then there's kind of everybody else, creatives with side practices who are teachers. Maybe they're lucky enough to be affiliated with the university. They might get small grants. They might have side jobs. They might be independently wealthy. Sadly, this is kind of how music works nowadays. And I would really want us to avoid a future like this in the sciences, where only a small few get to practice the work that we do.
Okay, so this is my last slide. It's a quote that I love from an artist who is talking about the integration of technology into her work, and she encourages other artists to think of working with technology like riding a bull, where you want to have some sense of control but you also want to be surprised in the process, and I think that there's something for us to learn from that. So she emphasizes we really need to move beyond illusions of control. We need to build instruments that transform us and allow us to hear different structures and sounds from unknown spaces. Inefficient and ineffective instruments which we learn to ride like wild bulls on a limitless plane. Thanks. [applause] MODERATOR: Thank you very much, Mariel. And for our last speaker in this panel, I can introduce Ioana Ciucă. Ioana is a research scientist here at Stanford University and she describes herself as always human, astronomer and world traveler, and she told me before I should introduce her as well as a hacker of the universe.
CIUCĂ: Thank you so much. [applause] So hi everyone, my name is Ioana Ciucă but everyone calls me Joe, and I am a researcher at the Center for Decoding the Universe at Stanford. And today I want to share with you very briefly a reflection on what happens when AI meets science, and what are some of the considerations and the questions we have to take into account when these artificial entities enter our individual private research lives and also the lives of our scientific teams. So to start with, I'm going to show this big sentence here. So this is the current narrative, especially here in the Bay Area, that AI will accelerate science, and I'm just a bit curious how many folks in the audience today actually agree with this. Please raise your hand just to give us some sense. Thank you so much. How many folks in the audience today don't agree with this? Yes, we have a few strong, courageous folks here. And how many folks in the audience today think that we need a little bit more information before we can make this assessment? Please raise your hand. So I am also in that third bucket, and I believe that in order for me to make an assessment on this statement, we first need to spend a bit of time meeting each other and asking each other hard questions to define, in the first place, what we mean by accelerating science. And once we get there, through many meetings and through many debates and being skeptical and maybe shouting at each other with love, once we get to that definition of what it means for science, at that point you also have to become aware of whose voices and whose values are reflected in that definition. But I think it's going to take a little bit of time until we get there, and I think that's okay.
And until we get there, we have to ask a question: how can AI actually help us do the science that we do today? So where do we start when we have to answer this question? So here at the Center for Decoding the Universe, led by a brilliant junior researcher called Christine, we did this project called ReplicationBench, where we pretty much talked to a lot of our human expert astronomers here, and we asked them to take tasks from the papers that they knew deeply and curate a set of 120 research tasks that were either very simple tasks — like this one, that would take minutes, pretty much just look at a very big data set and select a sample — a bit more convoluted tasks that take hours, where you have to fit a statistical model to your data, and tasks that were really really hard, where you have to implement some really big algorithm, like a force solver, an N-body solver like this one, and this would take an expert a month of some sort. Oh yeah, where's the mic? Oh, this mic — yes, sorry. I survived a tornado yesterday, so that's why I'm shouting. I'm very excited. So yes, just want to put out there, these tasks, some of them will take many weeks and months. So once we actually curated these tasks, we then asked AI agents to try to solve them. And one thing to mention is that each of these tasks had a verifiable, concrete answer at the end. We have the number of stars. We have the number of peaks. We know exactly the accuracy that we want to achieve with this solver. And we found that in October 2025 the models at the time were actually getting around 20%. But the more recent models are showing improvement. So I feel, with time, that this replication task will be solved by AI agents. But more importantly, what we found in that process were the failure modes of these agents. And why are failure modes interesting? Because then we can share with the community those failure modes, and the folks who are building these systems, which could be open-source systems — and I actually really encourage folks to think about that — can then improve them. And then maybe, just maybe, we can then use these methods and integrate them in the research workflows that we have, for example, in cosmology.
These are really advanced tasks that I believe AI can help us with. And in many conversations with Marcelo here, we're thinking all the time, how can AI actually help us develop these pipelines or change some components of the pipelines in a meaningful way, and not just in cosmology but across astrophysics? I think one thing that doesn't get talked about enough is that we can use these methods to develop methodology, and we can draw insights from different fields and make really good methodology. Okay. So here we're still in the domain where we can verify that the models are doing the right thing. And once we get that signal that they're not doing too badly, we can only then think of releasing these AI systems at discovery scale, at the discovery scale of Rubin. But what I've mentioned so far is only on the level of the AI capability. So we need to do something more than just creating very capable systems. We have to create systems that are deeply collaborative and that empower the teams that collaborate with these systems. So on the capability scale, before we release these systems at the Rubin scale, we need capability. We need these systems to create reproducible analysis. We need them to learn from experience and have a sense of memory and persistence. But not only that; on the human side, we need to make sure that the human is engaging with these systems actively, if the system can ask the human questions and so on. And finally, we have to create systems that enable team science. Why should we bring an artificial system inside the research team if that research team feels disempowered, that they're not doing really cool things, not learning and so on?
So just on my final slide before I go there, where do we go from here? So far I told you a little bit about the requirements, the design requirements that I think these systems should have before we release them at scale for discovery purposes. But what are some of our own principles that we should follow as we integrate the systems into our research life? So I think we should really care deeply about our curiosity. We should care deeply that once we are curious about something and we point a telescope somewhere and we find some really cool thing out there and we understand what's happening, the next thing that we can do is, one, we want to share that with our community. So I think we should care deeply about sharing our understanding with the folks around us, and I think we should care deeply about responsibility. So that means training a junior researcher into doing science is valuable in and of itself, and we should keep track of that. And more concretely — and I think this is a conversation I've had with Dennis a little bit — concretely, what can we do to really see how these systems work with us while following these big principles ahead? I think we should just explore. We should get in rooms. We should make mistakes. We should see how they respond to us. And at the end of the day, we should imagine — science has been done in many ways over time, and now we have an opportunity to really think deeply, how can the research team of the future look, and what is the role that AI should play inside our research team? Thank you very much. [applause] MODERATOR: So let's go to the first question. We have a lot of people working on foundation models currently in many different fields and disciplines and so on. I would like to ask you, do you think, looking into the future, we will have one foundation model for all of the physical sciences or for all of science, and will it come from an academia person or from an industry person? Maybe, Josh, you can try to answer first.
BLOOM: No. [laughter] Thank you. Hello. Yeah, that works. Yes, I will answer the question. The answer is no to the question. People that are practitioners now have been using bespoke models on bespoke training sets for very specialized problems, and I think that persists. We're working on a problem right now with Rubin Observatory to do the active optics system using AI, and the idea of using a very large foundation model that's good at looking at images from telescopes and then somehow plugging that into some little downstream task and then actuating a billion-dollar facility is kind of preposterous. It's even more preposterous when you realize we only have access to one CPU and a tiny amount of RAM at runtime. And so there's a very long tail of problems that foundation models can't and probably shouldn't touch. There's obviously a large class of problems where it probably will be extremely helpful. I guess my question to people in the audience — and I'd love to hear how people think about this — is the difference, as we try to make an analogy with foundation models in language, is that language seems to have this really nice, almost bijective mapping back to how we think and how we construct thought and how we think about the world. But I don't think it's the same thing when you look at images of galaxies, right? In every pixel, there's a lot of physics in there. And if you knew all the physics, you could generate the pixel and the flux from that pixel in all wavelengths. But going the other direction and trying to get back to that physics is not clear. So I'm not sure that even the data that we're using for foundation models is the right language of physics. Now that may be true in astronomy. It may not be true, and in fact it may actually be really exciting, in particle physics. It may be exciting and actually true in chemistry, where you have a one-to-one mapping between a SMILES representation and the actual state of the atom. But I think a lot of the data that we use now in foundation models, in astronomy at least, are not the level of the language that we need to be able to solve all problems.
MODERATOR: I saw a lot of head nodding. Do you both agree?
PETTEE: Yeah, I'd say I think it's clear that physics data is different from language in ways that we don't fully understand yet. I think some of that might be related to how efficiently language embeds a world model, or a sense of the world that it lives in. And physics data does that differently. I think my answer to your question is I hope that there's competition on all sides, and I think they have different purposes. I think a very multi-purpose foundation model could serve as its own object of study. It's a modification or reflection of the universe. And the fact that it might have diverse applications is interesting and valuable in itself. But if all we want is to really use a foundation model for one or two tasks, I can almost imagine another paradigm that's like disposable foundation models, where you understand very clearly the qualities of the downstream task that you want to target, and then you can quickly generate simulated data that captures those intrinsic properties well, and then you make your customized foundation model. But right now, I guess we have some evidence that even very large-scale generic image foundation models are not able to outperform domain-specific train-from-scratch foundation models if you have access to high quality data. So I don't think that adding more data in a generic sense is going to solve the problem.
CIUCĂ: I'm not very informed in this respect, but I want to say something. I'm kind of a scrappy person. So we have an opportunity now to train these very large-scale statistical models on our data sets. And I'm just generally curious to see how these large systems actually represent our data. And I think it's important to look a little bit into this mechanistic interpretability aspect, look a little bit into the representation of these data and look at their geometry, look at how they organize information, look how different modalities show up in those spaces — and there's a lot to explore. So I would say let's build foundation models to really understand a bit more about our data, and also the tools to probe their internal representations and things like that. Yeah, but I'm not an expert, so I don't want to talk more outside of my [lane].
MODERATOR: Thank you. MODERATOR: You already touched on the amount and the modality of data. One thing which I think a lot about is, if you for example have a fractal and you train a generative model to learn this fractal, you can train an infinite-parameter model to learn this fractal. However, the underlying structure is super simple. So when we train these foundation models — and currently all scaling laws seem to suggest that they get better and better at describing the underlying physics — are we just learning the universe by heart, or are we really discovering some underlying structure? Maybe start from the other side this time.
CIUCĂ: Oh gosh. Oh no, me. Oh god, oh lord. Okay. So I feel like, how can we even tell, right? I have a counter question to your question: how can we tell? Where do we look to make that distinction? Do we look in the — yeah, where do we look? And maybe start there. Sorry. [laughter]
MODERATOR: Good. I was not prepared to get a question back. But let me say that during lunch I had an interesting discussion with some students, and we were talking about whether the universe is intrinsically infinitely complex, which would mean you need an infinite amount of parameters to describe what is going on, or if there is indeed some underlying structure. When I started studying physics some years ago, I thought there would be an underlying structure, but maybe that's not true. And if there is an underlying structure, can that be encoded inside a very large statistical model? I don't know the answer to that. Maybe Josh has one.
BLOOM: Well, all I'll say is that we saw a talk yesterday about cellular automata, right? A very simple set of rules and you get this emergent phenomenon. So perhaps there is hope in being able to describe something that visually is as infinitely complex as you can imagine, but has simple sets of rules underlying it. That need for us as physicists to kind of reduce everything down to the simplest number of equations and appeal to Occam's razor is still very much not only implicit — I think it's quite explicit — in the way that many models are trained and built. In the architectures of models, when we think about sparse autoencoders and things, we are explicitly penalizing for complexity in representation space. And so we as physicists still believe that the right way to do things are with bottlenecks, and the bottlenecks with physical equations or bottlenecks in latent space have some deeper mapping with each other. But it doesn't mean it's the right representation.
PETTEE: Yeah, foundation models are only one part of the ecosystem, and that's why physicists steering them are critical, I think, because there wouldn't be constraints if there were infinite resources and we could train infinitely large foundation models. So I think having human understanding as a bottleneck is a choice that we've made in the field, and we've seen from history that that's a pretty effective prior to put onto the process. And also cost and computational limits are going to come into play too. So I think it's our job to help place constraints onto the process, because we believe that that will yield better results later.
BLOOM: Yeah, just to add to that, if we still also believe in the scientific method, then the falsifiability of predictions is a central part of what we're trying to do here. And so if you have a great representation and it can summarize all the data that you've seen but can also make predictions about the future on unseen data, and particularly in unseen regimes, then we would ascribe a lot of power to that, and I think there is still a lot of potential for that to actually be borne out. I don't know how much research is going into that in astronomy. There's some of it in the physical world of trying to do out-of-distribution testing on some models. The challenge we have in astronomy is we only have one universe, right? We have like 10 to the 10, 10 to the 11 pixels if every pixel is one arcsecond on the sky, and that's it, right? And we have a bunch of wavelengths and we've got time, but it's still a finite amount of photons that we could collect. And one of the problems that I have with foundation models in astronomy is that oftentimes if you're trying to do downstream tasks, you're not quite sure if the things that you're trying to do downstream tasks with are actually in your training data. If I'm trying to generate what would this part of the sky look like, it may have already been in the training data. If I'm trying to do shape analysis for dark matter, we've trained it on the data that we have — and yes, you could train it on simulated data, but if we're training on real data, then we have this challenge of just the finite nature of the amount of Shannon information bits that exist in the universe.
MODERATOR: Thanks. MODERATOR: With this I want to pivot a little bit away from the foundation model and go to the second topic of this session, which was agents. So if I walk through San Francisco and I talk to some people on the street, everyone seems to have enormous trust in scientists and especially in physicists, and people really believe that everyone here knows what they're doing and we are super sure about it. When we start using agents for our work, what do we have to do, and how do we have to make sure that we trust the agents, to keep the trust going? Mariel?
PETTEE: Yeah, you and I have had some pretty interesting back-and-forths on this. I don't have easy answers for it. But I think it's a little bit uncomfortable to realize that there's actually no way to have total trust in another person's work, let alone our own work. It's so easy to fool ourselves in our own process. So there is a bit of faith that comes into the research process, and that's why we encourage many people to be part of the field — is that we hope that even if individually we're fallible, in aggregate eventually the truth will prevail. And so I think the way that I think about trust with agents is similar to how I think about trust in my own work and in new collaborations that I've started, where if I feel like I can poke and prod the results and ask for new plots and ask for validation and verification, if I can read through the code myself, if I can compare that with writing about it too, eventually after some number of months or weeks of working with the process, I have to admit that I trust the work. And that's important, because as scientists I think we really have to advocate for our particular perspectives out in the courtroom of science. And so we need to feel very deeply that we believe in the quality of the work. So I think ultimately that's the bar — our own sense of taste and rigor, and we are ultimately just in control of that.
MODERATOR: Let me ask a follow-up question to you. I have the feeling that the way agents think and humans think has a very different regularization to it. Are the questions that you ask a human and the agent different, and in which way are they different, before you trust?
PETTEE: It's so true that the types of mistakes an agent would make are very different than the types of mistakes I would expect working with a student or a new collaborator. I had the experience the other day of realizing that an agent, when it was writing code, was explicitly optimizing for the thing that I was trying to measure. And if I hadn't looked through the code carefully, I could imagine missing something like that. And that's not something I would expect a student to try to slip into the process. So I don't think I've mastered the right questions. I think I have just tried to move very carefully.
CIUCĂ: I have a lot of thoughts on this one. Okay, let me start. So through the wonderful project we've had with Owen, who is today in the audience — Owen is building an amazing system that can help us implement methods that we care about deeply in astrophysics. And through the interaction with that system, we've learned that we have to reflect a little bit on ourselves in the way we even pose these tasks to the model, in such a way that the model can understand what we expect from it and proceed to implement the solution. So I think that is step one, just thinking deeply, how do we specify these tasks in such a way that an AI agent can act on it meaningfully? And then the second thing, something that I've learned working quite closely with Marcelo, and Marcelo's been thinking deeply about, is the context. When the agent goes and solves this task, it has its internal reasoning, it uses some tools, it makes some decisions, and we have to have a way to interpret what the agent is doing along the way, and we have to have some visibility into that process. So I think from a pure engineering perspective we need to have visibility into the process, and also, one thing that would help with the trust element would be for the agent to surface assumptions. I learned that from talking to Risa a lot about this, surfacing of assumptions and things like that. I think that's super key, and also, to not forget that these are artificial entities. They don't necessarily think about the problem like we do, and just being a bit more understanding that sometimes — should we even apply the same understanding of trust to an AI agent as we do to a human? That's a question in and of itself.
PETTEE: Thank you. Yeah, I wanted to add quickly to that that agents are fickle, and as soon as you push back on them, they'll say, “You're absolutely right. I was completely wrong. Now I've totally changed my opinion.” And fortunately, that's something that students are more hesitant to do. Real humans have an actual grounded sense of what is right and what is correct, what they understand. And that's why I sometimes chafe at the comparisons between agents and students, or agents and researchers, because we're not only measuring the quality of their code and their papers, but we're also thinking about their internal sense of research taste, and that's just, in my experience working with agents so far, you can't compare them.
BLOOM: I guess I would add, I agree with everything that's been said. We also need to think about community standards around this, and this is again where people in this room should have a big voice. We all have our own trust harnesses and our own developed practices. What's very strange, of course, is that what we didn't trust 6 months ago because everything was hallucinating, maybe now it's a lot better. And so we're also in a very strange moment where we're learning to interact with these artificial systems as if they were co-authors, as if they were collaborators. And that interface is the only thing we know how to do as people. But there are things we can do in code, right? There are community standards that we can implement. I hope many of you saw this, but arXiv, I think last week, said they're banning people for a year if they find that you have a hallucinated citation in your submission. And that's pretty harsh, right? So there are carrots and there are sticks. That's a pretty big stick. But what's interesting is that they're not penalizing people because there are hallucinations in the paper; they're penalizing them because they use that as a proxy for, you didn't write this paper and you didn't read it. And so it comes back to, do you stand behind, and does your reputation stand behind, the things that have your name on it? And now we're in this mode where it's not just you and your grad students and your collaborators putting your names on papers; it's you and your collaborators and your students and your agents putting names on papers. And so the trust still ultimately comes back to us. But I think it's up to us as a community to decide where we want to land on how much affordance we can have around using this kind of new set of tools.
MODERATOR: Thank you very much. MODERATOR: Now we have 15 minutes for questions from the audience. So please line up behind one of the two microphones and we'll get your question.
AUDIENCE: Hi guys. I just want to mention one thing. My partner actually works at arXiv, and that policy is apparently not official policy. Just one of the editors tweeted it and now everyone knows about it. So you're still okay for now. I guess I have a question: we've seen some breakthroughs recently, especially in math, that agents are now starting to discover things that we didn't discover. I guess a question I have is, in particle physics or in astronomy, what do you think the first ML-driven or agent-driven discovery would look like, and how do you think the community would receive that?
BLOOM: I think we'd embrace it with open arms, because it would be an exciting watershed moment for our field. I think it's more existential in math, because a large part of the math endeavor is thinking, right? But in astronomy, it's thinking and code and computation and confrontation with data and getting more data. It's a much more complex, much more messy scientific process. And frankly, we've all been kind of waiting for that moment to happen. There's been a lot of promises made for many years that we're going to discover new physics or something new is going to happen. There have been little tiny pockets of some lights that have happened, but one of the big challenges in astronomy is that we don't have an AlphaFold kind of data set. When you ask an astronomer what would be the best things to know, it's like, what's dark energy? What's dark matter? Is there life on other planets? Okay, we're not going to think our way to any of those answers. We're going to probably have to build billion-dollar facilities and be really smart about developing new algorithms over decades. And so, some of the astronomers in the room, your job is secure. But we would love, for the thinking types of axiomatic problems in astronomy, like solving something on the theory side — I think we're ripe for that, and there's a push that some of us have been thinking about, of trying to articulate what are those Erdős-like problems that, if we knew the answer to, that'd be pretty great.
AUDIENCE: Okay. So I'm wondering about the framework in your head for thinking about what science will look like 20 years from now, particularly with respect to the AI agents side of things. And so I'm wondering, what do you think are the qualities of humans as contrasted with the AI agents, for which they will maintain a fundamental advantage over the AI agents and will prevent the world where the agents replace huge amounts of the scientific labor? And then what are the implications of that for what things will look like 20 years from now?
PETTEE: I think there's a lot of value in the fact that humans are stubborn and curious and also are not limitless. I think sometimes agents are touted as saying, “Oh, it's amazing, they never sleep, they never get tired.” But I think a lot of the understanding of physics comes from the fact that we need things to be compressed and reduced, and we can't work infinite hours. So I think there's a sense of judgment that comes from humans having limited time on earth and resources and wanting to make something valuable with that time. And I guess I would also say that human collaborations, for the same reason that I just outlined, that it's hard for humans to change their minds, humans have a special value in collaborations that I think can't always be reflected with agents, even if we solved the idea that agents are less fickle.
MODERATOR: Let me ask back, why are you so sure that those are properties that cannot be modeled with an agent, like a sense of urgency, pushing back on thoughts?
PETTEE: Definitely we can imitate it. But I think that there's a special quality to a person choosing their life's work. And I think an agent doesn't care about its work in that way. And I do think that that's like a secret sauce of being a person.
CIUCĂ: I've wanted to share this quote forever, so thank you for your question, because now I get to share this quote. So there was this student who went on one of the open forums — he was in maths — and he asked on the open forum, hey, what can one such as myself contribute to mathematics when there are so many smarter people out there now, probably even agents in the future? And then what happened is that a Fields medalist actually jumped into the open forum and answered the student with the following. He said, it's not mathematics that you need to contribute to. It's deeper than that. How might you contribute to humanity, and even deeper, to the well-being of the world, by pursuing mathematics? So I think one thing that I think is deeply human is our desire to pay it forward, to sort of have this so we persist through time, and then maybe many generations from now we'll build on what we did. And I don't know if we can replicate that in an artificial system yet, but I think that's a deeply human value, at least.
MODERATOR: All right, thank you. MODERATOR: Then we have one more question on the right side.
AUDIENCE: Yeah, so thanks so much for all of that. That was great. I have a little bit of a philosophical question, but I think it's fair because some of the stuff you guys have been talking about has been pretty philosophical. So, as people who train and evaluate younger, earlier-career scientists, I wanted to get your opinion on whether or not you think there's value, and there will continue to be value, in having an identity as someone who is an AI-plus-physicist versus someone who just uses AI in their physics workflow. Do you see that continuing to be a difference? Because there's a lot of overlap between people who train models and people who develop architectures and people who think a lot and use agents. I'm wondering what the value of maintaining that identity, versus just a physicist that happens to use AI sometimes, is now and will be in the next 5 years or so.
BLOOM: I have a pretty rosy view of that, in the sense that I think AI just subsides into the background. We don't consider ourselves — I'm an astronomer who uses COBOL, right? Or I'm a physicist who knows how to use a database. AI is just a set of tools. All the companies that have AI in their last name — okay, you have a technique. AI is going to be pervasive in every single thing that we do. And once that is, then it ceases to be novel as a way to stand out. One of the things that I'm kind of excited about — maybe it's one of the only upsides of the disaster we have on the federal funding side of things — is that we get to reimagine where the money flows for doing science. And my hope is that it becomes less siloed than it is now. If you're an early-stage career scientist, or actually any career scientist, you are thinking about which are the agencies that you're applying to and what are the sub-agencies that you're going to get your money from. There's a hope, because now we don't have to necessarily say I am an astronomer working on this type of thing, but I'm somebody who thinks deeply about these types of problems, where we may also have enough of a training to be able to work across multiple domains, that we will break those silos, and now one can think of oneself as a scientist who's interested in this class of problems, whether it's in astronomy only, or astronomy plus physics, or astronomy plus physics plus bio.
PETTEE: Yeah, I think it's useful to worry less about labels and let the work speak for itself. So I try to practice that myself and not worry too much about how I'm framing my own work. But framing is important, because it conveys to other people what your context is, where your intuition comes from, who you admire in terms of their processes, and what biases you might bring to your work too. So I think if you have that larger framing, it's something you can take pride in. And I think we need physicists who happen to use AI, and we also need AI physicists.
MODERATOR: All right, then we have one more question on this side. AUDIENCE: Yeah, thanks for the nice discussion. I have two questions; you can answer any of them, I guess. So particle physics today has large global collaborations. So my question is, if we are thinking of integrating agents into how physics is done, then there might be two considerations. First is, how much would be dependent on the corporate token pricing policy? And the next one would be, HEP collaborations today keep pre-publication material private, and also the data is private. Unlike astronomy, it's open data, but particle physics, it's private. So I would just like the panel's opinion, because there were some conversations in our collaboration regarding, how really private are our conversations with these agents? Let's say you take some internal collaboration data, paste it to an agent and say, hey, do something — and now is that public? [laughter]
BLOOM: I am an astronomer talking about identity, not going to particle physics, but maybe — I'm happy to answer, but yeah.
PETTEE: I think they're both really cool questions. I think the particle physics community is actively wrestling with the question of open and closed data and how that interfaces with foundation models, and does that mean we pursue practices that keep the data closed, but can sort of preserve the use, like privacy-preserving methods or other techniques to train without having direct access to the data. Obviously astrophysics is taking a different approach, although there are still big questions around data attribution that Josh mentioned. So I don't think we've aligned on where we are. And it's challenging because, particle physics, all of that closed data is very valuable. There are a lot of companies that would love to have access to it, I think. And so I guess that leads to your first question about, how closely should we integrate our work with industry advances, and do we depend on tokenization strategies from companies? And yeah, it's a hard question.
BLOOM: I'm announcing here that I think we should have universal basic tokens, UBT, and so every year you get a billion tokens as a scientist, and it comes from the frontier models, and they should just give it to us, and we can sell it or use it or burn it. But you're asking a very deep set of questions about who funds science going forward, and how do we do this in a way that benefits the most number of people, not just close collaborations. I think it's still TBD, and again, we come back to where things are at with the federal funding situation in the United States. It's kind of wide open what the future looks like. But we will almost certainly be seeing other types of interaction models between industry and what we all do in this room.
AUDIENCE: Thank you.
MODERATOR: I think we have time for one more question on this side. AUDIENCE: Okay, sure. Thank you, and thank you to the panelists for your discussion so far. So I come from an area of physics and astrophysics where conceptual understanding of ideas and what is happening is preferred, or at least it's the state that we're trying to go to, more so than quantitative accuracy. And so in my field very few people use AI at all, except for fixing bugs in their code and whatnot. So I'm wondering what your opinion is on the relationship between using AI to understand physics concepts, like novel physics concepts, as opposed to using AI just to do quantitative data analysis and that sort of thing.
PETTEE: You're saying its value as a learning methodology, or a tool —
AUDIENCE: Including doing novel research on physics phenomena using AI, even if you're not trying to quantitatively match data exactly.
PETTEE: I feel like I can only really speak from experience about using it as an interlocutor for trying to process new ideas. And I think if you already have a well-honed sense of when you really understand something, it can be quite effective for that kind of thing. It's similar to talking with a colleague, but I think more dangerous than talking to a colleague, because a colleague can push back more easily on misunderstandings. But I think you're asking something deeper about how do you even maybe engage in research in that direction, and I can't speculate.
CIUCĂ: Yeah, me neither, but at least when it comes to conceptual understanding — the way I use these models, because I had to teach the transformer yesterday, and the architecture is pretty theoretical in the way I had to teach it, and I asked the models to push back. I instructed them to push back really strongly on my understanding of the transformer, and I found them to be quite good at that, to be honest with you. And because they're not human, if they say, “Hey, you're kind of silly,” right, it doesn't hurt me. So I can go quite a long time with them. So there's this sense of also psychological safety, in some weird way, with these models. You're not afraid to ask silly questions to them, and ask them to ask you silly questions. I think that maybe there's something in there about not being afraid to take the riskier conceptual jumps as you approach a problem. Yeah.
MODERATOR: Thank you very much for all the super interesting questions from the audience. To close this, I would like to have one statement from each of you. It is very hard for people to give some estimate of what will be going on in two years or five years or something. So I would like to ask you a pretty simple question. What will be one thing that will be true about your work 10 years from now? Can be a small thing. You guys start.
BLOOM: That's tough. I think it's fair to say that it is hard to see how any of us really go back to the way that we worked before LLMs came into our lives. I mean, I've been using AI in my research for two decades. It's been called different things, different methods, but we've been using these techniques to further science. And what's happened in the last 18 months has been remarkable, because it's not about the specific methods, or pulling this thing over from industry and pulling it into this way of changing our scientific workflow. It's changing and upending how we live and how we work professionally. I don't know what the details of that look like, but I'd be shocked if it's just not pervasive.
PETTEE: I feel pretty confident that in 10 years I will still be giving talks, because I think that is part of the fundamental duty of scientists, is to communicate their research and their discoveries to the relevant communities that can actually make use of it. That's my definition of what impactful research is — the type of research that inspires someone else to make new work on top of that. And something funny I realized about that recently is that that also means that actually doing bad work is potentially impactful research, because that can also inspire people to correct your mistakes and build on top of that. So yeah, I think I will still be giving talks, talking to people face to face. Maybe it'll be on Zoom, but I think it'll be largely in person. And I really hope I won't have an email address.
CIUCĂ: I think for me, in 10 years — hard one. My dream is to have this very nourishing, incredible, exciting interdisciplinary research collaboration. So I hope that in 10 years from now a few of us find a problem that we're really interested in and decide as a team to pursue it, working with AI agents or the systems at that time, and us as the humans, we can think slower. We can take our time deeply to think about the problem, to think about the approach, and then maybe have these AI systems act on our thinking and then help us implement and verify and all that stuff. And together we can maybe say things that we cannot say now about the evolution of our universe or the fundamental nature of the universe and so on. So maybe it's something about being together, working on problems that seem impossible today, and being even slower in our thinking, maybe. Yeah.
MODERATOR: Great vision. Thank you very much for your insightful answers. [applause]