How the New Black Hole Image Was(n't) Created With AI*

*At least not the kind of AI you're thinking of.

Last month, I saw a headline on the front page of the New York Times — about using AI to improve the first image ever produced of a black hole.

Being an AI researcher myself, that piqued my interest. So I emailed the lead scientist on that paper — and all the other papers building up to it — to talk about it. She, very generously, offered up some of her time.

This article has three parts: first, how this black hole image was actually created, and why making an image of a black hole isn’t anything like taking a photo with a normal camera. Second, what the New York Times and other media outlets got wrong. And third and most important, what we can learn from this image. 

Lia Medeiros is an astrophysicist at the Institute for Advanced Study, where Einstein, Oppenheimer, and Godel used to work, and where she studies computational approaches for understanding black holes. She was an author on the original paper revealing the first image of a black hole, and she’s been working for years with the Event Horizon Telescope, a network of telescopes spanning the globe, to publish many papers on black holes. This paper and its precursors are about a new algorithm called PRIMO.

MAXIME: What is PRIMO?

LIA MEDEIROS: What PRIMO does is we take the PCA, the two dimensional PCA of tens of thousands of images. And what that does in practice, looking at this from a linear algebra point of view, is that it creates a basis for this hyperspace of images. And we've gone out of our way to make this hyperspace of images as broad as it possibly can be, including, for example, simulations that are really not consistent with our theoretical expectations.

We go out of our way to include simulations that really are not well motivated for this source to make sure that that is not biasing our basis and our space of images.

Let me break that down: PCA stands for principal component analysis. And it’s a staple method in computer science. Given a dataset, PCA finds a small set of data points that can be combined to create any other point in the dataset.

A good way to think about this is in terms of primary colors. In art class, we learn that by mixing 3 colors of paint — red, blue, and yellow — we can create any other color. Red and blue make purple, blue and yellow make green, yellow and red make orange.

Those primary colors are the basis for all other colors. They can be combined to create any other color. That’s what PCA does — it finds such a basis for any dataset you give it: even if that dataset is made up of images. 

PRIMO uses a dataset of 30,000 images of black holes from simulations. As Lia points out, they’re careful to make that dataset of images diverse, so that it doesn’t bias this method toward what they expect — so that they’re not putting their fingers on the scientific scale, so to speak. 

They use PCA on that very diverse dataset to find the “primary colors” of black hole images. 

LIA MEDEIROS: Once we have this basis for this hyperdimensional space of images, then we can use that basis to reconstruct any image that is in this space.  We just fit a linear combination of the PCA components directly to the data in Fourier space.

Going back to our previous analogy, once we have the primary colors (red, yellow, blue), if we also have a particular color we want to match, we can use an iterative process to figure out how much red, yellow, and blue we need. We start with some red, adjust with some blue, add some yellow, and keep tweaking the mixture until it matches our source.

Similarly, the “matching” step of PRIMO seeks to match real observations from a specific black hole using the “primary colors” from PCA — tweaking the ratios of those “primary colors” until the mixture aligns with the real data more closely than any other mixture.

You might well ask, “Why do we need to do this at all?”

MAXIME: Can you explain why creating an image of a black hole is different from taking a photo with your camera or even looking through your telescope at home at the moon?

LIA MEDEIROS: We want to make an image of these black holes. We want to be able to resolve this little ring. If you actually do the math, you'll find that the size of the telescope you need is literally the size of the entire Earth. That's outside of our budget and also would be terrible for all kinds of other reasons. Half the Earth would just be dark all the time. It would be a disaster.

Okay, so we can't do that. So this idea of just taking a picture is just not something that humanity will ever realize in the sense of taking your phone and just like taking a picture. However, since we cannot do that and that's something that we will never be able to do, we need to use a technique called interferometry.

What interferometry does is it does not take an actual picture, but it learns information about the picture. And then we use that information that it learns to make our best guess of what that picture looks like. And again, this is the absolute best we will be able to do. 

Our picture will continue to get better as we increase the number of telescopes because we will have more information about it. But fundamentally, we will never have all of the information about all of the details of the image unless we can cover the entire Earth in telescopes.

Instead of covering the Earth in telescopes, Lia and other scientists use the Event Horizon Telescope — a collection of telescopes spread around the world that, as a whole, act as one Earth-sized telescope — with tons of blind spots.

PRIMO fills in those blind spots — in a way that is 1) consistent with the data, 2) dead simple — and that’s important.

When this paper came out, the media had a field day reporting on the new black hole image created with AI. But the AI used in this paper is nothing like ChatGPT or generative art systems. In fact, it’s not really AI at all.

LIA MEDEIROS: When I first wrote the original algorithm paper that came out earlier this year, I didn't even have the words machine learning in the text, to be completely honest. And the referee came back to me saying… you need to give machine learning some credit because this is technically machine learning. And so, of course, I obliged. I added a couple of paragraphs to that explanation, and those paragraphs made it into the follow up paper with the actual image, and it made it into the press release.

However, it's actually dictionary learning. It's not AI. And dictionary learning is something that's been around for quite some time. It's really just linear algebra. It's very easy to understand… And I actually think that that's what makes this so powerful is that we don't need something like a real AI or a convolutional neural network. It's built on concepts and ideas and equations that have been around for quite some time and are really well understood and easy to understand. 

I did an interview with The New York Times, for example. And I very clearly explained why it wasn't AI and why it wasn't a convolutional neural network. And of course, they print that it's an AI and it's a convolutional neural network.

MAXIME: What’s important to note here is that the AI, to call it that, that you're using is very different from what people think of. It's not anything like the AI art generators that have recently taken the Internet by storm: giant neural networks with billions of parameters, trained on millions of images. You use simulated images to train your approach, but PRIMO only “learns” around 20 parameters.

LIA MEDEIROS: It learns about 20 parameters. As you said, it's all built on principal component analysis, which I consider to be one of the main building blocks of modern machine learning and AI. It's all kind of built on this. And so we're using a building block of machine learning, but at the end of the day, we're nothing like ChatGPT or anything like that.  

PCA is an incredibly simple method. It’s bread and butter. I learned it in my very first semester of college — in a math class (linear algebra) mind you, not an AI class, and not a computer science class. 

It’s nothing like ChatGPT and it’s certainly nothing like AI art generators — which have billions of parameters and are trained on billions of images. This method has 20 parameters, and it’s not even right to say it’s “trained.” It just selects points from a dataset that summarize the dataset well.

But when newspapers write “astrophysicist releases black hole image generated by AI” — around the same time that they’re writing “Midjourney releases art generated by AI” — it makes the two sound equivalent. Which makes readers respond like this:

and I don’t blame them.  

In reality, this simple PCA approach and these giant AI models are about as different as this:

and this:

So what is right to say about this black hole? What can we learn from this image?

LIA MEDEIROS: Einstein's theory of gravity, general relativity, predicts that black holes in space should have a very specific shape. And when I say shape, I mean shape in 4D. The actual four-dimensional geometry of them… If you were to find a black hole that is not consistent with that, it would mean that either one of those assumptions is not valid or that there's something fundamentally wrong with our understanding of gravity itself.

There is a separate question of what we call exotic compact objects (ECOs). And there the idea is, “Okay, there is a way that this could be a black hole, but just not the specific black hole that we expect. Or maybe it's not a black hole at all. And if it's not a black hole at all and it has some sort of surface, then would we be able to tell?”

There are three critical measurements from the new image:

LIA MEDEIROS: The first thing I want to say is that the main thing we want is the ring diameter. A lot of my work focuses on using these images to test gravity, and what we use for that is just one measurement: it's just the diameter of the ring (with an error bar, of course.)

Second is this ring width. The fact that we can actually get a good ring width is super exciting and super important for a variety of reasons… We know that the ring width can act effectively as an upper bound on our uncertainty. And so the fact that you can constrain the ring to be thinner is actually really exciting and really important. 

The third thing… is that the dark spot in the center of the ring is really important for us. If Einstein is right, we think that this is a black hole and it should have an event horizon: it should not have a normal surface.

If, hypothetically, this wasn't a black hole and it was some other weird object that we don't understand, and it were to have a surface like a planet, for example, it might reflect some of the light that falls onto its surface. If it were to reflect light, the way that we would see that in our image would be that the dark spot at the center of the ring would be less dark.

If you look at the original dark spot in the original image, it's really not that dark, but in the new image, it's a lot darker.