The Adversarial Framework

The GAN framework, as introduced in Goodfellow et al. (2014_12ya) (see my paper review for a detailed discussion), sets up a two-player minimax game. The generator \(G\) takes random noise \(z \sim p_z(z)\) and produces a sample \(G(z)\). The discriminator \(D\) takes an input, either a real sample from the data distribution or a fake from the generator, and outputs a probability that it’s real. The objective is:

\[\min_G \max_D \mathbb{E}_{x \sim p_{data}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]\]

The discriminator wants to maximize this, assigning high probability to real data and low probability to fakes. The generator wants to minimize it, fooling the discriminator into assigning high probability to its fakes. In the idealized Nash equilibrium, \(G\) generates data exactly matching the true distribution, and \(D\) outputs 0.5 everywhere. It literally can’t tell real from fake.²

Goodfellow proved that under mild conditions, this minimax game has a unique solution where \(p_G = p_{data}\). The proof is elegant. For any fixed \(G\), the optimal \(D\) is \(D^*(x) = \frac{p_{data}(x)}{p_{data}(x) + p_G(x)}\), and substituting this back into the objective shows that \(G\) is minimizing the Jensen-Shannon divergence between \(p_G\) and \(p_{data}\), which is zero if and only if the distributions are equal.

The part of this that still strikes me as the real insight is captured well by an HN commenter named visarga, who once got to know Goodfellow on Reddit: “The great idea about GANs is that they replace one of the most hard to understand parts of a neural net, the loss function, with another neural net, thus making the loss function learnable.” That is the move. Instead of hand-designing a metric for “does this image look real,” you train a network to be that metric, and it adapts as the generator gets better. Beautiful theory. Brutal practice.

The Training Problem

GAN training is, notoriously, a nightmare. The theoretical convergence guarantee assumes perfect optimization at each step, with the discriminator fully converging before the generator takes a step. In practice you alternate single gradient steps for each network, and this alternating optimization of two coupled objectives is highly unstable. Several failure modes plagued GAN training throughout its history.

Mode collapse: The generator discovers a few outputs that consistently fool the discriminator and produces only those, ignoring the rest of the data distribution. Instead of learning to generate all of MNIST, the generator might produce nothing but 1s and 7s. In severe cases it collapses to generating a single image. The discriminator eventually catches on, the generator shifts to a different mode, the discriminator follows, and the system oscillates without converging.³ An HN commenter who builds with these models, sillysaurusx, put the practitioner’s version of it bluntly: mode collapse “is when the discriminator wins the game and the generator fails to generate anything that can fool it. Stylegan has some built in techniques to combat this.”

Training instability: The generator and discriminator can enter cycles where one overwhelms the other. If the discriminator becomes too strong too fast, the generator gets no useful gradient signal, because all its outputs are classified as fake with high confidence and the gradient saturates. If the generator gets too strong, the discriminator can’t learn. Balancing the two networks requires careful hyperparameter tuning across learning rates, architecture choices, and the number of discriminator steps per generator step.

Evaluation difficulty: How do you measure whether a GAN is good? You can’t compute the log-likelihood, because the generator doesn’t define an explicit density. The Inception Score (IS) and Fréchet Inception Distance (FID) became standard metrics, but both have known flaws. IS doesn’t detect mode collapse well, and FID assumes Gaussian statistics in feature space. The lack of a clean, reliable evaluation metric made GAN research partly a matter of aesthetic judgment, the unscientific question of whether the samples look good, which is an unusual situation for a quantitative field.⁴

A practitioner on HN who built image models with the architecture, MoonGhost, gave about the most honest description of the day-to-day experience I’ve seen: “One model produces output and tries to fool the second. Both are trained together. Simple, but requires a lot extra efforts to make it work. Unstable and falls apart (blows up to unrecoverable state). I found some ways to make it work by adding new loss functions, changing params, changing models’ architectures and sizes. Adjusting some coefficients through the training to gradually rebalance loss functions’ influence.” That is GAN engineering in one paragraph. The math fits on a napkin. Getting it to converge is a part-time job.

Taming It: DCGAN and the Architecture Tricks

The first paper that made GANs feel like an engineering tool rather than a curiosity was DCGAN, from Radford, Metz, and Chintala in late 2015_11ya. They didn’t change the objective. They figured out which convolutional architectures actually trained without exploding. Use strided convolutions instead of pooling, use batch normalization in both networks, drop the fully connected hidden layers, use ReLU in the generator and LeakyReLU in the discriminator. None of this is deep theory. It’s a recipe, hard-won from a lot of failed runs, and it’s the reason the next five years of GAN papers had a stable foundation to build on.

DCGAN also produced the first demonstration that the latent space had structure you could manipulate. The famous result was vector arithmetic on faces: take the latent code for “man with glasses,” subtract “man,” add “woman,” and you get “woman with glasses.” This is the same trick word2vec showed for language, and seeing it work on images was the moment a lot of people, me included, started taking GANs seriously. The network wasn’t memorizing. It had learned a smooth, navigable representation of the data.

The Loss-Function Wars: WGAN and Friends

The instability of the original objective drove a wave of research into replacing it. The standout was the Wasserstein GAN (WGAN), from Arjovsky, Chintala, and Bottou in 2017. Their diagnosis was that the Jensen-Shannon divergence the original GAN minimizes behaves badly when the real and generated distributions don’t overlap, which is almost always the case early in training. When the two distributions live on disjoint low-dimensional manifolds, the JS divergence is constant, so its gradient is zero, and the generator learns nothing. That is exactly the “discriminator wins and the generator gets no signal” failure mode practitioners kept hitting.

WGAN swapped in the Wasserstein distance, also called earth-mover distance, which gives a meaningful, non-vanishing gradient even when the distributions don’t overlap. The catch is that computing it requires the discriminator (now called a “critic”) to be a 1-Lipschitz function. The original WGAN enforced this with crude weight clipping, which the follow-up WGAN-GP replaced with a gradient penalty. For a while, “use WGAN-GP” was the standard advice for anyone whose GAN wouldn’t converge. It genuinely helped, though it never made training as boring and reliable as, say, training a classifier.

There’s a deeper point buried in this whole episode. The reason GANs were so finicky is that they’re solving a minimax optimization, and our optimizers (SGD, Adam) are built for minimization, not for finding saddle points of a two-player game. Every fix, from WGAN to spectral normalization to two-timescale update rules, is in some sense a patch on the mismatch between the problem (a game) and the tool (gradient descent). I think that mismatch is the real reason GANs were eventually overtaken, and I’ll come back to it.

The Golden Age: Progressive Growing to StyleGAN

If you only remember GANs for one thing, it’s probably faces. The line of work that got there ran through NVIDIA’s lab and a researcher named Tero Karras. Progressive GAN (2017) introduced a deceptively simple idea: don’t try to generate a 1024x1024 image from the start. Begin at 4x4, train until stable, then add layers to both networks to double the resolution, and repeat. Each stage only has to learn a small refinement on top of an already-working lower-resolution model. This stabilized high-resolution training in a way nothing before had, and it produced the first GAN faces that could genuinely fool people at a glance.

Then came StyleGAN in December 2018, and its successor StyleGAN2 a year later. StyleGAN rethought the generator architecture by borrowing from style transfer. Instead of feeding the latent code in at the bottom and letting it propagate, it mapped the latent code through a separate network into a “style” vector, then injected that style at every layer through adaptive instance normalization. The effect was that different layers controlled different scales of feature, coarse layers set pose and face shape, fine layers set skin texture and hair detail, and you could mix styles between images at different scales. This gave the model an unusual degree of controllable, disentangled generation.

StyleGAN2 is the model behind thispersondoesnotexist.com, the site that did more than any paper to make the public viscerally understand what generative models had become. As an HN commenter, hackingonempty, described it plainly: “StyleGAN2 is an unconditional image generation model. StyleGAN2 trained on faces from Flickr.” You hit refresh, and a person who has never existed looks back at you, lit and posed like a real portrait, indistinguishable from a photograph. For a couple of years that site was the single most effective demo of the technology’s power and its unsettling implications.⁵

The faces were so good that they became infrastructure for other things, some benign and some not. People used thispersondoesnotexist faces as placeholder avatars, as fake social media profile pictures, and, as one HN user candidly admitted, to defeat identity verification: a commenter described bypassing an age-verification system “using a picture of my drivers license with a picture from thispersondoesnotexist.com and some light image editing.” The technology that could synthesize a convincing human face was, inevitably, also a tool for fraud. That tension never went away.

Beyond Faces: Conditional and Image-To-Image GANs

Unconditional face generation was the showpiece, but the workhorse applications were conditional. Conditional GANs feed a label or other context to both networks, so you can ask for a specific class rather than a random sample. pix2pix (2016) turned this into general image-to-image translation: sketch to photo, map to satellite, day to night, edges to cat. You needed paired training data, an input and its desired output, which is expensive to collect.

CycleGAN (2017) removed that requirement, and it’s the application I find cleverest. It learns to translate between two domains, say horses and zebras, or photos and Monet paintings, without any paired examples. The key idea is cycle consistency: if you translate a horse to a zebra and then translate that zebra back, you should get your original horse. Adding this constraint as a loss term forces the two translation directions to be roughly inverse, which is enough to pin down a sensible mapping without supervision. One HN commenter who’d followed the space since the start, cbsudux, captured the era’s enthusiasm: “I’ve been a fan of Generative AI since the start (CycleGAN and pix2pix) back in 2016.” Another, elil17, summed up the mechanism in a sentence: “CycleGAN does some thing along those lines (it’s called ‘cycle consistency loss’).” The “reverse Monet” demos, where a painting was turned into a plausible photograph of the scene, made the rounds widely.

GANs also did real work outside art. Super-resolution GANs (SRGAN) upscaled low-resolution images with believable invented detail. Medical imaging researchers used GANs to synthesize training data when real labeled scans were scarce. And there was a quieter, more interesting use: training models on synthetic data. An HN commenter, rkaplan, explained why this mattered, discussing Apple’s SimGAN work: “Labeled data is very expensive. Historically attempts to learn on synthetic data has failed because ConvNets are very good at detecting small visual artifacts in the synthetic data and using those for classification during training. At test time on real data, those artifacts aren’t present so model fails.” Using a GAN to make synthetic images look real enough that a downstream classifier couldn’t cheat was a neat, practical trick.

Scale Changes Everything: BigGAN and the Stability Myth

By 2018, the conventional wisdom was hardening into a slogan: GANs don’t scale, they’re too unstable for large datasets. BigGAN (2018) was the counterexample. By throwing large batch sizes, more parameters, and careful regularization at the problem, it produced high-fidelity 512x512 ImageNet samples across all 1000 classes, the hardest unconditional-ish benchmark going. It worked, but it was famously brittle, prone to a “training collapse” you had to checkpoint around.

This is where Gwern has a take I keep coming back to, because it cuts against the received wisdom. Responding on HN to the claim that GANs are inherently unstable, he wrote: “if you’re wondering how it can look so good when ‘everyone knows GANs don’t work because they’re too unstable’, a widespread myth, repeated by many DL researchers who ought to know better, GANs can scale to high-quality realistic images on billion-image scale datasets, and become more, not less, stable with scale, like many things in deep reinforcement learning.” I think he’s right that the instability was overstated as a law of nature. But there’s a counterpoint from another HN user, GaggiX, that I find equally true: “obtaining good quality on complex distribution is much easier with a diffusion model than a GAN.” Both can be correct. GANs could be scaled. It was just more painful than the alternative that showed up next.

Diffusion Eats the World

By 2021 a different approach was producing better images with far less drama, and it did so by abandoning the adversarial setup entirely. Diffusion models define a fixed forward process that gradually adds Gaussian noise to an image until it’s pure static, then train a single network to reverse that process one small denoising step at a time. There’s no second network, no game, no equilibrium to chase. You’re just doing supervised learning on a well-defined regression target, which means you can throw a standard optimizer at it and it behaves.

The HN commenter oofbey called the shift early and cleanly: “Diffusion models seem like they’re poised to completely replace GANs. They obviously work super well, and you don’t have this super finicky minimax training problem.” That second sentence is the whole story. Diffusion didn’t win because its samples were dramatically prettier at first. It won because it was trainable by a normal person on a normal schedule, with a loss curve that went down instead of sideways. Another commenter, not2b, registered the same thing as a working assumption by the time it was obvious: “diffusion models seem to have replaced GANs for image synthesis.”

I want to be precise about what happened, because “diffusion replaced GANs” gets repeated as if GANs failed. They didn’t fail. They were outcompeted on the one axis that turns out to matter most in this field, which is how reliably a method delivers good results when a non-expert runs it. GANs gave you spectacular peaks and frequent crashes. Diffusion gave you a slightly slower, much steadier climb. For research that’s a tossup. For a product team that needs the thing to work on Tuesday, it isn’t close. The entire text-to-image wave that the public actually noticed, DALL-E 2, Stable Diffusion, Midjourney, is built on diffusion, not on the architecture that spent seven years learning to draw faces.

What GANs Left Behind

So is the adversarial idea dead? Not quite, and the reasons it survives are instructive. The one place GANs still have a clear, durable edge is speed. A GAN generates a sample in a single forward pass. A vanilla diffusion model needs dozens or hundreds of sequential denoising steps to produce one image, which makes it far slower and more expensive at inference time. That gap is the whole reason adversarial training keeps coming back through the side door.

The clearest example is the distillation work that makes diffusion fast enough for real-time use. When researchers want to compress a many-step diffusion model into a one or two step generator, the tool they reach for is, often, an adversarial loss. Methods like adversarial diffusion distillation (the technique behind SDXL Turbo) bolt a discriminator onto a distilled diffusion model precisely because a discriminator is the best known way to keep single-step samples from looking blurry. The minimax game that everyone supposedly abandoned is quietly doing essential work inside the models that replaced it. I find that genuinely funny. The idea didn’t die. It got demoted to a component.

GANs also remain competitive in narrow domains where their speed and their well-explored architectures still pay off: certain kinds of super-resolution, some medical and scientific imaging pipelines, audio synthesis vocoders like HiFi-GAN that need to run fast in production. The StyleGAN line in particular left behind an unusually rich understanding of how to build a controllable, editable latent space, and that body of knowledge fed directly into the editing and control techniques people now want from diffusion models. The research didn’t evaporate when the paradigm shifted. It diffused, so to speak, into everything around it.

And then there’s the cultural legacy, which outweighs the technical one. GANs are the reason the general public first understood, around 2018 and 2019, that “photo of a real person” had stopped being a reliable category. thispersondoesnotexist was a more effective piece of public education about machine learning than any explainer article, mine included. The word deepfake entered ordinary vocabulary on the back of this technology. Every conversation we’re now having about synthetic media, provenance, and the collapse of photographic evidence traces back to the moment a GAN got good enough to make a face you couldn’t catch.⁷

What the GAN Story Is Actually About

I keep coming back to the same lesson, and it isn’t a technical one. GANs were the more beautiful idea. The adversarial framing is genuinely elegant, the kind of thing that makes you sit up when you first understand it, and the proof that the optimal generator minimizes a real statistical divergence is the sort of result that feels like it should win. It didn’t. It lost to a method that is, conceptually, kind of dull: add noise, learn to remove it, repeat. Diffusion has no clever adversary, no game, no equilibrium poetry. It just trains.

That outcome shows up over and over in machine learning, and the GAN-to-diffusion transition is the cleanest illustration I know. The thing that wins is rarely the most elegant idea. It’s the idea that scales predictably, trains without babysitting, and does roughly what you expect when you hand it to someone who didn’t invent it. LSTMs lost to Transformers on a version of the same axis. Hand-tuned features lost to learned ones. Reliability and scalability beat cleverness, almost every time, and the people who internalize that early tend to bet better than the people chasing the prettiest math.

There’s a second lesson layered under the first, about how fields fall in love. For roughly seven years a huge fraction of generative-modeling research was, in effect, devoted to fighting one method’s pathologies. WGAN, spectral normalization, progressive growing, two-timescale updates, dozens of regularizers, all of it heroic engineering aimed at making an unstable thing stable. A lot of that effort was real progress, and some of it transferred. But a lot of it was the field paying interest on a loan it took out when it committed to the adversarial framing before knowing whether that framing was the right long-term bet. When diffusion arrived, much of that hard-won GAN-stabilization knowledge simply stopped being load-bearing. That’s not a tragedy. It’s how research works. But it’s worth remembering the next time everyone is certain the current paradigm is permanent.

If you want my honest summary: GANs were a brilliant, productive, ultimately transitional idea. They taught the field that you could learn a loss function, they gave us the first generative images that genuinely fooled people, and they seeded a generation of researchers and a public conversation that’s still unfolding. Then they handed the crown to something more boring and more reliable, and the most fitting epitaph is that the adversarial trick now lives on as a speed hack inside its own successor. I don’t think you could ask for a more on-brand ending for one of deep learning’s most beautiful, most frustrating ideas.

Further Reading

• The GANs paper, my detailed review of Goodfellow et al (2014_12ya)

• Diffusion Models, the approach that displaced GANs for image generation

• Convolutional Neural Networks, the backbone of every image GAN

• Batch Normalization, one of the tricks that made DCGAN trainable

• Gradient Descent and Its Variants and the Adam optimizer, the optimization machinery GANs strained against

• Recurrent Neural Networks and the Transformer architecture, for the same elegance-loses-to-reliability pattern in sequence modeling

• The original GAN paper and the StyleGAN2 paper, both very readable

• thispersondoesnotexist.com, still the best one-click demo of what this technology became

This piece draws on Goodfellow et al (2014_12ya) and its Wikipedia entry; the DCGAN, WGAN, Progressive GAN, StyleGAN, pix2pix, CycleGAN, and BigGAN papers; and a range of practitioner discussion on Hacker News. Images are from Wikimedia Commons under the licenses noted in each caption.