A Brain Made of Small Parts

By neuron I mean a small part of a brain that takes in shocks from wires and sends out shocks too. A brain is made of neurons.

Each neuron gets shocks from the rest through wires. If those shocks add up past a high mark, the neuron fires and sends out a shock too.

We change how a brain thinks by changing how strong each wire is. We can change the high mark too. Then one neuron may fire more, and one may fire less. That is what folk mean when they say, “the brain learns”.

Here is the prop for wires. Think of each wire as a small jar full of stones. The more stones, the stronger the wire. To make the wire strong, drop a stone in. To make it weak, take a stone out.

When the brain does well, we may add stones to the wires that helped. When it does ill, we may take stones out. Once you can flip wires from strong to weak and back, you can make a brain learn.

Try, Score, Change

A brain learns to do well at a task by try and miss. At each step, the brain looks at where it stands, picks a move, and the world hands back a score. High score for a good move. Low score, or no score, for a bad one. The brain wants the most score it can get, on the whole, in the long run.

Think of a dog and a treat. You do not tell the dog all the rules. You just give it a treat when it sits. Soon, the dog sits to get the treat. A bot is the same: hand it score, and it learns to do more of what got score.

Or think of a small bot in a maze. It steps left and hits a wall. Bad score. It steps right and finds cheese. Good score. At first, the bot knows no path. It has to try, get score, and change.

By reinforcement learning I mean a way for a brain to learn from score.

Reinforcement learning is not a spell. It is not joy or pain in the bot. The score is just a count that tells us how to change the brain.

A Grid of Q

Now we can build our first way to learn, with no big brain at all: just a black board and chalk.

By table I mean a grid of rows and cells, like a calendar or Tic-Tac-Toe.

Draw a grid on the board. Down the left side, write each spot in the maze, one per row. On the top, write each move the bot can make: left, right, up, down. In each cell, write a count: the price tag for that move from that spot. At the start, write 0 in each cell.

By Q-learning I mean a way that fills a table with value guesses through play.

Q-learning goes like this. The bot lands on a spot. It looks at the row for that spot. It scans the row, finds the cell with the best count, and walks that way. Most of the time it picks the best cell; now and then it picks at chance, just to try. It does the move, gets score, and lands on a new spot.

Then it fixes the cell it just used. Look at the new spot’s row. What is the best count in that row? Add the score to that. Now blend the cell’s old count with this new one: rub out a bit of the old, write a bit of the new in its place.

Do this for a long stretch of play, and the counts on the board fill in to the right values. Then the bot just looks at its row, picks the best cell, and plays.

By value-based I mean a way that learns values first, then picks moves from them.

Q-learning is a value-based way. It fills in price tags first. Then to act, it just picks the move with the best price tag. The rule book is built on top of the price tags.

By tabular I mean made with a table, not with a brain.

This is tabular Q-learning. There is no brain of neurons here at all. Just a board and chalk. You have built a thing that learns to play a maze with just what is in your school room! Chalk boards were not meant to think, nor win or lose; try not to brag.

Map or No Map

By model I mean a map in the brain of how the world works.

Think of a model like this. The bot has, in its head, a small doll house of the maze. In the doll house lives a small doll bot. The real bot can poke the doll bot through the doll house and see what comes next. “If I send the doll left, then up, where does it land?” The doll house is the model. With it, the bot can plan a string of moves, then move a foot for real.

A bot with no model just plays in the real maze. No doll house. No plan in its head. It steps, gets score, and learns.

By model-free I mean a way that learns with no model. By model-based I mean a way that learns with a model.

Model-based ways can learn from less play, since the brain can plan in its doll house and squeeze more out of each run. But the doll house might be wrong. The doll house could say, “step left and you find cheese”, while the real maze says, “step left and you hit a wall.” Then the brain plans well in the doll house and fails in the real maze.

Model-free is slow but safe: no wrong doll house to fool you.

Tabular Q-learning is model-free. The chalk on the board holds price tags for moves, not a doll house of the maze.

Whose Play Counts?

Now a fine point. The brain is making moves, and it is learning. But is it learning from its own moves right now, or from old game tapes?

By on-policy I mean a way that learns from the policy it has now. By off-policy I mean a way that can learn from old play or a past policy.

If you are on-policy, you learn from fresh games. Once you change your policy, the old games may not fit. They came from the old policy, not the new one. You have to play more.

If you are off-policy, you keep a stack of old game tapes on your shelf. Some are tapes of you last week. Some are tapes of kin bots. You watch old tapes from the stack and learn from them too. This saves fresh play. But off-policy is a fair bit harder to make work right. Old play can point the new policy at the wrong thing.

Tabular Q-learning is off-policy. Why? When it fixes a cell, it blends in the best count from the next row, not the count of the cell the bot would in fact pick next. It learns the value of the best policy, not just the policy it used to get that play. That is what makes it off-policy.

We now have 3 splits to track: (1) price tag or rule book; (2) doll house or no doll house; (3)fresh games or old tapes. Each way to learn can be tagged on all 3.

Shake and Pick

How do we make a brain, not a board, learn? Here is one way that uses no math.

By evolution I mean change of a kind through births, shakes, wins, and deaths. By strategy I mean a plan for how to win. By evolution strategy I mean a strategy that trains a brain by small shakes and scores.

An evolution strategy goes like this. Take a brain. Make 100 twins. For each twin, give it a small shake: a few wires shift up, a few shift down, by chance. Now race all 100 brains through the maze. Each twin gets a run time or a score. Drop the, say, 90 worst. Then keep the 10 best.

From those 10, make 100 new twins. Shake each one. Race them through the maze. Drop the 90 worst. Keep the 10 best.

Do it once more. And once more…

Soon, the brains run the maze fast!

This is grim if you think too hard. But the brains do not feel a thing. This is in fact how you got here, just on a much longer time scale.

An evolution strategy is model-free: no doll house. The rule book brain is what gets a shake. It is plain on-policy: each score comes from what the brain just tried.

What is the good of an evolution strategy? It is plain. It works on huge brains. You can race all 100 twins at the same time on lots of chips, since each twin runs on its own.

What is bad? It wastes play. A twin has to run a whole maze just to get one score. One bad slip in a long run can make a great brain look bad. Worst, it is dumb to thanks and blame: it pays no heed to which moves in the run helped or hurt. The score is a big lump dropped on the brain as a whole. The next ways are sharper.

One Brain, Lots of Runs

By REINFORCE I mean a rule that makes moves in runs that beat a base more apt, and moves in runs that fall short less apt.

Set a shelf in front of you. On the shelf is a jar for each wire in the brain. The count of stones in a jar tells you how strong that wire is. A jar with lots of stones: a strong wire. A jar near dry: a weak wire.

REINFORCE goes like this. One brain plays a whole run. At each step, it picks a move with some chance. Note each move and which wires helped lead to it. At the end of the run, look at the score. If the score is high, walk down the shelf and drop stones in jars for the wires that helped make the moves. If the score is low, walk down the shelf and pluck stones out of those jars. Then play more runs and do it once more.

This is not fair to the wires in one run. A wire that helped make a smart move in a lost run still loses a stone. A wire that helped make a dumb move in a won run still gains a stone. The jar will not know why. The jar need not know. With lots of runs, the noise tends to wash out. On the whole, wires that help win pile up stones. Wires that help lose run dry. The brain gets good.

By policy-based I mean a way that changes the policy, not just values.

REINFORCE is a policy-based way. It does not fill in price tags first. It just shifts the rule book: make this move more apt, make that move less apt. Policy-based ways tend to work well on huge tasks, such as a chat brain that writes the next word, where there are far more moves than we could rate one by one. Value-based ways tend to work well on small tasks where the brain can rate each move with care.

REINFORCE is on-policy and model-free too. It is sharper than an evolution strategy, since it changes the parts that helped make the moves. But it is still loud. A whole run gives one blunt grade to lots of small moves.

Keep the Push Small

Lift the Fast Gut

So far, each way to learn has been fast: pick a move and play. But there is a slow way to pick too. You can think hard, then move.

First, use no net. Use the rules of the game. Draw a tree of what-ifs on a sketch pad. At the root is the board you see now. Each branch is a move. At the next row are the boards that could come next. Then the foe moves. Then you move. The tree grows till you stop. At each leaf, write a rough price tag. Then push the best tags back up the tree and pick the branch that looks best. This is slow, but it can be smart.

Now add a fast gut. The fast gut says which moves look worth a slow think. It is the policy. The price-tag head says how good a board looks. It is the value. The slow think uses both: the gut says which branches to search first; the price tag says how good a leaf looks, so the slow think need not play each line to the end.

Here is the loop. The slow think picks a move. Train the fast gut to pick that move with less thought. Then use the new fast gut to guide the next slow think. The slow pick trains the gut. The gut makes the next slow pick better. Round and round.

By expert iteration I mean a loop in which a slow good pick trains a fast gut, and that fast gut helps the next slow pick.

Expert iteration is not a new beast. It is the same old loop in a fine suit: try a strong slow pick, score it through play, change the fast gut to match it.

You can see the old price-tag and rule-book loop in it too. Better price tags make a better rule book. The better rule book makes new play. The new play makes better price tags.

When Folk Pick

AlphaZero had it good: a clear win-or-loss score from each game. But what if you want a brain to write a good note? There is no clear count for “good note”. A note can be true and dull. A note can be wrong and slick. A note can be sharp on the math but mean to read. You need a kind of grade that plain code can not give.

So we ask folk.

By human I mean one of us, not a bot. By feedback I mean a mark or word from a human that says good or bad. By RLHF I mean a way to train a bot with human feedback.

Think of a blind taste test. Two cakes on a plate. A bunch of folk take a bite of each, then point to which one tastes best. Do this lots of times, with lots of cake pairs and lots of folk. You end up with a thick stack of picks: this cake beat that cake.

Now switch from cakes to chat. A human sees two replies to the same prompt and picks the one they like more. Do this lots of times. Now we have a stack of picks: this reply beat that reply.

By reward model I mean a model that gives a score to a reply.

Train a reward model on the stack of picks. Its job is to guess which reply folk would pick. Soon the reward model can stand in for the folk. The folk go home. The score brain stays.

The chat brain writes replies. The reward model rates each reply. Use PPO or GRPO to push the chat brain to write replies the reward model rates high. The PPO leash keeps the chat brain from drifting too far from the chat brain you had at the start.

There is one trap. If you let the chat brain push too hard, it can learn to game the reward model. It might write a reply that scores high but is in truth junk food: full of fluff or the same stuff, but the reward model falls for it. The leash holds it back. Do well by the score brain, but do not drift too far from the chat brain you came from.

RLHF does not make the bot feel joy or pain. Score is just a count used to change wires.

How A Chat Thing Gets Built

By deep learning I mean a way to train a large net made of neurons. By language I mean words and ways folk speak or write. By ChatGPT I mean a chat bot that reads your words and writes words back.

Think of ChatGPT as a kid sent through cook school in 3 terms.

Term 1: read all the cook books in the world. The kid reads, and reads, and reads. At the end, you can hand the kid the start of a cook book line, and the kid can guess the next word. This term does not use reinforcement learning. It is deep learning on text. The brain learns to guess words. At the end of Term 1, the kid knows a lot of language, but does not yet know how to make a good meal for the task at hand. Cook books teach words. They do not teach taste.

Term 2: watch a great chef cook 100 dishes from start to end. Take notes. Try the chef’s moves on your own. The kid takes on the chef’s style. Now if you ask the kid to cook soup, the kid cooks soup that looks like the chef’s soup. But the kid is still not great. Some food comes out flat. Some comes out off. Some is not safe to eat.

Term 3: cook for guests. Put two plates of soup down at the same time: two takes on the same dish. Watch which plate the guests scrape clean. Train a reward model to guess which plate the guests would pick. Then use PPO or GRPO to shift the kid’s style toward plates the reward model rates high. That is RLHF, folded back in.

That is the plan. Term 1: read the world’s cook books. Term 2: watch a chef. Term 3: cook for guests, and take their picks to heart.

Most of ChatGPT’s skill comes from Term 1, from just text. The score loop in Term 3 is more like a last tune. It shapes the raw text brain toward help, truth, care, and a good chat style.

What to Keep

You now know the core parts: state, move, score, policy, and value.

You know the core hard task: which past move gets thanks or blame?

You know 3 splits:

1. Value-based or policy-based: price tags or rule book.

2. Model-free or model-based: no doll house or doll house.

3. On-policy or off-policy: fresh games or old tapes.

Here is a map of reinforcement learning we have seen:

Q-learning fills a table of values. An evolution strategy shakes whole brains and keeps the best. REINFORCE makes moves in high score runs more apt. The actor-critic trick pays for advantage, not raw score. PPO clips the push, so the new policy stays near the old one. GRPO uses a group mean in place of a critic. AlphaZero lifts a fast gut by self-play, tree search, and expert iteration. RLHF uses human feedback to train a reward model, then trains the chat policy.

The Shape of It

These 8 ways are not all there is. Folk who do this for a job have far more, with names and tricks this text did not touch. There is a long list of hard parts I have not shown you: how to learn when score is rare, how to plan with no map, how to share play with kin brains, how to fail safe.

But the heart of it is the same 3 words:

1. Try

2. Score

3. Change

This is one way to look at all life.

For as long as there has been life, this loop has run. The first cells did this loop. So did the fish, the lungs, the bones, the hands, the brains. Each kind of life was a try. The world gave back a score: did the kind live and breed, or not? The kinds that won made more of their own kind. The kinds that lost did not. Slow, slow, slow—but that loop built each life on Earth, you and me with the rest.

Brains do this loop too, on a much faster clock. A child learns to walk in a year. A dog learns to fetch in a day. A chat brain learns a new style in hours. The same 3 words, run on a chip in place of a cell.

Once you see the loop, you start to see it all through life. A child stands, falls, gets up, and stands more. A bird hears kin sing, tries its own song, hears it fall short, and tries once more. A kid does math, gets praise, does more math, and gets good. A shop posts a price, folk buy or pass, and the price shifts. A tale is told, some folk tell it on, and some let it die. A word catches on, or fades. A law works, or it breaks, and the law is changed.

The score need not be wise. It can be false, thin, or cruel. That is why we must choose scores with care. A bad score can make a bad world. A good score can grow skill, grace, and trust.

A raw old line says “endless forms most beautiful”. That line fits here. From blind tries and hard scores came fins, wings, eyes, songs, hands, tools, jokes, proofs, games, and minds. From fast loops on chips come nets that can read, spell, search, and chat.

No one planned all of it at the start. No one held the full map. There was just the loop: try, score, change; try, score, change; try, score, change.

That is why this field is more than a set of tricks. It is a lens. Look through it, and the world looks less like a fixed thing, and more like a vast work in flight: forms tried, forms scored, forms changed, and new forms born from old.