#alphazero

I just watched this video and it got me thinking on chess AI.

For those unfamiliar, chess AI will evaluate possible future positions of the current board, then evaluate which of them are most desirable, and then it’ll play the moves to get it there.

If a future branch for example has a lot of good but also a couple really bad boards, it’ll go with a safer branch instead. It always plays the move that maximizes it’s future state. Once it finds a forced checkmate sequence, of course it will follow that branch until it’s opponent is defeated.

~~~

However, back when chess AI was in it’s infancy and just did this, humans could still beat it for a while. The positions in the future were evaluated on if there was a forced checkmate sequence as well as material points. Different chess pieces are worth different so called material points, and by evaluating future boards on material point advantage, the AI could take all it’s opponents options and eventually win that way. But humans still sometimes won. Why?

Well, they understood something that the AI didn’t, but they couldn’t formulate what exactly that was.

So the solution: Make an AI that emulates humans.

~~~

The second generation of chess AI then proceeded to use a combination of the brute force “look for a forced checkmate sequence and material advantage“ approach, but mixed it with emulating actual human played games too, emulating the winning side, of course.

This made the AI much better at openings and endgames. And the mid-game was already it’s strong suit, no human is better at tactics than an AI that can calculate into the future that far.

After that advancement, AI completely outpaced humans. No human alive can beat such an AI anymore, and those are used to evaluate positions of human games. The most prevalent and powerful of that type of AI is called StockFish, and it has gone through countless iterations, each one improving further and further away from the human skill-cap.

But a problem remains: The developers have no real idea what it is we humans understand better, that enabled us to beat the first generation of chess AI.

They knew it was some “positional advantage“ but had no way of quantifying that, no way of making the AI understand that. Except by letting it emulate humans, who do understand and utilize that concept.

~~~

And then this AI from the video i linked came on to the scene, AlphaZero is it’s name. And from how it plays, i think it’s creators have finally found what humans did differently that allowed them to best 1st generation AI. I believe it simply changed the way it evaluates/rates how good future boards are!

And the measure it rates them by is the amount of possible legal moves the AI will have at that future board vs. the amount of possible moves their opponent will have.

The extreme case is a forced checkmate sequence: The enemy has no moves except those that lead to the king being unable to move, as well as attacked.

A checkmate.

So first of all, if AlphaZero evaluates future positions like that, it doesn’t have to make an exception for forced checkmate sequences. Those just happen to be favored naturally. No if-case needed. And also no more evaluation based on material points.

This enables the AI to apply pressure on it’s opponent, because pressure means less possibilities for them, more for the AI.

Many things GothamChess said in the video point to this being the case:

it likes long bishop diagonals

it likes it’s own king to be mobile

it doesn’t care about sacrificing pawns, and will gladly do so

it closes down positions, but usually on the enemy side

The first two points are simple to understand if it evaluates future boards on how mobile it’s pieces are supposed to be.

The third also makes sense, as pawns are just in the way of it’s own piece moves. A pawn can only move one square, a piece in way more ways. So it’ll gladly sacrifice a pawn just to make it’s own pieces able to move more freely.

The fourth point can be understood as minimizing the enemy movement options.

This way of evaluation allows AlphaZero to incorporate things like trapping pieces and developing moves into it’s natural behavior. Developing moves were previously relegated to emulating human openings, but that was because the material point evaluation didn’t account for them. They don’t have to be handled like that anymore, possible move evaluation naturally leads to the chess AI playing developing moves.

612 Frozen Hellscape Links for All Your Frozen Hellscape Needs

* In case you missed it, I posted my syllabi for the spring last week: Classics of Science Fiction, Game Studies, and Methods of Inquiry: The Mind. And just in time for my games course: Marquette announces that esports — competitive video gaming — will be a varsity sport next year.

* Another just-in-case-you-missed-it: I was on the most recent episode of Random Trek talking about Voyagerepisode…

Planes fly differently from birds. Cars move differently from horses. And computers think differently from humans. That had always been my assumption but if you needed more proof, Google’s AlphaZero program, which had previously shown novel ways of playing Go, has just learned how to play chess incredibly well. It did so in 24 hours and without studying any prior games. Instead it just played games following the rules of the movement of pieces and learned from that.

I encourage everyone to check out the technical paper on Arxiv, as it contains many fascinating insights. But today I want to focus instead on a key high level implication: Computers are not constrained to learning and thinking like humans. And for many, many tasks that will give them at an extraordinary advantage over humans. Just like mechanized transport turned out to be superior to horses in almost all circumstances.

Our big brain that was shaped by the forces of evolution is a marvel in its complexity. But it has evolved to let us deal with a great many different scenarios and try as we might, we cannot apply more than a small fraction of our brain to a specific problem (such as playing chess). And even then our speed of learning is constrained by extremely slow clock cycles (see my previous post about AlphaGo).

AlphaZero’s success should be a startling wake up call. When we developed motorized transport, we went from 25 million employed horses in the US in 1915 to 3 million by 1960 and then we stopped tracking as the number fell further. We now have the technology to free ourselves from the ridiculous demands on humans to spend their lives as machines. We can have computers and robots carry out many of those tasks. We can be free to excel at those things that make us distinctly human, such as caring for each other.

33rd Square, DeepMind Proves Its AI System's General Purposefulness

📖 What Is Reinforcement Learning (RL)?

Reinforcement Learning is an AI training method where models learn by interacting with their environment — receiving rewards or penalties based on their actions. Instead of being told what to do, RL agents discover optimal strategies through trial and error, improving performance over time.

This makes RL ideal for complex decision-making tasks like robotics, gaming, autonomous driving, and dynamic resource allocation.

⚙️ How It Works

An RL system involves three key components:

Agent: The decision-maker (AI model).

Environment: The world the agent interacts with.

Reward Signal: Feedback that helps the agent learn what’s good or bad.

The agent observes the environment, takes an action, and receives a reward. Over many iterations, it refines its policy — the strategy that maps observations to actions — to maximize long-term reward.

Popular algorithms like DQN (Deep Q-Network), PPO (Proximal Policy Optimization), and AlphaZero use deep learning to make this process scalable and efficient.

💡 Where It’s Used

🎮 Gaming: AlphaZero mastered chess, Go, and shogi purely through self-play — without human data.

🚗 Autonomous Driving: RL agents learn to make safe, adaptive driving decisions in simulated environments.

🤖 Robotics: Used to teach robots tasks like grasping, navigation, or object manipulation.

📈 Finance: Optimizing trading strategies and dynamic portfolio management based on feedback signals.

🌐 Operations & Networking: Used in traffic control, server optimization, and energy management systems.

⚖️ Why It Matters

Reinforcement Learning represents true autonomous intelligence — systems that learn to act optimally through experience, not instruction. It’s a core pillar of Artificial General Intelligence (AGI), powering breakthroughs in self-learning systems and adaptive decision-making.

🚀 Examples

🔹 DQN (Deep Q-Network): Combines Q-learning with deep neural networks for playing Atari games at human-level performance.

🔹 PPO (Proximal Policy Optimization): A stable and efficient RL algorithm used in OpenAI’s robotic and gaming experiments.

🔹 AlphaZero: Developed by DeepMind, AlphaZero taught itself chess and Go, achieving superhuman skill purely through reinforcement learning.

🧠 Pro Tip

✅ Use RL for environments with clear feedback loops or dynamic decision-making. ❌ Avoid it for tasks with limited data or no measurable reward signal — supervised or unsupervised learning may fit better there.

🔍 Summary

Reinforcement Learning is the science of learning through interaction. It enables AI to adapt, improve, and master complex real-world environments — from playing games to driving cars and controlling robots.

New Post has been published on https://thedigitalinsider.com/the-sequence-opinion-730-reinforcement-learning-a-street-smart-guide-from-go-boards-to-gpt-alignment/

The Sequence Opinion #730: Reinforcement Learning: a Street-Smart Guide from Go Boards to GPT Alignment

The walkthrough the history of RL.

Reinforcement learning (RL) is the part of AI that learns by doing. Not from a teacher with answer keys (supervised learning), and not by free-associating the web (self-supervised pretraining), but by poking the world, seeing what happens, and tweaking itself to do better next time. Think of a curious agent in a loop:

look at the world

pick an action

get a little pat on the back (or a slap on the wrist)

update itself

repeat forever

That’s RL. It’s simultaneously powerful and annoying: powerful because it can discover strategies nobody wrote down; annoying because it’s sample-hungry, finicky, and loves to “hack” whatever score you give it.

Below is a guided tour: where RL came from, the big algorithmic building blocks, the AlphaGo → AlphaZero → MuZero leap, the migration from games to reality, and how today’s frontier models use RL after pretraining (RLHF, RLAIF) to become useful and safe(-ish).

Origins: trial-and-error with a feedback loop

Early psychology noticed a simple rule: actions that lead to good outcomes get repeated. Control theory made it operational: if you can estimate how “good” a situation is, you can plan and optimize decisions over time. Computer science then wrapped this into the agent loop above and asked: how do we learn a good way to act when the world is messy, delayed, and sometimes random?