#information entropy

there's a concept called "information entropy". basically, the more complicated and alien a concept is to us (or whatever system is processing it), the more information and data is needed to understand it, so it has higher information entropy. for example: imagine one of the infamous tumblr comics. rats in a subway station, not being used to receiving love. these things are high entropy. we need the entirety of these things to know what they are and to understand them. if i showed you a random panel of the rats in a subway station comic, unless you were already personally familiar with the whole work, you wouldn't know what it was. in contrast, something like Loss, which has been twisted into probably thousands if not tens of thousands of formats and versions, is low entropy. even a simple 7 lines can convey it, because it is so entrenched in people's minds. another example of low entropy is a lot of literary themes that draw on aspects of the human experience. these are low entropy because most every person that reads them has already experienced the feelings that it draws upon.

the concept of information entropy is one that humans are fundamentally not wired to understand. because our own brains are subject to this as well. memories are not a lossless compression format. they are imperfect. we forget details, or tack on imagined pieces to fill in gaps. hell, we don't even remember actual events, save the first time we think abt them afterwards. most of the time (with a few exceptions), we remember the last time we remembered something. it's like repeatedly compressing a jpeg until it becomes a blur of pixels. eventually, we forget.

The Informational Shield: Civilization, AI, and the Deflection of Chaos

We have to distinguish thermodynamic entropy from information entropy, because they are related but not identical.

Thermodynamic entropy, from physics, concerns the dispersal of energy and the tendency of physical systems toward equilibrium. Information entropy, introduced by Claude Shannon, concerns uncertainty. A message has high information entropy when there are many possible outcomes and you do not know which one will occur. Information reduces uncertainty. Living systems seem to negotiate both.

A bacterium reduces thermodynamic entropy locally by maintaining its own organization. At the same time, it reduces informational uncertainty by sensing where nutrients are and swimming toward them rather than randomly. Humans do the same on a vastly larger scale. Suppose you hear a rustle in the bushes. Your brain immediately predicts. Is it the wind? A rabbit? A wolf? The prediction reduces informational uncertainty. Then you act. If the prediction is correct, you may avoid injury, find food, or save energy. Better information improves thermodynamic regulation.

This is why prediction is so important. Information is not collected for its own sake. In living organisms, it is almost always in the service of regulating the body.

Now consider AI. An AI system also reduces informational uncertainty. Given a question, it estimates which continuation is most probable. More generally, AI constructs models that make future observations or outputs more predictable. The AI however is not reducing informational uncertainty for itself. It is reducing uncertainty for the task or for the user.

Humans, by contrast, evolved predictive brains because prediction increased the probability that their bodies would remain alive. Of course people will build machines to help negotiate entropy. That is exactly what civilization has always done. A coat negotiates thermal entropy. A refrigerator negotiates food spoilage. A bridge negotiates gravity. Medicine negotiates biological decay. Agriculture negotiates seasonal scarcity. Writing negotiates informational loss across generations. Libraries negotiate cultural forgetting. Computers negotiate computational complexity. Artificial intelligence negotiates informational uncertainty. Seen from this perspective, technology is not something separate from evolution. It is an extension of evolution.

Evolution first produced muscles to negotiate physical challenges. Then brains to negotiate informational challenges. Then language to negotiate social challenges. Then science to negotiate explanatory challenges. Then computers to negotiate computational challenges. Then AI to negotiate informational complexity beyond unaided human cognition. Each step expands the ways organisms manage the constraints imposed by reality.

Civilization itself can be understood as a gigantic anti-entropic process. Not anti-entropic in the sense of violating the Second Law, because civilization still consumes enormous amounts of energy and produces even more entropy overall. Rather, civilization creates local islands of increasing organization by exporting greater entropy elsewhere. A city is an island of extraordinary order, maintained only through continuous flows of energy, materials, food, and information. A university is an island of organized knowledge. A hospital is an island where biological order is restored. An AI system is an island where informational uncertainty is reduced.

From that perspective, information becomes almost as fundamental as energy. Energy allows work. Information guides work. Without energy, nothing can happen. Without information, energy is wasted.

Information Entropy:

H(X) = E[I(X)] = E[-ln(P(X))]