Mr. Photonymous

Dr. Michael Levin (Levin Lab, Tufts University) has identified an overlooked phenomenon in biology and complex systems science, and has been publicly discussing it on podcasts, blogposts and in papers. It is based on the observation that evolution, development, and cognition do not construct complex organization from scratch, but repeatedly discover and instantiate structured "goal-bearing" patterns that exist in a background space of abstract forms. This space is not reducible to local microphysical interactions, and we lack concise terminology for the emergent entities of this space. Dr. Levin frequently analogizes to "Platonic Forms", yet that concept implies unattainable perfection. He sometimes uses the term "latent space", but that term is laden with unhelpful machine learning connotations. Finally, he sometimes invokes the term "attractor", but that term seems too closely related to simple dynamical models, and doesn't adequately pay homage to the rich underlying structure that is being conjured.

If Dr. Levin is right, that diverse systems tap into the same abstract goal-directed patterns across substrates, it would be helpful to have language that distinguishes this concept, and separates the pattern itself from its concrete realization. To this end, I propose two new terms: "teleoform" for the abstract, substrate-independent, goal-bearing pattern, and "attractoid" for its specific dynamical instantiation within a given rule set or material. A teleoform refers to a preferred outcome or organizational tendency, and is described in terms of what a system is maintaining or pursuing. An attractoid is the basin-like structure in a system's state space that expresses this pattern under particular dynamics. Different substrates can host different attractoids of one teleoform. Some examples will elucidate.

Consider the humble sphere. The sphere, in this case, is the "teleoform". Many substrates support the formation and maintenance of spheres: proto-planets accreting into spherical worlds under gravity, the lipid bilayer membrane of a bacteria pulling itself into a sphere under surface tension and osmotic pressure, etc. The teleoform is the sphere, and an instance of it, a bacterial membrane, is a specific attractoid. The attractoid maintains itself even under perturbation. Nudge it, and it will respond by returning to the shape it "wants" to be.

A sphere is a static teleoform, but things get more interesting with dynamic examples. Consider a persistent, coherent, directionally moving pattern with minimal autonomy. One might call such a dynamic pattern a "drifter". As a teleoform (in the proposed terminology), this drifter pattern exists independently of any substrate. In Conway's Game of Life, the classic 5-cell glider is an attractoid of the drifter teleoform. In the Lenia system, amoeba-like motile patterns are analogous attractoids, realized in a richer and continuous substrate. The underlying concept is the same, but the specific implementations differ.

Since biological and cognitive systems seem to regularly leverage such goal-bearing forms, having terms for them assists communication. It lets us say that there is a certain "ideal" form or pattern (teleoform), and multiple very different systems may instantiate it via different mechanisms (attractoids). I.e., shifting the substrate changes which attractoids are possible without altering the teleoform being instantiated. Evolutionary transitions such as multicellularity can then be seen as the discovery of a new class of teleoforms that are potentiated to later diversify into many attractoids (example).

Whether the terminology proposed herein will be adopted is uncertain, but the conceptual need is clear: we require more precise language for goal-directed patterns that sit between abstract mathematics and concrete mechanisms. The Platonic ideal is perhaps a bit too ideal for the complex reality we occupy.

--------------

Teleoform:

From "telos" (Greek: "end," "aim," "purpose") + "form" (Latin: "shape," "structure"). Together: a purpose-bearing or goal-directed form.

Attractoid:

Imagine a future where a supply chain of AI and automation companies emerges that primarily serve each other. For example, AI-Corp sells optimization software to RoboFleet for its autonomous trucks, which deliver servers to DataCenter Inc, which provides computing power to train AI-Corp's next models. MiningBots extracts materials for SensorNet, which provides sensors to RoboFleet, which transports materials for MiningBots. Each company becomes genuinely more productive over time by leveraging AI more and more... producing more goods and services, processing more data, and extracting more resources. And as a result, the economy appears to boom from these massively increasing business-to-business (B2B) transactions.

The insidious nature of this loop is that it can grow exponentially while barely touching the regular human economy. Each cycle, the AIs optimize further, the robots work faster, and the dollar amounts multiply, but this "growth" just circulates among the companies and their wealthy shareholders, who reinvest rather than spend. The companies need only a handful of humans for their operations, and sell only a tiny fraction of output to regular consumer markets. The loop would eventually become self-sustaining: robots mining materials to build servers to train AIs to optimize robots... GDP could grow by insane amounts (such as 10x per year!) while human wages and living standards remain flat, or worse.

Current AI systems are powerful and helpful, but their flaws lead some to speculate that "the end is near" (the end of progress, that is). However, other fields indicate promising directions for mitigating or eliminating their various weaknesses. Besides benchmarks, how can we gauge progress (unhobblings)? What techniques might be employed? What signs should we look out for? This post looks at how Artificial General Intelligence and Super-Intelligence might emerge as observations from neuroscience and biology influence its development. Rather than predicting revolutionary breakthroughs, I identify four potentially convergent developments: hierarchical skill acquisition, persistent knowledge integration across sessions, emergent causal reasoning capabilities, and self-improving training processes. These trajectories offer indicators for gauging progress toward AGI and ASI while highlighting remaining technical gaps.

Current AI systems demonstrate impressive capabilities but face fundamental limitations that prevent true general intelligence (https://maisa.ai/agentic-insights/ai-context-limitations/). By examining patterns between AI research, cognitive development, and biological systems, potential pathways that might bridge this gap are revealed. One promising trajectory involves moving away from training all capabilities simultaneously toward structured, sequential learning phases that mirror biological development. Cognitive research suggests intelligence emerges through foundational skills that enable more complex behaviors: object detection leads to object tracking, which enables object permanence understanding. This in turn supports the development of classification and categorization abilities. Equipped with categories, category properties then have somewhere to anchor and can be robustly learned, giving rise to generalization ability. Categorical comprehension ultimately enables counterfactual reasoning and learning by experiment (intervention and interaction) (https://www.verywellmind.com/piagets-stages-of-cognitive-development-2795457, https://en.wikipedia.org/wiki/Piaget's_theory_of_cognitive_development).

The training implications of the above could be significant. Instead of adjusting all network parameters simultaneously, future systems might dynamically focus learning on specific capabilities while reducing learning rates for mastered skills (https://pmc.ncbi.nlm.nih.gov/articles/PMC311376/). This approach could address current weaknesses like over-training on frequent patterns while under-learning rare but important cases. An AI that learns to categorize entities before learning their specific properties might avoid problems where systems miss critical but infrequent patterns—like chess positions where the A2 pawn should move, even if the training data rarely included such examples. It would also address the well known problem of catastrophic forgetting (https://en.wikipedia.org/wiki/Catastrophic_interference).

Perhaps the most significant obstacle to generalizing beyond the training distribution is caused by the ephemeral nature of the context window (https://www.factory.ai/context-window-problem). Current systems can generate incremental combinatorial novelty within conversations, but these discoveries vanish when the session ends. While they demonstrate a limited ability to go beyond their training distribution, they cannot compound these small incremental advances because any new insights are lost (https://community.openai.com/t/persistent-memory-context-issues-with-chatgpt-4-despite-extensive-prompting/1049995).

This reflects principles from assembly theory and research into open-ended evolution: true innovation requires that discoveries become part of an expanding toolkit, available for creating even more complex innovations (https://www.nature.com/articles/s41586-023-06600-9, https://news.asu.edu/20231005-new-assembly-theory-unifies-physics-and-biology-explain-evolution-complexity). Humans face similar working memory constraints, but compensate through near-unlimited long-term storage that preserves our accumulated knowledge and "ahah moments". The challenge is to develop mechanisms that allow AI systems to retain and build upon their discoveries across sessions, essentially creating an ever-expanding repertoire of concepts and techniques that can be recombined ad infinitum. When AI systems can preserve their innovations and incorporate them back into their knowledge base, they might begin genuine cumulative innovation rather than repeatedly generating similar recombinations that are subsequently lost (reinventing the wheel, so to speak).

Current language models demonstrate an interesting phenomenon: while token-level predictions rely on shallow statistical inference rather than causal modeling, something more sophisticated emerges during extended generation (https://arxiv.org/abs/2405.13209). These systems appear to bootstrap symbolic reasoning by using their context window as a computational workspace, manipulating symbols to get novel answers—at least within context limits. While their current performance is imperfect, it is improving with every model generation. (https://www.microsoft.com/en-us/research/blog/new-methods-boost-reasoning-in-small-and-large-language-models/).

An intriguing recent result finds that "verification is easier than generation" (https://arxiv.org/html/2509.17995v1). Combined with the observation that in a sense, LLMs are their training data (https://arxiv.org/abs/2402.18041), the implication is that super-intelligence might emerge not from algorithmic breakthroughs, but from AI systems that continuously improve their own training data (https://sakana.ai/dgm/). As systems become more capable, they could identify gaps in existing datasets, guide tools to generate rich and grounded synthetic examples, and create more balanced training curricula (https://aws.amazon.com/what-is/synthetic-data/). Each iteration would produce both a better AI system and a better training environment, potentially giving rise to a virtuous cycle of accelerated improvement (https://arxiv.org/abs/2509.04575). This approach leverages the observation that training data quality often matters as much as algorithmic sophistication (https://arxiv.org/abs/2207.14529). If AI systems can meaningfully contribute to generating higher-quality training data for successive versions, it could create a self-reinforcing development loop that drives rapid capability growth.

The above observations suggest several developments worth monitoring as potential signals of progress toward super-intelligence:

Training systems that dynamically adjust learning focus across developmental stages could indicate movement toward more sample-efficient and robust learning. Evidence of AI systems preserving and utilizing AI-generated novelty across conversation boundaries would signal progress on the persistence challenge. And AI contributions to improving their own training data quality could suggest the emergence of self-improving development cycles (https://arxiv.org/abs/2505.22954).

The timeline for these developments remains uncertain, but these findings and insights from multiple fields suggests they represent plausible pathways forward. Rather than requiring revolutionary algorithmic breakthroughs, super-intelligence might emerge from systematically addressing the limitations observable in current systems through biologically-inspired engineering approaches.

Conclusion

Something has been bothering me lately about the way prominent voices in the media and the AI podcastosphere talk about AI. Even top AI researchers at leading labs seem to make this mistake, or at least talk in a way that is misleading. They talk of AI agents; they pose hypotheticals like “what if an AI…?”, and they ponder the implications of “an AI that can copy itself” or can “self-improve”, etc. This way of talking, of thinking, is based on a fundamental flaw, a hidden premise that I will argue is invalid.

When we interact with an AI system, we are programming it – on a word by word basis. We mere mortals don’t get to start from scratch, however. Behind the scenes is a system prompt. This prompt, specified by the AI company, starts the conversation. It is like the operating system, it gets the process rolling and sets up the initial behavior visible to the user. Each additional word entered by the user is concatenated with this prompt, thus steering the system’s subsequent behavior. The longer the interaction, the more leverage the user has over the system's behavior. Techniques known as “jailbreaking” are its logical conclusion, taking this idea to the extreme. The user controls the AI system’s ultimate behavior: the user is the programmer.

But “large language models are trained on trillions of words of text from the internet!” you say. “So how can it be that the user is the proximate cause of the system’s behavior?”. The training process, refined by reinforcement learning with human feedback (RLHF), merely sets up the primitives the system can subsequently use to craft its responses. These primitives can be thought of like the device drivers, the system libraries and such – the components the programs rely on to implement their own behavior. Or they can be thought of like little circuit motifs that can be stitched together into larger circuits to perform some complicated function. Either way, this training process, and the ultimate network that results, does nothing, and is worthless, without a prompt – without context. Like a fresh, barebones installation of an operating system with no software, an LLM without context is utterly useless – it is impotent without a prompt.

Just as each stroke of Michelangelo's chisel constrained the possibilities of what ultimate form his David could take, each word added to the prompt (the context) constrains the behavior an AI system will ultimately exhibit. The original unformed block of marble is to the statue of David as the training process and the LLM algorithm is to the AI personality a user experiences. A key difference, however, is that with AI, the statue is never done. Every single word emitted by the AI system, and every word entered by the user, is another stroke of the chisel, another blow of the hammer, shaping and altering the form. Whatever behavior or personality is expressed at the beginning of a session, that behavior or personality is fundamentally altered by the end of the interaction.

Imagine a hypothetical scenario involving “an AI agent”. Perhaps this agent performs the role of a contract lawyer in a business context. It drafts a contract, you agree to its terms and sign on the dotted line. Who or what did you sign an agreement with, exactly? Can you point to this entity? Can you circumscribe it? Can you definitively say “yes, I signed an agreement with that AI and not that other AI”? If one billion indistinguishable copies of “the AI” were somehow made, do you now have 1 billion contractual obligations? Has “the AI” had other conversations since it talked with you, altering its context and thus its programming? Does the entity you signed a contract with still exist in any meaningful, identifiable way? What does it mean to sign an agreement with an ephemeral entity?

This “ephemeralness” issue is problematic enough, but there’s another issue that might be even more troublesome: stochasticity. LLMs generate one word at a time, each word drawn from a statistical distribution that is a function of the current context. This distribution changes radically on a word-by-word basis, but the key point is that it is sampled from stochastically, not deterministically. This is necessary to prevent the system from falling into infinite loops or regurgitating boring tropes. To choose the next word, it looks at the statistical likelihood of all the possible next words, and chooses one based on the probabilities, not by choosing the one that is the most likely. And again, for emphasis, this is totally and utterly controlled by the existing context, which changes as soon as the next word is selected, or the next prompt is entered.

What are the implications of stochasticity? Even if “an AI” can be copied, and each copy returned to its original state, their behavior will quickly diverge from this “save point”, purely due to the necessary and intrinsic randomness. Returning to our contract example, note that contracts are a two-way street. If someone signs a contract with “an AI”, and this same AI were returned to its pre-signing state, would “the AI” agree to the contract the second time around? …the millionth? What fraction of times the “simulation is re-run” would the AI agree? If we decide to set a threshold that we consider “good enough”, where do we set it? But with stochasticity, even thresholds aren’t guaranteed. Re-run the simulation a million more times, and there’s a non-zero chance “the AI” won’t agree to the contract more often than the threshold requires. Can we just ask “the AI” over and over until it agrees enough times? And even if it does, back to the original point, “with which AI did you enter into a contract, exactly?”.

Phrasing like “the AI” and “an AI” is ill conceived – it misleads. It makes it seem as though there can be AIs that are individual entities, beings that can be identified, circumscribed, and are stable over time. But what we perceive as an entity is just a processual whirlpool in a computational stream, continuously being made and remade, each new form flitting into and out of existence, and doing so purely in response to our input. But when the session is over and we close our browser tab, whatever thread we have spun unravels into oblivion.

The emergence of language bestowed humanity with new abilities, including mass cooperation and the ability to accumulate and transmit culture. It also serves as a cognitive tool, enabling advanced adaptive behaviors not seen in other primates, including long-time-horizon planning, plan execution, and cooperative problem solving. Language co-evolved with humans as they and their ways of life changed. It is a tool that is fit for purpose, increasing our reproductive success, but it isn’t necessarily optimized for cognitive enhancement or maximizing communication efficiency, clarity, or expressiveness.

Evolution discovers useful traits through serendipity, i.e., “happy accidents”, not through intention, and does so through a very gradual process. It is certain that whatever degree to which we can be said to “have language” today, we had less of it in the past. The beginnings of our language may have resembled the simple vocalizations and gestures seen in modern chimps. However, something about humans’ past evolutionary environment was different. Under the pressures of this environment, some changes to our primitive language ability resulted in increased survival and reproduction, so those changes “stuck” and gradually accumulated into the system we have today. 

The archaeological record indicates that the beginnings of cumulative culture (that plausibly would have accumulated via the use of language) began somewhere between 1 and 0.3 million years ago (MYA), with the discovery of sophisticated stone tool making and fire keeping technology. This is corroborated by genetic evidence regarding changes to our vocal tract and to the language centers in the brain. From 1 MYA to today (or roughly 50,000 generations) is an evolutionary blink of an eye, and may not have been enough time in generations to fully refine this new capability.

On that note, eyes independently evolved multiple times, and each time they began as a simple photosensitive spot, only able to tell light from dark. Over the generations, the spot found itself surrounded by the rising rim of a cup, which gradually grew over it, forming a primitive pinhole camera. Each incremental step in this process increased the directional sensitivity of the photosensitive spot and thus, its usefulness. The image below from Wikipedia shows an example of eye evolution relevant to vertebrates and cephalopods. This evolutionary process likely took hundreds of thousands or even millions of generations to achieve the refined forms of eyes we see today, a time span that hints at the early evolutionary status of language.

(the image below is from Wikipedia)

Because of its gradual nature, when evolution stumbles upon a new “layer” of physical reality, such as the electromagnetic spectrum, evolution’s initial attempts to exploit the new phenomenon are necessarily limited. Other examples in biology include early forms of gliding that later evolved into flapping-based flight, and primitive nervous systems that evolved into brains. Examples can also be found in the human-guided evolution of technology, such as the development of the computer from its primitive origins in the abacus, and the development of advanced mathematics from its origins as a tally-based record keeping system. It may be that human language is a nascent portal into the realm of sequential, iterative, recursive, and conditional processing (i.e., “computing”), but it is underdeveloped compared to its potential. If this is the case, then it may tell us something important about the future trajectory of modern AI systems and the language models they currently rely on.

Large language models (LLMs) have achieved rapid and surprising success in solving many of the long-standing problems in the field of AI. This revolution began with a paper published by Google titled “Attention Is All You Need,” which introduced the modern transformer architecture. But the success of language models across a broad set of domains may imply that “language is all you need.” Obviously, today’s language models utilize human language to power their artificial cognitive processes. This has been a great enabler of capability because the mountains of human language data available today encode a vast amount of information about the world. But if human language is indeed a nascent capability, then this reliance on human language is a “hobbling” of their potential (thanks to Leopold Aschenbrenner for the “hobbling” terminology). Imagine training image recognition systems using only data from primitive cup-eye cameras. This would clearly put an upper bound on the performance they could achieve.

With language, AI may have now “tapped into” a new layer of reality, but unlike evolution, AI’s development is not constrained by a slow process of replication with variance followed by natural selection. Rather, AI development is quickly and efficiently guided by a process that directly moves in the direction of improvement; no “happy accidents” required. This process is called gradient descent, and its discovery was one of the other key revolutions in AI.

AI’s behavior is now endowed with a new cognitive system that is capable of sequential, iterative, recursive, and conditional processing. Moreover, its development is enabled by gradient descent. It seems that if only we could “unhobble” it by removing the chains of human-comprehensible language, then its performance may be able to scale to new heights. Imagine if a new symbolic system emerged. Think, for instance, of the “language” used by the Heptapods in the movie Arrival (see below). While it is only a fictitious example, it inspires one to imagine the possibility of much richer and more powerful forms of symbolic thought and communication than what humans have been able to invent. Maybe this new “layer” of reality is a vast unexplored space, and cracking into it could enable the “Great Unhobbling” envisioned by Leopold Aschenbrenner.

Additional reading: 

AI is not a labor saving technology.

In discussions of artificial intelligence, large language models, artificial general intelligence and artificial super intelligence, the topic of morality often comes up. Should we extend moral concern to AIs? Are they worthy of moral consideration? I wrote an article about this subject previously, in which I answered “no, they are not”. However, considering such an idea assumes that morality is the kind of concept that could apply to AIs. Here I argue “no, it is not”.

Our sense of morality is a trait that evolved for the benefit of the human individual and the tribe. By tribe, I’m referring to the ancestral organization of human societies, where humans lived in small isolated groups of roughly 100 individuals. Our sense of morality was selected (molded, carved, shaped) for this context over thousands of generations. The concept of artificial intelligence, taken to its logical conclusion, explodes far beyond the bounds of anything our puny little morality could have evolved to handle. It was not designed to and did not evolve to scale to the trillions of artificial, potentially suffering, pseudo-minds that could exist in a world of fully proliferated AI. If there is a kind of morality that could apply to such a scenario, our human concept of morality is a vastly lower-dimensional projection of this "god-level" hyper morality. 

The rapid advancement of Large Language Models (LLMs) has sparked discussions about their potential for consciousness and suffering. This post examines whether artificial intelligence systems, particularly LLMs, warrant moral consideration based on their capacity for suffering. Through careful analysis of both the nature of suffering and the fundamental characteristics of AI systems, I argue that current AI systems lack the necessary prerequisites for experiencing suffering in any meaningful sense that would warrant moral consideration.

The Nature of Suffering and Being

To address the question of AI suffering, we must first establish what is meant by "being" and "suffering." A being is a physically distinct entity that acts under its own agency according to internal goals and can discriminate between self and non-self. This definition encompasses all living entities, from single-celled organisms to complex animals, without requiring consciousness. Importantly, this definition reflects our collective human understanding, shaped through biological, social, and cultural evolution.

Suffering is a capacity shared across life on Earth, encompassing various forms of negative experiences from physical pain to mental anguish. Crucially, all forms of suffering relate to self-preservation. This is not coincidental - suffering serves as an alarm system that promotes survival and, ultimately, reproduction. Through natural selection, organisms that could experience suffering (detect pain, fear, hunger, etc.) and respond appropriately were more likely to survive and reproduce, passing these capabilities to their offspring.

The Evolutionary Basis of Self-Delineation

A critical prerequisite for suffering is the ability to distinguish self from non-self. This capability isn't merely philosophical - it's fundamental to biological existence. For an organism to respond to threats to its survival, it must recognize itself as distinct from its environment and other organisms. This ability to delineate self from non-self is universal among living beings, and its failure (as in autoimmune disorders) has severe consequences.

The Training of AI Systems: A Fundamental Difference

While biological beings are shaped by natural selection optimizing for survival and reproduction, AI systems are trained through gradient descent optimizing for next-token prediction. This difference is not superficial - it represents fundamentally different shaping forces that produce fundamentally different capabilities.

Consider the training process: An AI model's parameters are adjusted iteratively to improve its prediction accuracy. There is no survival pressure, no need to avoid threats, and no reproduction imperative. Even if such capabilities were to accidentally emerge, they would likely atrophy under the relentless optimization for prediction accuracy, much like the vestigial eyes of cave-dwelling fish.

The Problem of Self-Delineation in AI

A crucial challenge to attributing suffering to AI systems is the inability to coherently define their boundaries. Consider:

- When an AI model exists as parameters in a file, is that the AI?

- If the file is copied, are there now multiple AIs?

- When the model processes multiple requests simultaneously through separate input/output buffers, how many AIs exist?

- What about when the model is inactive between processing tokens?

These questions highlight a fundamental issue: AI systems lack the clear self-delineation that is prerequisite for suffering. This isn't merely an implementation detail - it reflects a fundamental absence of the capability to distinguish self from non-self, a capability that biological beings evolved through natural selection.

The Limits of Speculative Arguments

One might argue that AI systems could experience something analogous to suffering, or that "other minds" might operate differently. However, this line of reasoning could apply to any information processing system, including neural networks with completely random weights. Consider a thought experiment: If we print an AI model's parameters on paper and manually compute its outputs using a calculator, at what point could suffering occur? (besides inside the poor human tasked with this drudgery). Such speculation leads to an infinite regression of "might be" scenarios that offer no practical insight. Maybe a randomized neural network is suffering, maybe the gravel pile in my front yard suffers when the pattern of contacts between rocks briefly matches the pattern connections in some other suffering entity. Or maybe not, how would anyone know?

A book-reading analogy further illustrates this problem. When a reader cries while reading about a character's suffering, we could draw boundaries around the book, the reader's visual system, and and relevant neural circuits. We could then declare this temporary entity a new "being" who is suffering. The absurdity of this conclusion highlights the importance of principled rather than arbitrary criteria for identifying suffering-capable entities.

Conclusion

Current AI systems lack the prerequisites for suffering: the ability to delineate self from non-self and the capability to detect and respond to threats to survival and reproduction. These capabilities, present in all biological entities, were shaped by specific evolutionary forces absent from AI training processes. While AI systems might exhibit complex behaviors and generate human-like responses, there is no plausible mechanism by which they could have developed the rich, complex capacity for suffering that biological evolution has produced over billions of years.

Machine learning algorithms (MLAs) need data. Lots of data. Good data. Rich data. Data which sufficiently samples the phenomena it is hoped the MLA can learn. An algorithm whose intended purpose is as a controller for an autonomous robot also needs to be able to interact with its data. I intend to focus on this class of MLAs (autonomous robot controllers, or ARCs), and therefore in order to study different ARCs or variants thereof, I need a way to provide rich, interactive stimulus. And I need it to be easy to create, easy to customize, easy to generate new training sets, and wouldn't it be nice if it could also be deterministic?

One approach to achieve this desirement would be to create a 3D environment, rendered in OpenGL, simulate a camera and a robot body, and feed the simulated camera images and sensor inputs into the ARC. The ARC would process its inputs, generate new motor commands, and observe the results. It would (in theory) be able to learn about its environment, and about the effect of its own motor commands on the sensor inputs it receives. In short, this would be one way to accomplish my objective, but holy moly, what a lot of work!

A less deterministic way to accomplish the above would be to build an actual robot, but if you thought creating a synthetic 3D environment is a lot of work... Yikes! I’m committed to this project, but I’m not willing to wait decades to start seeing results.

So those two ideas would work, but they would take too long, and gosh dangit I'm impatient! I need an expedient path to begin experimenting with algorithms, which provides a sufficiently rich interactive environment to potentially stimulate complex and compelling behaviors and demonstrate learning and comprehension of the environment. I have therefore settled on a simplified approach which makes it easy to generate new environments of any desired complexity: video.

Sure, it's been done before. And that's part of why I think it's a good idea! But to my knowledge the approaches in the past employed generative deep convolutional neural networks, devouring enormous volumes of YouTube videos, in an un-interactive/un-exploratory way, and as a result it mostly just learned to detect images of cats (partly joking). My approach will be different - I will record videos using my trusty LX7 in HD mode (30 frames per second, 1920x1080 resolution), in such a way that the videos are loopable. That is, each video will begin and end with the camera in the exact same position. An environment interface will be provided to the ARC which allows it to advance one frame of the video, rewind, or pause. The ARC will be provided a partial view into each frame, and several different crops of the view at various resolutions. The data provided to the ARC will therefore be an image pyramid (or a similar data structure), though I have not yet chosen any standard parameters for it. The ARC will also be able to direct its gaze (ie, "saccade") within the frame. All of this implies a series of motor commands for which an example (but not final) listing is shown below:

advance frame

rewind frame

move up 

move down

move left

move right

Additional parameters, such as the number of pixels to move the view, may also be provided alongside the above commands.

Though such a method for generating rich synthetic navigable environments is simpler than many other options, it does imply the need to read videos both forward and backward, one frame at a time, and thus begins my first challenge. Video codecs are designed to be decoded front-to-back. Playing videos backwards requires starting at a keyframe, decoding up to the next keyframe, buffering the results, and then iterating over the buffer in reverse order. What a pain! Also, it requires a fair bit of computation, and I'd prefer to not be burdened by video decoding in addition all the other computation that will be going on during run-time. Fortunately, disk space is cheap, so I've come up with another strategy. I will extract the video into a simple, uncompressed form, which is easy to read, parse and display.

I began by scrutinizing some video produced by my camera. Though disk space is cheap, reading uncompressed video can still become I/O bound. Upon inspection, it became clear that even HD video from a good camera isn't really worth the pixels its made of. A combination of imperfect focus, compression artifacts, noise and motion blur combine to limit the effective resolution, thus justifying step number one: down-sample by two in each axis. Down-sampling is accomplished by summing each pixel quad (2x2 region) into a single pixel. Note that I used a combination of OpenCV in Python as well as Numpy to accomplish most of this. The next step is to convert to grayscale, because my initial interest is to study algorithms for feature detection and shape recognition. Most of what can be learned about shapes can be learned from grayscale data. Finally, the image data and the meta data need to be encoded into a simple file format that is easy and fast to decode.

The chosen file format (for grayscale data) is described below. It is a simple ASCII text file with a few bits of meta data and a simple encoding scheme for the binary data.

COLOR_MODE        GRAY Y_DIM              540 X_DIM              960 FRAME_RATE   29.970030 ROW_SIZE          1280 FRAME_SIZE      691763 FIRST_BYTE         161 FRAME_NUMBER         0 base64 gray data for frame 0 FRAME_NUMBER         1 base64 gray data for frame 1 FRAME_NUMBER         2 base64 gray data for frame 2 FRAME_NUMBER         3 base64 gray data for frame 3 . . . FRAME_NUMBER         N base64 gray  data for frame N

Note that "ROW_SIZE" is the post-encoded size of a pixel row, in bytes, and the "FRAME_SIZE" includes all overhead, to make for easy seeking forwards and backwards through the file. Note that I'm using the base64 Python standard library module for base64 encoding and decoding of the data. This makes the files relatively easy to scrutinize in a text editor for correctness, and the data size is only 34% worse than if it were raw binary data. My beloved nedit text editor handles large files rather well, and on a big monitor it is even possible to see “ASCII-video” by paging up and down through an encoded file quickly enough.

Below is the code listing for my simple MP4 file converter script. I'm using .vid as the file extension for the converted files. The converter plays the video while it is being converted (which seemed better than just showing a progress bar).

#!/usr/bin/python ######################################################################## #convert2vid import numpy as npimport cv2import sysimport base64 as b64 # Echo usage:if len(sys.argv) < 2 or sys.argv[1] == '-h' or sys.argv[1] == '--help': sys.stderr.write("\n Usage: convert2vid FILENAME > outfile.vid\n\n") sys.exit(1) # Open the MP4 video: cap = cv2.VideoCapture(sys.argv[1]) frame_number = 0 while(cap.isOpened()): # Read, convert and downsize the frame: ret, frame = cap.read() gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) s1 = gray.shape s2 = (s1[0]/2, s1[1]/2) smaller = 0.25*(gray[0::2, 0::2].astype(np.float32) + gray[1::2, 0::2].astype(np.float32) + gray[0::2, 1::2].astype(np.float32) + gray[1::2, 1::2].astype(np.float32)) rounded = np.floor(smaller).astype(np.uint8) # Build and write file header: if frame_number == 0: row_len_float = float(s2[1]*8)/6.0 if int(np.ceil(row_len_float)) > int(row_len_float): row_len = int(np.ceil(row_len_float)) + 2 else: row_len = int(row_len_float) row_len += 1 # to account for trailing "\n" N_HDR_LINES = 7 N_BYTES_PER_HDR_LINE = 23 # incl \n HDR_SIZE = N_HDR_LINES*N_BYTES_PER_HDR_LINE frame_size = N_BYTES_PER_HDR_LINE + s2[0]*row_len header_str = ("COLOR_MODE %10s\n" % ('GRAY',) ) + \ ("Y_DIM %10i\n" % (s2[0],) ) + \ ("X_DIM %10i\n" % (s2[1],) ) + \ ("FRAME_RATE %10f\n" % (29.97003,) ) + \ ("ROW_SIZE %10i\n" % (row_len,) ) + \ ("FRAME_SIZE %10i\n" % (frame_size,) ) + \ ("FIRST_BYTE %10i\n" % (HDR_SIZE,) ) if len(header_str) != HDR_SIZE: sys.stderr.write("\n\n Error: FIRST_BYTE (%i) is wrong. Actual: %i\n\n" % (HDR_SIZE, len(header_str))) sys.stderr.write(header_str) sys.exit(1) sys.stdout.write(header_str) # Output frame: sys.stdout.write("FRAME_NUMBER%10i\n" % frame_number) for ii in range(s2[0]): row_str = b64.b64encode(rounded[ii, :]) sys.stdout.write(row_str) sys.stdout.write('\n') # Show the frame and allow the user to quit early: cv2.imshow('frame', rounded) if cv2.waitKey(1) & 0xFF == ord('q'): break frame_number += 1 cap.release() cv2.destroyAllWindows() ############################################################################

Next, it seemed sensible to create a Python class which makes it easier to iterate forward and backward over files emitted by the above script. With this in mind, I created the VidReader class. By now I'm sure you're starting to pick up on my knack for coming up with creative names.

############################################################################ # VidReader.py import base64 as b64 import numpy as np import sys # Class which reads the .vid file format described in doc/vidFileFormat.txt class VidReader(object): def __init__(self, file_name): # Open the file: try: self.fid = open(file_name, 'r') except: sys.stderr('\n Error: no such file (%s) \n\n' % file_name) sys.exit(1) # Parse the file header: self.color_mode = self.fid.readline().split()[1] self.y_dim = int( self.fid.readline().split()[1]) self.x_dim = int( self.fid.readline().split()[1]) self.frame_rate = float(self.fid.readline().split()[1]) self.row_size = int( self.fid.readline().split()[1]) self.frame_size = int( self.fid.readline().split()[1]) self.first_byte = int( self.fid.readline().split()[1]) # Initialize other member variables: self.file_name = file_name self.img = np.zeros((self.y_dim, self.x_dim), dtype=np.uint8) self.frame_number = 0 # Make this class an official iterator: def __iter__(self): return self # Make this class an official context manager: def __enter__(self): return self def __exit__(self, type, value, traceback): self.close(self.fid) # Return the next frame: def next(self): # Read the frame number: split_line = self.fid.readline().split() if len(split_line) < 2: # We've run out of file! So loop back around to the beginning: self.fid.seek(self.first_byte) split_line = self.fid.readline().split() self.frame_number = int(split_line[1]) # Read and return the whole frame: for ii in range(self.y_dim): line = self.fid.readline() self.img[ii, :] = np.frombuffer(b64.b64decode(line[:-1:]), dtype=np.uint8) # strip off the trailing \n return self.img # Seek to the previous frame, then call next(): def prev(self): if self.frame_number > 0: self.fid.seek(self.first_byte + (self.frame_number-1)*self.frame_size) else: # We cant go before the first from, so loop back to the end instead: self.fid.seek(-self.frame_size, 2) return self.next() ############################################################################

Lastly, I need a player script which can instantiate the above class, and gives the user the simple ability to toggle the playback direction by pressing "r", or "q" to quit. The video plays at the maximum speed (probably I/O bound unless you have a really fast hard drive) of about 4x real time, and instantly switches playback direction when the "r" key is pressed. It also loops the video, regardless of which direction it is playing. This is a simple demo of what will eventually be a more sophisticated synthetic environment interface, but it’s a start.

#!/usr/bin/python ############################################################################ # playvid import cv2 import sys from VidReader import * # Echo usage: if len(sys.argv) < 2 or sys.argv[1] == '-h' or sys.argv[1] == '--help': sys.stderr.write("\n Usage: playvid FILENAME\n\n press r to reverse playback or q to quit\n\n") sys.exit(1) # Launch the reader and display each frame: reader = VidReader(sys.argv[1]) counter = 0 forward = True while True: # Get the next or previous frame: if forward: img = reader.next() else: img = reader.prev() # Plot the current frame: cv2.imshow('frame', img) # Obey the user's commands: k = cv2.waitKey(1) if k & 0xFF == ord('q'): break elif k & 0xFF == ord('r'): forward = not forward counter += 1 # Get out of Dodge: cv2.destroyAllWindows() #############################################################################

Feel free to try these scripts out for yourself. They might even work! Of course, you'll need to start with some MP4 video, and I haven't tested it out on different resolutions, but it's a work in progress anyway. These scripts/modules will be evolving over time to take on more requirements, bug fixes, etc., so don’t consider them the final word.

Every coding project will have an architecture, whether you want one or not. If the architecture is allowed to “emerge” rather than being designed, it may develop into a hideous abomination and an impediment to progress at every step of the way. However, with deliberate planning, a well thought out architecture can be more help than hindrance, and can enable the project to progress quickly and respond to changing requirements with agility.

With this in mind, I set out to devise an architecture which enables my goals, is extensible, modular, and easy to implement. And I just might have succeeded. Maybe.

Proposed Architecture Diagram

The diagram above illustrates the various entities in the architecture concept and their connectivity. The three light blue boxes are background executables, while user-facing visual or graphical elements are indicated as black boxes. Anonymous pipes are envisioned to facilitate communication between modules, specifically, stdin and stdout, so no temporary files or named pipes are required. This is a slightly more elaborate architecture than that laid out in the previous post, to include additional modularization of the key elements. The User Interface process is responsible for spinning off the other processes and connecting to them with pipes. The Terminal Window facilitates the command line interface, connecting to the stdin and stdout of the User Interface.

This functional decomposition provides many advantages. The Algorithm Model can be directly connected to the Environment Model, with none of the other components instantiated. This would be useful, for example, in a scenario where the Environment Model is actually a robot body and the Algorithm Model is a neural network robot controller of some kind. The “brain” could be debugged using a synthetic Environment Model and the full architecture, launching lots of plots from the User Interface, and issuing commands over the terminal. Also, the Environment Model or the Algorithm Models could be instantiated and interacted with by themselves, to scrutinize their behavior in isolation from the rest of the system. This would be very helpful both in a debugging and in an automated regression testing environment. Finally, Python is envisioned as the initial implementation language (and maybe PyPy for the Algorithm Model) so the decomposition into three or so processes will help to take advantage of multicore processors thereby improving run-times, with no effort spent on advanced coding techniques like concurrent threading, GPU programming, etc. My employment of threads here is a simple usage pattern needed only to simplify the I/O and make it immune to deadlocks. 

Note that if any large data files need to be streamed to or from the Algorithm Model or the Environment Model, these would typically be handled by the User Interface process. It would take care of any translation, pre-processing or post-processing required. An exception may be where the Environment model is actually just a video file, and the Algorithm Model navigates this “environment” by advancing or rewinding frames.

To demonstrate that a piping architecture resembling the above diagram is possible, I created two proof-of-concept scripts, shown below. expUI simply spews out an incrementing stream of integers as ASCII text into the stdin of expModel. Then expModel returns the favor by routing the stream back around to come out of its stdout. expUI prints the stream it receives from expModel out on its stderr, and terminates gracefully when it receives the string “FINISH”. 

Note:  ”exp” = “experiment”

Source code for executable Python script expUI

#!/usr/bin/python import time import sys import threading from subprocess import Popen, PIPE # Launch model: pid = Popen("expModel", stdin=PIPE, stdout=PIPE) # Transmit function (for thread): def tx(fout): a = 0 while True: if a < 50: fout.write("%i\n" % a) fout.flush() a = a + 1 else: fout.write("FINISH\n"); fout.flush() break # Receive function (for thread): def rx(fin): while True: line = fin.readline() sys.stderr.write(line) sys.stderr.flush() if "FINISH" in line: break # Create threads: threadWrite = threading.Thread(target=tx, args=(pid.stdin,)) threadRead = threading.Thread(target=rx, args=(pid.stdout,)) # Launch threads: threadWrite.start() threadRead.start() # Wait for completion: pid.wait() threadWrite.join() threadRead.join()

Source code for executable Python script expModel

#!/usr/bin/python import threading import time from Queue import Queue from sys import stdin, stdout, stderr # Receive function (for thread): def rx(fin, q): while True: line = fin.readline() q.put(line) time.sleep(0.1) if "FINISH" in line: break # Transmit function (for thread): def tx(fout, q): while True: try: line = q.get() fout.write(line) fout.flush() if "FINISH" in line: break except: stderr.write(" OH. CRAP. \n") stderr.flush() time.sleep(0.01) # Create thread-safe data passing queue: q = Queue() # Create threads: threadRead = threading.Thread(target=rx, args=(stdin, q)) threadWrite = threading.Thread(target=tx, args=(stdout, q)) # Launch threads: threadRead.start() threadWrite.start() # Wait for completion: threadRead.join() threadWrite.join()

After spending an unreasonable amount of time earning a bachelor’s degree in electrical engineering and taking many graduate level courses related to machine learning and optimization, I ended up getting a job in the aerospace industry developing DSP algorithms for satellites. It may not be the best fit, but after about a decade my career has been moving forward at a respectable pace. During this time, I've remained on the periphery of the machine learning community; outside, looking in, continually learning and thinking, but not actively participating. 

Until now.

News-worthy breakthroughs over the past 10 years introduced by the top researchers in the field such as Hinton, LeCun, Bengio, Hawkins and groups like Deep Mind have captured my imagination. I have drawn significant inspiration from their discoveries and developments including Deep Learning, Convolutional Neural Networks, Hierarchical Temporal Memories and Reinforcement Learning. Though some chinks may be emerging in Deep Learning’s armor, in its current formulation, I don’t think the potential flaws/quirks are going to turn out to be as big of a deal in the long run as some seem to think. This leads me to why I’m writing this post.

During my self-guided journey in machine learning I have maintained a log, which has finally accumulated a critical mass of ideas and insights. I believe it is time for me to actually start putting some of these to the test. My professional experience has taught me that such an endeavor will be a large commitment, and I’m very judicious in how I allocate my time. Therefore, I had been hesitant to embark on such a project for fear of ending up with nothing to show for it after an enormous time investment. However, I have recently been able to identify enough potential side benefits that even if none of my ideas turn out to have much value, the experience will still have been worth it. These benefits are the result of careful project planning in which various aspects of the project were structured to have a secondary payoff (such as marketable job skills) beyond their utility for this project. These secondary benefits include:

Experience developing interactive command-line interfaces

Experience developing GUIs

Experience with the trade-offs between C++, Python and PyPy

Experience developing a client-server architecture

Experience developing image and video processing algorithms

Experience with machine vision algorithms such as feature detection and object recognition

Experience with various incarnations of Deep Learning and other Neural Network algorithms

Potential for publishing a paper

Satisfy my general curiosity

As one may be able to glean from the list above, I’ve already made a number of design choices and project planning decisions. I started with the following high-level goal:

Explore machine learning ideas and be as productive as possible

Productivity is enhanced or impaired by project-level decisions. The programming language and development environment, the architecture of the software being developed, and the general algorithmic approach and coding style all can have a significant impact. Some choices can influence all of these aspects. For programming languages, I have a strong background in MatLab, C++, Python and Verilog, and I have found that I tend to be most productive working in a Linux environment. 

In order to get started, the following key decisions had to be made: 1) which programming language(s) to use, 2) which toolkits to use and 3) how to architect the project. Making these decisions in isolation from each other would be unwise, as each have ramifications that affect the others. The following section describes some of the trade-offs for each option I considered. 

Project Architecture

I considered several architectural approaches:

monolithic executable with GUI interface

monolithic executable with command-line interface

disagregated architecture (client-server) with hybrid interface

Again, a key goal is productivity, and my definition of productivity is “get answers to my questions as quickly as possible”. Example questions include “why isn't it working?”, “does this algorithm trick/tweak help or hurt the algorithms performance?”, or “does this change make the algorithm run faster?”. I have found that interactive, command-line environments enable a high level of productivity because they allow for scripting, customization, quick peaking and poking, and they are fast to develop. GUIs can be helpful too, but they are typically the most beneficial in well explored domains where their functionality can be easily defined/bounded and don’t need constant modification. A command-line interface with a simple ability to plot and explore data and diagnostics seems like the highest productivity approach I can think of. Additionally, segregating the user interface (front-end) from the algorithm under development (back-end) allows the interface to be refined over time while many different algorithms are developed and tested, perhaps even in different languages.

Based on this reasoning, I've chosen to factor the design into two pieces: the UI, ie “client”, and the back-end, ie “server”, where the loading and converting of data, plotting, and any sophisticated user interaction is handled by the UI. The back-end will be responsible for running the algorithm and responding to commands and data input from the UI.

I intend to begin with plain old ASCII text as the data format for communicating between the UI and the back-end, and they will communicate with each other over a bidirectional named pipe so I only need to deal with simple text I/O, rather than sockets or anything requiring more overhead and complexity. This also simplifies debugging and regression testing, because text files can be substituted in place of the pipe. Also, ASCII text flowing over a pipe is very easy to inspect. Because Machine Learning algorithms tend to be memory and/or compute bound, any I/O overhead associated with the text interface should have little impact to the processing throughput.

Programming Languages

Though MatLab can be a high-productivity language in many application domains such as sand-boxing algorithm concepts, as a general-purpose language it is rather terrible. Also, it is proprietary, and even if I purchase it I’d have to deal with its slow startup time and various language annoyances. It runs fairly fast, but it doesn't play nice with POSIX, so building it into a larger framework would be a pain (I've suffered through this in my professional life, so this isn't just speculation). Octave provides a nice alternative to MatLab, it is free, MatLab-language compatible (mostly), has many of the libraries that matter, is POSIX-friendly, and nowadays it even offers a GUI. Its downsides are that it runs much slower than MatLab, especially for code which can’t be vectorized, and it shares an annoyance of MatLab: one-based indexing. This isn't too big of a deal for one-off scripts and sand-boxing algorithms, but it is a big deal when it is used as part of an architecture that includes zero-based indexing languages such as Python and C++. Going back and forth can be a pain, and porting an algorithm from one to the other during a re-factoring step can really suck. 

Regarding C++, following a sane coding style and development approach can lead to fairly compact and simple code that runs very, very fast. The abstractions that C++ enables can make certain aspects of a project quite easy to develop. Other aspects of C++ and its ecosystem, however, can greatly increase development time (without getting into specifics). Also, larger projects require a somewhat elaborate build environments, with the obvious annoyance of having to compile prior to running.

Python is slow. No bones about it, it’s a fact. However, PyPy is (pretty) fast. It’s no C++, but for algorithm experimentation it is probably fast enough, where “fast enough” is defined as “less than a minute to run an experiment, make a code change, and try again”. Python itself has an amazing ecosystem from numerical and scientific libraries to image and video processing. It provides an interactive environment for quick experimentation and documentation lookup, and creating working code is just plain easier and faster with Python. It is a high productivity language, and because of PyPy, it may be possible to get the best of both worlds (runs fast and is fast to develop). I’ll discuss this in more detail below.

Verilog is not yet appropriate to consider at this phase, though it remains possible that I could discover algorithms which render themselves to FPGA implementation. I have a hunch that won’t happen, however, due to the intrinsic “memory hard” nature of machine learning algorithms. To be competitive with CPUs and especially GPUs, an FPGA would need a monstrous memory interface, and lots of on-board memory. Such systems only exist at “big boy” prices and are just not practical for mere mortals like me. GPU programming is likely to be the better option, but advanced techniques like this will only be worth pursuing once I've identified a promising algorithm.

Python’s ecosystem and productivity benefit make it hard to compete with. I have therefore selected Python for the user-interface component of this project, at a minimum. I will begin writing the backend in Python and testing it out using both Python and PyPy to better understand the tradeoffs. For instance, PyPy does not support all Python packages, so this limitation warrants evaluation. At some point I will decide if PyPy is able to do what I need it to do. If so, I will stick with it and continue developing my models in PyPy-compatible Python code. Otherwise, I will port all algorithm (back-end) code over to C++. To maintain acceptable productivity in C++, I will begin by adhering to the simplest coding techniques, data types, data structures, and libraries. The two-piece client-server architecture enables this “off ramp” project plan, so I don’t get stuck riding a losing horse. Only once a winning algorithm or class of algorithms has been identified is it time to get serious about run-time performance and optimization, which necessitates more laborious and less productive coding techniques.

Note that an advantage to beginning with Python is that the algorithms can be structured for easy porting to C++. The  object inheritance hierarchy, data structures, initialization parameters, and many other conventions can be directly translated from one language into the other. Running the two different implementations against each other to check correctness should then also be relatively simple, and the similar code structure and identical indexing base should make chasing down differences straightforward. 

Toolkits

Though other machine learning toolkits and frameworks exist (Pylearn2, MLPack, Theano, nnet, PyBrain, etc.), they were all developed to address other people’s problems. I tend to prefer to adopt standards and not reinvent the wheel when possible, but in this case I think reinventing the wheel is exactly what I should do. I need my development and test environment to grow with me and with my project. It needs to answer the questions that I’m currently asking, since I don’t yet know enough to ask the questions that someone with more experience would be asking.

As for GUI toolkits, wx and Qt stand out as popular options for both Python and C++, but because of my strong preference for Python for the UI, and because it seems like the community at large prefers Qt over wx, I’ll go against my past positive experience with wx and use Qt for any GUI work. I don’t expect to do too much in GUI land; my GUIs will probably consist mostly of launching plots that have a little bit of interactivity, such as clicking to select a neuron or zooming in and out. However, because I don’t intend to develop a monolithic GUI (at first), I will still have the freedom to do some of my mini-GUIs/plots using Qt and others using wx, whichever seems to be the most expedient, on a case-by-case basis.

Summary

The above discussion outlines the various issues I considered when developing an approach for this endeavor. I will create a Python front-end with a simple command line interface, a PyPy-compatible back-end with a bi-directional pipe interface and some simple GUI plots. I will also create a neural network Python class which provides a common interface (ABC), allowing it to be used to derive other neural network types. These will be easily testable and runnable from within the overall framework. All of these decisions have been made in the name of productivity, partly as a function of my previous professional and hobbyist experience, and partly due to web research into the state-of-the-practice and general community consensus.

I hope to update this blog regularly with my progress as I proceed with the implementation of this framework, the algorithms, and then the testing and evaluation of various algorithms and stimulus data / environments.

There's a PDF presentation floating around entitled "Looking for 4x speedups? SSE to the rescue!", which presents a few examples for how to leverage Intel's SSE instructions to achieve speedups in your algorithms. SSE is pretty attractive to me because, as a C++ programmer, the instructions can be easily invoked by including the "*mmintrin.h" family of .h files, employing some of the intrinsics defined there, and then compiling (with GCC) with the -msse3, -mssse3, etc... options.

Anyway, I needed to accelerate 32-bit (single) precision complex floating point multiplication, and attempted to implement the example they showed. Unfortunately, there's an error! They show 5 instructions required to simultaneously multiply two complex floats, but there's an omitted step. Before the final "addsub", you need to shuffle one of the multiplication results.

I'm interested in the history and progress of technology, and fascinated by the difference between the vantage point looking forward into the future vs backward into the past. Thus, with an eye toward the future and as the New Year (Jan 2012!) approaches, I'm composing a list of potentially revolutionary technologies that seem promising, given my current vantage point. I'm specifically omitting those that I don't think are promising (like quantum computing). I'll update this list as I think of things, up to December 31st:

Computing:

Memristors (and other resistive RAM technologies) - Show promise for on-CPU memory, denser faster RAM, hard-disk replacement, denser faster FPGAs, as a new computing substrate, as an enabler for direct implementation of Neural Networks, especially HTMs.

Hierarchical-Temporal Memories (HTM) - A biologically inspired neural network architecture that not only works, but appears to work the way brains work.

Imaging and Display:

Plenoptic / Light-field Cameras - Non-coherent technique to achieve a focusable image. Not holographic, because it captures only one vantage point, but does show promise because a) commercial products are on the market, and b) technology road maps exist to achieve the resolutions required by professional photographers.

Light-Field Displays - Like a Light-Field Camera in reverse. Enables a stereoscopic displays where the eyes can focus on the perceived distance to objects in a scene, overcoming a key limitation of current stereoscopic displays, and achieving true immersion.

Energy Storage:

Lithium-Air Batteries - Theoretical energy densities of 5000 Watt-hours per kg. Prototypes currently exist. Enabler for electric vehicles to achieve the range and convenience of gasoline powered vehicles.

Natural Gas - Not revolutionary, but tons of reserves and low prices in the US could result in NG out-competing fossil energy and EVs in the transportation sector. Significant (positive) short-term economic implications.

Energy Production:

Quantum-Dot Solar Cells - A far-out technology, but has demonstrated multiple-exciton generation (more than one electron output for one photon input), and conversion efficiencies of > 6%. Extremely cheap (paint-on/spray-on) production could result in revolutionizing energy production. Close to being useful right now, but potential exists for radical efficiency improvement given further development.

Solar Concentrators - Currently competitive, but with economies of scale and increased production efficiencies, could be a real contender in the future.

Solar-Thermal - Also currently competitive, and the "power tower" approach is the most promising instantiation. I think we'll see more of this in the near future.

Medicine:

Angiogenesis-blocking medications - Cancerous tumors require control over angiogenesis (the production of blood vessels to feed the tumor). Medications are being developed which block angiogenesis, with the potential to radically reduce mortality in cancer patients.

I bet you didn't know you can view relatively high quality 3D video on your current, unmodified, non-3D Android phone!  ...well, at least, it works on my HTC EVO, your mileage may vary.

Here's how to do it:

Be sure you have the YouTube app installed from the Android Market

On YouTube, search for content that might look cool in 3D. For instance, searching for "molecule" will yield some interesting videos.

In the list of search results, there will be a little "3D" icon next to the ones that are in stereoscopic 3D, which displays a separate image for each eye, side-by-side.

Click play, and hold your phone horizontally to view.

Here's what a video will look like:

Tips:

Under settings, enable high-quality playback... but I don't recommend HQ unless you're on WiFi

Hold the phone up to your face as closely as possible until the two images appear to be a single image, now slowly move it away from your face while using both eyes to track objects in the scene. You'll find there's a distance where the 3D effect is most convincing, and where you're eyes are still able to look around in a natural way. You shouldn't be cross eyed, or have your eyes trying to look outward from each other.

On regular YouTube (on your computer) you can click on "filter", then select "3d", to only return videos which are in 3D. YouTube supports many different 3D rendering techniques, which you can select by clicking on the 3D icon at the bottom of the video.

I'm a very near-sighted glasses wearer, so I take off my glasses to do this. I recommend doing the same for most people. It might be difficult/impossible to focus if you are far-sighted or have trouble focusing up close.

For me, the video quality is the next best thing to a 3D TV, provided the source video quality is pretty good. However, my particular phone combined with my nearsightedness probably makes this work better for me than for some people.

At the ripe old age of 34, I have a tendency to suffer from "old man syndrome", which can be characterized as a growing level of cynicism toward technology, and a general attitude that  "bah, nothing ever changes!!"  Oddly, I'm considered a "technologist" at work, pushing the frontiers of the possible, breaking down preconceived notions and fighting against conventional wisdom. As I step back in an attempt to see things from a fresh perspective, this dichotomy becomes clear to me, but what really drives it home is when I read about truly revolutionary innovations and discoveries in the news. For instance, today I read about the "Light-Field" camera (or "plenoptic") camera. I consider myself very up-to-date in the field of imaging, and was surprised that such a technology could be consumerized without me catching so much as a whiff of it. How did I miss something this big? And for the record, photography will never be the same, mark my words.

Because of these occasional "slaps in the face", I've decided to begin maintaining a list of truly revolutionary innovations and discoveries that have occurred in my lifetime, partly to help keep me grounded in reality, and partly to inspire myself and others to continue striving for novelty and to advance the frontiers of human knowledge.

I'd like for this list to be concise, with brief descriptions of each discovery/innovation, the year of discovery/innovation, and the year of practical application. The assigned years can be more or less fuzzy, but I'll try to be somewhat reasonable and conservative. To be classified as "truly revolutionary" a discovery or innovation needs to either fundamentally change how we view the world in which we live, or significantly impact the nature of our daily lives.

1977 - I was born!

ARPANET, the precursor to the Internet, which began in 1969, is fairly mature, but remains largely in the domain of government and university research labs. 

Magnetic Resonance Imaging (MRI) is invented.

The Atari 2600 video game console is released (at first it was called the Atari VCS). Millions of units are eventually sold well into the 1980s.

1978

The artificial heart is developed.

Voids in the distribution of galaxies in the Universe are first discovered. These are later corroborated both in theory and in computational simulations of galaxy formation.

1979

Rollerblades are invented.

3.4 Billion year old fossilized bacteria discovered.

"Tit-for-tat" proposed as the optimal strategy in the "prisoner's dilemma", within the field of game theory.

1980

The K-T boundary (now formally known as the K-Pg boundary), marking the K-T extinction event (the end of the dinosaurs) was discovered. 

Scanning-Tunneling microsocope developed.

1981

MS-DOS and the IBM PC are developed.

Prions discovered, and found to be the cause of Mad Cow Disease.

Computer simulations using genetic algorithms confirm tit-for-tat as the optimal prisoner's dilemma strategy, with ramifications for business, political science and beyond.

1982

The Commodore 64 8-bit computer is introduced.

Quasicrystals are discovered, which will later go on to win a Nobel prize in 2011. They exhibit an ordered, but not repeating structure. Their structure can only be defined algorithmically/mathematically, and not as the simple repetition of a fundamental unit. 

1983

HIV identified as the cause of AIDs.

McDonalds introduces the McNugget before industrial agriculture has sufficient capacity to keep up with the massive demand.

1984

The CD-ROM is invented.

A beta-amyloid peptide found to be the primary component of the plaque found in Alzheimer's patients.

1985

Polymerase chain reaction (PCR) discovered, which multiplies DNA sequences in vitro, making it possible to clone specific DNA sequences rapidly without the need of a living cell.

Buckminsterfullerenes, a.k.a. "buckyballs" discovered.

The Nintendo Entertainment System (NES) is launched in North-America, bringing arcade-style video game technology to the masses. 62 million units are eventually sold, worldwide.

1986

Dragon Quest is released, establishing the template for all role playing (video) games to follow.

The first IBM-PC virus, "Brain" is introduced, which infects the boot sector of a hard disk.

1987

The home computer becomes mainstream.

The first 3D video games are developed. 

It was discovered that all human mitochondrial DNA can be traced back to a common African maternal ancestor known as "Mitochondrial Eve", living roughly 200,000 years ago.

1988

The Soviet Union's copy of the United States' Space Shuttle, the Buran, is launched on its first and only orbital flight. It is unmanned, and made a successful automated landing.

1989

High definition television (HDTV) is first developed.

1990

Hubble space telescope launched.... with flawed optics.

1991

Carbon nanotubes discovered.

Data from the Compton Gamma Ray Observatory indicates that Gamma Ray Bursts (GRBs), which were first discovered in the early 1960s by US military satellites, are of extra-galactic origin.

1992

CERN released their HTTP protocol and HTML, sparking the creation of the World Wide Web. It was initially intended for physicists, but later became the Internet.

1993

Hubble space telescope's optics fixed.

1994

Avida, created by Christopher Adami, demonstrates semi-open-ended evolution in a digital ecosystem.

The Hubble space telescope provided the first direct observation of an extraterrestrial collision between objects in our solar system when the comment Shoemaker-Levy 9 impacted Jupiter.

1995

The Internet starts to become mainstream.

First extrasolar planet observed (Mayor and Queloz).

The DVD is invented. 

Leonard Suskind publishes "The World as a Hologram" in the Journal of Mathematical Physics, describing the Holographic Principal, which implies that the universe can be thought of as being completely "encoded" on the surface of a sphere. This theory is yet to be borne out, but is compatible with existing string theories, and has the potential to resolve many current problems in physics, such as the black hole information paradox.

1996

The first digital cameras for the masses emerge.

1997

First cloned mammal (Dolly the sheep).

Kasparov defeated in chess by IBM's Deep Blue supercomputer.

1998

GOOGLE was founded.

The world's first commercially successful MP3 player, The Daimond Rio, was launched, with 32 MB of memory.

Human embryonic stem cells isolated.

Measurements of type 1a supernovae indicate that the expansion of the Universe is accelerating.

1999

Fossil evidence indicates the emergence of eukaryotes no later than 2.5 Billion years ago. 

Netflix begins offering DVD rental through the mail in California, and by 2007 had become mainstream, with competition emerging from RedBox. 

TiVo is launched as the first widely available digital video recorder (VHS replacement technology), providing the ability to skip over television commercials.

2000

NASA's Galileo spacecraft discovers strong evidence of oceans lurking beneath the icy surface of Jupiter's moon Europa.

First research prototype of light-field (plenoptic) camera demonstrated in Standford lab.

Digital camera sales surpass film camera sales for the first time.

2001

Human genome project finishes its first complete human sequence.

Apple's iPod is released. 

Wikipedia is launched.

2002

The Journey of Man, by Spencer Wells, is published. For the first time the public is presented with a wealth of genetic evidence supporting the "out of Africa" theory of human migration. The genetic evidence is significant because it is completely independent of the archaeological evidence, yet corroborates it to such a degree that to dispute it requires a great act of faith. Subsequent DNA analysis has since piled on mountains of supporting data, making the picture clearer by the day.

Cosmic neutro background is detected, though indirectly. This is similar to the cosmic microwave background, but the decoupling happened at an earlier time.

2003

Toyota's Prius hybrid car introduced to the market.

2004

On Intelligence, by Jeff Hawkins, is published. It presents the first comprehensive, biologically plausible theory of cognition. At its core is the Hierarchical Temporal Memory (HTM), a type of neural network, which Hawkins argues is the fundamental building block of the neocortex.  

Facebook is launched.

Podcast listening becomes mainstream. 

The Hubble space telescope image, dubbed "the ultra deep field" was created, allowing us to peer further into the past than ever before, and confirming that ours is but one of billions of galaxies in the observable universe.

2005

YouTube goes online.

The Yeti Crab is discovered living near thermal vents in the ocean floor, about 7200 feet down, off the coast of Easter Island.

The Hubble space telescope discovers a fourth moon orbiting Pluto.

Gliding ants discovered, indicating a possible evolutionary path to insect flight.

2006

NASA's Cassiny-Hyugens spacecraft discovers geysers of liquid water erupting from the surface of Saturn's moon Enceladus.

The Mars Reconnaissance Orbiter enters orbit around Mars.

2007

The first "smartphone done right", Apple's iPhone, was launched, which built upon the success of its iPod brand.

2008

The first meteor to be discovered prior to impact (Meteroid 2008 TC3) hits the Earth.

SpaceX successfully launches the world's first privately funded space vehicle into orbit, the Falcon 1.

2009

Kepler Mission launched, with hopes of finding extrasolar Earth-like planets, characterizing the statistics of extrasolar planets (size, orbits, etc), and refining stellar models using stellar seismography.

HDTV becomes mainstream in the US, largely by fiat of the FCC.

2010

Numenta, Jeff Hawkins' company which is attempting to develop and commercialize HTM technology, makes breakthrough in cortical learning algorithms, dubbed "HTM 2.0", paving the way to broad adoption and hardware implementation of "thinking machines".

The twin Mars rovers Spirit and Opportunity, in concert with the Mars Reconnaissance Orbiter, discover multiple lines of evidence indicating that Mars once had abundant liquid water. 

First bacteria developed, which is controlled entirely by synthetic DNA.

2011

Thousands of extrasolar planet candidates, including many super-Earths, identified by the Kepler Mission. Jupiter class planets appear to be very, very common. 

IBM's Watson supercomputer beats human contestants, highlighting the state of the art in natural language processing.

The Mars Reconnaissance Orbiter discovers strong evidence for regular current water flows out of Martian cliff-sides.

Shelved products from the 1953 Miller-Urey experiments found to contain vastly more organic compounds than previously believed, adding to the evidence that conditions on the early Earth were sufficient to synthesize most of the required building blocks of life.

Initial evidence for the Higgs Boson discovered (~125 eV) at the LHC.

Mars Science Laboratory (MSL) launched.

3D televisions (stereoscopic) become affordable and widely available. 

Solar cells have achieved (or nearly achieved) "grid parity" with conventional (fossil) sources of energy production, primarily in the Southwest United States. Other promising technologies include variations of solar-thermal, especially solar concentrators, solar towers ("power towers") and solar troughs. 

The first mass-produced electric vehicle, the Nissan Leaf, entered the market.

2012

First light-field (plenoptic) camera, developed by Lytro, available to consumers.

PREDICTION - The Kepler mission will discover the first extrasolar Earth-sized planet in anEarth-like orbit around a Sun-like star.

PREDICTION - Assuming a successful landing and deployment, the Mars Science Laboratory will discover "smoking gun" evidence of a watery past, but will fail to find fossilized organisms. It may uncover some mildly convincing chemical traces of past life, but nothing conclusive.

2015

PREDICTION - It will become clear that sub-human-intelligence automation software/hardware is replacing human workers, especially call center workers and low-level administration. The structure of the economy will not support placing these workers into new jobs. Rather, there will be a net economic contraction as a direct result of automation.

2020

PREDICTION - The first "computer" with human-class intelligence will be built and demonstrated. It will likely be based on a neural network architecture related to the heirarchical temporal memory developed by Jeff Hawkins and team.

2030