Siyuan Xu RC11

The Markov chain from a movie cannot be directly used in the city context.

This is because the state classification of a movie clip and the state classification of city space are not the same object. Shot size, scene category, and basic component from a movie cannot be directly converted into streets, squares, or bridges in a city.

The correct logic should be:

First, establish a sequence of city spaces.

Then, define the state for each clip in the city sequence.

After completing the first round of semantic classification and semantic Markov chains, based on the teacher's suggestion, we reclassified all existing slices according to the main colors of the scene, and then trained a second set of Markov chains.

We believe this suggestion is very important because it shifts the project from "what's in the shot" to "what the shot looks like visually."

In other words:

The first set is a content-driven state system

The second set is a color and atmosphere-driven state system

This allows us to compare the two different organizational logics:

The semantic chain organizes content relationships

The result of natural clustering of all words in the subtitles of six movies after being projected into a 20×30 semantic space.

This is the result of semantic matching after I put this movie keyframe into the code.

I am building a city model deformation system based on the semantic features of six movies.

The core idea is:

To map the semantic structure of different movies into parameters using SOM (Self-Organizing Map), driving the geometric animation of 17 city models.

The entire pipeline is currently running, and I have completed the deformable animation structure of the model (GeoNodes + point cloud buildings).

The next step is to use the generated control CSV to drive all of this.

Training Data Logic (Data Source → SOM)

(1) Training data comes from the original subtitles (.srt) of six movies.

Read the .srt files of the six movies.

Clean the text: remove timestamps, serial numbers, and punctuation, keeping only English words.

Obtain a set of all words (approximately several thousand words).

Use the CLIP text encoder ViT-B/32 to encode each word into a 512-dimensional vector.

This ensures that the input to the SOM is completely consistent with the text content to be mapped, resulting in a more accurate semantic space.

SOM Training Logic: Semantic Map

(1) The SOM uses a 20×30 grid I wrote myself.

Input: CLIP 512D vectors of all words

Output: A 20×30 semantic topology space (word_som)

Each SOM cell stores a semantic "center vector"

(2) The meaning of SOM

Each word has a semantic position (som_x, som_y)

Similar words are close together (e.g., "memory", "dream", "past"), and words with different semantics are far apart.

This map is the basis for driving the animation later.

Extending SOM to "Sentence-level Semantics"

I needed to find the SOM position for "each line of dialogue", so I did the following:

(1) Parse each line of subtitles

Remove the subtitle number and timestamp

Extract line by line

A total of several thousand lines of dialogue (six-film corpus) were obtained.

(2) For a single line → All words → Find their positions on the SOM

Cleanse a sentence into multiple words

Find each word's position on the SOM (som_x, som_y)

Take the average of these word positions as the semantic center of the sentence (float is normal)

(3) Obtain the final sentence-level SOM annotation table

The final generated data is:

movie, line_id, line_text, som_x, som_y, emotion

This CSV is the "semantic feature library driving the animation".

City Model Deformation Animation (Completed)

I have completed this part:

I built 17 city modules (point cloud building form) in Blender

I created a deformable system (Geo Nodes)

I introduced various animation control parameters such as noise, scaling, rotation, and translation

All models can be uniformly controlled on the timeline

It is now possible to generate dynamic effects of city morphology changing with parameters

That is to say: the city model can deform, push, rotate, and breathe, but it is not yet semantically bound.

The core next step: Driving animation with CSV

The next task is clear:

I need to map the previously generated SOM+emotion table (movie semantics) into animation control parameters.

This includes:

scale (zoom in/out)

loc_x / loc_y / loc_z (displacement)

rot_z (rotation)

noise_val (Geo Node noise intensity)

The logic is as follows:

Normalize som_x, som_y, and emotion

Set different parameter ranges for each movie (determining style)

Generate a movie_driving_params.csv file

Batch read using Blender + automatic keyframe insertion

This achieves:

Different movies → different semantics → different deformation methods for city models

A. Technical route (Methodology)

Feature Extraction (Feature Extraction):

Abandon the traditional RGB histogram and adopt the OpenAI CLIP (ViT-B/32) model.

Map each video as a 512-dimensional semantic vector, which contains high-dimensional semantic information such as objects, scenes, light and shadow.

Unsupervised Clustering:

10 \times 10 SOM (self-organizing mapping neural network) has been built.

Automatically map thousands of frames of images to 100 topological nodes to achieve automatic aggregation of similar scenes.

B. Key Achievements

1.Semantic Overview Grid

Output: A 10 *10 SOM overview puzzle.

Effect: Successfully classify all the scenes in the video by content.

All the wide-angle/landscape shots are gathered in the lower left corner. All the dark light/in-car lenses are gathered in the upper right corner. Character close-ups and dialogue shots are gathered in the middle area.

Visualization optimization: Use the Count label with white letters on a red background to clearly show the distribution density of each type of lens.

2.Intelligent Detail Analysis Card (AI-Annotated Detail Sheets)

Output: A high-definition detail puzzle is generated for each cluster node. AI automatic graphic and text analysis has been introduced.

Use CLIP's graphic and text matching ability to automatically tag each set of pictures (such as "Analysis: Contains [close-up face] & [dark indoor]"). not only the classification, but also the explanation of "why it is so divided".

3.Narrative Rhythm Analysis in the Time Dimension (Temporal Narrative Analysis)

Outpot: Semantic Barcode and Scene Step Chart.

In-depth insight: Map the cluster results back to the timeline to visualize the editing rhythm of the film.

Long strip color block: Identify the long shots and stable narrative paragraphs in the film.

High-frequency color switching: Identify intense action scenes or montage clipping paragraphs.

Color reproduction: It reveals how the director controls the visual rhythm of the film by periodically repeating specific scenes (such as empty mirrors).

Phase 1: Color-based SOM (Baseline)

Method: Initially, I extracted the RGB histogram data for each frame and input it into the SOM.

Results: The SOM successfully separated the images according to "hue".

All "blue images" clustered together (whether it's an ocean or a blue wall).

All "black images" clustered together (whether it's a night scene or a person wearing black).

The Limitations:

"Semantic Gap": The machine only understood the color, not the content.

Misclassification Cases: A red car and a red apple were grouped together because their pixel colors were similar. However, from a cinematic narrative perspective, these two are unrelated.

Research Objective: To analyze the "invisible infrastructure" of Seattle. I hypothesize that, in a cyberpunk context, the physical space of a city has been reconstructed by capital and logistical control. My work is not merely based on geography, but on a new map based on "power weights."

Data Construction: I constructed a [200, 4] vector matrix with four dimensions as input to the SOM:

Physical Layer:

Latitude/Longitude: Coordinates of 200 real-world tech companies based on the T200 GeekWire list.

Invisible Layer:

Capital Power: Based on company headcount and valuation, with adjustments made for giants like Amazon ($2.2T) and Microsoft ($3.1T) based on their actual market capitalization. (Metaphor: The Blood of the City)

Amazon Gravity: Calculates the distance weight of each company to Amazon's four major control nodes (HQ, Kent, Airport, SoDo). (Metaphor: The Skeleton and Shackles of the City)

Algorithmic Process Using the SOM (Self-Organizing Map) algorithm, these 200 discrete data points are mapped onto a continuous topological grid. The algorithm clusters companies with similar geographical locations and similar power attributes together.

Key Findings & Narrative By comparing the generated Component Planes (heatmaps), I discovered three "power strata" in Seattle:

A. The Singularity – [High Capital + High Control]

Location: South Lake Union (SLU).

Characteristics: In the SOM heatmap, this is the area where the brightest orange (capital) and the brightest red (control) perfectly overlap.

Space, capital, and logistical control collapse into a single point here.

B. The Resistance – [High Capital + Low Control]

Location: Eastside (Bellevue / Redmond) & North (Everett)

Members: Microsoft, T-Mobile, Helion Energy

Characteristics: Possesses extremely high capital brightness, but appears dark on the "Amazon gravity map."

Meaning: This is an "independent sovereign zone" separated by Lake Washington. They do not need to rely on the Amazon's logistical network and possess their own ecological defenses. This serves as a reference for your design of "defensive structures" or "energy arks."

C. The Periphery – [Low Capital + High Control]

Location: SoDo / Kent Valley

Characteristics: Dim capital map, but bright red gravity map.

Meaning: This is a bottom-level region "locked up" by the system. They must depend on logistical pipelines for survival but cannot accumulate wealth. This is a spatial manifestation of the cyberpunk concept of "High Tech, Low Life."

Extraction of Amazon-Related Buildings

Within the captured downtown model, I isolated all buildings that contain Amazon offices. This step included: • identifying them from address and massing • separating them into standalone meshes • preparing them for annotation, environmental overlays, and later cyberpunk-theoretical analysis

This allows Amazon’s urban dominance to be visually expressed in a clear, analytical way.

Next step:

To support later spatial analysis and diagramming, I captured 3D models of Seattle’s downtown core using RenderDoc and Google Maps capture techniques. So far I have:

reconstructed a partial but not a very high-resolution block-level model

verified geometric accuracy for later integration into GIS and Rhino

This provides a manipulable base model for visualizing invisible infrastructures and tech-urban dynamics.

Given Amazon’s centrality to the (techno-environmental anxiety, cyberpunk origin, and invisible infrastructures), I conducted a deeper layer of data collection:

scraped all official and third-party records of Amazon’s Seattle office locations, including the downtown campus towers and scattered satellite offices

extracted address, building name, function type (office / R&D / specialty), and public information about each site

I collected and structured the latest list of Seattle’s top 200 technology and startup companies. For each company, I organized:

company name

sector/type

brief description

founded

employees

employees_1yr_change_pct

linkedin_followers

Technical:

The operational and mechanical systems underlying perception, communication, and control.

Architectural / Cinematic Meaning： Data centers, AI networks, urban automation, algorithmic governance.（The Creator, Blade Runner (1982/2049), Her）

Psychological：

The invisible emotional and cognitive mechanisms shaping human experience within technological systems.

Architectural / Cinematic Meaning： Interfaces, intimacy, cloning ethics, memory, disembodied empathy. （Swan Song, Her, Blade Runner 2049）

Ecological：

The metabolic, climatic, and environmental feedback loops sustaining or decaying the world.

Architectural / Cinematic Meaning： Living systems, post-natural environments, biotechnological decay. （Vesper, Blade Runner 2049）

VESPER

The film literalizes the idea that ecology is infrastructure — life-support systems that persist autonomously, even when human governance collapses. The aesthetic of overgrown laboratories and living machines resonates with hidden metabolic layers of the city (e.g., hydroponic systems, vertical agriculture, data-center heat loops).

Extract frames showing living architecture and biomechanical textures for material ecology mapping.

2. THE CREATER

The Creator exposes the cognitive infrastructure of global computation: networks that decide, surveil, and simulate life at planetary scale. It mirrors the hidden logics of Amazon Web Services and cloud infrastructures that underlie Seattle’s identity as a techno-capital.

scenes depicting AI control centers and sensor networks as spatial metaphors for TECH cities.

3. SWAN SONG

It articulates an emotional infrastructure — invisible affective systems (memory, empathy, grief) that mediate our relationship with both architecture and the digital. The duplication of self parallels my concept of spatial duplication and emotional mirroring in post-digital environments.

Correlate “substitution scenes” with design scenarios where built form replaces or extends human emotion.

4. Blade Runner 2049

It operates across all three dimensions:

Technical: AI, holograms, data.

Emotional: loneliness and constructed memory.

Ecological: atmospheric control, desaturation. The film’s cityscapes parallel the “Seattle-as-server” metaphor — architectures of constant processing.

5. Blade Runner (1982)

It frames the very discourse my research extends: the city as a living computation of power, emotion, and decay. Its city — often cited as the prototype for Snow Crash’s and Her’s Seattle-like megacities — establishes the cinematic grammar of infrastructural invisibility.

Compare with 2049 to trace evolution of infrastructural atmosphere.

6. Her

It translates Seattle’s corporate digital culture into an aesthetic of invisible communication: architecture as operating system. The built environment becomes responsive but intangible — precisely the condition my “implicit infrastructure” theory describes.

I extracted three key points from the article that I think are very interesting and worth further study.

The first is that the cloud is not an illusion. What appears as an ethereal “cloud” is in fact a global, material system — a chain of mineral extraction, manufacturing, transport, data-centre energy, and waste that produces every AI interaction. The apparent immateriality is a political concealment of material costs.

The second point is that behind the interface is the human and logistics chain

A single voice command or data request depends on distributed human labor (data-labelers, repair crews, logistics workers) and extensive shipping/logistics infrastructures: the “AI” is inseparable from these grounded, political economies.

The third point is particularly important, directly explaining why I've chosen Amazonian capitalism as a focus of my research. We all know that large-scale AI systems consume enormous amounts of energy. However, the specific details of these costs remain murky in public understanding. A major obstacle is Amazon Web Services, the world's largest cloud computing company, which remains almost completely silent about the energy footprint of its vast operations. Among global cloud service providers, AWS alone refuses to disclose even the most basic details about its energy performance and environmental impact.

At the same time, the popularity of artificial intelligence and machine learning gives people the illusion of democratization, but in fact all user labels and behavioral data are used to train neural networks. Therefore, in this era of data extractivism, the true value of this data is controlled and utilized by a very small number of people at the top of the pyramid.