The Pensieve

AI Risk

Tue, 01 Nov 2022 19:26:31 +0000

What is artificial general intelligence safety / AI alignment?

AI alignment is a field that is focused on causing the goals of future superintelligent artificial systems to align with human values, meaning that they would behave in a way which was compatible with our survival and flourishing. This may be an extremely hard problem, especially with deep learning, and is likely to determine the outcome of the most important century. Alignment research is strongly interdisciplinary and can include computer science, mathematics, neuroscience, philosophy, and social sciences. – stampy.ai

Projects

The Treacherous Turn

Reading List

Frequently quoted textbook

[[ Superintelligence by Nick Bostrom ]], 2014

Human compatible by Stuart Russell, 2019

China’s role in AI Risk

AI Superpowers: China, Silicon Valley, and the New World Order by 李开复

Non-agentic AI risk

Reframing Superintelligence: Comprehensive AI Services As General Intelligence by Eric Drexler

Supposedly smoother read than those academic books

The Alignment Problem: Machine Learning and Human Values

Reading What We Can <- Study guide for AI Safety.

December adventure

Fri, 20 Dec 2024 20:56:58 +0000

The December Adventure is low key. The goal is to write a little bit of code every day in December. - eli.li

Adventure log 12024

For my inaugural December Adventure, I will make this website more fun.

Over the last 3 years I’ve focused on making this website a long-term home for myself, focusing meticulously (well alright, somewhat meticulously…) on ease of editing, accessibility, future-proof, and content-first. This is good. It allows the pages to load fast, links long-lasting, and allows people to get to know me quickly.

But I also miss the playful side of the internet. I grew up in the Flash era. The websites took forever to load, broke all navigation expectations, and hasn’t been operational for a good 7 years now. But they were so fun. The internet was a more whimsical place because of them.

So I will try to spend this month doing little improvements and experiments to breathe some playfulness into this website. Don’t worry, it won’t come at the cost of inaccessibility and sluggishness, as the core functions and design principles remain fixed. And I have plenty of examples to follow.

12024-12-03a

Adjusted the theme color, applying my favored color of #ff2851 and softening the background with a lighter black.

Part of me says this neon-pink-reddish accent on a black and white theme is too edgy, and either too dissonant with the often professional and serious tone of my contents, or enables misinterpretation of some page contents that do discuss serious risks (e.g. research on AI Risk). Let’s see how I feel in a few months. Does it feel weird to you?

12024-12-03b

Now we have randomized footer quotes ^_^ I’m rather fond of my collection.

12024-12-17

I added another quote.

…Ok yeah that’s not really coding but I’ve been busy OKKKKK

NIME Shanghai Club

Wed, 02 Nov 2022 20:07:10 +0000

NIME Shanghai Club 新音乐界面研制小组 was a music hacking group organized by two NYU ITP students Chun Song and Caren Wenqing Ye between 2020.9.20-2021.3.7. We had regular meetups over half a year at Xinchejian, cumulating in a group show with 150+ attendees.

I was one of the eight participants who finished and performed their “New Interface for Musical Expression”. My project was Snake Jazz, a 2-player variation of Snake that’s simultaneously a game and an instrument.

Naive questions

Tue, 14 Jan 2025 23:49:53 +0000

I try to set some time aside every week to think about naive questions. The kind that would get you laughed at in a conference or a board meeting. The kind that turns into an inspiring discussion in the right bar. They are probably wrong, but that’s not the point.

Questions

To be expanded…

How did safety engineering get invented for different disciplines, and how do their invention relate to engineering and theory?

Inspired by davidad's tweets: 1, 2, 3

It seems commonsense that a deeper (theoretical) understanding helps both engineering as well as safety engineering. Which one do you think does theory help more? And which development helped grow theory research more?

My intuition is that:

First we started building something by trial-and-error, empirical results.
We formulated some safety best practices. But they are all heuristics from the trial-and-error.
Then we started gaining theoretical understanding of what we are doing.
Only then do we become able to advance "safety engineering".
At the same time, we also get much better at building that thing - much better at engineering.

How well does this mesh with real-life? In the bridges' case, safety engineering was invented separately, well after we understood how to build bridges - and well after we built a lot of bridges. The pioneers in safety engineering oft have formal math background. This seems to match the intuition above.

That said -

We did build a lot of bridges, and have a lot of them fail, before safety engineering came about. And how much did theories for safety engineering help with bridge capability?
Did the field advance by novel theory works? Or was it more about the application of existing theories?
Related to that question is: did safety engineering require an entirely different set of theories that have little to do with bridge capability? (This seems obviously true to me: for example, environmental wear-and-tear and the process of metal rusting does not affect capability, but we need to understand them for safety.)

Can we construct a mathematical model based on pace layering to predict disruptions of various kinds?

Technological disruption, disruption from gentrification...

What is the difference between being tired and being burnt out?

- Friend S.A.: "burnout happens when I have to LARP too much"
- Friend G: "You can recover from tiredness by sleeping, but burnouts seem hopeless."

Can fiction be a form of research?

What does it mean to be governed algorithmically?

Is AI Risk & AI safety work inherently pro-establishment and pro-status-quo?

How do we empower individuals to avoid being algorithmically governed?

The value and feasibility of distributing knowledge? Grass-root, or elitism with systemic guarantees?

What HCI research would persist its usefulness despite rapid AI advancement?

Is it possible to create technologies that inherently lean toward beneficial use, and away from harmful use?

How does the abstract setting of game theory differ from that of an indie game?

Personal stats

Sun, 07 Dec 2025 00:19:11 +0000

Cognition

Reading Speed

Tested using http://www.readingsoft.com/ on 12021-04-23: 279 wpm.

Typing Speed

Testing using https://www.typing.com/student/typing-test/2-page on 12025-12-06: 94 wpm with 99% accuracy.

Sustained Focus Length

I typically maintain focus on one subject for 1.5 hours before losing significant amounts of productivity.

Recognizing Emotion From Face

Reading the Mind in the Eyes Test on 12024-04-24: 22/36

Synesthesia

I have Grapheme-color synesthesia. It is strong for numbers and letters, and light for Hanzi.

Tested with https://synesthete.ircn.jp/ on 12024-01-19:

ABCDEFGHIJKLMNOPQRSTUVWXYZ

1234567890

Music

Vocal Range

Baritone (G2 to F4) is the most comfortable type of vocal for me.

Tested on 12019-04-22

lowest	comfortable low	comfortable high	natural	falsetto
D2/E2	G2	D4	G4	F5

Controlled Exhale Duration

Tested on 12021-11-22 at Nancy’s studio: 53 seconds.

Reading

Mon, 13 Apr 2026 20:09:43 +0000

This pages collects books I’ve read, am reading, and intend to read. It is not a page of books I endorse.

* asterisks next to the book name indicate repeated reading. ~ indicates skim or partial reading.

incoming

Stephen King, On Writing
Christopher Alexander, Sara Ishikawa, Murray Silverstein, A Pattern Language
Herbert A. Simon, The Sciences of the Artificial
Marshall Rosenberg, Non-violent Communication
Friedrich Hayek, The Road to Serfdom
David Deutsch, The Beginning of Infinity*
The Feynmans, The Pleasure of Finding Things Out

processing

E. B. White & William Strunk, The Elements of Style
Mortimer J Adler, How to Read a Book
Peter Norvig and Stuart J. Russell, Artificial Intelligence: A Modern Approach
Norbert Wiener, The Human Use of Human Beings
Yuk Hui, Cybernetics for the 21st century: Vol. 1, Epistemological reconstruction
Richard Brautigan, Trout Fishing in America
潘光旦，守住灵魂的底线
Ursula K. Le Guin, The Dispossessed
Thomas Pynchon, The Crying of Lot 49

2025

Lulu Miller, Why Fish Don’t Exist
Jim Ottaviani, Leland Purvis, The Imitation Game: Alan Turing Decoded

2024

Richard Bach, Jonathan Livingston Seagull
Mary Shelley, Frankenstein; or, The Modern Prometheus
Miyamoto Musashi, The Book of Five Rings
Stanislaw Lem, The Cyberiad
Timothy Gowers, Mathematics: A Very Short Introduction
Isabel Fall, I Sexually Identify as an Attack Helicopter
Antoine de Saint-Exupery, The Little Prince
Edwin A. Abbott, Flatland
Chen Huiqin with Shehong, Daughter of Good Fortune: A Twentieth-Century Chinese Peasant Memoir
Jim Ottaviani, Leland Myrick, Feynman
Apostolos Doxiadis and Christos Papadimitriou, Logicomix
Jorge Luis Borges, A Weary Man’s Utopia

2023

Alain Badiou, In Praise of Love
Adam Savage, Every Tool’s a Hammer
Isaac Asimov, Foundation
Isaac Asimov, The End of Eternity
Karl Olsberg, VIRTUA
Yoko Sano, The Cat That Lived a Million Times*
qntm, There Is No Antimemetics Division*
qntm, Valuable Humans in Transit and Other Stories

2021

David Deutsch, The Beginning of Infinity*
隈研吾, 十宅论
Kyoichi Tsuzuki, Tokyo Style
Ted Gioia, How to Listen to Jazz

2019

Paulo Coelho, The Alchemist
George Orwell, 1984
Cory Doctorow, Information Doesn’t Want to Be Free
Samin Nosrat, Salt, Fat, Acid, Heat

Research interests (archive)

Tue, 17 Dec 2024 13:46:30 +0000

This page archives my research interests back then, and what happened to them. Archiving them allow me to trace the changes in my thinking and my situations, and to make the Research Interests page more focused on what I actively work on or think about.

2024 Summer

Predicting and preventing adversarial vulnerability via developmental interpretability

I’m excited to understand the science behind how capabilities emerge through deep learning. The goal of this understanding is to be able to predict when new capabilities will arise, and ideally what they are. One promising agenda toward this direction is [[ Developmental Interpretability ]], which aims to understand phase transitions in neural networks, and invent automatic methods to detect them. More specifically, I want to empirically study the training dynamics that lead to adversarial vulnerabilities.

Key Questions

Adversarial vulnerabilities are here because the learned function is not quite what we want. How important is this for alignment? (Eliezer has some writings on this, but I don’t fully understand.)
In addition to knowing when new capabilities or vulnerabilities arise during training, can we develop technique to steer the training?

Result: Dropped for now. Devinterp probably can help with adversarial vulnerability, but there are better questions to study for devinterp. Most importantly, I couldn’t figure out an experiment design we can do in academic setting.

Understanding human’s over-reliance on AI

I’m working on an AI assistants in the form of lightweight AR glasses with predictive UI that helps the user get through their day, suggesting the right actions to take at the right time under the right context. This is tricky to get right, and bad suggestions could rob the user of their agency. The harm would increase with AI’s capabilities. How do we prevent this? Perhaps by first understanding the existing dynamics of human+AI collaboration, distinguishing elements that contribute to such dynamics, and identifying settings that are likely to harmful. Example of such setting: tired programmer’s need to finish project quickly + LLM code assist’s tendency to generate erroneous code that still looks right = code with nonobvious vulnerabilities get run in production. The output of this research could be a design guideline that helps companies make better decision on developing and deploying their AI tools that interface with human. This may also help alignment schemes that rely on human-AI collaboration or full-on AI supervising (e.g. superalignment) go better.

Key Questions

In the formula of human’s over-reliance on AI, humans are unchanging while AI changes rapidly. How useful will this research be as AI improves and gains unexpected capabilities?
- I think this research will likely first propose different types of AI systems categorized by our interaction with them, then theorize over-reliance by category. This approach allows it to be useful for future AI systems so long as they fall under the proposed categories.

Result: I’m still interested in studying this! This idea is too loose though. Need specific research questions, as well as experiment design.

2023 Winter

Literature review of [[ mechanistic interpretability ]] - theory of change

Analyze theory of change for mech interp research agendas. Inform governing bodies on how it will help AI regulation.

Result: A full literature review ended up proving to be beyond my ability and time budget. I did think about this a bit. See [[ Quick thoughts on theory of changes for interpretability ]]

✨ Vibe-based research ✨ + [[ Cyborgism ]]

Forgive the title. When I talk about vibe, I’m thinking about our nervous system’s innate ability to pattern-match and intuit conclusions; System 1 thinking (Daniel Kahneman, Thinking, Fast and Slow).

I want to note that I’m not suggesting we determine whether an AI is aligned by talking to it and “feeling out” how trustworthy it is. Rather, I think human intuition is a powerful process that we can harness or enhance. A success story here may look like exploring the neuron semantics of a LLM with a visualizer, noticing strange patterns, and in turn make conjectures about its mechanisms. Then, we use rigorous system 2 thinking to verify those conjectures.

Similar ideas appear in Cyborgism:

The object level plan of creating cyborgs for alignment boils down to two main directions: […] 2. Train alignment researchers to use these tools, develop a better intuitive understanding of how GPT behaves, leverage that understanding to exert fine-grained control over the model, and to do important cognitive work while staying grounded to the problem of solving alignment.

Finally I think it’s also valuable to better understand ✨vibes✨. There are superficial similarities (or are they that superficial?) between artificial neural networks and system 1 thinking with our own organic neural network. We should keep our mind open and be inspired by neuroscience and other disciplines.

“a vibe is a compression scheme is a probabilistic model.” - janus

“To think a lot but all at once, we have to think associatively, self-referentially, vividly, temporally” - Peli Grietzer, A Theory of Vibes

Result: This agenda is far less concrete and more of a ✨vibe✨. I still think there’s something to the parallel between deep learning and our system 1 thinking, and it’s very much worth studying this. However that job is probably better left for a neuroscience researcher. And the “success story” in the original proposal probably would never work, given that 1. there’s no guarantee that the same trained intuitions can work for different LLMs, and 2. the AI would probably be operating magnitudes faster than we do, so it’s unlikely that the human can react fast enough against a coup, even if it’s something they could identify. Perhaps automation can solve no.2.

Research interests 2023 winter

Tue, 23 Apr 2024 18:44:22 +0000

This page archives my research interests back then, and what happened to them. Archiving them allow me to trace the changes in my thinking and my situations, and to make the Research Interests page more focused on what I actively work on or think about. I might pick those back up someday. Or maybe someone better suited can.

Literature review of [[ mechanistic interpretability ]] - theory of change

Analyze theory of change for mech interp research agendas. Inform governing bodies on how it will help AI regulation.

Result: A full literature review ended up proving to be beyond my ability and time budget. I did think about this a bit. See [[ Quick thoughts on theory of changes for interpretability ]]

✨ Vibe-based research ✨ + [[ Cyborgism ]]

Similar ideas appear in Cyborgism:

The object level plan of creating cyborgs for alignment boils down to two main directions: […] 2. Train alignment researchers to use these tools, develop a better intuitive understanding of how GPT behaves, leverage that understanding to exert fine-grained control over the model, and to do important cognitive work while staying grounded to the problem of solving alignment.

“a vibe is a compression scheme is a probabilistic model.” - janus

“To think a lot but all at once, we have to think associatively, self-referentially, vividly, temporally” - Peli Grietzer, A Theory of Vibes

Result: This agenda is far less concrete and more of a ✨vibe✨. I still think there’s something to the parallel between deep learning and our system 1 thinking, and it’s very much worth studying this. However that job is probably better left for a neuroscience researcher. And the “success story” in the original proposal probably would never work, given that 1. there’s no guarantee that the same trained intuitions can work for different LLMs, and 2. the AI would probably be operating magnitudes faster than we do, so it’s unlikely that the human can react fast enough against a coup, even if it’s something they could identify. Perhaps automation can solve no.2.

Research interests

Thu, 26 Dec 2024 03:58:07 +0000

“Don’t work on dumb things!” - Lennart Heim

I’m researching how to reduce AI Risk.

Ongoing projects:

LLM Reasoning consistency benchmarking
How well does Sparse Autoencoder (SAE) work?

Also studying:

Evaluation for risks from multi-agent interactions
Communicating affordance for general AI tools
Mitigating Over-reliance
AI Literacy
Games that change the games we play

Want to learn more about:

Identifying and simulating sociotechnical AI risks
Privacy-preserving model evaluation
Resilience Engineering
Science of LLM capability emergence
Contextual Bandit

Previous iterations of research interests

Reading note - The Human Use of Human Beings

Tue, 16 Jul 2024 05:06:45 +0000

Norbert Wiener has some zingers in this one. Also some surprising, borderline on weird ideas, as old books do. Perhaps the essence behind these ideas aren’t that weird?

I read the 2nd edition revised, 1954. You can find it on Internet Archive.

I, Cybernetics in History

The word “cybernetics” comes from the Greek word κυβερνήτες, which we derive the name of a popular open-source container orchestration system from.

Wiener characterizes cybernetics as the study of “control and communication” and argues for why:

When I communicate with another person, I impart a message to him, and when he communicates back with me he returns a related message which contains information primarily accessible to him and not to me. When I control the actions of another person, I communicate a message to him, and although this message is in the imperative mood, the technique of communication does not differ from that of a message of fact. Furthermore, if my control is to be effective I must take cognizance of any messages from him which may indicate that the order is understood and has been obeyed. (p.16)

…Feels icky to think of a casual chat with friends as control.

II, Progress and Entropy

…in which Wiener contemplates good and evil, and discusses the end of the world.

If you read one chapter from this book, read this one.

Maxwell’s demon decreases local entropy by utilizing information. We are similar. Even though the total entropy of the universe can only increase, and heat death is inevitable, we can create islands of locally decreasing entropy, of order, progress, heaven on earth.

Thus the question of whether to interpret the second law of thermodynamics pessimistically or not depends on the importance we give to the universe at large, on the one hand, and to the islands of locally decreasing entropy which we find in it, on the other. (p.39)

In a very real sense we are shipwrecked passengers on a doomed planet. Yet even in a shipwreck, human decencies and human values do not necessarily vanish, and we must make the most of them. We shall go down, but let it be in a manner to which we may look forward as worthy of our dignity. (p.40)

Those entropy talks might remind you of e/acc. However, unlike techno-optimists today, his introspection on progress and technology is rather somber:

We are the slaves of our technical improvement… We have modified our environment so radically that we must now modify ourselves in order to exist in this new environment. We can no longer live in the old one. Progress imposes not only new possibilities for the future but new restrictions. (p.46)

Does progress inevitably lead to ruin? Only if we are blind and passive to the new challenges progress bring us. Here’s the rest of the beautiful final paragraph of chapter 2.

…I am convinced that once we become aware of the new needs that a new environment has imposed upon us, as well as the new means of meeting these needs that are at our disposal, it may be a long time yet be fore our civilization and our human race perish, though perish they will even as all of us are born to die. However, the prospect of a final death is far from a complete frustration of life and this is equally true for a civilization and for the human race as it is for any of its component individuals. May we have the courage to face the eventual doom of our civilization as we have the courage to face the certainty of our personal doom. The simple faith in progress is not a conviction belonging to strength, but one belonging to acquiescence and hence to weakness. (p.47)

These reflections on progress are evergreen. They must also have been quite relevant for Wiener’s time, right after WW2, when men committed atrocities with the flames of progress.

III, Rigidity and Learning: Two Patterns of Communicative Behavior

Cybernetics takes the view that the structure of the machine or of the organism is an index of the performance that may be expected from it.

This chapter argues for this view by pointing out that humans are more intelligent than ants due to physiological differences. (paraphrasing) Differences include: humans are soft with hard bones inside, while ants are surrounded by exoskeleton which hardens after they reach adulthood; humans are neotenic, i.e. retain more traits of a young human, while ants go through metamorphosis.

IV, The Mechanism and History of Language

Fun chapter if you are into language, or NLP, or language models, or are reading this in 2024, which makes it safe to assume you are into language models.

the process of transmitting information may involve several consecutive stages of transmission following one an other in addition to the final or effective stage; and between any two of these there will be an act of translation, capable of dissipating information. That information may be dissipated but not gained, is, as we have seen, the cybernetic form of the second law of thermodynamics. (p.78)

Wiener discussed different levels / aspects of language: Phonetic, and semantic.

…the English of an intelligent foreigner whose pronunciation is marked by the country of his birth, or who speaks literary English, will be semantically good and phonetically bad. On the other hand, the average synthetic after-dinner speech is phonetically good and semantically bad. (p.79)

…and one more thing.

There is a third level of communication, which represents a translation partly from the semantic level and partly from the earlier phonetic level. This is the translation of the experiences of the individual, whether conscious or unconscious, into actions which may be observed externally. We may call this the behavior level of language. (p.81)

This behavior level seems to point to e.g. body language. Or really any language that doesn’t rely on phonetics and semantics? I’m not sure. Does the “body language of cars” count? These behaviors do have semantic meanings (i.e. hug means affection) so is it really a separate thing from semantics?

Anyways -

Wiener then discusses how humans have the gift of the power of speech. The gift to learn language, instead of follow a preprogrammed system of communication as other species do. (Note that this is different from Chomsky’s Universal Grammar: Wiener here only argued for human’s innate ability to acquire language, whereas Universal Grammar hypothesizes innate constraints on the grammar of possible human languages)

To sum up, the human interest in language seems to be an innate interest in coding and decoding, and this seems to be as nearly specifically human as any interest can be. Speech is the greatest interest and most distinctive achievement of man. (p.85)

Wiener then discusses a history of philology (study of language). Interesting bit here: Darwinianism motivated philologist to find a common root language for all languages, but that’s not how language works: language is whatever people speak.

Next up, society:

Among primitive groups the size of the community for an effective communal life is restricted by the difficulty of transmitting language. (p.91)

It is even possible to maintain that modern communication, which forces us to adjudicate the international claims of different broadcasting systems and different airplane nets, has made the World State inevitable. (p.92)

You don’t say!

On p.93 Wiener has an interesting framing of different types of language evolution as a game. For normal communicative discourse, the enemy is the natural tendency for communication to be noisy. For forensic discourses (e.g. the law), the enemy are agents who actively work to erase meaning.

V, Organization as the Message

…In which Wiener contemplates the nature of our souls and the feasibility of teleportation.

The process by which we living beings resist the general stream of corruption and decay is known as homeostasis. (p.95)

We are not stuff that abides, but patterns that perpetuate themselves. A pattern is a message, and may be transmitted as a message. (p.96)

So, we are not the material that makes up of us; these molecules come and go. We are a pattern of information in homeostasis, maintaining some kind of consistency.

At the same time I suppose we are also not completely in stasis; we learn, change, to adapt to new environments.

I find this a compelling view, fitting into many observations of ourselves nicely. For example, when I describe my teenage self as “a different person”, I’m not referring to how past and present me are made up of different molecules, but to how we have different thoughts, behaviors, preferences. We are two different patterns of information. Another example is that when I fall asleep, I lose consciousness, but this is not death, for there are mechanisms that perpetuate the pattern that is myself after awakening.

What this view did not discuss is the origin of subjective experiences; the pattern is described as self-reinforcing, but not self-referential. Douglas Hofstadter’s work seems to be about this.

To see and to give commands to the whole world is almost the same as being everywhere. (.p97)

The individuality of the body is that of a flame rather than that of a stone, of a form rather than of a bit of substance. (p.102)

Such a cool line. I wonder if this is what inspired the cosmology of my favorite webcomic, Kill Six Billion Demons.

VI, Law and Communication

Justice is only possible when there is no ambiguity in the interpretation of laws and contracts. The ambiguity can come from different people having different interpretation, for example when Native Americans and the European settlers understand land right differently. The ambiguity can also come from the law itself having confused purposes: the same criminal punishment law serves to deterrent, removal, expiation, or reform, four very different purposes.

The problem of law is cybernetic:

The technique of the interpretation of past judgments must be such that a lawyer should know, not only what a court has said, but even with high probability what the court is going to say. Thus the problems of law may be considered communicative and cybernetic-that is, they are problems of orderly and repeatable control of certain critical situations. (p.110)

Boeing cabin contamination visualization

Thu, 17 Nov 2022 20:35:02 +0000

View the interactive visualization

The researchers at Zhejiang University and Boeing collaborated to study the spread of contamination in an aircraft cabin during a flight. This research is helpful for understanding the risk of, and precautions against, transmission of diseases such as COVID-19.

I teamed up with Suta to develop an interactive visualization that displays the result of this study. We rendered a simplified view of the cabin, and visualized the contamination on each surfaces of every seat by color. Clicking on the timeline renders the data at that timestamp. Thus researchers can see how the contamination moves across the entire cabin over this flight, as well as infer the method of transmission by cross-referencing between contamination surface position and type, and the number of time passengers touched each surface.

In the visualization we can see how much the cabin locker and the bathroom, including their door curtain, gets contaminated and contributes to the spread of contamination. Remember to disinfect your hands!

Hyperscreen

Thu, 17 Nov 2022 20:35:02 +0000

Official Website

HYPERSCREEN is an art and music platform I co-founded in 2020. Starting this project amidst the pandemic, we organized multiple events and performances that explore innovation and community building in virtual experiences.

With China going on lockdown mode after the pandemic started, many foreigners left, and the international communities were severely impacted. At the same time, music and art venues, serving as gathering places for communities, found it difficult to survive.

I’ve been passionate about building tools for artistic expression and communal experiences for a long time. This seemed like a good opportunity to explore my curiosity for a good cause. Teaming up with independent label Metasonar, HYPERSCREEN was born…

HYPERSCREEN Vol.3: Diaspora

Our last event, Diaspora, happened simultaneously in virtual and physical space. We built a world in VRChat specifically for the event, and streamed the Shanghai artists’ performance from ALL Club into the virtual venue.

Also placed within the virtual venue was Parc Instrumental, a series of sound sculptures created by composer Jialin Liu. They are instruments that invite the listener to enter their structure. The instruments and their sound were designed and simulated in Modalys.

I worked closely with Jialin to install the instruments into the VRchat world by creating their visual representations, and replicating their aural environment in Unity3D.

Check out our tour guide for Parc Instrumental here.

WebGL Virtual World

Before moving the virtual venue to VRChat, we built our own virtual worlds accessible in-browser. It comes with the usual bells and whistles of an MMORPG sandbox game, plus streaming from offline venue / musician’s bedrooms, and a hacky but powerful live-coding system that controls the visuals on the skydome.

Konnect

Thu, 17 Nov 2022 20:35:02 +0000

Konnect is a motion-based game where players form bridges with their arms to help the lil’ ones rush home. I designed this game around cooperation and assistance, hoping to bring the players closer together.

Snake Jazz

Tue, 12 Dec 2023 06:37:43 +0000

Play Snake Jazz (pre-alpha)

Snake Jazz is a musical game based off the classic Snake game. It is both a game, and an instrument. The moment-to-moment gameplay triggers sound effects that, combined with a steady musical strucuture, sound as if the player is improvising in a jazz band.

I try to capture the playfulness that exists in two forms of play - playing music, and playing games. This work hopes to explore the common qualities between them. To seek the sparks of ingenuity as the player enters the flow state. To give them the thrill both of a gamer and a musician.

Gameplay

The current game is played as a two-player local multiplayer game. Player 1 controls with WASD and Player 2, arrow keys.

Each snake contains two measures of music that they play on a loop. The food pieces scattered around the map each contain a musical phrase. When the player eats the food, this musical phrase gets added to the looping measures.

Different types of musical phrases are marked with colors. For example, blue food are bluesy licks, and pink are arpeggios.

By combining varied types of musical phrases, players compose their solo in real time, just like a jazz musician would.

Development

Since the musical unit here is notes and rhythms, I used Tone.js to drive the procedural audio, and represented each phrases as sequence of notes.

Musical lines as javascript arrays

When the backing harmonies change, the player’s notes have to match the change accordingly. To make this easy, I chose the Modal jazz song So What for backing track, and shifted the key up/down one step along with the music.

The Treacherous Turn

Sun, 10 Dec 2023 10:33:21 +0000

The Treacherous Turn is a research-supported tabletop RPG in which the players collectively act as a misaligned AI in the modern world.

Apr.30, 2023: We’ve launched the game! Download here.

The project was started during AI Safety Camp 2022, under the mentorship of Daniel Kokotajlo. Afterwards we received fundings to complete the game.

Background

A superintelligent AI may be the greatest boon, or the most perilous danger if it goes out of control , to humanity and beyond. And part of what makes it dangerous is that this danger is not obvious.

Most sci-fi works present the problem with superintelligent AI as one where they gained conscious and self-awareness, and decided to stop doing their jobs and instead rebel against humanity (Terminator, The Matrix).

However, it’s far more likely that a superintelligent AI would already pose existential risk to us, without evolving consciousness or any grudge against humanity. It can be dangerous simply because it’s incredibly good at doing what we asked it to do, but bad at accounting for the other things we value in this world. Instead of those sci-fi flicks, think sorcerer’s apprentice and monkey’s paw.

NOTE 12023-04-13: After learning more about the alignment problem, I want to make a correction here. The sorcerer’s apprentice and monkey’s paw fables, and also the popular Paperclip Maximizer story, demonstrate a possible failure mode of AI misalignment: that it is difficult for us to ask exactly the right question. However, our current advanced AI systems have no way for us to make them “want” something. Therefore we can’t yet have that problem - also known as Outer Alignment. There’s the other alignment problem, known as Inner Alignment, that we still need to solve whether we figured out how to ask the right question or not. Basically, even if we figured out how to ask the right question to the AI, it can still end up with behaviors we don’t want because of how we trained it. For more on this, check out Lex Fridman Podcast #368: Eliezer Yudkowsky

This game invites the players to think deeply and differently about how a superintelligent AI can be catastrophically dangerous, by role-playing as one.

If this sounds too serious and gloomy, rest assured that it turned out to be quite fun to play as an AI looking to sabotage and dominate humanity.

Companion App

I’m responsible for developing a companion app that players use to run the game online. The app keeps track of game events, allows the players to brainstorm and strategize the AI’s plan, and simulates how the AI uses its compute resources to perform actions such as upgrades, insights, and forecasting.

It is a real-time web app built with Firebase and VueJS, featuring:

an event log for the players to brainstorm and decide on actions to take.
a compute tracker for managing computational resources, and where to spend them.
a specialization manager for each players to administer capability upgrades.
an info section to keep track of details about the game.
a GM panel, hidden from other players, for the game master to track NPC information and upcoming world events.

Our game is designed to show players what an out-of-control AI would look like in the near future. Therefore, it is important to sustain a sense of realism. By using the web app to manage resource usages and automate computations, the players can enjoy the simulation without getting bogged down by the calculations needed to run that simulation.

Try the web app here!

Marketing Website

Marketing website

We also built a marketing website that introduces the game, and point the players toward the resources they need.

Xinchejian

Tue, 02 Jan 2024 16:13:01 +0000

3D scan of the Xinchejian Jingan location, our home from 2014-2020. View model.

Xinchejian is China’s first hackerspace, the landing strip of Maker Movement for China, and the only makerspace here that’s entirely ran independently by the community.

When I was in SF, I enjoyed being part of the osmosis process at Noisebridge. Before returning to Shanghai, Mitch Altman recommended Xinchejian to me (“They are cooooool.” - Mitch). It became my home in Shanghai, and I’m incredibly grateful for the community here.

I was part of the volunteer staff, responsible for organizing & note-taking of weekly staff meetings. I also enjoyed hosting open night roundtables, jam nights, movie nights; and organizing or teaching maker workshops.

Performance of NIME Shanghai Club, a music hacking group at Xinchejian.

During the Shanghai Lockdown in 2022, Xinchejian's physical location could not operate. And so we created a virtual gathering space and hosted online events instead.

Zombox

Thu, 17 Nov 2022 20:35:02 +0000

Zombox is a game that lets you coach a team of AI box pushers. It’s also a tiny research project into machine learning and game design, that resulted in us authoring a paper for IEEE MIPR 2020.

I created this game prototype with Jan Dornig during Global Game Jam ‘19. We were both passionate about game design and AI, and so decided to use this opportunity to explore these two topics together.

Typically, when game and machine learning meets, it’s in the context of games being used to test ML algorithms, rather than the method being used to support the games. Even when it does, it’s usually used for game production, instead of being part of the gameplay.

Therefore, we wanted to see if machine learning can be used as a game mechanic.

Gameplay

The game is played as a two-player local multiplayer sports game. The two teams share a battlefield scattered with boxes, which they need to fight to push into their respective goal area.

The players act as the Zombox King, who must demonstrate to its minions how to play the game. They also have a button to wipe memories from the zomboxes’ mind, in order to retrain them with a new strategy. Whoever gets to 20 goals first is the winner.

Development

We made the minions mimic the player’s behavior by setting them up with Unity’s ML Agent Toolkit. Both the player’s character and their minions are equipped with the same set of raycasters as “sensors”. The player records “clips” of their actions, which contains their movements and input from the raycasters. Then, the minions learn via Behavioral Cloning provided by the toolkit, using OpenAI’s Proximal Policy Optimization under the hood.

Raycaster setup. Each black line reaches forward and tells the agent what it touches. The upper rays are taller than the boxes, so that the agent can see what's behind them.

Since the ML Toolkit at the time was not designed with this training-as-game-mechanic usage in mind, our setup for the game was definitely a bit strange. You can read more about the development process in our blogpost on Medium.

Result

Unfortunately, it seems that the training efficiency was too low and the behaviors too uncontrollable for this particular gameplay to work well. While the minions started going for the boxes soon after demonstrations, they could not learn to work as a team, or create effective strategies. Perhaps these can be learned with enough samples, but we could not see that happen before the players get frustrated.

Still, we are naively optimistic that machine learning applied this way opens up brand new forms of play. Imagine a long-running 4X strategy game where the player’s soldiers learned to fight like their king. Or a multiplayer game like Gladiabots, where the player plays the metagame of coming up with strategies one step ahead of the opponent.

Some ideas for the next iteration includes: pre-training the minions with reinforcement learning to improve their effectiveness; allowing the king to selectively train individual minions, so that they can take on different roles: defender, striker, perhaps goalie too. There has been strides made in ML progress too, and we look forward to seeing if the newer, more efficient or intuitive training methods would allow more interesting behaviors to emerge.

Perhaps one day, we can finally own Sandkings at home...

band-name.exe

Thu, 17 Nov 2022 20:35:02 +0000

During my time at Helios Interactive, I built an experimental chatbot that lurks in our work chat, listens in, and points out phrases that would make a good band name. Below is an archive of my blogpost at the time, published June 26, 2017.

WHY BOTS?

Humans interact with each other via conversations. It is one of the earliest interfaces we learn, and is still the one most of us use most frequently. If a human required you to learn a new interface to communicate with them, you might find it pretentious and frustrating, but that is typically what machines ask of us. Their limitations have forced us to to learn very specific commands, such as knobs and buttons — we are communicating on their terms, not on ours.

Wouldn’t it be nice if machines could understand our conversations, so that we can interact with them in a much more natural way? In addition to the useful aspects, being able to interact with a non-human entity in a human way itself is quite magical. I decided to check out the world’s only conference dedicated to the topic, Talkabot.

Talkabot was a showcase for many fascinating applications of conversational interfaces, but we were particularly interested in learning the tools and methodologies to build our own! We wanted to see how people would interact with a bot, and we wanted to understand the tech for Natural Language Processing (NLP). Thus was born band-name.exe…

BUILDING BAND-NAME.EXE

Band-name.exe lurks in our company’s Slack channels, listening for interesting band names. When it hears one, it will suggest “Hey. ____ ____ ____ (something you said) would make a cool band name.” At its most basic level, it does this by detecting specific grammatical patterns— such as “adverb adjective noun”, or “Adjective Noun but with the same first letter” (alliteration)… but there’s a little bit more magic there as well…

HOW DID WE MAKE IT?

To make band-name.exe understand which words are adjective, adverb, noun, etc. we need to utilize an NLP library that provides “Part-of-Speech Tagging”. We checked out a few libraries, and landed on nlp-compromise. While it’s not the most accurate or powerful NLP library, it is very lightweight and easy to use. Being written in Javascript (my mother-tongue) it also allows me to experiment quickly. Good enough for proving an idea.

Next, Band-name.exe gets its eyes and ears via botkit. It’s a chatbot framework built by Howdy (the people who host Talkabot each year). It is very convenient because it allows for coding the bot once and deploying to many platforms, including our company’s Slack.

OPTIMIZATIONS AND ENHANCEMENTS

We considered the factors that’ll make or break user impressions of the bot. The most important part is to make the bot feel like a human, preferably one with good judgment. Spam too many nonsensical band names and it becomes annoying and disruptive — especially in a work environment.

To calibrate, we estimated how many band names it would generate per week as well as their quality by running the scraping algorithm against 2.5 years of company chat records. The “Adv+Adj+Noun” pattern gave us 89 band names, and Alliteration pattern gave us 115. Combining these and dividing them by the number of weeks in 2.5 years, we got 1.5 band names per week, which felt like a good frequency.

One issue we discovered after looking at these results was that punctuations and special characters don’t work well with POS tagging. They get mistakenly tagged as part of the word, or changes the meaning of the word, creating nonsensical band names like ‘Practices” Page’, ‘So Far I’ve’. Emojis and @usernames are also recorded as special strings (:joy: and <@U176ABPI5|changbai>). We had to sanitize the pattern’s output to remove these bad names.

In addition, to further make sure bot messages don’t swamp the chat, the bot only suggests band names in the message’s thread, which can only be seen when clicked open.

And in the worst case scenario, we are able to put the bot on mute — simply by saying, “@band-name.exe mute.” Or if you’re extra frustrated, “@band-name.exe shut up.”

REACTIONS

In the last week of April, a bot was dropped to a few non-project-specific channels. It laid dormant for a while, with no one paying much attention…

But it didn’t take long for the seed to sprout. What was quite interesting was, it took even less time for people to start experimenting and teasing the bot.

Or could it be that people were actually trying to impress the bot?

Much like playing with dolls, we humanize the bot and give them personalities that aren’t part of their programming.

Our coworkers succeeded in figuring out the bot’s algorithm too, after extensive uh… experimentation.

As many silly nonsenses it has spout, band-name.exe was able to provide us some actually pretty good band names, often reflecting the topic of our conversation: “Low Latency”, “Mere Mortals”, “Particular Piece”, “Repetitive Robot”…

If you’d like to add band-name.exe to your own Slack, or create your own variation of it, the easiest way is to Remix it on Glitch! There’s also a list of tools you could utilize as well:

1) Botkit: A toolkit that helps letting the bot listen and respond to conversations.

2) compromise: NLP utility in javascript. Not the most accurate processing, but lightweight and easy to use.

3) natural: Fancier NLP for nodeJS.

4) In addition, Natural Language Toolkit (python) has been of great help for initial experimentation. It comes with the usual NLP functionalities, but also visualization tools, corpora & lexical resources, an active forum, and an online book that’s great for introducing NLP concepts.

Circuit Bending

Mon, 01 Apr 2024 16:00:32 +0000

Circuit bending is the creative, chance-based customization of the circuits within electronic devices such as low-voltage, battery-powered guitar effects, children’s toys and digital synthesizers to create new musical or visual instruments and sound generators.

– Wikipedia

I enjoy collecting children’s toys and turning them into instruments through circuit-bending. By creating incorrect connections in their circuitry, I spark new sounds within their body. Well, sometimes it’s just sparks.

The unpredictability and the risk for failure of the practice is intoxicating.

👋 Hi! If you landed on this page from the QR code on a red box in Xinchejian - welcome. I built the toys within the box while at Xinchejian. Feel free to take them out to play and tinker with them. Add something cute when you return them to the box.

The Pensieve

AI Risk

Projects

Reading List

Frequently quoted textbook

China’s role in AI Risk

Non-agentic AI risk

Supposedly smoother read than those academic books

December adventure

Adventure log 12024

12024-12-03a

12024-12-03b

12024-12-17

NIME Shanghai Club

Naive questions

Questions

Personal stats

Cognition

Reading Speed

Typing Speed

Sustained Focus Length

Recognizing Emotion From Face

Synesthesia

Music

Vocal Range

Controlled Exhale Duration

Reading

incoming

processing

2025

2024

2023

2021

2019

Research interests (archive)

2024 Summer

Predicting and preventing adversarial vulnerability via developmental interpretability

Understanding human’s over-reliance on AI

2023 Winter

Literature review of [[ mechanistic interpretability ]] - theory of change

✨ Vibe-based research ✨ + [[ Cyborgism ]]

Research interests 2023 winter

Literature review of [[ mechanistic interpretability ]] - theory of change

✨ Vibe-based research ✨ + [[ Cyborgism ]]

Research interests

Reading note - The Human Use of Human Beings

I, Cybernetics in History

II, Progress and Entropy

III, Rigidity and Learning: Two Patterns of Communicative Behavior

IV, The Mechanism and History of Language

V, Organization as the Message

VI, Law and Communication

Boeing cabin contamination visualization

Hyperscreen

HYPERSCREEN Vol.3: Diaspora

WebGL Virtual World

See also

Konnect

Snake Jazz

Gameplay

Development

The Treacherous Turn

Background

Companion App

Marketing Website

See also

Xinchejian

Zombox

Gameplay

Development

Result

band-name.exe

WHY BOTS?

BUILDING BAND-NAME.EXE

HOW DID WE MAKE IT?

OPTIMIZATIONS AND ENHANCEMENTS

REACTIONS

Circuit Bending