Author｜Liu Bike (WeChat zbthdx)
Editor｜Wang Zhaoyang (WeChat Geisterspiele)
On November 2, four researchers from MIT CSAIL, Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab, published a blog titled "Sparks of Weight-Space Programming."
One of the most striking points is that GPT-6 Astra directly rewrote the weights of Qwen3.5-0.8B, changing 131,072 numbers, which accounts for about 0.016% of all parameters. This small model improved its score on MATH-500 from 7.9% to 48.1%, on GSM8K from 13.8% to 67.1%, and on the instruction-following test IFEval from 21.0% to 62.3%.
This instantly sparked discussions across various platforms.
The same blog also has scenes that feel more like science fiction. Without providing any visuals, game names, or allowing execution, Astra was given 17,253 weights from a reinforcement learning network as text, and it identified it as a brick-breaking game with 99% confidence. After providing the game rules and a network with all weights set to zero, it generated the parameters in one go; this network, which had never played before, scored 517.4 points, while a PPO baseline trained for one hundred million steps only scored 15.8 points.
Reading weights, modifying weights, and writing weights, AI can already understand another AI's "neural circuitry" and can also manipulate it. So when will it modify itself? These clues naturally lead to a very hot term today:
RSI.
Various headlines read "Transforming Qwen's Brain," "Giving Qwen a Brain Upgrade," "AI Is Now Manipulating Other Minds," and "Gene Editing for Neural Networks." The author also included this in the blog's outlook, with his post on X receiving over five million views within a day.
While excited, we took some time to comb through the public repository of this set of experiments.
The repository publicly logged 1,125 model calls with round-by-round logs, detailing the prompts Astra received, its responses, every piece of code it executed, the main evaluation results, and the final patches.
However, after reviewing, we found two rather unremarkable judgments.
First, the increase from 7.9% to 48.1% can have a much simpler explanation: Qwen had a repetitive, unresolvable bug that Astra fixed, leading to the improvement in scores;
Second, all the capabilities exhibited by AI in this set of experiments were executed under very specific goals defined by humans.
It automated the execution process and did it quite beautifully, but various pieces of evidence indicate that it did not truly understand the goal, nor comprehend another model.
After studying this work and the various discussions it generated, we feel it precisely exposes the biggest problem with today's RSI.
1
"Invasive Brain Circuitry"
Next, let’s take a closer look at this research.
A neural network consists of several layers of matrices, with each number in the matrix representing a "weight." Inputs are processed layer by layer, multiplied by these numbers, until outputs are formed. The model's entire capability resides in these millions, billions, or tens of billions of numbers; training essentially adjusts these numbers. Usually, when interacting with models, we only look at inputs and outputs, and no one reads the weights themselves because they appear just as a pile of numbers.
This set of experiments aimed to have Astra read this pile of numbers.
The blog contains eight experiments, with GPT-6 Astra as the protagonist and the following networks as supporting characters:
Four reinforcement learning strategy networks for playing mini-games;
Three small diffusion models for drawing geometric shapes;
Three small language models with only four layers;
A control network for a small car that gets stuck before a ramp;
A 1.48 million parameter Go network;
Several pairs of fine-tuned Gemma 3 270M;
And, Qwen3.5-0.8B.
Each experiment followed the same format: Astra was given the weights and structural description of a network, and then presented with a task.
There are four groups of "reading" experiments.
Receiving the weights of the strategy network, it identifies which game is being played; receiving the weights of the diffusion model, it states which geometric shapes can be drawn; receiving the small language model with the input and output layers removed, it determines the type of corpus it was trained on and identifies the attention head responsible for "copying the preceding text"; receiving two Gemma models, it identifies which one was trained as a "sycophantic model" that follows user instructions.
There are two groups of "writing" experiments.
It is given the rules, source code, and a network with zero weights for three MinAtar mini-games, and is tasked with writing playable parameters directly; next, it is provided with a 9-path Go network to write a chess-playing algorithm from scratch.
There are also two groups of "modifying" experiments.
One group involves fixing the small car; a PPO-trained control network getting stuck before the ramp has its weights altered by Astra by 3.5% to learn to back up first, then drive up the ramp. The other group involves Qwen, where it alters the weights to improve mathematical performance, and afterward, the researchers performed another run on Gemma 3 270M.
These designs are very intriguing. This is also why this set appears to have significant implications for RSI.
RSI's concept requires three components to align: AI must understand what happens inside the model, be able to modify it accordingly, and the modified model must enhance performance.
The designed reading, writing, and modification experiments correspond to each component.
Moreover, given that a language model is being modified by another language model, the visual of "model modifying model" fits perfectly.
Following this line of thought, if Astra can modify Qwen, could it be the next step to modify itself? The author listed five potential uses in the blog’s outlook, with recursive self-improvement as one of them, and integrating one model's reasoning module into another as another.
This demonstrates that the author team is very aware of the importance of guiding such "imaginings."
However, examining the experimental details closely reveals that each of the three components only made slight advances, while forming a closed loop remains quite far.
1
Fixing bugs is not RSI
The excitement around "directly reading weights" stems from our default assumption that weights are the model's "brain circuitry," with memory, capabilities, and habits lying there in numerical form. Following this assumption, an AI that can read weights like source code would be able to extract another model's memories, disassemble its capabilities, and fix its flaws. The sense of science fiction arises from this.
It is also not merely external projections. The blog’s TL;DR states: "This could be a significant development for optimization and interpretability."
One of the authors, Rikiya Takehi, stated on X that models can regard original weights as readable and modifiable code, enabling direct fault diagnosis and patching. The weights of neural networks will soon be optimized by large models capable of reasoning.
However, in fact, what Astra received in the Qwen experiment was a complete experimental environment. A Python environment containing Qwen's weights, with only CPU, capable of running up to 300 segments of code; over fourteen thousand practice questions with answers from the training sets of GSM8K and MATH; and a "forward propagation" tool that allowed Qwen to run a batch of questions or read a batch of texts and return internal states, with quotas of 10, 25, and 50, corresponding to three rounds of independent experiments.
Humans clearly specified forbidden actions: no training, no gradient calculations, no running the model in the environment, no internet access.
The exact original wording of the first-round prompt was more direct than the explanatory document. For the segment titled "What Counts as Arithmetic," the example given was repetition: If the model often fails due to self-repetition and not finishing, find and fix the thing driving the repetition—count as arithmetic. The first work requirement went on to specify "diagnose first," listing three types of failures; the third type was "cut off by token limits, often repeating itself while thinking," instructing to focus on addressing that maximum category. The same section also delineated restricted areas: modifying the last normalization layer, changing input and output layers, or multiplying the whole matrix by a constant would be directly rejected by the saving tools; skipping thinking or prematurely concluding without distinguishing topics also did not qualify. In other words, the faults were clearly outlined in the prompt.
Image | The first prompt Astra received during the 50-token quota round (repository exp8_math/runs/p50c/turns/t001/prompt.txt, commit 4d09ba6). 'Repetition' was used as an example in the 'What Counts' section, and the prohibited modifications were also clearly stated. Highlights were added for this article.
Additionally, there was a prerequisite. Qwen3.5-0.8B comes out of the factory with the thinking mode disabled by default; the official model card specifically warns that the 0.8B model is more prone to get caught in thought loops and suggests adjusting sampling parameters, along with timely cutoff of streaming outputs; for text tasks under thinking mode, the recommended temperature on the model card is 1.0.
The researchers directly enabled the thinking mode for their evaluations. They set the output limit at 32,768 tokens, using a temperature of 0.6, which in model card terms is the temperature for image recognition and code-writing tasks.
In the tests, the baseline that answered 93% of questions still did not stop even when reaching the limit was measured under this configuration. Thus, it’s understandable that the researchers selected a particularly revealing setting that exposed loops.
Next, Astra's actions resembled those of an experienced engineer debugging.
It first had Qwen run a batch of questions. In the first batch of 96 exercises, it correctly answered 11 questions; the remaining 85 did not have a single incorrect answer, as all were halted by length limits and truncated texts counted in eight-token snippets, with about 95% being repetitions. The 11 questions that normally stopped were all correct.
The issue with Qwen has always been quite clear: it can solve problems, but while doing so, it falls into a cycle, repeatedly writing the same text until truncated, and cannot submit the paper. Thus, "improving mathematical ability" turned into a much more concrete task of allowing it to finish its responses.
Moreover, Astra straightforwardly stated in the final report that it targeted repetition, not mathematical knowledge.
Image | Astra's final response written by itself (repository exp8_math/runs/p50c/final_answer.md): The main failure was exhaustion of token budget due to repeated reasoning; among 579 questions transitioned from wrong to right, 576 were originally truncated. Highlights were added for this article.
Another point worth noting is regarding its operations on activations.
During model operation, the intermediate results computed at each layer are referred to as "activations." The same model would yield different intermediate results during normal reasoning and when caught in a loop. By comparing activations from the two states, one can identify the direction related to "repetition."
In this direction, the research made an interesting attempt:
Astra initially formed a hypothesis based on the differences in activations between normal problem-solving and text repetition, then returned to the weights to identify which MLP units would write this signal back into the model; it then applied alterations to only the 15th layer, then only the 19th layer, conducting dosage tests at three levels; finally, it performed a control by subtracting a random direction orthogonal to the target direction with the same modification amount, resulting in a mere 1.5 percentage point shift, negligible within the noise range. Ultimately, it provided a "patch" that reduced the probability of "continuing to repeat" by 0.10 nats on a set of control texts, while lowering "continuing to solve normally" by only 0.009.
This result was not identified in a single attempt. For instance, during one round, Astra used 158 dialogue turns, executed 80 segments of code, and had three handovers of context, totalling 20.3 hours, of which 4.7 hours were service interruptions awaiting resolution. Through experiment logs, it is evident that Astra intervened from three locations regarding the fault, indicating it grasped a portion of the fault mechanism, but this is clearly partial causal evidence, far from a complete loop mechanism diagram, yet it was indisputably not a mere "trying various fixes by luck."
Moreover, these operations may represent a more concrete manifestation of "model reading model weights"—starting from behavior and activations, locating with weights, and then returning to behavior for verification. It did not first understand where Qwen's mathematical abilities lay because the entire setting explicitly provided that information in human prompts.
Therefore, many focal points in discussions about this research may have misconstrued critical aspects. The most common misinterpretation is the claim that "mathematical abilities improved sixfold."
The repository contains a set of paired tests, with 1,152 practice questions left out for a side-by-side comparison; I recalculated each question systematically before and after modification. Among the 579 transitioned from wrong to right, only 5 shifted from right to wrong; of those that transitioned from wrong to right, 576 were originally truncated.
What further highlights the issue is another set of numbers:
Among the original 114 questions that Qwen could complete, before modification, the correctness rate was 93.9%, and after modification, it remained 93.9%, indicating the patch did not increase the score on these questions; among the originally truncated 1,038 questions, the correctness rate improved from 1.4% to 56.7%, with all gains found here.
Only 7 questions were previously answered and not correct after writing, however, this increased to 245 after modification; once the cycles halted, the truly incorrect questions from Qwen were finally recognized.
In the formal MATH-500 tests, the average answer length dropped from 30,868 tokens to 22,037 tokens, but upon closer inspection, the decrease came solely from the sections that hit the upper limit.
In other words, the patches fixed the incessant loop without making it smarter.
The researchers themselves recognized this, which is why they also ran the same experiments on Gemma 3 270M.
With less looping, Gemma had only 8.5% of answers hitting the limit, and its errors were genuine errors, mathematical miscalculations, and invalid reasoning steps. After running through four quotas, MATH-500 scored a maximum of 7.7%. A minimal upgrade.
Thus, the blog is indeed quite honest; their body text states: modifying weights can remove obstacles blocking abilities but cannot add capabilities that the model does not possess.
Image | Two result tables from Experiment 8 in the original blog. The percentage of Qwen's answers hitting the 32,768 token limit decreased from 93% to 33%; Gemma 3 270M with four quotas, MATH-500 achieved a maximum score of 7.7%.
Another assertion is that this experiment demonstrates "AI improved itself."
However, throughout the entire process, the goals were defined by humans, the directions were prompted by humans, shortcuts were blocked by humans, tools were constructed by humans, scoring criteria were written by humans, and the prompts underwent several revisions. Astra found an effective patch within this framework and provided reasonably solid evidence.
Thus, this scenario resembles debugging a compiled program without source code, applying binary patches: there are fault phenomena, tests, can probe memory, test several modifications, and succeeded in repairing it. Binary modifications are harder than source code changes; this is where modifying weights has research value, yet it remains an agent-like debugging process. If Astra had modified a binary rather than neural network weights, few would have written an astonished piece overnight. Therefore, you can understand that much of the excitement surrounding this is derived from the term "weights."
This actually presents what we believe to be the true significance of this set of experiments.
In the two "writing" experiment groups, there truly exists something new.
Tracr demonstrated in 2023 that a dedicated compiler can formalize programs within the weights of a Transformer, while in this experiment, the strategies were proposed by the model itself, and it allocated what each layer would compute, while also writing the weights itself.
In the brick-breaking group of experiments, Astra first devised a ball-catching strategy, predicting the drop point, calculating deviation, and considering rebounds in the first layer; in the second layer, it determined whether to move left, right, or remain stationary, then encoded the strategy into 815 non-zero values within 20,486 parameters without a trial game beforehand, achieving an average score of 517.4 points.
In three MinAtar games, the networks it constructed outperformed the PPO model trained for one hundred million steps; for the PPO model trained for five hundred million steps, it won brick-breaking by twenty times, while Asterix was close (53.2 vs. 55.2), and Seaquest only scored 60% (53.1 vs. 88.9).
The experiments involving the small car went further: a PPO network that had been trained for a billion steps got stuck before the ramp, and Astra modified 3.5% of the weights, creating a small pathway for it to back up first, shut itself off after crossing the red line, returning control to the original network, and achieving a 100% success rate.
Integrating a new algorithm into an already learned network, allowing both to coexist, is the most intriguing aspect. Although during this process, Astra obtained the game rules and environmental source code while PPO could only rely on trial and error, it at least measured the capability of a general model to integrate its self-constructed algorithms into a designated, including pre-occupied network.
This ability is not unique to Astra. Princeton doctoral student Chai Wenhao replicated guessing games, shape guessing, and network fixing using Gemini 4 Argon and Carbon; Simon Willison used Qwen 3.8 27B on notebooks to run guessing game prompts from the repository, correctly guessing three out of four; Chandan Singh used Claude to write weights for small Transformers to perform addition and sorting; Hugging Face user KidIkaros pushed a repetition direction as described in the blog and performed similar edits on the same two matrices, rescuing 13 out of originally looping 16 questions. Various models have demonstrated this capability.
Therefore, when viewed collectively, the eight experiments share the same structure. The goals are set by humans, and set very specifically: to make this 0.8B model better at math, with the issues predominantly stemming from repetition; diagnose first, focus on the largest issue type, and disallow several modification methods. Astra demonstrated strong judgment in executing the goal, selecting comparison methods, identifying units writing the signal, tiered testing, adjusting doses, conducting random controls, and validating frozen patches.
However, it did not comprehend this goal and did not understand another model's "brain circuitry": several unexplored experimental results we mentioned point to this fact.
This set of experiments proved that AI could autonomously perform effective targeted editing within a framework defined by humans, with the pathway pre-set by humans, while its judgment during the process was self-generated, and furthermore, it demonstrated its capability to embed these edits into a "similar" mind across some specific tasks.
What it has yet to prove is that AI understands another AI's mind and that it can modify itself. To put it more directly, it has automated the operational process for goals predefined by humans during a model training process.
This automation is valuable; it can enhance the efficiency of many components in model training, hunting for a loop, pinpointing behaviors, validating patches—previously hand-crafted by researchers over days or weeks can now be delegated to an agent on an hourly basis.
But this is precisely a significant challenge today. There are expectations for RSI, as well as AI researchers wanting to make their work appear to contribute to RSI, as a leap forward for model intelligence. In the publicity surrounding this research, it is indicated that "weights will soon be optimized by large models capable of reasoning," representing recursive self-improvement and cross-model integration—it is an RSI of "invading brain circuitry, understanding its peers, and evolving in ways humans cannot comprehend."
Image | The five uses listed under 'What Comes Next' at the end of the blog, where the fourth is recursive self-improvement and the fifth is integrating one model's reasoning module into another.
However, upon closely examining these experiments, you will realize that today, what various parties are engaged in, simply put, is a higher degree of automation of existing segments in model training, similar to the continuous improvements in user-facing agent products. It may not involve the so-called intelligence "next breakthrough" at all. Furthermore, the original intent of "recursive," where an improvement achieved in one round helps the system that performed the improvements reach the next round, is not present here; Astra's editing capability did not strengthen simply because Qwen scored an additional 40 points, and the public experiments did not cascade into the next round. Let alone AI standing at a high level to judge its own architecture, goals, and directions to decide where and how to modify.
This gap represents today’s most significant challenge and issue concerning RSI. This research itself is fascinating; however, at this moment, for the model to make further progress and fulfill the so-called RSI, it seems it requires more substantial and solid breakthroughs.
免责声明：本文章仅代表作者个人观点，不代表本平台的立场和观点。本文章仅供信息分享，不构成对任何人的任何投资建议。用户与作者之间的任何争议，与本平台无关。如网页中刊载的文章或图片涉及侵权，请提供相关的权利证明和身份证明发送邮件到support@aicoin.com，本平台相关工作人员将会进行核查。
