On Tuesday, X (formerly Twitter) made good on Elon Musk’s latest promise to open-source the code behind its recommendation engine.
The big picture
While Musk framed the release as a “new era of transparency,” the move comes as the platform faces mounting regulatory pressure in the EU and a growing crisis over AI-generated content.
What’s new in the code
Unlike the 2023 release, which critics called “transparency theater,” this update shows a fundamental shift in how X builds your feed.
- Grok at the core: The recommendation system now relies on the same “transformer” architecture used by Musk’s AI, Grok.
- End of “hand-tuning”: X claims it has eliminated “manual feature engineering.” Instead of engineers hard-coding rules for what’s relevant, the AI learns entirely from user engagement sequences.
- The “Phoenix” stream: The algorithm uses a new retrieval module called Phoenix to find “out-of-network” content—posts from people you don’t follow—to keep the “For You” feed populated.
How the feed is built
The algorithm follows a four-step process to decide what you see:
- Candidate Sourcing: The system pulls roughly 1,500 candidate posts from two pools: people you follow (In-Network) and accounts the AI thinks you’ll like (Out-of-Network).
- Initial Filtering: Before scoring, the system scrubs posts from blocked or muted accounts, recently viewed content, and “low-quality” or “spam-like” posts.
- AI Ranking: The Grok-based transformer predicts the probability that you will like, reply, or repost. Each action is weighted; for instance, a reply is now reportedly worth significantly more than a simple like.
- Diversity & Final Polish: The system applies “heuristics” to ensure your feed isn’t just 50 posts from the same person.
Why it matters for creators
Early analysis of the code suggests the “meta” for reaching a wide audience has shifted:
- Engagement > Reach: “Author replies” are heavily weighted. If a creator responds to a comment on their own post, the algorithm gives that thread a massive boost.
- The “Link Tax”: The algorithm continues to penalize posts with external links, as X tries to keep users from leaving the platform.
- The Block Penalty: Being blocked or muted by users carries a heavy negative weight, meaning “hate-posting” for engagement can backfire by tanking your overall account reputation.
The “Transparency” catch
While X is sharing the logic of the code, it is not sharing the model weights—the specific numbers that tell the AI how much to care about one variable versus another.
- The “Dumb” Admission: Musk himself acknowledged the current system’s flaws on Tuesday, posting: “We know the algorithm is dumb and needs massive improvements, but at least you can see us struggle to make it better in real-time.”
- Regulatory pressure: The timing isn’t accidental. X is currently under a “retention order” from the EU to preserve algorithm data through 2026 and is facing scrutiny in California over Grok’s role in generating non-consensual deepfake images.
