AI Technology, Operating System AI Research, Decentralized AI (DeAI) Efforts and Open Source History

Author: @yb_effect

Compiled by: Baihua Blockchain

Imagine this: you've been staring at this super cool puzzle for a long time and finally decide to buy it. It's right in front of you, and all the pieces are there. You've got your coffee ready, music playing in the background, and the house is empty—just you and this puzzle. What are your thoughts?

The problem is—it's just so overwhelming. Where do I even start? Am I being foolish… why do none of these pieces seem to fit?

However, after calming down and thinking rationally, I realized that I first needed to categorize the pieces based on which area they belong to. Then, I would connect some simple parts to gain small insights and explore deeper from there. This has basically been my experience over the past week.

This frustrating yet enjoyable puzzle is called "Decentralized and Open Source AI."

My goal for the next few months is to become an expert in "Decentralized AI (DeAI)," and I want to be a thematic expert who can glue these two fields together by deeply understanding the latest developments at the forefront of open source AI and decentralized AI.

This means I need both depth and breadth—I don't want to have any knowledge gaps in this field.

In today's post, I want to share the framework for how I plan to achieve this goal. Below are the "puzzle areas" that I find most meaningful, along with brief thoughts and to-do lists for each area.

Mainly divided into the following sections:

AI Basics: Catching Up Isn't Hard
Open Source AI Community: How Far Behind Are Closed Source Competitors?
Decentralized AI: What is the Golden Basket?
Open Source History: What Key Learning Experiences Are There?

Let's dive deeper.

1. AI Basics: Catching Up Isn't Hard

This week, I started trying to write an article about Nous Psyche. Obviously, I began by reading the documentation and even got a rough understanding of most of it at a high level.
But I didn't really understand it.

So, I decided to read the paper on Nous Distro, thinking it might help me. Even then—with the help of ChatGPT—I could only grasp it at a high level.
But I still didn't truly get it.

This made me realize that I have a significant gap in my AI knowledge. Without mastering the basic AI concepts, I can't derive new insights. So, I decided to jump into the deep end. I started with Karpathy's famous introductory lecture on LLMs (Large Language Models). After taking notes, I felt much more confident. Many of the terms I had heard on Twitter and podcasts began to make more sense.

As a natural math enthusiast, I decided to delve deeper into how neural networks and transformers work. So, I binge-watched the complete series on LLMs by 3blue1brown.

By this point, I was very excited and began to have a clear understanding of many AI concepts. If you're like me, you can surely relate to that frustrating feeling when you can't fully grasp the entire system of a new concept (I still hadn't completely figured it out, but I had finally entered that rabbit hole).

Then! It was as if the god of technology was listening to my thoughts. Just as I finished watching those videos, Andrej Karpathy released a 3.5-hour video covering all the intermediate knowledge you might want to know about LLMs—perfect timing.

Believe me when I say this… if you're feeling imposter syndrome in the AI field, I guarantee this video can single-handedly help you tackle that issue. I've only watched 2 hours of it (progress is slow, still taking notes), but I can already say it's one of the best tutorial videos I've ever seen.

Seeing the tools and resources he used in the video really helped me a lot:

• FineWeb: Essentially "internet data" compressed for you via web crawling.
• Common Crawler: An open repository for scraping all internet data.
• Tiktokenizer: Choose a model, and you can convert between plain text and tokens.
• BBY Croft: An excellent LLM visualization tool to help understand the training process.
• GPT-2 from scratch: Karpathy's repository to do everything from scratch.
• Hyperbolic: Access pre-hosted models without setup.
• UltraChat: A synthetic data improvement repository for post-training processes.
• Nielsen's Textbook: Follow along with 3blue1brown's examples for hands-on practice. After completing the last hour of the above LLM tutorial, my next plans are as follows:
  • √ Familiarize myself with Hugging Face
  • √ Follow the ChatGPT-2 tutorial and run it from scratch
  • √ Briefly read the Llama and Deepseek papers to understand open source SOTA models
  • √ Read Leopold's situational awareness paper to clarify that the AI basics section is a phase that requires more warming up; I am currently in this phase. Once my foundational knowledge is clear, the next step is to keep up with the latest papers and releases. Just like learning about cryptocurrency, you have to enter that "rabbit hole" first.

2. Open Source AI Community: How Far Behind Are Closed Source Competitors?

I know there's been a lot of hype around Deepseek and open source lately, but I'm still trying to figure out where we stand in the specific progress of open source AI development.

Fortunately, Lex Fridman's podcast this week introduced me to Nathan Lambert and his work at Ai2. Nathan has been an advocate for open source AI for the past few years, and he wrote an excellent Substack article titled "Interconnects," covering many dynamics in the open source AI world.
Just yesterday, he published an article explaining why he believes the recent Deepseek news should serve as a huge wake-up call for Americans to increase their investment in open source.

The core argument of the article is that for the past 30 years, China has been replicating Western technology and improving its marginal benefits. But in the current AI race, it is clear that they are trying to be the ones spreading innovation. So, if American companies do not make an effort to open source their models, other countries (including Americans) will quickly adopt Chinese technology.

A passage in the article that made me realize the reality is:

Building powerful AI models is much easier than establishing a sustainable open source ecosystem around AI.
Creating a better, truly open AI ecosystem has been my lifelong pursuit over the past few years, and I obviously want it to thrive further, but the closer you get to the core of the current open source ecosystem, the more you realize that it is not a given, as the costs of relevant AI training are skyrocketing (look, I know DeepSeek has low computational costs, but these organizations are not easy to establish), and many regulatory bodies are acting quickly to seize the initiative in the AI field, which may inadvertently hinder the open process.
Yes, efficiency is constantly improving, and costs will decrease, as demonstrated by DeepSeek V3, but training truly open models at the forefront has not become easier.

Specifically, "the closer you get to the core of the current open source ecosystem, the more you realize that the costs of relevant AI training are not a given."

To be honest, I am the first to admit that I do not fully understand the specifics of the open source AI field. But! I must say, I was surprised that in the articles I read this week from open source AI researchers like Nathan and Tim Dettmers, there was absolutely no mention of cryptocurrency.

They may not value the crypto industry, thinking it is filled with scams, schemes, and various issues. But I do not agree with that view.

While reading Nathan's article, I couldn't help but notice that the core issue he raised is actually a cost issue. If that's the case, then undoubtedly, crypto incentive mechanisms could significantly enhance the efficiency of open source research.

So, have they considered cryptocurrency and concluded that "it's not worth the investment"? Or are they just generally avoiding the crypto space?

If it's the former, I would like to see some concrete examples. If it's the latter, then that's their blind spot, and I hope to reinforce it.
In any case, I've only just scratched the surface of the open source AI field, but Lambert's work is undoubtedly a great starting point.

One thing is very clear to me: these open source AI researchers and practitioners in the crypto industry are actually discussing similar issues. As I mentioned in last Friday's article, the challenges faced by open source AI are very similar to those of Bitcoin, and without a doubt, these two communities (crypto and open source AI) need more collaboration. I hope I can contribute to bridging this gap.

This quadrant's task list:

• √ Watch Lambert's talk on open source AI at Harvard Law School
• √ Understand Ai2's work and current progress
• √ Find companies and researchers similar to Nathan/Ai2 to gain a comprehensive understanding of the current state of open source AI (and build a Twitter list)
• √ Understand different types of licenses and their implications (e.g., Amazon needs to pay Meta for Llama, while Deepseek uses the MIT license, which is more "free")
• √ Form a perspective on the current state of open source AI from the viewpoint of Lambert-type researchers. Try to clarify the main bottlenecks currently and understand which solutions they consider effective (excluding biases against cryptocurrency)

3. Decentralized AI: What is the "Golden Basket"?

Before starting this section, I recommend spending 10 minutes reading this article to understand why AI particularly needs cryptocurrency. I've discussed the core points in previous articles, but I think @dbarabander did an excellent job clearly explaining the relationship between the past, present, and future.

Source: https://variant.fund/articles/why-good-ai-needs-crypto/

He accurately pointed out the resource issues facing open source AI and explained why cryptocurrency can solve this problem.

One thing that caught my attention was the list of valuable companies I received in the acknowledgments section from those who reviewed and edited this article. This will serve as a starting point for me to delve into the field of decentralized AI after mastering the basics of AI. It’s worth noting that I know this information may have some bias due to personal interests, as Daniel works at Variant, but nonetheless, this information is quite helpful to me.

Recently, I've been pondering a question: How well do people in the AI field understand decentralized AI (what we refer to as cryptocurrency) companies?

I don't have an answer yet, but I must admit it was really cool to see Andrej Karpathy using Hyperbolic in his new LLM tutorial video (thanks to @Yuchenj_UW for pointing this out).

You might be wondering… YB, why is it taking you so long to understand these decentralized AI companies? Well, as I mentioned earlier, like Nous Psyche, I am limited by my knowledge bottleneck in AI. For example, 48 hours ago, I couldn't understand the meaning of the tweet below at all.

But! I can confidently say that as I complete my rapid learning of AI basics, I will quickly familiarize myself with these companies in the next two weeks and form my own opinions.

My goal is to create a "Golden Basket" of decentralized AI companies. If I could only invest in these 5 startups, which ones would I choose? Why? Next, my plan is to become the biggest supporter of these teams. So far, I have already mentioned two companies multiple times in my articles over the past few months—Nous and Prime, which may make it onto my investment list.

@0xPrismatic, a friend and peer in the crypto and AI fields, is doing a fantastic job introducing these companies through his newsletter, and I can't wait to start reading his introductory articles. If you haven't read "Chain of Thought," you must check it out; it's definitely worth it!

Overall, the core argument of decentralized AI is: Our task is to make it all a reality.

Specifically, this is a perfect summary by @krybharat, with two specific areas being:
1) Decentralized and incentive-based open source model aggregators
2) Decentralized data providers offering near real-time data access

Therefore, here is the to-do list for this section:

• √ Browse Nous, Prime, and Hyperbolic next week and post a Twitter summary
• √ Read Ronan's article "Can Decentralized AI Compete?"
• √ Check if there are other DeAI companies mentioned or used by AI researchers (e.g., Hyperbolic-Karpathy type examples. I know Nous is widely discussed, but also check other companies)
• √ Start reading resources in the CoT introductory article
• √ Learn more about Teleport and Nous's work in TEEs
• √ Write my own perspective on "Why Crypto Needs AI" and create the Golden Basket

4. Open Source History: What Key Learning Experiences Are There?

Finally, let's talk about open source. I'll keep this part brief. While it's not strictly necessary to understand open source history, I believe that spending time learning about the history of open source can help me better understand why the struggle for DeAI is so important.

To be honest, I'm not old enough to remember the Linux era or the early 2000s when open source was the norm in computing. I grew up in the era of Facebook, Twitter, and Uber, where centralized platforms have always been the default choice.

So, I want to learn about what happened in the past with open source and what lessons can be learned from those examples.

A few days ago, while listening to the BG2 podcast, Bill Gurley compared Meta's Llama strategy with the history of companies like MongoDB. I found this comparison very interesting and really want to delve into the details. Many long-time readers know that I have a keen interest in technology history, so this part allows me to draw some comparisons between the past and present.

I believe this will not only help me gain unique insights in this field but also align me more closely as I become an expert at the intersection of crypto and AI.

I also want to read "Working in Public" and "Cathedral and Bazaar."

If you've made it this far, I guess you, like me, are passionate and excited about the DeAI field. If you want to join a discussion group and learn about DeAI with us enthusiasts, feel free to DM the author.

Article link: https://www.hellobtc.com/kp/du/02/5672.html

Source: https://x.com/yb_effect/status/1889792237216870445

免责声明：本文章仅代表作者个人观点，不代表本平台的立场和观点。本文章仅供信息分享，不构成对任何人的任何投资建议。用户与作者之间的任何争议，与本平台无关。如网页中刊载的文章或图片涉及侵权，请提供相关的权利证明和身份证明发送邮件到support@aicoin.com，本平台相关工作人员将会进行核查。