Your website might be invisible to ChatGPT without this new file 👀 LL...

There's a new file you might have to start including on your website if you want large language models like ChatCPT to crawl you. It's called llms.txt and it could be very important for ranking in these large language models. So I just want to talk about everything we know so far and also what we don't know. So you might have heard of a robots.txt file that tells web crawlers like Google whether or not they're allowed to crawl certain pages of your website. llms.txt is like that for large language models, except instead of telling them what pages they're allowed to crawl, it shows them where the important information on your website is. So this is a markdown file you place in the root directory of your website, and it just has details about all the most important pages and what information can be found there. And this might be helpful because large language models aren't going to crawl every page of your website, and so you want to make sure you're directing them to the pages that are most important. I'll talk about how to add this to your website in a little bit, but first let's talk about if it actually helps. Currently llms.txt is just a proposed web standard. One of the biggest proponents of it is Anthropic. They're the makers of one of my favorite models, Claude. And so it's a pretty safe bet that they've already adopted it. On the other end of the spectrum is Google, who has just recently come out and said they are not adopting llms.txt, and that makes sense since they already have the most sophisticated search engine. Google actually recommended adding a noindex header to your llms.txt file so it doesn't show up in search results and confuse users. And then in the middle, and perhaps the most important, is ChatsBT, who to my knowledge have not made a public statement about it, but there have been some people reporting that they've seen ChatsBT crawling their llms.txt in their log files. Now just because they're looking at your llms.txt doesn't mean they're actually adopting the protocol, so we'll just have to wait and see on that one. But if you're wondering if you should add this to your website, you might as well. It's really easy. If you're on a WordPress site, you can just download a plugin. I think the most popular right now is websitellms.txt. The SEO plugin Yoast is now also offering one-click llms.txt creation. I'm sure other website builders like Wix will be following suit soon. But if your website builder doesn't support it yet, it's just a simple markdown file. Open up your favorite large language model. I would recommend ChatsB3's 03 model. And just have it walk you through the process of creating and adding the llms.txt file to your website. There's also something called an llms.full.txt, which also includes the important content on all those important pages. This is all happening incredibly fast, but if you're watching this, it means that you're keeping up. And I think there are going to be huge opportunities for those of us keeping up with it.