Whether you like it or not, SEO (Search Engine Optimization) is changing. People have already started shifting to the newest and most powerful tools like ChatGPT, Gemini, Claude, and Grok, just to name a few, and among them, website owners who want to get indexed by those.
Some call it GEO (Generative Engine Optimization), others AIO (AI Optimization), and many LLMO (Large Language Model Optimization). However you like to call it, the AI era is here, and the need for optimizing for AI chatbots and assistants is becoming higher and higher.
I mean, let’s be honest, who’s not using at least one or two AI chatbots nowadays? You ask a question, you get an answer. It’s that simple. And although they’re not quite as powerful and smart as we may want them to be, they’re improving and growing by the minute.
Even my mother uses ChatGPT for recipes and various questions, while my best friend uses ChatGPT’s Voice feature to translate in real time whenever he visits another country.
My father loves asking about conspiracy theories and political topics in general, or other times, it amazes me with absolutely stunning prompts like, “Why do we say ‘sleep like a baby’ when babies wake up crying every two hours?”
The importance of optimizing our content for AI
As the days go by, more and more people are transitioning from the traditional search engines to more robust and way smarter AI chatbots and assistants. You can either start optimizing your content for those AI bots now or deny them any access and block them from scraping your content.
Blocking AI bots won’t affect anything to your SEO, and you can continue working like AI chatbots never existed. But by doing that, you may also be losing a lot of visitors, or even potential clients.
But, just how important have they become?
Based on DemandSage, OpenAI’s ChatGPT has now over 800,000 weekly visitors, and over 10 million users who have subscribed to ChatGPT Plus. It also leads in the world of LLMs with an impressive 81.47% market share. That’s huge for all the alternatives that currently exist.
Google, on the other hand, doesn’t quite share its usage statistics, but approximately, it is expected to have 1.5 to 2 billion weekly visits. Although when it comes to monthly visits, it’s extremely higher than its rival, ChatGPT.
With that in mind, it makes sense for website owners to start optimizing their content for AI chatbots and assistants by giving them a quick way to discover and easily understand their most valuable content.
That includes ChatGPT, Perplexity, Claude, DeekSeek, Grok, Microsoft CoPilot, Google Gemini, Meta AI, and the hundreds of others that you can currently use.
If you’re using WordPress as your CMS of choice, then you’ll probably already know Rank Math SEO, which is being used by over 2 million websites all over the world.
Why? Besides a powerful free version, their premium option is fairly cheap (at $6.99 per month, excluding VAT), and gives you 500 credits that you can use for their Content AI feature.
It has so many built-in features and options that in the end of the day, it also saves you from having multiple different plugins installed for specific optimizations, which is a security and maintenance nightmare.
Learn more about Rank Math SEO from these articles:
- CSS Styles that’ll make your Rank Math FAQ blocks stand out.
- How I’ve Made $10,000+ with WordPress and Rank Math SEO.
- You can now get a 100 SEO score using Rank Math’s Content AI.
- Yoast SEO vs. Rank Math SEO – Is it even a question anymore?
- 10 Things I wish my clients knew about Rank Math SEO.
Using the llms.txt file to help AI Chatbots index your site
Not long ago, Rank Math SEO announced a new feature for their premium versions, called “LLMS Txt“. It’s available for everyone, and you can easily enable it by going from your WordPress Dashboard to Rank Math SEO -> Dashboard.
What is the llms.txt file?
Basically, the llms.txt is located in your website’s root directory (example.com/llms.txt), and it’s a simple file in markdown format (.md) with a simple structure that’s human-readable.
This is what AI chatbots and assistants will search for whenever they need to scrape your content and give their users sources.
If that reminds you of something, you may have the robots.txt file on your mind. For people who don’t know, the robots.txt it’s a file (that Rank Math SEO helps you create with 1 click by the way), which website owners use to tell search engines which of their content they don’t want to crawl.
The llms.txt file, on the other hand, is basically like saying AI chatbots and assistants, “Hey, here’s the best content on my website.”
Here’s a comparison to better understand the difference between those two files:
| Feature | robots.txt | llms.txt |
|---|---|---|
| Primary Purpose: | To prevent or allow crawling of specific URLs by bots. | To guide LLMs to the most important, high-value content on the site. |
| Audience: | Search engine crawlers (e.g., Googlebot, Bingbot) and other web bots. | Large Language Models (e.g., Google Gemini, OpenAI’s GPT, Anthropic’s Claude). |
| Function: | Exclusion/Permission. It’s about what a bot can or can’t access. | Guidance/Curation. It’s about what a bot should focus on for better understanding. |
| Format: | A simple text file with specific User-agent and Disallow directives. | A Markdown file with headings, lists, and links, designed to be easily parsed by AI. |
| Standard: | A well-established, widely recognized protocol for decades. | An emerging and unofficial standard that is still in the early stages of adoption. |
The one is designed for traditional search engines, while the other is for optimizing the AI’s understanding of your website’s content.
And if you want your content to get indexed by everything, you need both of those available and configured the right way.
How to install Rank Math SEO and enable ContentAI
Downloading and installing Rank Math SEO is an easy, fast, and mostly automated process thanks to the team behind it, which made sure to introduce an installation Wizard that handles our setup easily.
First and foremost, head over to your WordPress site and sign in to your account, and then:
- From the WordPress Dashboard, click on “Plugins“.
- Search for Rank Math SEO.
- Click on “Install”.
- Click on “Activate”.
And there you have it, the free version of Rank Math SEO is now installed.
But before you proceed with the installation Wizard, keep in mind that if you’re going to be getting the Pro version, it actually unlocks even more features during our first setup.
Grab the Pro version of Rank Math SEO:
..or learn more about its Content AI.
Once you select the plan and complete the purchase, follow these instructions:
- Visit Rank Math’s ‘My Account’ page and sign in to your account.
- Go to the “Dashboard” page if you aren’t already there.
- Click on “Download” and choose where to save the .zip file.
- Go to your WordPress Dashboard.
- Click on Plugins → Add New.
- Click on “Browse” and select the .zip file, and click on “Install“.
A new option called “Rank Math SEO” will appear in the left sidebar on WordPress; if not, simply refresh the page. If the plugin didn’t automatically redirect you to the installation Wizard, you’ll find it there.
If you’re using Yoast SEO, All In One SEO, or any other WordPress SEO plugin, Rank Math will give you the option to automatically migrate all the data with 1 click.
You’ll also get the ability to restrict other authors or managers from accessing specific Rank Math settings or features, enable the auto-redirections feature, and much more.
How to enable the llms.txt file on WordPress using Rank Math SEO
Just for the record, I’ve been using the Pro version of Rank Math for many years now, and all the screenshots in the article show the plugin’s current latest version. The LLMS Txt module is provided in the free version, too.
Once you have the plugin installed, head over to Rank Math SEO → Dashboard, find the “LLMS Txt” module, and simply enable it.
If you want to configure the module, just click on the “Settings” button. Here are all the options that are currently available on the modules’ settings page:
Rank Math SEO gives us the option to select which types and taxonomies of our content we want to include or exclude, and its default limit for posts and terms is 50 (although I’ve changed it to 100). You can also include additional content that you may have, using text or links.
Your llms.txt URL will be available at the top of the page, letting you actually test it yourself.
Using the Content AI for research, ideas, post outlines, and writing
If you’re more of a person who prefers AI to do their complete job, Rank Math has you covered, as it gives you its own AI-integrated tool called Content AI, which gives you access to over 40 AI tools for any kind of work or need.
Once enabled from Rank Math’s Dashboard page, if you’re editing a blog post or page, you can find the AI icon in the top right corner.
It gives you a field in which you can give all your instructions (prompts), choose a tone (formal, informal, friendly, casual, creative, among many others), enter your main focus keywords, and finally choose the length (short, medium, or long) and click “Generate“.
If you click on the AI Tools section, you get access to a collection of pre-configured templates that you can use.
For example, if you want to organize your idea, select the “Blog Post Idea” template, or the “SEO Title” if you want help selecting something more “catchy”, etc.
Like most AI tools, this feature comes with a price. There are three different plans that you can choose from, starting with the Starter plan, which is the cheapest one at $4.99 per month (but billed manually), and gives you 12,000 monthly credits, 125+ curated AI prompt templates, access to RankBot (a personal assistant), long-form articles, and more.
If AI tools play a crucial role in how you generate content, do research, or even find new topic ideas, the Content AI module is one of the first you’ll want to enable from its Dashboard page.
What I found really useful was the “Paragraph Rewriter” template, which is sometimes very useful when I don’t quite like a specific paragraph, but I am not sure which words to change or use to describe something better, and it helps me by rewriting in a different way, in the tone I want.
Speaking of languages, currently, the Content AI by Rank Math SEO can generate content in 30 languages, so it’s pretty useful if you’re writing on a multilingual website. I’m pretty sure they’ll include more languages in the near future.
Avoid over-optimizing your website for AI
Look, optimizing our website’s structure and content for search engines and AI bots is a good way to increase our traffic, but overdoing it may lead to negative results. You need to keep your content original and as helpful as possible.
At the end of the day, all of those tools are only here to help us improve our content, decrease the hours we’re working on them, and make our research easier than ever before. If you’re content doesn’t meet some basic criteria, it won’t rank.
Use Rank Math SEO and its Content AI to help you understand, improve, and better optimize your content, but keep things simple and always focus on your readers rather than search engines and AI chatbots.
If you need further help, check out my previous article: The Ultimate SEO Checklist: Stay Ahead this year!
Until the next time,
Happy blogging!
Uh, oh! Legal stuff! Yikes.. Yet important! Affiliate Links Disclaimer: Many of the links on my articles are affiliate links. That means if you click on one and make a purchase, I might earn a small commission, and I'll keep doing what I'm doing, reviewing everything with absolute honesty. Sounds good? Good.
