The web has always needed little signs on the door
The internet looks like pages, apps, videos, stores, feeds, and endless buttons asking you to accept cookies like you are entering a diplomatic treaty. But underneath all of that, the web has always had a second layer: small, boring text files that tell machines how to behave.
They are not glamorous. Nobody launches a startup by saying, “We invented a tiny plain-text file.” Nobody’s keynote ends with confetti because a crawler discovered a sitemap. But these little files are some of the most important social contracts on the internet.
They are how site owners talk to bots. They are how developers protect themselves from accidentally publishing secrets. They are how search engines discover pages. And now, with AI assistants reading and summarizing the web, they are becoming the latest battlefield in the long argument between humans, machines, and “please do not scrape my entire life’s work before breakfast.”
If you read the earlier Notavello piece on the web’s machine-readable second layer, this is the origin story. The web did not suddenly become bot-friendly because AI arrived. It was already leaving notes for robots in the hallway.
First came robots.txt: the polite “do not enter” sign
In the early web, crawlers were not giant AI data vacuums yet. They were spiders, wanderers, indexers, and experimental bots built by people trying to map a strange new network. The problem was that even a well-meaning bot could accidentally hammer a server, wander into useless duplicate pages, or crawl places the site owner did not want crawled.
So the web needed a convention. Not a lock. Not a login. Not a lawyer in a blazer. Just a plain-text file called robots.txt.
The idea was beautifully naive in the way early internet ideas often were. A website could place a file at its root:
https://example.com/robots.txt
Inside, the site owner could write instructions like:
User-agent: *
Disallow: /private/
Translated from robot to human: “Hey, all crawlers, please do not crawl this folder.”
That “please” is doing a lot of work. Robots.txt is not a security wall. It does not stop a bad actor. It does not hide secret files. It is more like a sign on a hiking trail that says, “Don’t walk here; the ground is being restored.” Decent people obey it. Raccoons and venture-backed scraping companies may have mixed feelings.
Then came sitemaps: less “stay out,” more “start here”
Robots.txt tells machines where not to go. But as websites grew larger, search engines needed another kind of note: a map.
That is what sitemap.xml does. A sitemap gives crawlers a structured list of pages that exist on a site. Instead of waiting for a crawler to discover everything by following links like a very patient intern, the site can hand it a list:
Here are the important URLs. Here is when they changed. Please do not miss them.
This changed the relationship. Robots.txt was defensive. Sitemap.xml was cooperative. It said: “The site is big. Crawlers are busy. Let’s not make this weird.”
That pattern kept repeating: machines got more important, so websites started adding machine-readable side channels. Not because humans wanted to read XML. Humans do not want to read XML. XML looks like paperwork got into a fight with angle brackets. But machines are fine with it, and that was the point.
Developers needed notes to themselves too
Not every tiny file is for outside bots. Some are for the most dangerous user of all: the person building the thing.
That is where .gitignore enters the story.
Git tracks files in a project. This is wonderful until it tracks the wrong files: temporary folders, build junk, local settings, secret keys, giant dependency folders, operating-system crumbs, and that one experimental file named final-final-actually-working-copy-2.js.
A .gitignore file tells Git which files should stay untracked. It is a developer’s way of saying:
Do not commit this. Future me is not emotionally prepared.
That might sound minor, but it is one of the most practical safety rails in software. It protects repositories from clutter. It reduces accidents. It keeps local machine garbage from becoming team history. It does not fix every mistake — files already tracked by Git are not magically forgotten — but it prevents a lot of self-inflicted wounds.
This is a different kind of machine instruction. Robots.txt talks to crawlers. Sitemap.xml talks to search engines. .gitignore talks to the toolchain. But the core idea is the same: small plain-text rules keep fast systems from doing exactly the wrong thing very efficiently.
The family got bigger: env files, dockerignore, npmignore, and friends
Once developers got used to writing little instruction files, they appeared everywhere.
- .env files keep local configuration and secrets out of the code itself.
- .dockerignore tells Docker not to copy unnecessary files into a container build.
- .npmignore controls what gets left out of a package published to npm.
- .editorconfig helps different editors agree on formatting rules.
- humans.txt was a charming attempt to credit the humans behind a website.
Some of these files are about privacy. Some are about speed. Some are about consistency. Some are about not humiliating yourself in public.
They also reveal a funny truth about software: the computer is rarely the only thing being controlled. These files control humans too. They reduce the number of decisions a person has to remember. They turn “please don’t forget” into “the system will help you not forget.”
That is why they matter. A good little file does not just instruct a machine. It protects a workflow from human tiredness.
Then AI crawlers made the old handshake awkward
For years, the basic deal was simple enough: search engines crawled pages, indexed them, and sent traffic back. The arrangement was imperfect, but at least the trade was visible. A site allowed crawling. Google, Bing, or another engine helped users find it. The site got visitors. Everyone pretended this was a clean arrangement and moved on.
AI complicated that bargain.
When large language models and AI search tools began consuming huge amounts of web content, site owners started asking a new question: “Is this crawler indexing my page, summarizing it, training on it, replacing the visit, or all of the above?”
That is why robots.txt returned to the spotlight. A file created for polite crawlers in the 1990s suddenly had to carry the emotional weight of the AI age. Site owners began adding specific AI user agents. Some allowed search crawlers but blocked training crawlers. Others blocked everything that smelled like a bot with a venture capital budget.
This is where the old system started to creak. Robots.txt can say “do not crawl.” It does not explain the site. It does not summarize the best pages. It does not say, “If an AI assistant is trying to understand this site, read these pages first and don’t hallucinate that we sell refrigerators.”
llms.txt is the newest note on the door
That is where llms.txt fits in.
The idea is simple: put a Markdown file at the root of a website, usually here:
https://example.com/llms.txt
Instead of telling crawlers only what not to crawl, the site gives AI systems a clean, short guide:
- What the site is
- Which pages matter most
- Where the documentation lives
- What tone or context matters
- Which links are canonical
It is not magic. It is not an official universal law. It may or may not become a major standard. But it is very much in the same tradition as the files before it: a tiny, readable note for machines that are moving too fast to infer everything correctly.
The interesting part is that llms.txt is not just about blocking. It is about helping. Robots.txt says, “Don’t go there.” Sitemap.xml says, “Here are the pages.” llms.txt says, “Here is how to understand the place.”
The timeline is really a story about trust
These files tell a quiet history of the internet’s changing relationship with automation.
At first, the web assumed bots would be polite. Then the web realized bots needed maps. Then developers realized they needed guardrails against their own tools. Then platforms realized the whole system needed metadata, schemas, manifests, ignore files, feeds, package rules, and endless tiny declarations just to keep the machinery from chewing through the furniture.
Now AI has made the machine-readable layer more visible. Not new. Just visible.
When an assistant summarizes a page, when a search engine builds an answer box, when a crawler decides whether to index or skip a page, when a developer avoids committing a password by accident — somewhere nearby, there is usually a tiny file doing thankless work.
It is tempting to think the future of the web will be giant models, agents, browsers that click for you, and answer engines that reduce whole websites into three bullet points. Some of that is probably true. But the future will also be small text files with boring names. The internet loves glamour, but it runs on notes.
What site owners should actually do
If you run a website, you do not need to chase every new file format like it is a limited-edition sneaker drop. But you should understand the basics.
Use robots.txt to express crawler access preferences. Use sitemap.xml to help search engines discover your public pages. Use structured data where it genuinely describes the page. Use .gitignore and related ignore files to keep your own tools from publishing junk. And if your site has articles, documentation, tools, or explanations that AI systems might summarize, consider adding a modest llms.txt.
Not because it guarantees traffic. It does not. Not because Google will throw a parade. It will not. But because the web is increasingly read by machines before humans ever see it. A clean note at the door is cheap insurance.
For more on the crawler side of the story, the Notavello explainer What Is a Bot, Really? breaks down how bots became such a large part of internet traffic. And if you want the newer AI-search angle, How AI Actually Searches the Web explains why this is no longer just “Google with a nicer paragraph.”
The boring files won
The funny ending is that the most durable web technologies are often the least impressive ones. Plain text. Simple rules. Files with names that sound like someone forgot to finish them.
Robots.txt. Sitemap.xml. .gitignore. .dockerignore. llms.txt.
They are not the show. They are the stage directions. And as the web fills with more bots, agents, crawlers, assistants, build tools, deploy scripts, and automated everything, stage directions matter more than ever.
The machines are coming. Actually, they already came. The least we can do is leave them a note — and occasionally tell them not to touch the good towels.