[tech] robots.txt format

René Wagner rwagner at rw-net.de
Wed Jan 27 09:40:39 GMT 2021


Hi,

simple question: 
is the following robots.txt format valid in a form that the "disallow" is applied to all User-agents mentioned before?
---
User-agent: researcher
User-agent: indexer
User-agent: archiver
Disallow: about
---

or do i need to be more chatty?
---
User-agent: researcher
Disallow: about
User-agent: indexer
Disallow: about
User-agent: archiver
Disallow: about
---

kind regards
René


More information about the Gemini mailing list