[tech] robots.txt format

Stephane Bortzmeyer stephane at sources.org
Wed Jan 27 10:27:41 GMT 2021


On Wed, Jan 27, 2021 at 10:40:39AM +0100,
 René Wagner <rwagner at rw-net.de> wrote 
 a message of 23 lines which said:

> simple question:

Complicated answers:

> is the following robots.txt format valid in a form that the
> "disallow" is applied to all User-agents mentioned before?

1) There is no standard for robots.txt.

2) There is not yet an "official" adaptation to Gemini, just
proposals.


More information about the Gemini mailing list