[tech] robots.txt format
stephane at sources.org
Wed Jan 27 10:27:41 GMT 2021
On Wed, Jan 27, 2021 at 10:40:39AM +0100,
René Wagner <rwagner at rw-net.de> wrote
a message of 23 lines which said:
> simple question:
> is the following robots.txt format valid in a form that the
> "disallow" is applied to all User-agents mentioned before?
1) There is no standard for robots.txt.
2) There is not yet an "official" adaptation to Gemini, just
More information about the Gemini