Re: Download a whole gopherhole using wget/curl?

To: gopher-project@other.debian.org
Subject: Re: Download a whole gopherhole using wget/curl?
From: "Mr. Leveck" <leveck@leveck.us>
Date: Thu, 28 Nov 2019 14:51:40 -0700
Message-id: <[🔎] 2738af6a-18d2-1c8e-d0e7-820d5bf572f8@leveck.us>
In-reply-to: <[🔎] 20191128174525.11D43101923F3@r-36.net>
References: <[🔎] 16eb08176ef.11eef69dc482927.3075840238943420488@kiwidev.xyz> <[🔎] 20191128103133.GB30671@brevard.conman.org> <[🔎] 20191128182223.0117879b@aluminium.mobile.teply.info> <[🔎] 20191128174525.11D43101923F3@r-36.net>

On 11/28/19 10:39 AM, Christoph Lohmann wrote:

Greetings.

For the crawling I already proposed for eomyidae and what if
follows some robots.txt for gopherspace.

See gopher://gopherproject.org/1/eomyidae for the details. All
automatic crawlers or archivers should keep to this standard.


Sincerely,

Christoph Lohmann

Thank you for bringing this up again. I have (re)added a robots.txt toRPoD.

The one I had was lost in a server migration about a year ago!

--
Nathaniel Leveck
gopher://1436.ninja
https://leveck.us

Reply to:

References:
- Download a whole gopherhole using wget/curl?
  - From: kiwidevelops <kiwidev@kiwidev.xyz>
- Re: Download a whole gopherhole using wget/curl?
  - From: Sean Conner <sean@conman.org>
- Re: Download a whole gopherhole using wget/curl?
  - From: Florian Teply <usenet@teply.info>
- Re: Download a whole gopherhole using wget/curl?
  - From: Christoph Lohmann <20h@r-36.net>

Prev by Date: Re: Download a whole gopherhole using wget/curl?
Next by Date: robots.txt (was Re: Download a whole gopherhole using wget/curl?)
Previous by thread: Re: Download a whole gopherhole using wget/curl?
Next by thread: robots.txt (was Re: Download a whole gopherhole using wget/curl?)
Index(es):
- Date
- Thread