You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Jeff Cohen <ap...@gej-it.com> on 2003/01/02 09:37:49 UTC
[users@httpd] robots.txt
Hi All,
Does anyone of you get the robots.txt file requests from the internet??
It seems that I'm starting to get it almost every 3-4 hours a day, every
time from a different IP.
Is there a Virus going outta there that I'm not aware of?
How can I block these requests for this file on the root directory?
Help would be very appreciated,
Jeff Cohen
Re: [users@httpd] robots.txt
Posted by Chris Taylor <ch...@x-bb.org>.
robots.txt-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
robots.txt is a file placed in the webroot used to control how
spiders "crawl" through your site for search-engine indexing. It's
nothing to worry about, all search engines look for it as they index
your site.
HTH :)
Chris Taylor - chris@x-bb.org - The guy with the PS2 WebServer -
http://www.x-bb.org/chris.asc
- ----- Original Message -----
From: Jeff Cohen
To: Apache Group
Sent: Thursday, January 02, 2003 8:37 AM
Subject: [users@httpd] robots.txt
Hi All,
Does anyone of you get the robots.txt file requests from the
internet??
It seems that I'm starting to get it almost every 3-4 hours a day,
every time from a different IP.
Is there a Virus going outta there that I'm not aware of?
How can I block these requests for this file on the root directory?
Help would be very appreciated,
Jeff Cohen
-----BEGIN PGP SIGNATURE-----
Version: PGPfreeware 7.0.3 for non-commercial use <http://www.pgp.com>
iQA+AwUBPhP9tCqf8lmE2RZkEQKEHwCeMpNY8p9rPMLbujJRHXbWzT43hh0Al2WB
lSk9t4IpcOePAc9VV8HxOrE=
=SiLQ
-----END PGP SIGNATURE-----
Re: [users@httpd] robots.txt
Posted by Jurgen <ap...@squarehosting.com>.
Hi Jeff,
Chris is completly right. If you went for example to the google web site you find for example information how to prevent google from caching your web site via an entry in a robots.txt file.
Jurgen
On Thu, 2 Jan 2003 05:26:10 -0500
"Jeff Cohen" <ap...@gej-it.com> wrote:
> That's the problem, I don't have such file robots.txt. I guess the
> content should be text isn't? :)
> But Chris says not to be worried, so I'm not.
> Thanks Chris.
>
> Jeff Cohen
>
> > -----Original Message-----
> > From: Dharmendra.T [mailto:dharmu@nsecure.net]
> > Sent: Thursday, January 02, 2003 4:09 AM
> > To: users@httpd.apache.org
> > Subject: Re: [users@httpd] robots.txt
> >
> > On Thu, 2003-01-02 at 14:07, Jeff Cohen wrote:
> > > Hi All,
> > >
> > > Does anyone of you get the robots.txt file requests from the
> internet??
> > > It seems that I'm starting to get it almost every 3-4 hours a day,
> every
> > > time from a different IP.
> > > Is there a Virus going outta there that I'm not aware of?
> > > How can I block these requests for this file on the root directory?
> > >
> > > Help would be very appreciated,
> > >
> > > Jeff Cohen
> >
> > What is the format of the file? Try
> > #file robots.txt
> >
> > What does the file contains?
> > --
> > Dharmendra.T
> > Linux Enth
> >
> >
> > ---------------------------------------------------------------------
> > The official User-To-User support forum of the Apache HTTP Server
> Project.
> > See <URL:http://httpd.apache.org/userslist.html> for more info.
> > To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> > " from the digest: users-digest-unsubscribe@httpd.apache.org
> > For additional commands, e-mail: users-help@httpd.apache.org
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] robots.txt
Posted by "Dharmendra.T" <dh...@nsecure.net>.
On Thu, 2003-01-02 at 15:56, Jeff Cohen wrote:
> That's the problem, I don't have such file robots.txt. I guess the
> content should be text isn't? :)
> But Chris says not to be worried, so I'm not.
> Thanks Chris.
>
> Jeff Cohen
>
> > -----Original Message-----
> > From: Dharmendra.T [mailto:dharmu@nsecure.net]
> > Sent: Thursday, January 02, 2003 4:09 AM
> > To: users@httpd.apache.org
> > Subject: Re: [users@httpd] robots.txt
> >
> > On Thu, 2003-01-02 at 14:07, Jeff Cohen wrote:
> > > Hi All,
> > >
> > > Does anyone of you get the robots.txt file requests from the
> internet??
> > > It seems that I'm starting to get it almost every 3-4 hours a day,
> every
> > > time from a different IP.
> > > Is there a Virus going outta there that I'm not aware of?
> > > How can I block these requests for this file on the root directory?
> > >
> > > Help would be very appreciated,
> > >
> > > Jeff Cohen
> >
> > What is the format of the file? Try
> > #file robots.txt
> >
> > What does the file contains?
> > --
> > Dharmendra.T
> > Linux Enth
> >
> >
> > ---------------------------------------------------------------------
> > The official User-To-User support forum of the Apache HTTP Server
> Project.
> > See <URL:http://httpd.apache.org/userslist.html> for more info.
> > To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> > " from the digest: users-digest-unsubscribe@httpd.apache.org
> > For additional commands, e-mail: users-help@httpd.apache.org
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
This link should help you:
http://www.searchengineworld.com/robots/robots_tutorial.htm
--
Dharmendra.T
Linux Enthu
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] robots.txt
Posted by Rich Bowen <rb...@rcbowen.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Thu, 2 Jan 2003, Jeff Cohen wrote:
> That's the problem, I don't have such file robots.txt. I guess the
> content should be text isn't? :)
> But Chris says not to be worried, so I'm not.
See http://www.robotstxt.org/wc/robots.html for a complete description
of what you should put in this file. You should not be "worried" about
it, but you should learn about it, and create one to put on your site,
if you want your site to be indexed by search engines, and you want them
to do it in a friendly manner.
- --
Oh I have slipped the surly bonds of earth
And danced the sky on laughter-silvered wings
--High Flight (John Gillespie Magee)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)
Comment: Made with pgp4pine 1.75-6
iD8DBQE+FCOsXP03+sx4yJMRArVTAKCAc4E5UZmoHh/0s6MtUwDBFAziswCfTRDU
DI16Lc3spdKJokOMd0YBkAE=
=LtEw
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] robots.txt
Posted by Jeff Cohen <ap...@gej-it.com>.
That's the problem, I don't have such file robots.txt. I guess the
content should be text isn't? :)
But Chris says not to be worried, so I'm not.
Thanks Chris.
Jeff Cohen
> -----Original Message-----
> From: Dharmendra.T [mailto:dharmu@nsecure.net]
> Sent: Thursday, January 02, 2003 4:09 AM
> To: users@httpd.apache.org
> Subject: Re: [users@httpd] robots.txt
>
> On Thu, 2003-01-02 at 14:07, Jeff Cohen wrote:
> > Hi All,
> >
> > Does anyone of you get the robots.txt file requests from the
internet??
> > It seems that I'm starting to get it almost every 3-4 hours a day,
every
> > time from a different IP.
> > Is there a Virus going outta there that I'm not aware of?
> > How can I block these requests for this file on the root directory?
> >
> > Help would be very appreciated,
> >
> > Jeff Cohen
>
> What is the format of the file? Try
> #file robots.txt
>
> What does the file contains?
> --
> Dharmendra.T
> Linux Enth
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server
Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] robots.txt
Posted by "Dharmendra.T" <dh...@nsecure.net>.
On Thu, 2003-01-02 at 14:07, Jeff Cohen wrote:
> Hi All,
>
> Does anyone of you get the robots.txt file requests from the internet??
> It seems that I'm starting to get it almost every 3-4 hours a day, every
> time from a different IP.
> Is there a Virus going outta there that I'm not aware of?
> How can I block these requests for this file on the root directory?
>
> Help would be very appreciated,
>
> Jeff Cohen
What is the format of the file? Try
#file robots.txt
What does the file contains?
--
Dharmendra.T
Linux Enth
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org