You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Markus Jelsma <ma...@openindex.io> on 2011/11/24 17:58:25 UTC
Incorrect identifying agent
Hi devs,
I stumbled upon the following user agent string:
TestNutch/Nutch-1.2 (testing Nutch; http://nutch.apache.org;
info@nutch.apache.org)
It says it's from Apache but it seems is not, since ASF does not operate a
Nutch spider last time i checked. Should we allow this?
Cheers
--
Markus Jelsma - CTO - Openindex
Re: Incorrect identifying agent
Posted by Markus Jelsma <ma...@openindex.io>.
On Thursday 24 November 2011 18:05:30 Mattmann, Chris A (388J) wrote:
> Hey Markus,
>
> On Nov 24, 2011, at 8:58 AM, Markus Jelsma wrote:
> > Hi devs,
> >
> > I stumbled upon the following user agent string:
> >
> > TestNutch/Nutch-1.2 (testing Nutch; http://nutch.apache.org;
> > info@nutch.apache.org)
> >
> > It says it's from Apache but it seems is not, since ASF does not operate
> > a Nutch spider last time i checked.
>
> It doesn't, yet. But, as soon as I figure out how to get Nutch running
> for this stinkin' site I'm trying to crawl I am going to head over to
> infra@ and propose to set up a Nutch crawler. I'd also like to solidify
> some of the REST control URL stuff that Andrzej started to work on and get
> that going a little bit more, but I think it would be very useful to
> crawl, e.g., all the ASF sites.
We do that too. Take care, it's a _lot_ of data and quickly changing. :D
>
> > Should we allow this?
>
> Do you mean should we block the use of Apache by some trickery
> in code?
>
> If that's the suggestion, I am not sure I'd be in favor of it. I'd rather
> make it easy to find out where the spider came from and help
> in the identification of the agent, before restricting what user's can
> put in that area.
Yes. Hardcode prevent use of Apache in one of the strings. It's easy to take
out but i guess a lot of users never actually compile the thing. Well, i think
restriction is indeed not very proper within ASF.
Thanks
>
> Cheers,
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--
Markus Jelsma - CTO - Openindex
Re: Incorrect identifying agent
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Markus,
On Nov 24, 2011, at 8:58 AM, Markus Jelsma wrote:
> Hi devs,
>
> I stumbled upon the following user agent string:
>
> TestNutch/Nutch-1.2 (testing Nutch; http://nutch.apache.org;
> info@nutch.apache.org)
>
> It says it's from Apache but it seems is not, since ASF does not operate a
> Nutch spider last time i checked.
It doesn't, yet. But, as soon as I figure out how to get Nutch running
for this stinkin' site I'm trying to crawl I am going to head over to infra@
and propose to set up a Nutch crawler. I'd also like to solidify some of the
REST control URL stuff that Andrzej started to work on and get that going
a little bit more, but I think it would be very useful to crawl, e.g., all the
ASF sites.
> Should we allow this?
Do you mean should we block the use of Apache by some trickery
in code?
If that's the suggestion, I am not sure I'd be in favor of it. I'd rather
make it easy to find out where the spider came from and help
in the identification of the agent, before restricting what user's can
put in that area.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++