You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Graziano Aliberti <gr...@eng.it> on 2010/03/11 15:54:33 UTC
Proxy Authentication
Hi everyone,
I'm trying to use nutch ver. 1.0 on a system under squid proxy control.
When I try to fetch my website list, into the log file I see that the
authentication was failed...
I've configured my nutch-site.xml file with all that properties needed
for proxy auth, but my error is "httpclient.HttpMethodDirector - No
credentials available for BASIC 'Squid proxy-caching web
server'@proxy.my.host:my.port"
The error in fetch phase is "http 407".
Can someone help me?
Best regards,
--
-----------
Graziano Aliberti
Engineering Ingegneria Informatica S.p.A
Via S. Martino della Battaglia, 56 - 00185 ROMA
*Tel.:* 06.49.201.387
*E-Mail:* graziano.aliberti@eng.it
RE: invertlinks: Input path does not exist
Posted by Ar...@csiro.au.
Hi Patricio,
It seems to be quite a lot, but whether it is enough depends on your data size.
Regards,
Arkadi
> -----Original Message-----
> From: Patricio Galeas [mailto:pgaleas@yahoo.de]
> Sent: Sunday, March 21, 2010 1:40 AM
> To: nutch-user@lucene.apache.org
> Subject: AW: invertlinks: Input path does not exist
>
> Hello Arkadi,
>
> I ran the crawl setting hadoop.tmp.dir to a partition with 1,5 TB.
> Do you think that it is enough space for a web-crawl?
>
> Thanks
> Pato
>
>
>
> ----- Ursprüngliche Mail ----
> Von: "Arkadi.Kosmynin@csiro.au" <Ar...@csiro.au>
> An: nutch-user@lucene.apache.org
> Gesendet: Freitag, den 19. März 2010, 3:56:23 Uhr
> Betreff: RE: invertlinks: Input path does not exist
>
> I had similar problems caused by lack of space in temp directory. To
> solve, edited hadoop-site.xml and set hadoop.tmp.dir to a directory with
> plenty of space.
>
> > -----Original Message-----
> > From: kevin chen [mailto:kevinchen@bdsing.com]
> > Sent: Friday, March 19, 2010 1:42 PM
> > To: nutch-user@lucene.apache.org
> > Subject: Re: invertlinks: Input path does not exist
> >
> > Sounds like the last segment is corrupted.
> > Did you try to remove the last segment?
> >
> > On Wed, 2010-03-17 at 16:10 +0000, Patricio Galeas wrote:
> > > Hello all,
> > >
> > > I crawling the web using the
> > > LanguageIdentifier plugin, but I get an error by running nutch
> > > invertlinks.
> > > The error occurs always by processing
> > > the last segment (20100317010313-81).
> > >
> > > The problem is the same described in
> > > http://www.mail-archive.com/nutch-user@lucene.apache.org/msg14776.html
> > > With both syntax variants of
> > > invertlinks I get the same error:
> > > a) nutch invertlinks crawl/linkdb -dir
> > > crawl/segments
> > > b) nutch invertlinks crawl/linkdb
> > > crawl/segments/*
> > >
> > > I applied
> > > https://issues.apache.org/jira/browse/NUTCH-356 to avoid some java
> > > heap problems by using the Language Identifier, but I got the same
> > error. ;-(
> > >
> > > I set the NUTCH_HEAPSIZE with 6000
> > > (physical memory) and I merged the segments using slice=50000
> > >
> > > Any idea where to look for ?
> > >
> > > Thanks
> > > Pato
> > >
> > > --------------------hadoop.log----------------------------------
> > > ..
> > > 2010-03-17 02:33:25,107 INFO crawl.LinkDb - LinkDb: adding segment:
> > file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-47
> > > 2010-03-17 02:33:25,107 INFO crawl.LinkDb - LinkDb: adding segment:
> > file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-68
> > > 2010-03-17 02:33:25,108 INFO crawl.LinkDb - LinkDb: adding segment:
> > file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-56
> > > 2010-03-17 02:33:25,108 INFO crawl.LinkDb - LinkDb: adding segment:
> > file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-12
> > > 2010-03-17 02:33:25,108 INFO crawl.LinkDb - LinkDb: adding segment:
> > file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-26
> > > 2010-03-17 02:33:25,109 INFO crawl.LinkDb - LinkDb: adding segment:
> > file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-73
> > > 2010-03-17 02:33:25,109 INFO crawl.LinkDb - LinkDb: adding segment:
> > file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-59
> > > 2010-03-17 02:33:25,109 INFO crawl.LinkDb - LinkDb: adding segment:
> > file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-30
> > > 2010-03-17 02:33:25,110 INFO crawl.LinkDb - LinkDb: adding segment:
> > file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-2
> > > 2010-03-17 02:33:25,110 INFO crawl.LinkDb - LinkDb: adding segment:
> > file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-34
> > > 2010-03-17 02:33:25,111 INFO crawl.LinkDb - LinkDb: adding segment:
> > file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-52
> > > 2010-03-17 02:33:25,111 INFO crawl.LinkDb - LinkDb: adding segment:
> > file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-29
> > > 2010-03-17 02:33:25,111 INFO crawl.LinkDb - LinkDb: adding segment:
> > file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-24
> > > 2010-03-17 02:33:25,610 FATAL crawl.LinkDb - LinkDb:
> > org.apache.hadoop.mapred.InvalidInputException:
> > > Input path does not exist: file:/mnt/nutch-
> > 1.0/crawl_al/segments/20100317010313-81/parse_data
> > > at
> >
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:1
> > 79)
> > > at
> >
> org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileIn
> > putFormat.java:39)
> > > at
> >
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:19
> > 0)
> > > at
> > org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:797)
> > > at
> > org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1142)
> > > at org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:170)
> > > at org.apache.nutch.crawl.LinkDb.run(LinkDb.java:285)
> > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > > at org.apache.nutch.crawl.LinkDb.main(LinkDb.java:248)
> > >
> > > __________________________________________________
> > > Do You Yahoo!?
> > > Sie sind Spam leid? Yahoo! Mail verfügt über einen herausragenden
> Schutz
> > gegen Massenmails.
> > > http://mail.yahoo.com
>
> __________________________________________________
> Do You Yahoo!?
> Sie sind Spam leid? Yahoo! Mail verfügt über einen herausragenden Schutz
> gegen Massenmails.
> http://mail.yahoo.com
AW: invertlinks: Input path does not exist
Posted by Patricio Galeas <pg...@yahoo.de>.
Hello Arkadi,
I ran the crawl setting hadoop.tmp.dir to a partition with 1,5 TB.
Do you think that it is enough space for a web-crawl?
Thanks
Pato
----- Ursprüngliche Mail ----
Von: "Arkadi.Kosmynin@csiro.au" <Ar...@csiro.au>
An: nutch-user@lucene.apache.org
Gesendet: Freitag, den 19. März 2010, 3:56:23 Uhr
Betreff: RE: invertlinks: Input path does not exist
I had similar problems caused by lack of space in temp directory. To solve, edited hadoop-site.xml and set hadoop.tmp.dir to a directory with plenty of space.
> -----Original Message-----
> From: kevin chen [mailto:kevinchen@bdsing.com]
> Sent: Friday, March 19, 2010 1:42 PM
> To: nutch-user@lucene.apache.org
> Subject: Re: invertlinks: Input path does not exist
>
> Sounds like the last segment is corrupted.
> Did you try to remove the last segment?
>
> On Wed, 2010-03-17 at 16:10 +0000, Patricio Galeas wrote:
> > Hello all,
> >
> > I crawling the web using the
> > LanguageIdentifier plugin, but I get an error by running nutch
> > invertlinks.
> > The error occurs always by processing
> > the last segment (20100317010313-81).
> >
> > The problem is the same described in
> > http://www.mail-archive.com/nutch-user@lucene.apache.org/msg14776.html
> > With both syntax variants of
> > invertlinks I get the same error:
> > a) nutch invertlinks crawl/linkdb -dir
> > crawl/segments
> > b) nutch invertlinks crawl/linkdb
> > crawl/segments/*
> >
> > I applied
> > https://issues.apache.org/jira/browse/NUTCH-356 to avoid some java
> > heap problems by using the Language Identifier, but I got the same
> error. ;-(
> >
> > I set the NUTCH_HEAPSIZE with 6000
> > (physical memory) and I merged the segments using slice=50000
> >
> > Any idea where to look for ?
> >
> > Thanks
> > Pato
> >
> > --------------------hadoop.log----------------------------------
> > ..
> > 2010-03-17 02:33:25,107 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-47
> > 2010-03-17 02:33:25,107 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-68
> > 2010-03-17 02:33:25,108 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-56
> > 2010-03-17 02:33:25,108 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-12
> > 2010-03-17 02:33:25,108 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-26
> > 2010-03-17 02:33:25,109 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-73
> > 2010-03-17 02:33:25,109 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-59
> > 2010-03-17 02:33:25,109 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-30
> > 2010-03-17 02:33:25,110 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-2
> > 2010-03-17 02:33:25,110 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-34
> > 2010-03-17 02:33:25,111 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-52
> > 2010-03-17 02:33:25,111 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-29
> > 2010-03-17 02:33:25,111 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-24
> > 2010-03-17 02:33:25,610 FATAL crawl.LinkDb - LinkDb:
> org.apache.hadoop.mapred.InvalidInputException:
> > Input path does not exist: file:/mnt/nutch-
> 1.0/crawl_al/segments/20100317010313-81/parse_data
> > at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:1
> 79)
> > at
> org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileIn
> putFormat.java:39)
> > at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:19
> 0)
> > at
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:797)
> > at
> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1142)
> > at org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:170)
> > at org.apache.nutch.crawl.LinkDb.run(LinkDb.java:285)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > at org.apache.nutch.crawl.LinkDb.main(LinkDb.java:248)
> >
> > __________________________________________________
> > Do You Yahoo!?
> > Sie sind Spam leid? Yahoo! Mail verfügt über einen herausragenden Schutz
> gegen Massenmails.
> > http://mail.yahoo.com
__________________________________________________
Do You Yahoo!?
Sie sind Spam leid? Yahoo! Mail verfügt über einen herausragenden Schutz gegen Massenmails.
http://mail.yahoo.com
RE: invertlinks: Input path does not exist
Posted by Ar...@csiro.au.
I had similar problems caused by lack of space in temp directory. To solve, edited hadoop-site.xml and set hadoop.tmp.dir to a directory with plenty of space.
> -----Original Message-----
> From: kevin chen [mailto:kevinchen@bdsing.com]
> Sent: Friday, March 19, 2010 1:42 PM
> To: nutch-user@lucene.apache.org
> Subject: Re: invertlinks: Input path does not exist
>
> Sounds like the last segment is corrupted.
> Did you try to remove the last segment?
>
> On Wed, 2010-03-17 at 16:10 +0000, Patricio Galeas wrote:
> > Hello all,
> >
> > I crawling the web using the
> > LanguageIdentifier plugin, but I get an error by running nutch
> > invertlinks.
> > The error occurs always by processing
> > the last segment (20100317010313-81).
> >
> > The problem is the same described in
> > http://www.mail-archive.com/nutch-user@lucene.apache.org/msg14776.html
> > With both syntax variants of
> > invertlinks I get the same error:
> > a) nutch invertlinks crawl/linkdb -dir
> > crawl/segments
> > b) nutch invertlinks crawl/linkdb
> > crawl/segments/*
> >
> > I applied
> > https://issues.apache.org/jira/browse/NUTCH-356 to avoid some java
> > heap problems by using the Language Identifier, but I got the same
> error. ;-(
> >
> > I set the NUTCH_HEAPSIZE with 6000
> > (physical memory) and I merged the segments using slice=50000
> >
> > Any idea where to look for ?
> >
> > Thanks
> > Pato
> >
> > --------------------hadoop.log----------------------------------
> > ..
> > 2010-03-17 02:33:25,107 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-47
> > 2010-03-17 02:33:25,107 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-68
> > 2010-03-17 02:33:25,108 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-56
> > 2010-03-17 02:33:25,108 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-12
> > 2010-03-17 02:33:25,108 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-26
> > 2010-03-17 02:33:25,109 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-73
> > 2010-03-17 02:33:25,109 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-59
> > 2010-03-17 02:33:25,109 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-30
> > 2010-03-17 02:33:25,110 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-2
> > 2010-03-17 02:33:25,110 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-34
> > 2010-03-17 02:33:25,111 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-52
> > 2010-03-17 02:33:25,111 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-29
> > 2010-03-17 02:33:25,111 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-24
> > 2010-03-17 02:33:25,610 FATAL crawl.LinkDb - LinkDb:
> org.apache.hadoop.mapred.InvalidInputException:
> > Input path does not exist: file:/mnt/nutch-
> 1.0/crawl_al/segments/20100317010313-81/parse_data
> > at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:1
> 79)
> > at
> org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileIn
> putFormat.java:39)
> > at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:19
> 0)
> > at
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:797)
> > at
> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1142)
> > at org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:170)
> > at org.apache.nutch.crawl.LinkDb.run(LinkDb.java:285)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > at org.apache.nutch.crawl.LinkDb.main(LinkDb.java:248)
> >
> > __________________________________________________
> > Do You Yahoo!?
> > Sie sind Spam leid? Yahoo! Mail verfügt über einen herausragenden Schutz
> gegen Massenmails.
> > http://mail.yahoo.com
AW: invertlinks: Input path does not exist
Posted by Patricio Galeas <pg...@yahoo.de>.
Hi,
I'm ran the crawl with the script from http://wiki.apache.org/nutch/Crawl
This time, I defined a break (exit) before the invertlinks is executed and ran the rest nutch commands manually *without* errors.
I suppose that something is going wrong by moving the MERGEDsegments to segments:
mv $MVARGS $crawldir/MERGEDsegments $crawldir/segments
I'm running the crawl in a VirtualMachine (Debian) and execute the crawl-script using "nohup":
nohup runbot.sh > foo.out 2> foo.err < /dev/null &
Is it possible that invertlinks is executed before the "mv" command is finished?
or
are there some problems to run the runbot.sh using "nohup"?
Thanks
Pato
----- Ursprüngliche Mail ----
Von: kevin chen <ke...@bdsing.com>
An: nutch-user@lucene.apache.org
Gesendet: Freitag, den 19. März 2010, 3:41:40 Uhr
Betreff: Re: invertlinks: Input path does not exist
Sounds like the last segment is corrupted.
Did you try to remove the last segment?
On Wed, 2010-03-17 at 16:10 +0000, Patricio Galeas wrote:
> Hello all,
>
> I crawling the web using the
> LanguageIdentifier plugin, but I get an error by running nutch
> invertlinks.
> The error occurs always by processing
> the last segment (20100317010313-81).
>
> The problem is the same described in
> http://www.mail-archive.com/nutch-user@lucene.apache.org/msg14776.html
> With both syntax variants of
> invertlinks I get the same error:
> a) nutch invertlinks crawl/linkdb -dir
> crawl/segments
> b) nutch invertlinks crawl/linkdb
> crawl/segments/*
>
> I applied
> https://issues.apache.org/jira/browse/NUTCH-356 to avoid some java
> heap problems by using the Language Identifier, but I got the same error. ;-(
>
> I set the NUTCH_HEAPSIZE with 6000
> (physical memory) and I merged the segments using slice=50000
>
> Any idea where to look for ?
>
> Thanks
> Pato
>
> --------------------hadoop.log----------------------------------
> ..
> 2010-03-17 02:33:25,107 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-47
> 2010-03-17 02:33:25,107 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-68
> 2010-03-17 02:33:25,108 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-56
> 2010-03-17 02:33:25,108 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-12
> 2010-03-17 02:33:25,108 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-26
> 2010-03-17 02:33:25,109 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-73
> 2010-03-17 02:33:25,109 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-59
> 2010-03-17 02:33:25,109 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-30
> 2010-03-17 02:33:25,110 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-2
> 2010-03-17 02:33:25,110 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-34
> 2010-03-17 02:33:25,111 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-52
> 2010-03-17 02:33:25,111 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-29
> 2010-03-17 02:33:25,111 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-24
> 2010-03-17 02:33:25,610 FATAL crawl.LinkDb - LinkDb: org.apache.hadoop.mapred.InvalidInputException:
> Input path does not exist: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-81/parse_data
> at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:179)
> at org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:39)
> at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:190)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:797)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1142)
> at org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:170)
> at org.apache.nutch.crawl.LinkDb.run(LinkDb.java:285)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.nutch.crawl.LinkDb.main(LinkDb.java:248)
>
> __________________________________________________
> Do You Yahoo!?
> Sie sind Spam leid? Yahoo! Mail verfügt über einen herausragenden Schutz gegen Massenmails.
> http://mail.yahoo.com
__________________________________________________
Do You Yahoo!?
Sie sind Spam leid? Yahoo! Mail verfügt über einen herausragenden Schutz gegen Massenmails.
http://mail.yahoo.com
Re: invertlinks: Input path does not exist
Posted by kevin chen <ke...@bdsing.com>.
Sounds like the last segment is corrupted.
Did you try to remove the last segment?
On Wed, 2010-03-17 at 16:10 +0000, Patricio Galeas wrote:
> Hello all,
>
> I crawling the web using the
> LanguageIdentifier plugin, but I get an error by running nutch
> invertlinks.
> The error occurs always by processing
> the last segment (20100317010313-81).
>
> The problem is the same described in
> http://www.mail-archive.com/nutch-user@lucene.apache.org/msg14776.html
> With both syntax variants of
> invertlinks I get the same error:
> a) nutch invertlinks crawl/linkdb -dir
> crawl/segments
> b) nutch invertlinks crawl/linkdb
> crawl/segments/*
>
> I applied
> https://issues.apache.org/jira/browse/NUTCH-356 to avoid some java
> heap problems by using the Language Identifier, but I got the same error. ;-(
>
> I set the NUTCH_HEAPSIZE with 6000
> (physical memory) and I merged the segments using slice=50000
>
> Any idea where to look for ?
>
> Thanks
> Pato
>
> --------------------hadoop.log----------------------------------
> ..
> 2010-03-17 02:33:25,107 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-47
> 2010-03-17 02:33:25,107 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-68
> 2010-03-17 02:33:25,108 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-56
> 2010-03-17 02:33:25,108 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-12
> 2010-03-17 02:33:25,108 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-26
> 2010-03-17 02:33:25,109 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-73
> 2010-03-17 02:33:25,109 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-59
> 2010-03-17 02:33:25,109 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-30
> 2010-03-17 02:33:25,110 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-2
> 2010-03-17 02:33:25,110 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-34
> 2010-03-17 02:33:25,111 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-52
> 2010-03-17 02:33:25,111 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-29
> 2010-03-17 02:33:25,111 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-24
> 2010-03-17 02:33:25,610 FATAL crawl.LinkDb - LinkDb: org.apache.hadoop.mapred.InvalidInputException:
> Input path does not exist: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-81/parse_data
> at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:179)
> at org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:39)
> at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:190)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:797)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1142)
> at org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:170)
> at org.apache.nutch.crawl.LinkDb.run(LinkDb.java:285)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.nutch.crawl.LinkDb.main(LinkDb.java:248)
>
> __________________________________________________
> Do You Yahoo!?
> Sie sind Spam leid? Yahoo! Mail verfügt über einen herausragenden Schutz gegen Massenmails.
> http://mail.yahoo.com
invertlinks: Input path does not exist
Posted by Patricio Galeas <pg...@yahoo.de>.
Hello all,
I crawling the web using the
LanguageIdentifier plugin, but I get an error by running nutch
invertlinks.
The error occurs always by processing
the last segment (20100317010313-81).
The problem is the same described in
http://www.mail-archive.com/nutch-user@lucene.apache.org/msg14776.html
With both syntax variants of
invertlinks I get the same error:
a) nutch invertlinks crawl/linkdb -dir
crawl/segments
b) nutch invertlinks crawl/linkdb
crawl/segments/*
I applied
https://issues.apache.org/jira/browse/NUTCH-356 to avoid some java
heap problems by using the Language Identifier, but I got the same error. ;-(
I set the NUTCH_HEAPSIZE with 6000
(physical memory) and I merged the segments using slice=50000
Any idea where to look for ?
Thanks
Pato
--------------------hadoop.log----------------------------------
..
2010-03-17 02:33:25,107 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-47
2010-03-17 02:33:25,107 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-68
2010-03-17 02:33:25,108 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-56
2010-03-17 02:33:25,108 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-12
2010-03-17 02:33:25,108 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-26
2010-03-17 02:33:25,109 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-73
2010-03-17 02:33:25,109 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-59
2010-03-17 02:33:25,109 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-30
2010-03-17 02:33:25,110 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-2
2010-03-17 02:33:25,110 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-34
2010-03-17 02:33:25,111 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-52
2010-03-17 02:33:25,111 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-29
2010-03-17 02:33:25,111 INFO crawl.LinkDb - LinkDb: adding segment: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-24
2010-03-17 02:33:25,610 FATAL crawl.LinkDb - LinkDb: org.apache.hadoop.mapred.InvalidInputException:
Input path does not exist: file:/mnt/nutch-1.0/crawl_al/segments/20100317010313-81/parse_data
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:179)
at org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:39)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:190)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:797)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1142)
at org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:170)
at org.apache.nutch.crawl.LinkDb.run(LinkDb.java:285)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.LinkDb.main(LinkDb.java:248)
__________________________________________________
Do You Yahoo!?
Sie sind Spam leid? Yahoo! Mail verfügt über einen herausragenden Schutz gegen Massenmails.
http://mail.yahoo.com
Re: Proxy Authentication
Posted by Susam Pal <su...@gmail.com>.
On Tue, Mar 16, 2010 at 12:55 AM, Susam Pal <su...@gmail.com> wrote:
> On Mon, Mar 15, 2010 at 2:32 PM, Graziano Aliberti
> <gr...@eng.it> wrote:
>> Il 13/03/2010 22.55, Susam Pal ha scritto:
>>>
>>> On Fri, Mar 12, 2010 at 3:17 PM, Susam Pal<su...@gmail.com> wrote:
>>>
>>>>
>>>> On Fri, Mar 12, 2010 at 2:09 PM, Graziano Aliberti
>>>> <gr...@eng.it> wrote:
>>>>
>>>>>
>>>>> Il 11/03/2010 16.20, Susam Pal ha scritto:
>>>>>
>>>>>>
>>>>>> On Thu, Mar 11, 2010 at 8:24 PM, Graziano Aliberti
>>>>>> <gr...@eng.it> wrote:
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> I'm trying to use nutch ver. 1.0 on a system under squid proxy
>>>>>>> control.
>>>>>>> When
>>>>>>> I try to fetch my website list, into the log file I see that the
>>>>>>> authentication was failed...
>>>>>>>
>>>>>>> I've configured my nutch-site.xml file with all that properties needed
>>>>>>> for
>>>>>>> proxy auth, but my error is "httpclient.HttpMethodDirector - No
>>>>>>> credentials
>>>>>>> available for BASIC 'Squid proxy-caching web
>>>>>>> server'@proxy.my.host:my.port"
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> Did you replace 'protocol-http' with 'protocol-httpclient' in the
>>>>>> value for 'plugins.include' property in 'conf/nutch-site.xml'?
>>>>>>
>>>>>> Regards,
>>>>>> Susam Pal
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> Hi Susam,
>>>>>
>>>>> yes of course!! :) Maybe I can post you the configuration file:
>>>>>
>>>>> <?xml version="1.0"?>
>>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>>>>
>>>>> <!-- Put site-specific property overrides in this file. -->
>>>>>
>>>>> <configuration>
>>>>>
>>>>> <property>
>>>>> <name>http.agent.name</name>
>>>>> <value>my.agent.name</value>
>>>>> <description>
>>>>> </description>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>plugin.includes</name>
>>>>>
>>>>> <value>protocol-httpclient|urlfilter-regex|parse-(text|html|js)|index-(basic|anchor)|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
>>>>> <description>
>>>>> </description>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>http.auth.file</name>
>>>>> <value>my_file.xml</value>
>>>>> <description>Authentication configuration file for
>>>>> 'protocol-httpclient' plugin.
>>>>> </description>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>http.proxy.host</name>
>>>>> <value>ip.my.proxy</value>
>>>>> <description>The proxy hostname. If empty, no proxy is
>>>>> used.</description>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>http.proxy.port</name>
>>>>> <value>my.port</value>
>>>>> <description>The proxy port.</description>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>http.proxy.username</name>
>>>>> <value>my.user</value>
>>>>> <description>
>>>>> </description>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>http.proxy.password</name>
>>>>> <value>my.pwd</value>
>>>>> <description>
>>>>> </description>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>http.proxy.realm</name>
>>>>> <value>my_realm</value>
>>>>> <description>
>>>>> </description>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>http.agent.host</name>
>>>>> <value>my.local.pc</value>
>>>>> <description>The agent host.</description>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>http.useHttp11</name>
>>>>> <value>true</value>
>>>>> <description>
>>>>> </description>
>>>>> </property>
>>>>>
>>>>> </configuration>
>>>>>
>>>>> Only another question: where i must put the user authentication
>>>>> parameters
>>>>> (user,pwd)? In nutch-site.xml file or in my_file.xml that I use for
>>>>> authentication?
>>>>>
>>>>> Thank you for your attention,
>>>>>
>>>>>
>>>>> --
>>>>> -----------
>>>>>
>>>>> Graziano Aliberti
>>>>>
>>>>> Engineering Ingegneria Informatica S.p.A
>>>>>
>>>>> Via S. Martino della Battaglia, 56 - 00185 ROMA
>>>>>
>>>>> *Tel.:* 06.49.201.387
>>>>>
>>>>> *E-Mail:* graziano.aliberti@eng.it
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> The configuration looks okay to me. Yes, the proxy authentication
>>>> details are set in 'conf/nutch-site.xml'. The file mentioned in
>>>> 'http.auth.file' property is used for configuring authentication
>>>> details for authenticating to a web server.
>>>>
>>>> Unfortunately, there aren't any log statements in the part of the code
>>>> that reads the proxy authentication details. So, I can't suggest you
>>>> to turn on debug logs to get some clues about the issue. However, in
>>>> case you want to troubleshoot it yourself by building Nutch from
>>>> source, I can tell you the code that deals with this.
>>>>
>>>> The file is: src/java/org/apache/nutch/protocol/httpclient/Http.java :
>>>>
>>>> http://svn.apache.org/viewvc/lucene/nutch/trunk/src/plugin/protocol-httpclient/src/java/org/apache/nutch/protocol/httpclient/Http.java?view=markup
>>>>
>>>> The line number is: 200.
>>>>
>>>> If I get time this weekend, I will try to insert some log statements
>>>> into this code and send a modified JAR file to you which might help
>>>> you to troubleshoot what is going on. But I can't promise this since
>>>> it depends on my weekend plans.
>>>>
>>>> Two questions before I end this mail. Did you set the value of
>>>> 'http.proxy.realm' property as: Squid proxy-caching web server ?
>>>>
>>>> Also, do you see any 'auth.AuthChallengeProcessor' lines in the log
>>>> file? I'm not sure whether this line should appear for proxy
>>>> authentication but it does appear for web server authentication.
>>>>
>>>> Regards,
>>>> Susam Pal
>>>>
>>>>
>>>
>>> I managed to find some time to insert more logs into
>>> protocol-httpclient and create a JAR. I have attached it with this
>>> email.
>>>
>>> Please replace your
>>> 'plugins/protocol-httpclient/protocol-httpclient.jar' file with the
>>> one that I have attached. Also, edit your 'conf/log4j.properties' file
>>> to add these two lines:
>>>
>>> log4j.logger.org.apache.nutch.protocol.httpclient=DEBUG,cmdstdout
>>> log4j.logger.org.apache.commons.httpclient.auth=DEBUG,cmdstdout
>>>
>>> When you run a crawl now, you should see more logs in
>>> 'logs/hadoop.log' than before. I hope it helps you in providing some
>>> clues. In case you want to compare the logs with how the control flows
>>> from the source code, I have attached the JAVA file as well.
>>>
>>> Regards,
>>> Susam Pal
>>>
>>
>> Hi Susam,
>>
>> first of all I want to thank you for your support :). I've tried your
>> solution and I've seen in the log file the that the authentication
>> parameters was correctly read by the application.
>>
>> In the log file I've finded these lines about auth.AuthChallengeProcessor:
>>
>>
>> 2010-03-15 09:52:33,140 DEBUG auth.AuthChallengeProcessor - Challenge for
>> ntlm authentication scheme not available
>> 2010-03-15 09:52:33,140 DEBUG auth.AuthChallengeProcessor - Challenge for
>> digest authentication scheme not available
>> 2010-03-15 09:52:33,140 INFO auth.AuthChallengeProcessor - basic
>> authentication scheme selected
>> 2010-03-15 09:52:33,140 DEBUG auth.AuthChallengeProcessor - Using
>> authentication scheme: basic
>> 2010-03-15 09:52:33,140 DEBUG auth.AuthChallengeProcessor - Authorization
>> challenge processed
>> 2010-03-15 09:52:33,140 INFO httpclient.HttpMethodDirector - No credentials
>> available for BASIC 'Squid proxy-caching web server'@my.proxy:my.port
>>
>
> 'Squid proxy-caching web server'@my.proxy:my.port - should be the
> authentication details mentioned in the proxy configuration.
>
> It means that the 'http.proxy.realm' should be specified as: Squid
> proxy-caching web server
>
> You can also try omitting the value for 'http.proxy.realm' property.
>
> I was also wanted to confirm whether you got the following line in
> 'logs/hadoop.log':
>
> Custom logs for troubleshooting authentication (set 4)
>
> If you have got this line and your configuration is correct, I don't
> see a reason why AuthChallengeProcessor should complain about missing
> credentials. It could be a bug either in Nutch or in the Jakarta
> Commons HttpClient library which is being used in Nutch to do the
> authentication. It could also be a mistake in the configuration.
>
> In case, you find a way to resolve it, please let us know what the
> problem was and how you resolved it.
>
> Regards,
> Susam Pal
>
Here is an update. It is most likely a configuration problem. I just
tested the proxy authentication feature with a Squid proxy server with
the same realm as yours. It works well.
Also, the issue you are facing seems to be due to an incorrect realm
specified. I would suggest that you omit the realm and see if it works
fine. When you omit the realm, the corresponding XML code for the
configuration should look like this:
<property>
<name>http.proxy.realm</name>
<value></value>
<description></description>
</property>
In case you do want to specify the realm, your XML code should look like this:
<property>
<name>http.proxy.realm</name>
<value>Squid proxy-caching web server</value>
<description></description>
</property>
Note that this is the exact string appearing in the log message:
INFO httpclient.HttpMethodDirector - No credentials available for
BASIC 'Squid proxy-caching web server'@my.proxy:my.port
There should be no quotes around the string.
If everything goes fine, the logs should appear like the following.
These logs are from my system.
2010-03-16 02:45:28,280 DEBUG auth.AuthChallengeProcessor - Supported
authentication schemes in the order of preference: [ntlm, digest,
basic]
2010-03-16 02:45:28,280 DEBUG auth.AuthChallengeProcessor - Challenge
for ntlm authentication scheme not available
2010-03-16 02:45:28,280 DEBUG auth.AuthChallengeProcessor - Challenge
for digest authentication scheme not available
2010-03-16 02:45:28,280 INFO auth.AuthChallengeProcessor - basic
authentication scheme selected
2010-03-16 02:45:28,280 DEBUG auth.AuthChallengeProcessor - Using
authentication scheme: basic
2010-03-16 02:45:28,281 DEBUG auth.AuthChallengeProcessor -
Authorization challenge processed
2010-03-16 02:45:28,282 DEBUG auth.AuthChallengeProcessor - Supported
authentication schemes in the order of preference: [ntlm, digest,
basic]
2010-03-16 02:45:28,283 DEBUG auth.AuthChallengeProcessor - Challenge
for ntlm authentication scheme not available
2010-03-16 02:45:28,283 DEBUG auth.AuthChallengeProcessor - Challenge
for digest authentication scheme not available
2010-03-16 02:45:28,283 INFO auth.AuthChallengeProcessor - basic
authentication scheme selected
2010-03-16 02:45:28,283 DEBUG auth.AuthChallengeProcessor - Using
authentication scheme: basic
2010-03-16 02:45:28,283 DEBUG auth.AuthChallengeProcessor -
Authorization challenge processed
2010-03-16 02:45:28,284 DEBUG auth.BasicScheme - enter
BasicScheme.authenticate(Credentials, HttpMethod)
2010-03-16 02:45:28,286 DEBUG auth.BasicScheme - enter
BasicScheme.authenticate(UsernamePasswordCredentials, String)
2010-03-16 02:45:28,286 DEBUG auth.AuthChallengeProcessor - Supported
authentication schemes in the order of preference: [ntlm, digest,
basic]
2010-03-16 02:45:28,287 DEBUG auth.AuthChallengeProcessor - Challenge
for ntlm authentication scheme not available
2010-03-16 02:45:28,287 DEBUG auth.AuthChallengeProcessor - Challenge
for digest authentication scheme not available
2010-03-16 02:45:28,287 INFO auth.AuthChallengeProcessor - basic
authentication scheme selected
2010-03-16 02:45:28,287 DEBUG auth.AuthChallengeProcessor - Using
authentication scheme: basic
2010-03-16 02:45:28,288 DEBUG auth.AuthChallengeProcessor -
Authorization challenge processed
2010-03-16 02:45:28,288 DEBUG auth.BasicScheme - enter
BasicScheme.authenticate(Credentials, HttpMethod)
2010-03-16 02:45:28,288 DEBUG auth.BasicScheme - enter
BasicScheme.authenticate(UsernamePasswordCredentials, String)
2010-03-16 02:45:28,284 DEBUG auth.BasicScheme - enter
BasicScheme.authenticate(Credentials, HttpMethod)
2010-03-16 02:45:28,289 DEBUG auth.BasicScheme - enter
BasicScheme.authenticate(UsernamePasswordCredentials, String)
2010-03-16 02:45:28,330 DEBUG httpclient.Http - url:
http://en.wikipedia.org/robots.txt; status code: 200; bytes received:
4853; Content-Length: 4853; Content-Encoding: gzip; extracted to 26147
bytes
I hope this helps.
If you still face issues, please send me the complete log file
(logs/hadoop.log) and the complete configuration file
(conf/nutch-site.xml). It is easier to spot configuration mistakes if
you send the complete files. Please do remove the existing hadoop.log
file before starting a new crawl so that that the log file you send us
isn't too large.
Regards,
Susam Pal
Re: Proxy Authentication
Posted by Susam Pal <su...@gmail.com>.
On Mon, Mar 15, 2010 at 2:32 PM, Graziano Aliberti
<gr...@eng.it> wrote:
> Il 13/03/2010 22.55, Susam Pal ha scritto:
>>
>> On Fri, Mar 12, 2010 at 3:17 PM, Susam Pal<su...@gmail.com> wrote:
>>
>>>
>>> On Fri, Mar 12, 2010 at 2:09 PM, Graziano Aliberti
>>> <gr...@eng.it> wrote:
>>>
>>>>
>>>> Il 11/03/2010 16.20, Susam Pal ha scritto:
>>>>
>>>>>
>>>>> On Thu, Mar 11, 2010 at 8:24 PM, Graziano Aliberti
>>>>> <gr...@eng.it> wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> I'm trying to use nutch ver. 1.0 on a system under squid proxy
>>>>>> control.
>>>>>> When
>>>>>> I try to fetch my website list, into the log file I see that the
>>>>>> authentication was failed...
>>>>>>
>>>>>> I've configured my nutch-site.xml file with all that properties needed
>>>>>> for
>>>>>> proxy auth, but my error is "httpclient.HttpMethodDirector - No
>>>>>> credentials
>>>>>> available for BASIC 'Squid proxy-caching web
>>>>>> server'@proxy.my.host:my.port"
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> Did you replace 'protocol-http' with 'protocol-httpclient' in the
>>>>> value for 'plugins.include' property in 'conf/nutch-site.xml'?
>>>>>
>>>>> Regards,
>>>>> Susam Pal
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> Hi Susam,
>>>>
>>>> yes of course!! :) Maybe I can post you the configuration file:
>>>>
>>>> <?xml version="1.0"?>
>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>>>
>>>> <!-- Put site-specific property overrides in this file. -->
>>>>
>>>> <configuration>
>>>>
>>>> <property>
>>>> <name>http.agent.name</name>
>>>> <value>my.agent.name</value>
>>>> <description>
>>>> </description>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>plugin.includes</name>
>>>>
>>>> <value>protocol-httpclient|urlfilter-regex|parse-(text|html|js)|index-(basic|anchor)|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
>>>> <description>
>>>> </description>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>http.auth.file</name>
>>>> <value>my_file.xml</value>
>>>> <description>Authentication configuration file for
>>>> 'protocol-httpclient' plugin.
>>>> </description>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>http.proxy.host</name>
>>>> <value>ip.my.proxy</value>
>>>> <description>The proxy hostname. If empty, no proxy is
>>>> used.</description>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>http.proxy.port</name>
>>>> <value>my.port</value>
>>>> <description>The proxy port.</description>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>http.proxy.username</name>
>>>> <value>my.user</value>
>>>> <description>
>>>> </description>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>http.proxy.password</name>
>>>> <value>my.pwd</value>
>>>> <description>
>>>> </description>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>http.proxy.realm</name>
>>>> <value>my_realm</value>
>>>> <description>
>>>> </description>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>http.agent.host</name>
>>>> <value>my.local.pc</value>
>>>> <description>The agent host.</description>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>http.useHttp11</name>
>>>> <value>true</value>
>>>> <description>
>>>> </description>
>>>> </property>
>>>>
>>>> </configuration>
>>>>
>>>> Only another question: where i must put the user authentication
>>>> parameters
>>>> (user,pwd)? In nutch-site.xml file or in my_file.xml that I use for
>>>> authentication?
>>>>
>>>> Thank you for your attention,
>>>>
>>>>
>>>> --
>>>> -----------
>>>>
>>>> Graziano Aliberti
>>>>
>>>> Engineering Ingegneria Informatica S.p.A
>>>>
>>>> Via S. Martino della Battaglia, 56 - 00185 ROMA
>>>>
>>>> *Tel.:* 06.49.201.387
>>>>
>>>> *E-Mail:* graziano.aliberti@eng.it
>>>>
>>>>
>>>>
>>>>
>>>
>>> The configuration looks okay to me. Yes, the proxy authentication
>>> details are set in 'conf/nutch-site.xml'. The file mentioned in
>>> 'http.auth.file' property is used for configuring authentication
>>> details for authenticating to a web server.
>>>
>>> Unfortunately, there aren't any log statements in the part of the code
>>> that reads the proxy authentication details. So, I can't suggest you
>>> to turn on debug logs to get some clues about the issue. However, in
>>> case you want to troubleshoot it yourself by building Nutch from
>>> source, I can tell you the code that deals with this.
>>>
>>> The file is: src/java/org/apache/nutch/protocol/httpclient/Http.java :
>>>
>>> http://svn.apache.org/viewvc/lucene/nutch/trunk/src/plugin/protocol-httpclient/src/java/org/apache/nutch/protocol/httpclient/Http.java?view=markup
>>>
>>> The line number is: 200.
>>>
>>> If I get time this weekend, I will try to insert some log statements
>>> into this code and send a modified JAR file to you which might help
>>> you to troubleshoot what is going on. But I can't promise this since
>>> it depends on my weekend plans.
>>>
>>> Two questions before I end this mail. Did you set the value of
>>> 'http.proxy.realm' property as: Squid proxy-caching web server ?
>>>
>>> Also, do you see any 'auth.AuthChallengeProcessor' lines in the log
>>> file? I'm not sure whether this line should appear for proxy
>>> authentication but it does appear for web server authentication.
>>>
>>> Regards,
>>> Susam Pal
>>>
>>>
>>
>> I managed to find some time to insert more logs into
>> protocol-httpclient and create a JAR. I have attached it with this
>> email.
>>
>> Please replace your
>> 'plugins/protocol-httpclient/protocol-httpclient.jar' file with the
>> one that I have attached. Also, edit your 'conf/log4j.properties' file
>> to add these two lines:
>>
>> log4j.logger.org.apache.nutch.protocol.httpclient=DEBUG,cmdstdout
>> log4j.logger.org.apache.commons.httpclient.auth=DEBUG,cmdstdout
>>
>> When you run a crawl now, you should see more logs in
>> 'logs/hadoop.log' than before. I hope it helps you in providing some
>> clues. In case you want to compare the logs with how the control flows
>> from the source code, I have attached the JAVA file as well.
>>
>> Regards,
>> Susam Pal
>>
>
> Hi Susam,
>
> first of all I want to thank you for your support :). I've tried your
> solution and I've seen in the log file the that the authentication
> parameters was correctly read by the application.
>
> In the log file I've finded these lines about auth.AuthChallengeProcessor:
>
>
> 2010-03-15 09:52:33,140 DEBUG auth.AuthChallengeProcessor - Challenge for
> ntlm authentication scheme not available
> 2010-03-15 09:52:33,140 DEBUG auth.AuthChallengeProcessor - Challenge for
> digest authentication scheme not available
> 2010-03-15 09:52:33,140 INFO auth.AuthChallengeProcessor - basic
> authentication scheme selected
> 2010-03-15 09:52:33,140 DEBUG auth.AuthChallengeProcessor - Using
> authentication scheme: basic
> 2010-03-15 09:52:33,140 DEBUG auth.AuthChallengeProcessor - Authorization
> challenge processed
> 2010-03-15 09:52:33,140 INFO httpclient.HttpMethodDirector - No credentials
> available for BASIC 'Squid proxy-caching web server'@my.proxy:my.port
>
'Squid proxy-caching web server'@my.proxy:my.port - should be the
authentication details mentioned in the proxy configuration.
It means that the 'http.proxy.realm' should be specified as: Squid
proxy-caching web server
You can also try omitting the value for 'http.proxy.realm' property.
I was also wanted to confirm whether you got the following line in
'logs/hadoop.log':
Custom logs for troubleshooting authentication (set 4)
If you have got this line and your configuration is correct, I don't
see a reason why AuthChallengeProcessor should complain about missing
credentials. It could be a bug either in Nutch or in the Jakarta
Commons HttpClient library which is being used in Nutch to do the
authentication. It could also be a mistake in the configuration.
In case, you find a way to resolve it, please let us know what the
problem was and how you resolved it.
Regards,
Susam Pal
Re: Proxy Authentication
Posted by Graziano Aliberti <gr...@eng.it>.
Il 13/03/2010 22.55, Susam Pal ha scritto:
> On Fri, Mar 12, 2010 at 3:17 PM, Susam Pal<su...@gmail.com> wrote:
>
>> On Fri, Mar 12, 2010 at 2:09 PM, Graziano Aliberti
>> <gr...@eng.it> wrote:
>>
>>> Il 11/03/2010 16.20, Susam Pal ha scritto:
>>>
>>>> On Thu, Mar 11, 2010 at 8:24 PM, Graziano Aliberti
>>>> <gr...@eng.it> wrote:
>>>>
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> I'm trying to use nutch ver. 1.0 on a system under squid proxy control.
>>>>> When
>>>>> I try to fetch my website list, into the log file I see that the
>>>>> authentication was failed...
>>>>>
>>>>> I've configured my nutch-site.xml file with all that properties needed
>>>>> for
>>>>> proxy auth, but my error is "httpclient.HttpMethodDirector - No
>>>>> credentials
>>>>> available for BASIC 'Squid proxy-caching web
>>>>> server'@proxy.my.host:my.port"
>>>>>
>>>>>
>>>>>
>>>> Did you replace 'protocol-http' with 'protocol-httpclient' in the
>>>> value for 'plugins.include' property in 'conf/nutch-site.xml'?
>>>>
>>>> Regards,
>>>> Susam Pal
>>>>
>>>>
>>>>
>>>>
>>> Hi Susam,
>>>
>>> yes of course!! :) Maybe I can post you the configuration file:
>>>
>>> <?xml version="1.0"?>
>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>>
>>> <!-- Put site-specific property overrides in this file. -->
>>>
>>> <configuration>
>>>
>>> <property>
>>> <name>http.agent.name</name>
>>> <value>my.agent.name</value>
>>> <description>
>>> </description>
>>> </property>
>>>
>>> <property>
>>> <name>plugin.includes</name>
>>> <value>protocol-httpclient|urlfilter-regex|parse-(text|html|js)|index-(basic|anchor)|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
>>> <description>
>>> </description>
>>> </property>
>>>
>>> <property>
>>> <name>http.auth.file</name>
>>> <value>my_file.xml</value>
>>> <description>Authentication configuration file for
>>> 'protocol-httpclient' plugin.
>>> </description>
>>> </property>
>>>
>>> <property>
>>> <name>http.proxy.host</name>
>>> <value>ip.my.proxy</value>
>>> <description>The proxy hostname. If empty, no proxy is used.</description>
>>> </property>
>>>
>>> <property>
>>> <name>http.proxy.port</name>
>>> <value>my.port</value>
>>> <description>The proxy port.</description>
>>> </property>
>>>
>>> <property>
>>> <name>http.proxy.username</name>
>>> <value>my.user</value>
>>> <description>
>>> </description>
>>> </property>
>>>
>>> <property>
>>> <name>http.proxy.password</name>
>>> <value>my.pwd</value>
>>> <description>
>>> </description>
>>> </property>
>>>
>>> <property>
>>> <name>http.proxy.realm</name>
>>> <value>my_realm</value>
>>> <description>
>>> </description>
>>> </property>
>>>
>>> <property>
>>> <name>http.agent.host</name>
>>> <value>my.local.pc</value>
>>> <description>The agent host.</description>
>>> </property>
>>>
>>> <property>
>>> <name>http.useHttp11</name>
>>> <value>true</value>
>>> <description>
>>> </description>
>>> </property>
>>>
>>> </configuration>
>>>
>>> Only another question: where i must put the user authentication parameters
>>> (user,pwd)? In nutch-site.xml file or in my_file.xml that I use for
>>> authentication?
>>>
>>> Thank you for your attention,
>>>
>>>
>>> --
>>> -----------
>>>
>>> Graziano Aliberti
>>>
>>> Engineering Ingegneria Informatica S.p.A
>>>
>>> Via S. Martino della Battaglia, 56 - 00185 ROMA
>>>
>>> *Tel.:* 06.49.201.387
>>>
>>> *E-Mail:* graziano.aliberti@eng.it
>>>
>>>
>>>
>>>
>> The configuration looks okay to me. Yes, the proxy authentication
>> details are set in 'conf/nutch-site.xml'. The file mentioned in
>> 'http.auth.file' property is used for configuring authentication
>> details for authenticating to a web server.
>>
>> Unfortunately, there aren't any log statements in the part of the code
>> that reads the proxy authentication details. So, I can't suggest you
>> to turn on debug logs to get some clues about the issue. However, in
>> case you want to troubleshoot it yourself by building Nutch from
>> source, I can tell you the code that deals with this.
>>
>> The file is: src/java/org/apache/nutch/protocol/httpclient/Http.java :
>> http://svn.apache.org/viewvc/lucene/nutch/trunk/src/plugin/protocol-httpclient/src/java/org/apache/nutch/protocol/httpclient/Http.java?view=markup
>>
>> The line number is: 200.
>>
>> If I get time this weekend, I will try to insert some log statements
>> into this code and send a modified JAR file to you which might help
>> you to troubleshoot what is going on. But I can't promise this since
>> it depends on my weekend plans.
>>
>> Two questions before I end this mail. Did you set the value of
>> 'http.proxy.realm' property as: Squid proxy-caching web server ?
>>
>> Also, do you see any 'auth.AuthChallengeProcessor' lines in the log
>> file? I'm not sure whether this line should appear for proxy
>> authentication but it does appear for web server authentication.
>>
>> Regards,
>> Susam Pal
>>
>>
> I managed to find some time to insert more logs into
> protocol-httpclient and create a JAR. I have attached it with this
> email.
>
> Please replace your
> 'plugins/protocol-httpclient/protocol-httpclient.jar' file with the
> one that I have attached. Also, edit your 'conf/log4j.properties' file
> to add these two lines:
>
> log4j.logger.org.apache.nutch.protocol.httpclient=DEBUG,cmdstdout
> log4j.logger.org.apache.commons.httpclient.auth=DEBUG,cmdstdout
>
> When you run a crawl now, you should see more logs in
> 'logs/hadoop.log' than before. I hope it helps you in providing some
> clues. In case you want to compare the logs with how the control flows
> from the source code, I have attached the JAVA file as well.
>
> Regards,
> Susam Pal
>
Hi Susam,
first of all I want to thank you for your support :). I've tried your
solution and I've seen in the log file the that the authentication
parameters was correctly read by the application.
In the log file I've finded these lines about auth.AuthChallengeProcessor:
2010-03-15 09:52:33,140 DEBUG auth.AuthChallengeProcessor - Challenge
for ntlm authentication scheme not available
2010-03-15 09:52:33,140 DEBUG auth.AuthChallengeProcessor - Challenge
for digest authentication scheme not available
2010-03-15 09:52:33,140 INFO auth.AuthChallengeProcessor - basic
authentication scheme selected
2010-03-15 09:52:33,140 DEBUG auth.AuthChallengeProcessor - Using
authentication scheme: basic
2010-03-15 09:52:33,140 DEBUG auth.AuthChallengeProcessor -
Authorization challenge processed
2010-03-15 09:52:33,140 INFO httpclient.HttpMethodDirector - No
credentials available for BASIC 'Squid proxy-caching web
server'@my.proxy:my.port
Best regards,
--
-----------
Graziano Aliberti
Engineering Ingegneria Informatica S.p.A
Via S. Martino della Battaglia, 56 - 00185 ROMA
*Tel.:* 06.49.201.387
*E-Mail:* graziano.aliberti@eng.it
Re: Proxy Authentication
Posted by Susam Pal <su...@gmail.com>.
On Fri, Mar 12, 2010 at 3:17 PM, Susam Pal <su...@gmail.com> wrote:
> On Fri, Mar 12, 2010 at 2:09 PM, Graziano Aliberti
> <gr...@eng.it> wrote:
>> Il 11/03/2010 16.20, Susam Pal ha scritto:
>>>
>>> On Thu, Mar 11, 2010 at 8:24 PM, Graziano Aliberti
>>> <gr...@eng.it> �wrote:
>>>
>>>>
>>>> Hi everyone,
>>>>
>>>> I'm trying to use nutch ver. 1.0 on a system under squid proxy control.
>>>> When
>>>> I try to fetch my website list, into the log file I see that the
>>>> authentication was failed...
>>>>
>>>> I've configured my nutch-site.xml file with all that properties needed
>>>> for
>>>> proxy auth, but my error is "httpclient.HttpMethodDirector - No
>>>> credentials
>>>> available for BASIC 'Squid proxy-caching web
>>>> server'@proxy.my.host:my.port"
>>>>
>>>>
>>>
>>> Did you replace 'protocol-http' with 'protocol-httpclient' in the
>>> value for 'plugins.include' property in 'conf/nutch-site.xml'?
>>>
>>> Regards,
>>> Susam Pal
>>>
>>>
>>>
>>
>> Hi Susam,
>>
>> yes of course!! :) Maybe I can post you the configuration file:
>>
>> <?xml version="1.0"?>
>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>
>> <!-- Put site-specific property overrides in this file. -->
>>
>> <configuration>
>>
>> <property>
>> <name>http.agent.name</name>
>> <value>my.agent.name</value>
>> <description>
>> </description>
>> </property>
>>
>> <property>
>> <name>plugin.includes</name>
>> <value>protocol-httpclient|urlfilter-regex|parse-(text|html|js)|index-(basic|anchor)|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
>> <description>
>> </description>
>> </property>
>>
>> <property>
>> <name>http.auth.file</name>
>> <value>my_file.xml</value>
>> <description>Authentication configuration file for
>> �'protocol-httpclient' plugin.
>> </description>
>> </property>
>>
>> <property>
>> <name>http.proxy.host</name>
>> <value>ip.my.proxy</value>
>> <description>The proxy hostname. �If empty, no proxy is used.</description>
>> </property>
>>
>> <property>
>> <name>http.proxy.port</name>
>> <value>my.port</value>
>> <description>The proxy port.</description>
>> </property>
>>
>> <property>
>> <name>http.proxy.username</name>
>> <value>my.user</value>
>> <description>
>> </description>
>> </property>
>>
>> <property>
>> <name>http.proxy.password</name>
>> <value>my.pwd</value>
>> <description>
>> </description>
>> </property>
>>
>> <property>
>> <name>http.proxy.realm</name>
>> <value>my_realm</value>
>> <description>
>> </description>
>> </property>
>>
>> <property>
>> <name>http.agent.host</name>
>> <value>my.local.pc</value>
>> <description>The agent host.</description>
>> </property>
>>
>> <property>
>> <name>http.useHttp11</name>
>> <value>true</value>
>> <description>
>> </description>
>> </property>
>>
>> </configuration>
>>
>> Only another question: where i must put the user authentication parameters
>> (user,pwd)? In nutch-site.xml file or in my_file.xml that I use for
>> authentication?
>>
>> Thank you for your attention,
>>
>>
>> --
>> -----------
>>
>> Graziano Aliberti
>>
>> Engineering Ingegneria Informatica S.p.A
>>
>> Via S. Martino della Battaglia, 56 - 00185 ROMA
>>
>> *Tel.:* 06.49.201.387
>>
>> *E-Mail:* graziano.aliberti@eng.it
>>
>>
>>
>
> The configuration looks okay to me. Yes, the proxy authentication
> details are set in 'conf/nutch-site.xml'. The file mentioned in
> 'http.auth.file' property is used for configuring authentication
> details for authenticating to a web server.
>
> Unfortunately, there aren't any log statements in the part of the code
> that reads the proxy authentication details. So, I can't suggest you
> to turn on debug logs to get some clues about the issue. However, in
> case you want to troubleshoot it yourself by building Nutch from
> source, I can tell you the code that deals with this.
>
> The file is: src/java/org/apache/nutch/protocol/httpclient/Http.java :
> http://svn.apache.org/viewvc/lucene/nutch/trunk/src/plugin/protocol-httpclient/src/java/org/apache/nutch/protocol/httpclient/Http.java?view=markup
>
> The line number is: 200.
>
> If I get time this weekend, I will try to insert some log statements
> into this code and send a modified JAR file to you which might help
> you to troubleshoot what is going on. But I can't promise this since
> it depends on my weekend plans.
>
> Two questions before I end this mail. Did you set the value of
> 'http.proxy.realm' property as: Squid proxy-caching web server ?
>
> Also, do you see any 'auth.AuthChallengeProcessor' lines in the log
> file? I'm not sure whether this line should appear for proxy
> authentication but it does appear for web server authentication.
>
> Regards,
> Susam Pal
>
I managed to find some time to insert more logs into
protocol-httpclient and create a JAR. I have attached it with this
email.
Please replace your
'plugins/protocol-httpclient/protocol-httpclient.jar' file with the
one that I have attached. Also, edit your 'conf/log4j.properties' file
to add these two lines:
log4j.logger.org.apache.nutch.protocol.httpclient=DEBUG,cmdstdout
log4j.logger.org.apache.commons.httpclient.auth=DEBUG,cmdstdout
When you run a crawl now, you should see more logs in
'logs/hadoop.log' than before. I hope it helps you in providing some
clues. In case you want to compare the logs with how the control flows
from the source code, I have attached the JAVA file as well.
Regards,
Susam Pal
Re: Proxy Authentication
Posted by Susam Pal <su...@gmail.com>.
On Fri, Mar 12, 2010 at 2:09 PM, Graziano Aliberti
<gr...@eng.it> wrote:
> Il 11/03/2010 16.20, Susam Pal ha scritto:
>>
>> On Thu, Mar 11, 2010 at 8:24 PM, Graziano Aliberti
>> <gr...@eng.it> wrote:
>>
>>>
>>> Hi everyone,
>>>
>>> I'm trying to use nutch ver. 1.0 on a system under squid proxy control.
>>> When
>>> I try to fetch my website list, into the log file I see that the
>>> authentication was failed...
>>>
>>> I've configured my nutch-site.xml file with all that properties needed
>>> for
>>> proxy auth, but my error is "httpclient.HttpMethodDirector - No
>>> credentials
>>> available for BASIC 'Squid proxy-caching web
>>> server'@proxy.my.host:my.port"
>>>
>>>
>>
>> Did you replace 'protocol-http' with 'protocol-httpclient' in the
>> value for 'plugins.include' property in 'conf/nutch-site.xml'?
>>
>> Regards,
>> Susam Pal
>>
>>
>>
>
> Hi Susam,
>
> yes of course!! :) Maybe I can post you the configuration file:
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
>
> <property>
> <name>http.agent.name</name>
> <value>my.agent.name</value>
> <description>
> </description>
> </property>
>
> <property>
> <name>plugin.includes</name>
> <value>protocol-httpclient|urlfilter-regex|parse-(text|html|js)|index-(basic|anchor)|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
> <description>
> </description>
> </property>
>
> <property>
> <name>http.auth.file</name>
> <value>my_file.xml</value>
> <description>Authentication configuration file for
> 'protocol-httpclient' plugin.
> </description>
> </property>
>
> <property>
> <name>http.proxy.host</name>
> <value>ip.my.proxy</value>
> <description>The proxy hostname. If empty, no proxy is used.</description>
> </property>
>
> <property>
> <name>http.proxy.port</name>
> <value>my.port</value>
> <description>The proxy port.</description>
> </property>
>
> <property>
> <name>http.proxy.username</name>
> <value>my.user</value>
> <description>
> </description>
> </property>
>
> <property>
> <name>http.proxy.password</name>
> <value>my.pwd</value>
> <description>
> </description>
> </property>
>
> <property>
> <name>http.proxy.realm</name>
> <value>my_realm</value>
> <description>
> </description>
> </property>
>
> <property>
> <name>http.agent.host</name>
> <value>my.local.pc</value>
> <description>The agent host.</description>
> </property>
>
> <property>
> <name>http.useHttp11</name>
> <value>true</value>
> <description>
> </description>
> </property>
>
> </configuration>
>
> Only another question: where i must put the user authentication parameters
> (user,pwd)? In nutch-site.xml file or in my_file.xml that I use for
> authentication?
>
> Thank you for your attention,
>
>
> --
> -----------
>
> Graziano Aliberti
>
> Engineering Ingegneria Informatica S.p.A
>
> Via S. Martino della Battaglia, 56 - 00185 ROMA
>
> *Tel.:* 06.49.201.387
>
> *E-Mail:* graziano.aliberti@eng.it
>
>
>
The configuration looks okay to me. Yes, the proxy authentication
details are set in 'conf/nutch-site.xml'. The file mentioned in
'http.auth.file' property is used for configuring authentication
details for authenticating to a web server.
Unfortunately, there aren't any log statements in the part of the code
that reads the proxy authentication details. So, I can't suggest you
to turn on debug logs to get some clues about the issue. However, in
case you want to troubleshoot it yourself by building Nutch from
source, I can tell you the code that deals with this.
The file is: src/java/org/apache/nutch/protocol/httpclient/Http.java :
http://svn.apache.org/viewvc/lucene/nutch/trunk/src/plugin/protocol-httpclient/src/java/org/apache/nutch/protocol/httpclient/Http.java?view=markup
The line number is: 200.
If I get time this weekend, I will try to insert some log statements
into this code and send a modified JAR file to you which might help
you to troubleshoot what is going on. But I can't promise this since
it depends on my weekend plans.
Two questions before I end this mail. Did you set the value of
'http.proxy.realm' property as: Squid proxy-caching web server ?
Also, do you see any 'auth.AuthChallengeProcessor' lines in the log
file? I'm not sure whether this line should appear for proxy
authentication but it does appear for web server authentication.
Regards,
Susam Pal
Re: Proxy Authentication
Posted by Graziano Aliberti <gr...@eng.it>.
Il 11/03/2010 16.20, Susam Pal ha scritto:
> On Thu, Mar 11, 2010 at 8:24 PM, Graziano Aliberti
> <gr...@eng.it> wrote:
>
>> Hi everyone,
>>
>> I'm trying to use nutch ver. 1.0 on a system under squid proxy control. When
>> I try to fetch my website list, into the log file I see that the
>> authentication was failed...
>>
>> I've configured my nutch-site.xml file with all that properties needed for
>> proxy auth, but my error is "httpclient.HttpMethodDirector - No credentials
>> available for BASIC 'Squid proxy-caching web server'@proxy.my.host:my.port"
>>
>>
> Did you replace 'protocol-http' with 'protocol-httpclient' in the
> value for 'plugins.include' property in 'conf/nutch-site.xml'?
>
> Regards,
> Susam Pal
>
>
>
Hi Susam,
yes of course!! :) Maybe I can post you the configuration file:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>http.agent.name</name>
<value>my.agent.name</value>
<description>
</description>
</property>
<property>
<name>plugin.includes</name>
<value>protocol-httpclient|urlfilter-regex|parse-(text|html|js)|index-(basic|anchor)|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
<description>
</description>
</property>
<property>
<name>http.auth.file</name>
<value>my_file.xml</value>
<description>Authentication configuration file for
'protocol-httpclient' plugin.
</description>
</property>
<property>
<name>http.proxy.host</name>
<value>ip.my.proxy</value>
<description>The proxy hostname. If empty, no proxy is used.</description>
</property>
<property>
<name>http.proxy.port</name>
<value>my.port</value>
<description>The proxy port.</description>
</property>
<property>
<name>http.proxy.username</name>
<value>my.user</value>
<description>
</description>
</property>
<property>
<name>http.proxy.password</name>
<value>my.pwd</value>
<description>
</description>
</property>
<property>
<name>http.proxy.realm</name>
<value>my_realm</value>
<description>
</description>
</property>
<property>
<name>http.agent.host</name>
<value>my.local.pc</value>
<description>The agent host.</description>
</property>
<property>
<name>http.useHttp11</name>
<value>true</value>
<description>
</description>
</property>
</configuration>
Only another question: where i must put the user authentication
parameters (user,pwd)? In nutch-site.xml file or in my_file.xml that I
use for authentication?
Thank you for your attention,
--
-----------
Graziano Aliberti
Engineering Ingegneria Informatica S.p.A
Via S. Martino della Battaglia, 56 - 00185 ROMA
*Tel.:* 06.49.201.387
*E-Mail:* graziano.aliberti@eng.it
Re: Proxy Authentication
Posted by Susam Pal <su...@gmail.com>.
On Thu, Mar 11, 2010 at 8:24 PM, Graziano Aliberti
<gr...@eng.it> wrote:
> Hi everyone,
>
> I'm trying to use nutch ver. 1.0 on a system under squid proxy control. When
> I try to fetch my website list, into the log file I see that the
> authentication was failed...
>
> I've configured my nutch-site.xml file with all that properties needed for
> proxy auth, but my error is "httpclient.HttpMethodDirector - No credentials
> available for BASIC 'Squid proxy-caching web server'@proxy.my.host:my.port"
>
Did you replace 'protocol-http' with 'protocol-httpclient' in the
value for 'plugins.include' property in 'conf/nutch-site.xml'?
Regards,
Susam Pal