You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by Tolga <to...@ozses.net> on 2012/05/10 08:10:04 UTC

HTTP error 400

Hi,

This will sound like a duplicate, but actually it differs from the other 
one. Please bear with me. Following 
http://wiki.apache.org/nutch/NutchTutorial, I first issued the command

bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3 -topN 5

Then when I got the message

Exception in thread "main" java.io.IOException: Job failed!
     at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
     at 
org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:373)
     at 
org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:353)
     at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
     at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)

I issued the commands

bin/nutch crawl urls -dir crawl -depth 3 -topN 5

and

bin/nutch solrindex http://127.0.0.1:8983/solr/ crawldb -linkdb crawldb/linkdb crawldb/segments/*

separately, after which I got no errors. When I browsed to 
http://localhost:8983/solr/admin and attempted a search, I got the error


    HTTP ERROR 400

Problem accessing /solr/select. Reason:

     undefined field text

------------------------------------------------------------------------
/Powered by Jetty://

/What am I doing wrong?

Regards,/
/

Re: HTTP error 400

Posted by Ferdy Galema <fe...@kalooga.com>.

It indeed seems to be caused by some sort of (schema) configuration issue.
We are currently trying to resolve this issue. Although the browse page
shows an error, it is still possible to search. Use:
/solr/select/?q=content%3Athe

On Thu, May 10, 2012 at 3:45 PM, Markus Jelsma
<ma...@openindex.io>wrote:

> On Thursday 10 May 2012 14:35:03 Lewis John Mcgibbney wrote:
> > Hi Michael,
> >
> > As I'm also not using most recent stable Solr distribution (3.6.0), I
> > can only comment (maybe unwisely) that the most recent version of Solr
> > that Nutch supports is maybe 3.4.0 as this is the dependency we pull
> > with ivy. It also looks like Solr and Solrj are released in parallel
> > so maybe try upgrading your solrj dependency if you wish to use Solr
> > 3.6.0...
>
> This should not be a version issue. We happily index from trunk or 1.4 to
> Solr
> versions > 3.0. There must be some schema thing or bad Solr request handler
> defined.
>
> >
> > If the above is correct, then this is why 3.1.0 works fine when you
> > roll back as I would imagine backwards compatibility is always of key
> > importance.
> >
> > I would be pleased to know that the above is not correct and that
> > Nutch is above to index to Solr 3.6.0, however if not then maybe we
> > should upgrade accordingly in trunk.
> >
> > Thanks
> >
> > Lewis
> >
> > On Thu, May 10, 2012 at 1:56 PM, Michael Erickson
> >
> > <er...@gmail.com> wrote:
> > > On May 10, 2012, at 1:42 AM, Markus Jelsma wrote:
> > >> Hi,
> > >>
> > >> On Thu, 10 May 2012 09:10:04 +0300, Tolga <to...@ozses.net> wrote:
> > >>> Hi,
> > >>>
> > >>> This will sound like a duplicate, but actually it differs from the
> > >>> other one. Please bear with me. Following
> > >>> http://wiki.apache.org/nutch/NutchTutorial, I first issued the
> command
> > >>>
> > >>> bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3
> -topN 5
> > >>>
> > >>> Then when I got the message
> > >>>
> > >>> Exception in thread "main" java.io.IOException: Job failed!
> > >>>    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> > >>>    at
> > >>>
> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDupli
> > >>> cates.java:373) at
> > >>>
> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDupli
> > >>> cates.java:353) at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
> > >>>    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > >>>    at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
> > >>
> > >> Please include the relevant part of the log. This can be a known
> issue.
> > >>
> > >>> I issued the commands
> > >>>
> > >>> bin/nutch crawl urls -dir crawl -depth 3 -topN 5
> > >>>
> > >>> and
> > >>>
> > >>> bin/nutch solrindex http://127.0.0.1:8983/solr/ crawldb -linkdb
> > >>> crawldb/linkdb crawldb/segments/*
> > >>>
> > >>> separately, after which I got no errors. When I browsed to
> > >>> http://localhost:8983/solr/admin and attempted a search, I got the
> > >>> error
> > >>>
> > >>>
> > >>>   HTTP ERROR 400
> > >>>
> > >>> Problem accessing /solr/select. Reason:
> > >>>
> > >>>    undefined field text
> > >>
> > >> But this is a Solr thing, you have no field named text. Resolve this
> in
> > >> Solr or on the Solr mailing list.>
> > > I will say that I had similar issues last week when I tried the Nutch
> > > tutorial.  I went to the #Solr IRC channel and got no response.  The
> > > quick answer was that I had to go back to Solr version 3.1.0 for the
> > > instructions in the Nutch tutorial to work.
> > >
> > > The longer answer is that following the existing Nutch tutorial gave me
> > > two errors.
> > >
> > > 1) SolrDeleteDuplicates exception as mentioned by Tolga above.
> > >
> > > To fix this I:
> > >
> > > 1.a) Stop Solr.
> > > 1.b) Delete Solr index.
> > > 1.c) Copy the Nutch-provided schema.xml into the proper Solr directory
> > > (example/solr/conf/). 1.d) Replace Nutch's solr-solrj-xxx.jar with the
> > > appropriate version from Solr: ( solr/dist/apache-solr-solrj-xxx.jar
>  -->
> > > nutch/runtime/local/lib/solr-solrj-xxx.jar ) 1.e) Restart Solr.
> > >
> > > The first two steps may only be necessary if you had Solr running
> already
> > > using the default schema that they provided as I did because I had done
> > > the Solr tutorial first.
> > >
> > > 2) The HTTP 400 Error "undefined field text" issue.
> > >
> > > This appears to be the same as:
> > > https://issues.apache.org/jira/browse/SOLR-3416.  Log output from Solr
> > > output is here: http://pastebin.com/YWdPnXpv and the Nutch provided
> > > schema is here: http://pastebin.com/LQDDKC5B
> > >
> > > The only way I got this working was to move Solr from version 3.6.0
> back
> > > to version 3.1.0.
> > >
> > > I'm *totally* new to Solr/Nutch, but I might suggest a versioning
> > > mismatch?
> > >
> > >
> > > Regards,
> > > --mike
> > >
> > > Michael Erickson
> > > erickson.michael@gmail.com
> --
> Markus Jelsma - CTO - Openindex
>
>

Re: HTTP error 400

Posted by Markus Jelsma <ma...@openindex.io>.

On Thursday 10 May 2012 14:35:03 Lewis John Mcgibbney wrote:
> Hi Michael,
> 
> As I'm also not using most recent stable Solr distribution (3.6.0), I
> can only comment (maybe unwisely) that the most recent version of Solr
> that Nutch supports is maybe 3.4.0 as this is the dependency we pull
> with ivy. It also looks like Solr and Solrj are released in parallel
> so maybe try upgrading your solrj dependency if you wish to use Solr
> 3.6.0...

This should not be a version issue. We happily index from trunk or 1.4 to Solr 
versions > 3.0. There must be some schema thing or bad Solr request handler 
defined.

> 
> If the above is correct, then this is why 3.1.0 works fine when you
> roll back as I would imagine backwards compatibility is always of key
> importance.
> 
> I would be pleased to know that the above is not correct and that
> Nutch is above to index to Solr 3.6.0, however if not then maybe we
> should upgrade accordingly in trunk.
> 
> Thanks
> 
> Lewis
> 
> On Thu, May 10, 2012 at 1:56 PM, Michael Erickson
> 
> <er...@gmail.com> wrote:
> > On May 10, 2012, at 1:42 AM, Markus Jelsma wrote:
> >> Hi,
> >> 
> >> On Thu, 10 May 2012 09:10:04 +0300, Tolga <to...@ozses.net> wrote:
> >>> Hi,
> >>> 
> >>> This will sound like a duplicate, but actually it differs from the
> >>> other one. Please bear with me. Following
> >>> http://wiki.apache.org/nutch/NutchTutorial, I first issued the command
> >>> 
> >>> bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3 -topN 5
> >>> 
> >>> Then when I got the message
> >>> 
> >>> Exception in thread "main" java.io.IOException: Job failed!
> >>>    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> >>>    at
> >>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDupli
> >>> cates.java:373) at
> >>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDupli
> >>> cates.java:353) at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
> >>>    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >>>    at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
> >> 
> >> Please include the relevant part of the log. This can be a known issue.
> >> 
> >>> I issued the commands
> >>> 
> >>> bin/nutch crawl urls -dir crawl -depth 3 -topN 5
> >>> 
> >>> and
> >>> 
> >>> bin/nutch solrindex http://127.0.0.1:8983/solr/ crawldb -linkdb
> >>> crawldb/linkdb crawldb/segments/*
> >>> 
> >>> separately, after which I got no errors. When I browsed to
> >>> http://localhost:8983/solr/admin and attempted a search, I got the
> >>> error
> >>> 
> >>> 
> >>>   HTTP ERROR 400
> >>> 
> >>> Problem accessing /solr/select. Reason:
> >>> 
> >>>    undefined field text
> >> 
> >> But this is a Solr thing, you have no field named text. Resolve this in
> >> Solr or on the Solr mailing list.> 
> > I will say that I had similar issues last week when I tried the Nutch
> > tutorial.  I went to the #Solr IRC channel and got no response.  The
> > quick answer was that I had to go back to Solr version 3.1.0 for the
> > instructions in the Nutch tutorial to work.
> > 
> > The longer answer is that following the existing Nutch tutorial gave me
> > two errors.
> > 
> > 1) SolrDeleteDuplicates exception as mentioned by Tolga above.
> > 
> > To fix this I:
> > 
> > 1.a) Stop Solr.
> > 1.b) Delete Solr index.
> > 1.c) Copy the Nutch-provided schema.xml into the proper Solr directory
> > (example/solr/conf/). 1.d) Replace Nutch's solr-solrj-xxx.jar with the
> > appropriate version from Solr: ( solr/dist/apache-solr-solrj-xxx.jar  -->
> > nutch/runtime/local/lib/solr-solrj-xxx.jar ) 1.e) Restart Solr.
> > 
> > The first two steps may only be necessary if you had Solr running already
> > using the default schema that they provided as I did because I had done
> > the Solr tutorial first.
> > 
> > 2) The HTTP 400 Error "undefined field text" issue.
> > 
> > This appears to be the same as:
> > https://issues.apache.org/jira/browse/SOLR-3416.  Log output from Solr
> > output is here: http://pastebin.com/YWdPnXpv and the Nutch provided
> > schema is here: http://pastebin.com/LQDDKC5B
> > 
> > The only way I got this working was to move Solr from version 3.6.0 back
> > to version 3.1.0.
> > 
> > I'm *totally* new to Solr/Nutch, but I might suggest a versioning
> > mismatch?
> > 
> > 
> > Regards,
> > --mike
> > 
> > Michael Erickson
> > erickson.michael@gmail.com
-- 
Markus Jelsma - CTO - Openindex

Re: HTTP error 400

Posted by Lewis John Mcgibbney <le...@gmail.com>.

Hi Michael,

As I'm also not using most recent stable Solr distribution (3.6.0), I
can only comment (maybe unwisely) that the most recent version of Solr
that Nutch supports is maybe 3.4.0 as this is the dependency we pull
with ivy. It also looks like Solr and Solrj are released in parallel
so maybe try upgrading your solrj dependency if you wish to use Solr
3.6.0...

If the above is correct, then this is why 3.1.0 works fine when you
roll back as I would imagine backwards compatibility is always of key
importance.

I would be pleased to know that the above is not correct and that
Nutch is above to index to Solr 3.6.0, however if not then maybe we
should upgrade accordingly in trunk.

Thanks

Lewis

On Thu, May 10, 2012 at 1:56 PM, Michael Erickson
<er...@gmail.com> wrote:
>
> On May 10, 2012, at 1:42 AM, Markus Jelsma wrote:
>
>> Hi,
>>
>> On Thu, 10 May 2012 09:10:04 +0300, Tolga <to...@ozses.net> wrote:
>>> Hi,
>>>
>>> This will sound like a duplicate, but actually it differs from the
>>> other one. Please bear with me. Following
>>> http://wiki.apache.org/nutch/NutchTutorial, I first issued the command
>>>
>>> bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3 -topN 5
>>>
>>> Then when I got the message
>>>
>>> Exception in thread "main" java.io.IOException: Job failed!
>>>    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
>>>    at
>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:373)
>>>    at
>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:353)
>>>    at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
>>>    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>    at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
>>
>> Please include the relevant part of the log. This can be a known issue.
>>
>>>
>>> I issued the commands
>>>
>>> bin/nutch crawl urls -dir crawl -depth 3 -topN 5
>>>
>>> and
>>>
>>> bin/nutch solrindex http://127.0.0.1:8983/solr/ crawldb -linkdb
>>> crawldb/linkdb crawldb/segments/*
>>>
>>> separately, after which I got no errors. When I browsed to
>>> http://localhost:8983/solr/admin and attempted a search, I got the
>>> error
>>>
>>>
>>>   HTTP ERROR 400
>>>
>>> Problem accessing /solr/select. Reason:
>>>
>>>    undefined field text
>>
>> But this is a Solr thing, you have no field named text. Resolve this in Solr or on the Solr mailing list.
>
>
> I will say that I had similar issues last week when I tried the Nutch tutorial.  I went to the #Solr IRC channel and got no response.  The quick answer was that I had to go back to Solr version 3.1.0 for the instructions in the Nutch tutorial to work.
>
> The longer answer is that following the existing Nutch tutorial gave me two errors.
>
> 1) SolrDeleteDuplicates exception as mentioned by Tolga above.
>
> To fix this I:
>
> 1.a) Stop Solr.
> 1.b) Delete Solr index.
> 1.c) Copy the Nutch-provided schema.xml into the proper Solr directory (example/solr/conf/).
> 1.d) Replace Nutch's solr-solrj-xxx.jar with the appropriate version from Solr:
>       ( solr/dist/apache-solr-solrj-xxx.jar  --> nutch/runtime/local/lib/solr-solrj-xxx.jar )
> 1.e) Restart Solr.
>
> The first two steps may only be necessary if you had Solr running already using the default schema that they provided as I did because I had done the Solr tutorial first.
>
> 2) The HTTP 400 Error "undefined field text" issue.
>
> This appears to be the same as: https://issues.apache.org/jira/browse/SOLR-3416.  Log output from Solr output is here: http://pastebin.com/YWdPnXpv and the Nutch provided schema is here: http://pastebin.com/LQDDKC5B
>
> The only way I got this working was to move Solr from version 3.6.0 back to version 3.1.0.
>
> I'm *totally* new to Solr/Nutch, but I might suggest a versioning mismatch?
>
>
> Regards,
> --mike
>
> Michael Erickson
> erickson.michael@gmail.com
>
>



-- 
Lewis

Re: HTTP error 400

Posted by Michael Erickson <er...@gmail.com>.

On May 10, 2012, at 1:42 AM, Markus Jelsma wrote:

> Hi,
> 
> On Thu, 10 May 2012 09:10:04 +0300, Tolga <to...@ozses.net> wrote:
>> Hi,
>> 
>> This will sound like a duplicate, but actually it differs from the
>> other one. Please bear with me. Following
>> http://wiki.apache.org/nutch/NutchTutorial, I first issued the command
>> 
>> bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3 -topN 5
>> 
>> Then when I got the message
>> 
>> Exception in thread "main" java.io.IOException: Job failed!
>>    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
>>    at
>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:373)
>>    at
>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:353)
>>    at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
>>    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>    at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
> 
> Please include the relevant part of the log. This can be a known issue.
> 
>> 
>> I issued the commands
>> 
>> bin/nutch crawl urls -dir crawl -depth 3 -topN 5
>> 
>> and
>> 
>> bin/nutch solrindex http://127.0.0.1:8983/solr/ crawldb -linkdb
>> crawldb/linkdb crawldb/segments/*
>> 
>> separately, after which I got no errors. When I browsed to
>> http://localhost:8983/solr/admin and attempted a search, I got the
>> error
>> 
>> 
>>   HTTP ERROR 400
>> 
>> Problem accessing /solr/select. Reason:
>> 
>>    undefined field text
> 
> But this is a Solr thing, you have no field named text. Resolve this in Solr or on the Solr mailing list.

I will say that I had similar issues last week when I tried the Nutch tutorial.  I went to the #Solr IRC channel and got no response.  The quick answer was that I had to go back to Solr version 3.1.0 for the instructions in the Nutch tutorial to work.

The longer answer is that following the existing Nutch tutorial gave me two errors.

1) SolrDeleteDuplicates exception as mentioned by Tolga above.

To fix this I:

1.a) Stop Solr.
1.b) Delete Solr index.
1.c) Copy the Nutch-provided schema.xml into the proper Solr directory (example/solr/conf/).
1.d) Replace Nutch's solr-solrj-xxx.jar with the appropriate version from Solr:
       ( solr/dist/apache-solr-solrj-xxx.jar  --> nutch/runtime/local/lib/solr-solrj-xxx.jar )
1.e) Restart Solr.

The first two steps may only be necessary if you had Solr running already using the default schema that they provided as I did because I had done the Solr tutorial first.

2) The HTTP 400 Error "undefined field text" issue.

This appears to be the same as: https://issues.apache.org/jira/browse/SOLR-3416.  Log output from Solr output is here: http://pastebin.com/YWdPnXpv and the Nutch provided schema is here: http://pastebin.com/LQDDKC5B

The only way I got this working was to move Solr from version 3.6.0 back to version 3.1.0.

I'm *totally* new to Solr/Nutch, but I might suggest a versioning mismatch?

Regards,
--mike

Michael Erickson
erickson.michael@gmail.com

Re: HTTP error 400

Posted by keesp <ce...@wxs.nl>.

Stephan Kristyn wrote
> 
> Regarding the issue, I found out I was not using the right search syntax.
> 
> Make sure you specify a field and searchterm with a colon, like this:
> 
> XMLfieldname:SearchTerm
> 

I was having the same problem: http error 400: underfined field text, using
the most recent versions of solr and nutch (and having used the nutch
configuration files.
a search using:

      content:searchterm 

works fine. Is there any way to tell solr that text:searchtermis equivalent
to content:searchterm?

Thanks

--
View this message in context: http://lucene.472066.n3.nabble.com/HTTP-error-400-tp3976225p3985866.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: HTTP error 400

Posted by Stephan Kristyn <kr...@yahoo-inc.com>.

Regarding the issue, I found out I was not using the right search syntax.

Make sure you specify a field and searchterm with a colon, like this:

XMLfieldname:SearchTerm

Am 10.05.2012 09:07, schrieb Tolga:
> Thanks. *heads to the Solr list*
>
> On 5/10/12 9:42 AM, Markus Jelsma wrote:
>> Hi,
>>
>> On Thu, 10 May 2012 09:10:04 +0300, Tolga <to...@ozses.net> wrote:
>>> Hi,
>>>
>>> This will sound like a duplicate, but actually it differs from the
>>> other one. Please bear with me. Following
>>> http://wiki.apache.org/nutch/NutchTutorial, I first issued the command
>>>
>>> bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3 -topN 5
>>>
>>> Then when I got the message
>>>
>>> Exception in thread "main" java.io.IOException: Job failed!
>>>     at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
>>>     at
>>>
>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:373) 
>>>
>>>     at
>>>
>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:353) 
>>>
>>>     at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
>>>     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>     at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
>> Please include the relevant part of the log. This can be a known issue.
>>
>>> I issued the commands
>>>
>>> bin/nutch crawl urls -dir crawl -depth 3 -topN 5
>>>
>>> and
>>>
>>> bin/nutch solrindex http://127.0.0.1:8983/solr/ crawldb -linkdb
>>> crawldb/linkdb crawldb/segments/*
>>>
>>> separately, after which I got no errors. When I browsed to
>>> http://localhost:8983/solr/admin and attempted a search, I got the
>>> error
>>>
>>>
>>>    HTTP ERROR 400
>>>
>>> Problem accessing /solr/select. Reason:
>>>
>>>     undefined field text
>> But this is a Solr thing, you have no field named text. Resolve this 
>> in Solr or on the Solr mailing list.
>>
>>>
>>> ------------------------------------------------------------------------
>>> /Powered by Jetty://
>>>
>>> /What am I doing wrong?
>>>
>>> Regards,/
>>> /

-- 

 

*stephan*
*kristyn*
partner operations manager
 
"The Internet? Is that thing still around?" - Homer Simpson
 
kristyns@yahoo-inc.com
direct +49 (0)89 231 97 207    mobile +49 (0) 162 28899 02
 
yahoo! deutschland gmbh theresienhoehe 12, munich, 80339, germany
phone (408) 349 3300    fax (408) 349 3301
 
http://us.i1.yimg.com/us.yimg.com/i/pt/i/buzzmktg/brand/logos/yahoo_email_sig_generic_v2.gif

Re: HTTP error 400

Posted by Tolga <to...@ozses.net>.

Thanks. *heads to the Solr list*

On 5/10/12 9:42 AM, Markus Jelsma wrote:
> Hi,
>
> On Thu, 10 May 2012 09:10:04 +0300, Tolga <to...@ozses.net> wrote:
>> Hi,
>>
>> This will sound like a duplicate, but actually it differs from the
>> other one. Please bear with me. Following
>> http://wiki.apache.org/nutch/NutchTutorial, I first issued the command
>>
>> bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3 -topN 5
>>
>> Then when I got the message
>>
>> Exception in thread "main" java.io.IOException: Job failed!
>>     at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
>>     at
>>
>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:373) 
>>
>>     at
>>
>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:353) 
>>
>>     at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
>>     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>     at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
>
> Please include the relevant part of the log. This can be a known issue.
>
>>
>> I issued the commands
>>
>> bin/nutch crawl urls -dir crawl -depth 3 -topN 5
>>
>> and
>>
>> bin/nutch solrindex http://127.0.0.1:8983/solr/ crawldb -linkdb
>> crawldb/linkdb crawldb/segments/*
>>
>> separately, after which I got no errors. When I browsed to
>> http://localhost:8983/solr/admin and attempted a search, I got the
>> error
>>
>>
>>    HTTP ERROR 400
>>
>> Problem accessing /solr/select. Reason:
>>
>>     undefined field text
>
> But this is a Solr thing, you have no field named text. Resolve this 
> in Solr or on the Solr mailing list.
>
>>
>>
>> ------------------------------------------------------------------------
>> /Powered by Jetty://
>>
>> /What am I doing wrong?
>>
>> Regards,/
>> /
>

Re: HTTP error 400

Posted by Tolga <to...@ozses.net>.

bin/nutch solrindex http://localhost:8983/solr/ crawldb -linkdb 
crawl/linkdb crawl/segments/*

SolrIndexer: starting at 2012-05-15 15:34:36
org.apache.hadoop.mapred.InvalidInputException: Input path does not 
exist: file:/root/apache-nutch-1.4-bin/runtime/local/crawl/current


On 5/15/12 2:05 PM, Markus Jelsma wrote:
> Please follow the step-by-step tutorial, it's explained there:
> http://wiki.apache.org/nutch/NutchTutorial
>
> On Tuesday 15 May 2012 13:40:26 Tolga wrote:
>> I'm a little confused. How can I not use the crawl command and execute
>> the separate crawl cycle commands at the same time?
>>
>> Regards,
>>
>> On 5/11/12 9:40 AM, Markus Jelsma wrote:
>>> Ah, that means don't use the crawl command and do a little shell
>>> scripting to execute the separte crawl cycle commands, see the nutch
>>> wiki for examples. And don't do solrdedup. Search the Solr wiki for
>>> deduplication.
>>>
>>> cheers
>>>
>>> On Fri, 11 May 2012 07:39:36 +0300, Tolga<to...@ozses.net>  wrote:
>>>> Hi,
>>>>
>>>> How do I exactly "omit solrdedup and use Solr's internal
>>>> deduplication" instead.? I don't even know what any of that means :D
>>>> I've just used bin/nutch crawl urls -solr http://localhost:8983/solr/
>>>> -depth 3 -topN 100 to get the error. I have to use all the steps?
>>>>
>>>> Regards,
>>>>
>>>> On 05/10/2012 11:38 PM, Markus Jelsma wrote:
>>>>> thanks
>>>>>
>>>>> This is a known issue:
>>>>> https://issues.apache.org/jira/browse/NUTCH-1100
>>>>>
>>>>> I have not been able find the bug nor do i know how to reproduce it
>>>>> from scratch. If you have a public site with which we can reproduce
>>>>> it please comment to the Jira ticket. Make sure you use either
>>>>> default config or little, a seed URL and the exact crawl&  dedup
>>>>> steps to reproduce.
>>>>>
>>>>> If you find it we might fix it. In any case we need to replace the
>>>>> dedup command with a more scalable tool which it currently is not.
>>>>>
>>>>> In the mean time you can omit solrdedup and use Solr's internal
>>>>> deduplication instead, it works similar and uses the same signature
>>>>> algorithm as Nutch has. Please consult the Solr wiki page on
>>>>> deduplication.
>>>>>
>>>>> Good luck
>>>>>
>>>>> On Thu, 10 May 2012 22:54:37 +0300, Tolga<to...@ozses.net>  wrote:
>>>>>> Hi Markus,
>>>>>>
>>>>>> On 05/10/2012 09:42 AM, Markus Jelsma wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> On Thu, 10 May 2012 09:10:04 +0300, Tolga<to...@ozses.net>  wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> This will sound like a duplicate, but actually it differs from the
>>>>>>>> other one. Please bear with me. Following
>>>>>>>> http://wiki.apache.org/nutch/NutchTutorial, I first issued the
>>>>>>>> command
>>>>>>>>
>>>>>>>> bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3
>>>>>>>> -topN 5
>>>>>>>>
>>>>>>>> Then when I got the message
>>>>>>>>
>>>>>>>> Exception in thread "main" java.io.IOException: Job failed!
>>>>>>>>
>>>>>>>>      at
>>>>>>>>
>>>>>>>> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
>>>>>>>>
>>>>>>>>      at
>>>>>>>>
>>>>>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDu
>>>>>>>> plicates.java:373)>>>>>>
>>>>>>>>      at
>>>>>>>>
>>>>>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDu
>>>>>>>> plicates.java:353)>>>>>>
>>>>>>>>      at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
>>>>>>>>      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>>>>>      at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
>>>>>>> Please include the relevant part of the log. This can be a known
>>>>>>> issue.
>>>>>> This is an excerpt from hadoop.log:
>>>>>>
>>>>>> 2012-05-10 22:26:30,349 INFO  crawl.Crawl - crawl started in:
>>>>>> crawl-20120510222629
>>>>>> 2012-05-10 22:26:30,350 INFO  crawl.Crawl - rootUrlDir = urls
>>>>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - threads = 10
>>>>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - depth = 3
>>>>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl -
>>>>>> solrUrl=http://localhost:8983/solr/
>>>>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - topN = 100
>>>>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: starting at
>>>>>> 2012-05-10 22:26:30
>>>>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: crawlDb:
>>>>>> crawl-20120510222629/crawldb
>>>>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: urlDir: urls
>>>>>> 2012-05-10 22:26:30,809 INFO  crawl.Injector - Injector: Converting
>>>>>> injected urls to crawl db entries.
>>>>>> 2012-05-10 22:26:34,173 INFO  plugin.PluginRepository - Plugins:
>>>>>> looking in: /root/apache-nutch-1.4-bin/runtime/local/plugins
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Plugin
>>>>>> Auto-activation mode: [true]
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered
>>>>>> Plugins:
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     the nutch
>>>>>> core extension points (nutch-extensionpoints)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic URL
>>>>>> Normalizer (urlnormalizer-basic)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Html
>>>>>> Parse Plug-in (parse-html)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic
>>>>>> Indexing Filter (index-basic)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     HTTP
>>>>>> Framework (lib-http)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -
>>>>>> Pass-through URL Normalizer (urlnormalizer-pass)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
>>>>>> Filter (urlfilter-regex)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Http
>>>>>> Protocol Plug-in (protocol-http)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
>>>>>> Normalizer (urlnormalizer-regex)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Tika
>>>>>> Parser Plug-in (parse-tika)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     OPIC
>>>>>> Scoring Plug-in (scoring-opic)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     CyberNeko
>>>>>> HTML Parser (lib-nekohtml)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Anchor
>>>>>> Indexing Filter (index-anchor)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
>>>>>> Filter Framework (lib-regex-filter)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered
>>>>>> Extension-Points:
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch URL
>>>>>> Normalizer (org.apache.nutch.net.URLNormalizer)
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>>>> Protocol (org.apache.nutch.protocol.Protocol)
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>>>> Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch URL
>>>>>> Filter (org.apache.nutch.net.URLFilter)
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>>>> Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     HTML
>>>>>> Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>>>> Content Parser (org.apache.nutch.parse.Parser)
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>>>> Scoring (org.apache.nutch.scoring.ScoringFilter)
>>>>>> 2012-05-10 22:26:35,439 INFO  regex.RegexURLNormalizer - can't find
>>>>>> rules for scope 'inject', using default
>>>>>> 2012-05-10 22:26:36,434 INFO  crawl.Injector - Injector: Merging
>>>>>> injected urls into crawl db.
>>>>>> 2012-05-10 22:26:36,710 WARN  util.NativeCodeLoader - Unable to load
>>>>>> native-hadoop library for your platform... using builtin-java classes
>>>>>> where applicable
>>>>>> 2012-05-10 22:26:37,542 INFO  crawl.Injector - Injector: finished at
>>>>>> 2012-05-10 22:26:37, elapsed: 00:00:06
>>>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: starting
>>>>>> at 2012-05-10 22:26:37
>>>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: Selecting
>>>>>> best-scoring urls due for fetch.
>>>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator:
>>>>>> filtering: true
>>>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator:
>>>>>> normalizing: true
>>>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: topN: 100
>>>>>> 2012-05-10 22:26:37,552 INFO  crawl.Generator - Generator: jobtracker
>>>>>> is 'local', generating exactly one partition.
>>>>>> 2012-05-10 22:26:37,820 INFO  crawl.FetchScheduleFactory - Using
>>>>>> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
>>>>>> 2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule -
>>>>>> defaultInterval=2592000
>>>>>> 2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule -
>>>>>> maxInterval=7776000
>>>>>> 2012-05-10 22:26:37,856 INFO  regex.RegexURLNormalizer - can't find
>>>>>> rules for scope 'partition', using default
>>>>>> ...
>>>>>> ...
>>>>>> INFO: [] webapp=/solr path=/update
>>>>>>
>>>>>>
>>>>>> params={waitSearcher=true&waitFlush=true&wt=javabin&commit=true&version
>>>>>> =2}
>>>>>>
>>>>>>
>>>>>> status=0 QTime=221
>>>>>> 2012-05-10 22:36:26,336 INFO  solr.SolrIndexer - SolrIndexer:
>>>>>> finished at 2012-05-10 22:36:26, elapsed: 00:00:05
>>>>>> 2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates -
>>>>>> SolrDeleteDuplicates: starting at 2012-05-10 22:36:26
>>>>>> 2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates -
>>>>>> SolrDeleteDuplicates: Solr url: http://localhost:8983/solr/
>>>>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>>>>>> INFO: [] webapp=/solr path=/select
>>>>>> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=220
>>>>>> status=0 QTime=74
>>>>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>>>>>> INFO: [] webapp=/solr path=/select
>>>>>> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=220
>>>>>> status=0 QTime=0
>>>>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>>>>>> INFO: [] webapp=/solr path=/select
>>>>>>
>>>>>>
>>>>>> params={fl=id,boost,tstamp,digest&start=0&q=id:[*+TO+*]&wt=javabin&rows
>>>>>> =220&version=2}
>>>>>>
>>>>>>
>>>>>> hits=220 status=0 QTime=9
>>>>>> 2012-05-10 22:36:27,656 WARN  mapred.LocalJobRunner - job_local_0020
>>>>>> java.lang.NullPointerException
>>>>>>
>>>>>>      at org.apache.hadoop.io.Text.encode(Text.java:388)
>>>>>>      at org.apache.hadoop.io.Text.set(Text.java:178)
>>>>>>      at
>>>>>>
>>>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.ne
>>>>>> xt(SolrDeleteDuplicates.java:270)>>>>
>>>>>>      at
>>>>>>
>>>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.ne
>>>>>> xt(SolrDeleteDuplicates.java:241)>>>>
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask
>>>>>> .java:192)>>>>
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:
>>>>>> 176)
>>>>>>
>>>>>>      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
>>>>>>      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>>>>>>      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177
>>>>>> )
>>>>>>
>>>>>>>> I issued the commands
>>>>>>>>
>>>>>>>> bin/nutch crawl urls -dir crawl -depth 3 -topN 5
>>>>>>>>
>>>>>>>> and
>>>>>>>>
>>>>>>>> bin/nutch solrindex http://127.0.0.1:8983/solr/ crawldb -linkdb
>>>>>>>> crawldb/linkdb crawldb/segments/*
>>>>>>>>
>>>>>>>> separately, after which I got no errors. When I browsed to
>>>>>>>> http://localhost:8983/solr/admin and attempted a search, I got the
>>>>>>>> error
>>>>>>>>
>>>>>>>>     HTTP ERROR 400
>>>>>>>>
>>>>>>>> Problem accessing /solr/select. Reason:
>>>>>>>>      undefined field text
>>>>>>> But this is a Solr thing, you have no field named text. Resolve
>>>>>>> this in Solr or on the Solr mailing list.
>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> ---
>>>>>>>>
>>>>>>>>
>>>>>>>> /Powered by Jetty://
>>>>>>>>
>>>>>>>> /What am I doing wrong?
>>>>>>>>
>>>>>>>> Regards,/
>>>>>>>> /
>>>>>> Regards,

Re: HTTP error 400

Posted by Tolga <to...@ozses.net>.

Hi,

I would like to report that the directory schema given in the command 
bin/nutch solrindex http://127.0.0.1:8983/solr/ crawldb -linkdb 
crawldb/linkdb crawldb/segments/* in Nutch FAQ doesn't match previous 
examples.

That said, I'm totally confused. How can I index to solr if I don't crawl?


On 5/15/12 2:05 PM, Markus Jelsma wrote:
> Please follow the step-by-step tutorial, it's explained there:
> http://wiki.apache.org/nutch/NutchTutorial
>
> On Tuesday 15 May 2012 13:40:26 Tolga wrote:
>> I'm a little confused. How can I not use the crawl command and execute
>> the separate crawl cycle commands at the same time?
>>
>> Regards,
>>
>> On 5/11/12 9:40 AM, Markus Jelsma wrote:
>>> Ah, that means don't use the crawl command and do a little shell
>>> scripting to execute the separte crawl cycle commands, see the nutch
>>> wiki for examples. And don't do solrdedup. Search the Solr wiki for
>>> deduplication.
>>>
>>> cheers
>>>
>>> On Fri, 11 May 2012 07:39:36 +0300, Tolga<to...@ozses.net>  wrote:
>>>> Hi,
>>>>
>>>> How do I exactly "omit solrdedup and use Solr's internal
>>>> deduplication" instead.? I don't even know what any of that means :D
>>>> I've just used bin/nutch crawl urls -solr http://localhost:8983/solr/
>>>> -depth 3 -topN 100 to get the error. I have to use all the steps?
>>>>
>>>> Regards,
>>>>
>>>> On 05/10/2012 11:38 PM, Markus Jelsma wrote:
>>>>> thanks
>>>>>
>>>>> This is a known issue:
>>>>> https://issues.apache.org/jira/browse/NUTCH-1100
>>>>>
>>>>> I have not been able find the bug nor do i know how to reproduce it
>>>>> from scratch. If you have a public site with which we can reproduce
>>>>> it please comment to the Jira ticket. Make sure you use either
>>>>> default config or little, a seed URL and the exact crawl&  dedup
>>>>> steps to reproduce.
>>>>>
>>>>> If you find it we might fix it. In any case we need to replace the
>>>>> dedup command with a more scalable tool which it currently is not.
>>>>>
>>>>> In the mean time you can omit solrdedup and use Solr's internal
>>>>> deduplication instead, it works similar and uses the same signature
>>>>> algorithm as Nutch has. Please consult the Solr wiki page on
>>>>> deduplication.
>>>>>
>>>>> Good luck
>>>>>
>>>>> On Thu, 10 May 2012 22:54:37 +0300, Tolga<to...@ozses.net>  wrote:
>>>>>> Hi Markus,
>>>>>>
>>>>>> On 05/10/2012 09:42 AM, Markus Jelsma wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> On Thu, 10 May 2012 09:10:04 +0300, Tolga<to...@ozses.net>  wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> This will sound like a duplicate, but actually it differs from the
>>>>>>>> other one. Please bear with me. Following
>>>>>>>> http://wiki.apache.org/nutch/NutchTutorial, I first issued the
>>>>>>>> command
>>>>>>>>
>>>>>>>> bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3
>>>>>>>> -topN 5
>>>>>>>>
>>>>>>>> Then when I got the message
>>>>>>>>
>>>>>>>> Exception in thread "main" java.io.IOException: Job failed!
>>>>>>>>
>>>>>>>>      at
>>>>>>>>
>>>>>>>> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
>>>>>>>>
>>>>>>>>      at
>>>>>>>>
>>>>>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDu
>>>>>>>> plicates.java:373)>>>>>>
>>>>>>>>      at
>>>>>>>>
>>>>>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDu
>>>>>>>> plicates.java:353)>>>>>>
>>>>>>>>      at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
>>>>>>>>      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>>>>>      at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
>>>>>>> Please include the relevant part of the log. This can be a known
>>>>>>> issue.
>>>>>> This is an excerpt from hadoop.log:
>>>>>>
>>>>>> 2012-05-10 22:26:30,349 INFO  crawl.Crawl - crawl started in:
>>>>>> crawl-20120510222629
>>>>>> 2012-05-10 22:26:30,350 INFO  crawl.Crawl - rootUrlDir = urls
>>>>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - threads = 10
>>>>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - depth = 3
>>>>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl -
>>>>>> solrUrl=http://localhost:8983/solr/
>>>>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - topN = 100
>>>>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: starting at
>>>>>> 2012-05-10 22:26:30
>>>>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: crawlDb:
>>>>>> crawl-20120510222629/crawldb
>>>>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: urlDir: urls
>>>>>> 2012-05-10 22:26:30,809 INFO  crawl.Injector - Injector: Converting
>>>>>> injected urls to crawl db entries.
>>>>>> 2012-05-10 22:26:34,173 INFO  plugin.PluginRepository - Plugins:
>>>>>> looking in: /root/apache-nutch-1.4-bin/runtime/local/plugins
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Plugin
>>>>>> Auto-activation mode: [true]
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered
>>>>>> Plugins:
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     the nutch
>>>>>> core extension points (nutch-extensionpoints)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic URL
>>>>>> Normalizer (urlnormalizer-basic)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Html
>>>>>> Parse Plug-in (parse-html)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic
>>>>>> Indexing Filter (index-basic)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     HTTP
>>>>>> Framework (lib-http)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -
>>>>>> Pass-through URL Normalizer (urlnormalizer-pass)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
>>>>>> Filter (urlfilter-regex)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Http
>>>>>> Protocol Plug-in (protocol-http)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
>>>>>> Normalizer (urlnormalizer-regex)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Tika
>>>>>> Parser Plug-in (parse-tika)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     OPIC
>>>>>> Scoring Plug-in (scoring-opic)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     CyberNeko
>>>>>> HTML Parser (lib-nekohtml)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Anchor
>>>>>> Indexing Filter (index-anchor)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
>>>>>> Filter Framework (lib-regex-filter)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered
>>>>>> Extension-Points:
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch URL
>>>>>> Normalizer (org.apache.nutch.net.URLNormalizer)
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>>>> Protocol (org.apache.nutch.protocol.Protocol)
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>>>> Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch URL
>>>>>> Filter (org.apache.nutch.net.URLFilter)
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>>>> Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     HTML
>>>>>> Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>>>> Content Parser (org.apache.nutch.parse.Parser)
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>>>> Scoring (org.apache.nutch.scoring.ScoringFilter)
>>>>>> 2012-05-10 22:26:35,439 INFO  regex.RegexURLNormalizer - can't find
>>>>>> rules for scope 'inject', using default
>>>>>> 2012-05-10 22:26:36,434 INFO  crawl.Injector - Injector: Merging
>>>>>> injected urls into crawl db.
>>>>>> 2012-05-10 22:26:36,710 WARN  util.NativeCodeLoader - Unable to load
>>>>>> native-hadoop library for your platform... using builtin-java classes
>>>>>> where applicable
>>>>>> 2012-05-10 22:26:37,542 INFO  crawl.Injector - Injector: finished at
>>>>>> 2012-05-10 22:26:37, elapsed: 00:00:06
>>>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: starting
>>>>>> at 2012-05-10 22:26:37
>>>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: Selecting
>>>>>> best-scoring urls due for fetch.
>>>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator:
>>>>>> filtering: true
>>>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator:
>>>>>> normalizing: true
>>>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: topN: 100
>>>>>> 2012-05-10 22:26:37,552 INFO  crawl.Generator - Generator: jobtracker
>>>>>> is 'local', generating exactly one partition.
>>>>>> 2012-05-10 22:26:37,820 INFO  crawl.FetchScheduleFactory - Using
>>>>>> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
>>>>>> 2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule -
>>>>>> defaultInterval=2592000
>>>>>> 2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule -
>>>>>> maxInterval=7776000
>>>>>> 2012-05-10 22:26:37,856 INFO  regex.RegexURLNormalizer - can't find
>>>>>> rules for scope 'partition', using default
>>>>>> ...
>>>>>> ...
>>>>>> INFO: [] webapp=/solr path=/update
>>>>>>
>>>>>>
>>>>>> params={waitSearcher=true&waitFlush=true&wt=javabin&commit=true&version
>>>>>> =2}
>>>>>>
>>>>>>
>>>>>> status=0 QTime=221
>>>>>> 2012-05-10 22:36:26,336 INFO  solr.SolrIndexer - SolrIndexer:
>>>>>> finished at 2012-05-10 22:36:26, elapsed: 00:00:05
>>>>>> 2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates -
>>>>>> SolrDeleteDuplicates: starting at 2012-05-10 22:36:26
>>>>>> 2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates -
>>>>>> SolrDeleteDuplicates: Solr url: http://localhost:8983/solr/
>>>>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>>>>>> INFO: [] webapp=/solr path=/select
>>>>>> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=220
>>>>>> status=0 QTime=74
>>>>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>>>>>> INFO: [] webapp=/solr path=/select
>>>>>> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=220
>>>>>> status=0 QTime=0
>>>>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>>>>>> INFO: [] webapp=/solr path=/select
>>>>>>
>>>>>>
>>>>>> params={fl=id,boost,tstamp,digest&start=0&q=id:[*+TO+*]&wt=javabin&rows
>>>>>> =220&version=2}
>>>>>>
>>>>>>
>>>>>> hits=220 status=0 QTime=9
>>>>>> 2012-05-10 22:36:27,656 WARN  mapred.LocalJobRunner - job_local_0020
>>>>>> java.lang.NullPointerException
>>>>>>
>>>>>>      at org.apache.hadoop.io.Text.encode(Text.java:388)
>>>>>>      at org.apache.hadoop.io.Text.set(Text.java:178)
>>>>>>      at
>>>>>>
>>>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.ne
>>>>>> xt(SolrDeleteDuplicates.java:270)>>>>
>>>>>>      at
>>>>>>
>>>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.ne
>>>>>> xt(SolrDeleteDuplicates.java:241)>>>>
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask
>>>>>> .java:192)>>>>
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:
>>>>>> 176)
>>>>>>
>>>>>>      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
>>>>>>      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>>>>>>      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177
>>>>>> )
>>>>>>
>>>>>>>> I issued the commands
>>>>>>>>
>>>>>>>> bin/nutch crawl urls -dir crawl -depth 3 -topN 5
>>>>>>>>
>>>>>>>> and
>>>>>>>>
>>>>>>>> bin/nutch solrindex http://127.0.0.1:8983/solr/ crawldb -linkdb
>>>>>>>> crawldb/linkdb crawldb/segments/*
>>>>>>>>
>>>>>>>> separately, after which I got no errors. When I browsed to
>>>>>>>> http://localhost:8983/solr/admin and attempted a search, I got the
>>>>>>>> error
>>>>>>>>
>>>>>>>>     HTTP ERROR 400
>>>>>>>>
>>>>>>>> Problem accessing /solr/select. Reason:
>>>>>>>>      undefined field text
>>>>>>> But this is a Solr thing, you have no field named text. Resolve
>>>>>>> this in Solr or on the Solr mailing list.
>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> ---
>>>>>>>>
>>>>>>>>
>>>>>>>> /Powered by Jetty://
>>>>>>>>
>>>>>>>> /What am I doing wrong?
>>>>>>>>
>>>>>>>> Regards,/
>>>>>>>> /
>>>>>> Regards,

Re: HTTP error 400

Posted by m2000hsf <m2...@hotmail.com>.

Ah, that means don't use the crawl command and do a little shell
 scripting to execute the separte crawl cycle commands, see the nutch
 wiki for examples. And don't do solrdedup. Search the Solr wiki for
 deduplication.

 cheers

 On Fri, 11 May 2012 07:39:36 +0300, Tolga <[hidden email]> wrote:

> Hi,
>
> How do I exactly "omit solrdedup and use Solr's internal
> deduplication" instead.? I don't even know what any of that means :D
> I've just used bin/nutch crawl urls -solr http://localhost:8983/solr/
> -depth 3 -topN 100 to get the error. I have to use all the steps?
>
> Regards,
>
> On 05/10/2012 11:38 PM, Markus Jelsma wrote:
>> thanks
>>
>> This is a known issue:
>> https://issues.apache.org/jira/browse/NUTCH-1100
>>
>> I have not been able find the bug nor do i know how to reproduce it
>> from scratch. If you have a public site with which we can reproduce it
>> please comment to the Jira ticket. Make sure you use either default
>> config or little, a seed URL and the exact crawl & dedup steps to
>> reproduce.
>>
>> If you find it we might fix it. In any case we need to replace the
>> dedup command with a more scalable tool which it currently is not.
>>
>> In the mean time you can omit solrdedup and use Solr's internal
>> deduplication instead, it works similar and uses the same signature
>> algorithm as Nutch has. Please consult the Solr wiki page on
>> deduplication.
>>
>> Good luck
>>
>>
>> On Thu, 10 May 2012 22:54:37 +0300, Tolga <[hidden email]> wrote:
>>> Hi Markus,
>>>
>>> On 05/10/2012 09:42 AM, Markus Jelsma wrote:
>>>> Hi,
>>>>
>>>> On Thu, 10 May 2012 09:10:04 +0300, Tolga <[hidden email]> wrote:
>>>>> Hi,
>>>>>
>>>>> This will sound like a duplicate, but actually it differs from
>>>>> the
>>>>> other one. Please bear with me. Following
>>>>> http://wiki.apache.org/nutch/NutchTutorial, I first issued the
>>>>> command
>>>>>
>>>>> bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3
>>>>> -topN 5
>>>>>
>>>>> Then when I got the message
>>>>>
>>>>> Exception in thread "main" java.io.IOException: Job failed!
>>>>>     at
>>>>> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
>>>>>     at
>>>>>
>>>>>
>>>>>
>>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:373)
>>>>>
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>>
>>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:353)
>>>>>
>>>>>
>>>>>     at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
>>>>>     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>>     at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
>>>>
>>>> Please include the relevant part of the log. This can be a known
>>>> issue.
>>>
>>> This is an excerpt from hadoop.log:
>>>
>>> 2012-05-10 22:26:30,349 INFO  crawl.Crawl - crawl started in:
>>> crawl-20120510222629
>>> 2012-05-10 22:26:30,350 INFO  crawl.Crawl - rootUrlDir = urls
>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - threads = 10
>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - depth = 3
>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl -
>>> solrUrl=http://localhost:8983/solr/
>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - topN = 100
>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: starting
>>> at
>>> 2012-05-10 22:26:30
>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: crawlDb:
>>> crawl-20120510222629/crawldb
>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: urlDir:
>>> urls
>>> 2012-05-10 22:26:30,809 INFO  crawl.Injector - Injector: Converting
>>> injected urls to crawl db entries.
>>> 2012-05-10 22:26:34,173 INFO  plugin.PluginRepository - Plugins:
>>> looking in: /root/apache-nutch-1.4-bin/runtime/local/plugins
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Plugin
>>> Auto-activation mode: [true]
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered
>>> Plugins:
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     the
>>> nutch
>>> core extension points (nutch-extensionpoints)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic
>>> URL
>>> Normalizer (urlnormalizer-basic)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Html
>>> Parse Plug-in (parse-html)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic
>>> Indexing Filter (index-basic)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     HTTP
>>> Framework (lib-http)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -
>>> Pass-through URL Normalizer (urlnormalizer-pass)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex
>>> URL
>>> Filter (urlfilter-regex)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Http
>>> Protocol Plug-in (protocol-http)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex
>>> URL
>>> Normalizer (urlnormalizer-regex)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Tika
>>> Parser Plug-in (parse-tika)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     OPIC
>>> Scoring Plug-in (scoring-opic)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -    
>>> CyberNeko
>>> HTML Parser (lib-nekohtml)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Anchor
>>> Indexing Filter (index-anchor)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex
>>> URL
>>> Filter Framework (lib-regex-filter)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered
>>> Extension-Points:
>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>> URL
>>> Normalizer (org.apache.nutch.net.URLNormalizer)
>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>> Protocol (org.apache.nutch.protocol.Protocol)
>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>> Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>> URL
>>> Filter (org.apache.nutch.net.URLFilter)
>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>> Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     HTML
>>> Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>> Content Parser (org.apache.nutch.parse.Parser)
>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>> Scoring (org.apache.nutch.scoring.ScoringFilter)
>>> 2012-05-10 22:26:35,439 INFO  regex.RegexURLNormalizer - can't find
>>> rules for scope 'inject', using default
>>> 2012-05-10 22:26:36,434 INFO  crawl.Injector - Injector: Merging
>>> injected urls into crawl db.
>>> 2012-05-10 22:26:36,710 WARN  util.NativeCodeLoader - Unable to
>>> load
>>> native-hadoop library for your platform... using builtin-java
>>> classes
>>> where applicable
>>> 2012-05-10 22:26:37,542 INFO  crawl.Injector - Injector: finished
>>> at
>>> 2012-05-10 22:26:37, elapsed: 00:00:06
>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: starting
>>> at 2012-05-10 22:26:37
>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator:
>>> Selecting
>>> best-scoring urls due for fetch.
>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator:
>>> filtering: true
>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator:
>>> normalizing: true
>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: topN:
>>> 100
>>> 2012-05-10 22:26:37,552 INFO  crawl.Generator - Generator:
>>> jobtracker
>>> is 'local', generating exactly one partition.
>>> 2012-05-10 22:26:37,820 INFO  crawl.FetchScheduleFactory - Using
>>> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
>>> 2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule -
>>> defaultInterval=2592000
>>> 2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule -
>>> maxInterval=7776000
>>> 2012-05-10 22:26:37,856 INFO  regex.RegexURLNormalizer - can't find
>>> rules for scope 'partition', using default
>>> ...
>>> ...
>>> INFO: [] webapp=/solr path=/update
>>>
>>>
>>> params={waitSearcher=true&waitFlush=true&wt=javabin&commit=true&version=2}
>>>
>>> status=0 QTime=221
>>> 2012-05-10 22:36:26,336 INFO  solr.SolrIndexer - SolrIndexer:
>>> finished at 2012-05-10 22:36:26, elapsed: 00:00:05
>>> 2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates -
>>> SolrDeleteDuplicates: starting at 2012-05-10 22:36:26
>>> 2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates -
>>> SolrDeleteDuplicates: Solr url: http://localhost:8983/solr/
>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>>> INFO: [] webapp=/solr path=/select
>>> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=220
>>> status=0 QTime=74
>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>>> INFO: [] webapp=/solr path=/select
>>> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=220
>>> status=0 QTime=0
>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>>> INFO: [] webapp=/solr path=/select
>>>
>>>
>>> params={fl=id,boost,tstamp,digest&start=0&q=id:[*+TO+*]&wt=javabin&rows=220&version=2}
>>>
>>> hits=220 status=0 QTime=9
>>> 2012-05-10 22:36:27,656 WARN  mapred.LocalJobRunner -
>>> job_local_0020
>>> java.lang.NullPointerException
>>>     at org.apache.hadoop.io.Text.encode(Text.java:388)
>>>     at org.apache.hadoop.io.Text.set(Text.java:178)
>>>     at
>>>
>>>
>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.next(SolrDeleteDuplicates.java:270)
>>>
>>>     at
>>>
>>>
>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.next(SolrDeleteDuplicates.java:241)
>>>
>>>     at
>>>
>>>
>>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
>>>
>>>     at
>>>
>>>
>>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
>>>
>>>     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
>>>     at
>>> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>>>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>     at
>>>
>>>
>>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
>>>>
>>>>>
>>>>> I issued the commands
>>>>>
>>>>> bin/nutch crawl urls -dir crawl -depth 3 -topN 5
>>>>>
>>>>> and
>>>>>
>>>>> bin/nutch solrindex http://127.0.0.1:8983/solr/ crawldb -linkdb
>>>>> crawldb/linkdb crawldb/segments/*
>>>>>
>>>>> separately, after which I got no errors. When I browsed to
>>>>> http://localhost:8983/solr/admin and attempted a search, I got
>>>>> the
>>>>> error
>>>>>
>>>>>
>>>>>    HTTP ERROR 400
>>>>>
>>>>> Problem accessing /solr/select. Reason:
>>>>>
>>>>>     undefined field text
>>>>
>>>> But this is a Solr thing, you have no field named text. Resolve
>>>> this in Solr or on the Solr mailing list.
>>>>
>>>>>
>>>>>   http://www.tradeput.com b2b web 
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> /Powered by Jetty://
>>>>>
>>>>> /What am I doing wrong?
>>>>>
>>>>> Regards,/
>>>>> /
>>>>
>>> Regards,
>>
... [show rest of quote]

--  http://www.tradeput.com *b2b web*  
 Markus Jelsma - CTO - Openindex 

--
View this message in context: http://lucene.472066.n3.nabble.com/HTTP-error-400-tp3976225p3984830.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: HTTP error 400

Posted by Jean-François Gingras <je...@gmail.com>.

Yes. Also take a look at this page
[1]<http://wiki.apache.org/nutch/Whole-Web%20Crawling%20incremental%20script>
for
script exemples.

[1]
http://wiki.apache.org/nutch/Whole-Web%20Crawling%20incremental%20script

On Thu, May 17, 2012 at 6:07 AM, Tolga <to...@ozses.net> wrote:

> I'm still confused. You mean to use http://wiki.apache.org/nutch/**
> NutchTutorial#A3.2_Using_**Individual_Commands_for_Whole-**Web_Crawling<http://wiki.apache.org/nutch/NutchTutorial#A3.2_Using_Individual_Commands_for_Whole-Web_Crawling>?
>
>
> On 5/15/12 2:05 PM, Markus Jelsma wrote:
>
>> Please follow the step-by-step tutorial, it's explained there:
>> http://wiki.apache.org/nutch/**NutchTutorial<http://wiki.apache.org/nutch/NutchTutorial>
>>
>> On Tuesday 15 May 2012 13:40:26 Tolga wrote:
>>
>>> I'm a little confused. How can I not use the crawl command and execute
>>> the separate crawl cycle commands at the same time?
>>>
>>> Regards,
>>>
>>> On 5/11/12 9:40 AM, Markus Jelsma wrote:
>>>
>>>> Ah, that means don't use the crawl command and do a little shell
>>>> scripting to execute the separte crawl cycle commands, see the nutch
>>>> wiki for examples. And don't do solrdedup. Search the Solr wiki for
>>>> deduplication.
>>>>
>>>> cheers
>>>>
>>>> On Fri, 11 May 2012 07:39:36 +0300, Tolga<to...@ozses.net>  wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> How do I exactly "omit solrdedup and use Solr's internal
>>>>> deduplication" instead.? I don't even know what any of that means :D
>>>>> I've just used bin/nutch crawl urls -solr http://localhost:8983/solr/
>>>>> -depth 3 -topN 100 to get the error. I have to use all the steps?
>>>>>
>>>>> Regards,
>>>>>
>>>>> On 05/10/2012 11:38 PM, Markus Jelsma wrote:
>>>>>
>>>>>> thanks
>>>>>>
>>>>>> This is a known issue:
>>>>>> https://issues.apache.org/**jira/browse/NUTCH-1100<https://issues.apache.org/jira/browse/NUTCH-1100>
>>>>>>
>>>>>> I have not been able find the bug nor do i know how to reproduce it
>>>>>> from scratch. If you have a public site with which we can reproduce
>>>>>> it please comment to the Jira ticket. Make sure you use either
>>>>>> default config or little, a seed URL and the exact crawl&  dedup
>>>>>>
>>>>>> steps to reproduce.
>>>>>>
>>>>>> If you find it we might fix it. In any case we need to replace the
>>>>>> dedup command with a more scalable tool which it currently is not.
>>>>>>
>>>>>> In the mean time you can omit solrdedup and use Solr's internal
>>>>>> deduplication instead, it works similar and uses the same signature
>>>>>> algorithm as Nutch has. Please consult the Solr wiki page on
>>>>>> deduplication.
>>>>>>
>>>>>> Good luck
>>>>>>
>>>>>> On Thu, 10 May 2012 22:54:37 +0300, Tolga<to...@ozses.net>  wrote:
>>>>>>
>>>>>>> Hi Markus,
>>>>>>>
>>>>>>> On 05/10/2012 09:42 AM, Markus Jelsma wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On Thu, 10 May 2012 09:10:04 +0300, Tolga<to...@ozses.net>  wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> This will sound like a duplicate, but actually it differs from the
>>>>>>>>> other one. Please bear with me. Following
>>>>>>>>> http://wiki.apache.org/nutch/**NutchTutorial<http://wiki.apache.org/nutch/NutchTutorial>,
>>>>>>>>> I first issued the
>>>>>>>>> command
>>>>>>>>>
>>>>>>>>> bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3
>>>>>>>>> -topN 5
>>>>>>>>>
>>>>>>>>> Then when I got the message
>>>>>>>>>
>>>>>>>>> Exception in thread "main" java.io.IOException: Job failed!
>>>>>>>>>
>>>>>>>>>     at
>>>>>>>>>
>>>>>>>>> org.apache.hadoop.mapred.**JobClient.runJob(JobClient.**java:1252)
>>>>>>>>>
>>>>>>>>>     at
>>>>>>>>>
>>>>>>>>> org.apache.nutch.indexer.solr.**SolrDeleteDuplicates.dedup(**
>>>>>>>>> SolrDeleteDu
>>>>>>>>> plicates.java:373)>>>>>>
>>>>>>>>>     at
>>>>>>>>>
>>>>>>>>> org.apache.nutch.indexer.solr.**SolrDeleteDuplicates.dedup(**
>>>>>>>>> SolrDeleteDu
>>>>>>>>> plicates.java:353)>>>>>>
>>>>>>>>>     at org.apache.nutch.crawl.Crawl.**run(Crawl.java:153)
>>>>>>>>>     at org.apache.hadoop.util.**ToolRunner.run(ToolRunner.**
>>>>>>>>> java:65)
>>>>>>>>>     at org.apache.nutch.crawl.Crawl.**main(Crawl.java:55)
>>>>>>>>>
>>>>>>>> Please include the relevant part of the log. This can be a known
>>>>>>>> issue.
>>>>>>>>
>>>>>>> This is an excerpt from hadoop.log:
>>>>>>>
>>>>>>> 2012-05-10 22:26:30,349 INFO  crawl.Crawl - crawl started in:
>>>>>>> crawl-20120510222629
>>>>>>> 2012-05-10 22:26:30,350 INFO  crawl.Crawl - rootUrlDir = urls
>>>>>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - threads = 10
>>>>>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - depth = 3
>>>>>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl -
>>>>>>> solrUrl=http://localhost:8983/**solr/ <http://localhost:8983/solr/>
>>>>>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - topN = 100
>>>>>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: starting at
>>>>>>> 2012-05-10 22:26:30
>>>>>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: crawlDb:
>>>>>>> crawl-20120510222629/crawldb
>>>>>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: urlDir: urls
>>>>>>> 2012-05-10 22:26:30,809 INFO  crawl.Injector - Injector: Converting
>>>>>>> injected urls to crawl db entries.
>>>>>>> 2012-05-10 22:26:34,173 INFO  plugin.PluginRepository - Plugins:
>>>>>>> looking in: /root/apache-nutch-1.4-bin/**runtime/local/plugins
>>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Plugin
>>>>>>> Auto-activation mode: [true]
>>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered
>>>>>>> Plugins:
>>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     the nutch
>>>>>>> core extension points (nutch-extensionpoints)
>>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic URL
>>>>>>> Normalizer (urlnormalizer-basic)
>>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Html
>>>>>>> Parse Plug-in (parse-html)
>>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic
>>>>>>> Indexing Filter (index-basic)
>>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     HTTP
>>>>>>> Framework (lib-http)
>>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -
>>>>>>> Pass-through URL Normalizer (urlnormalizer-pass)
>>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
>>>>>>> Filter (urlfilter-regex)
>>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Http
>>>>>>> Protocol Plug-in (protocol-http)
>>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
>>>>>>> Normalizer (urlnormalizer-regex)
>>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Tika
>>>>>>> Parser Plug-in (parse-tika)
>>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     OPIC
>>>>>>> Scoring Plug-in (scoring-opic)
>>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     CyberNeko
>>>>>>> HTML Parser (lib-nekohtml)
>>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Anchor
>>>>>>> Indexing Filter (index-anchor)
>>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
>>>>>>> Filter Framework (lib-regex-filter)
>>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered
>>>>>>> Extension-Points:
>>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch URL
>>>>>>> Normalizer (org.apache.nutch.net.**URLNormalizer)
>>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>>>>> Protocol (org.apache.nutch.protocol.**Protocol)
>>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>>>>> Segment Merge Filter (org.apache.nutch.segment.**SegmentMergeFilter)
>>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch URL
>>>>>>> Filter (org.apache.nutch.net.**URLFilter)
>>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>>>>> Indexing Filter (org.apache.nutch.indexer.**IndexingFilter)
>>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     HTML
>>>>>>> Parse Filter (org.apache.nutch.parse.**HtmlParseFilter)
>>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>>>>> Content Parser (org.apache.nutch.parse.**Parser)
>>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>>>>> Scoring (org.apache.nutch.scoring.**ScoringFilter)
>>>>>>> 2012-05-10 22:26:35,439 INFO  regex.RegexURLNormalizer - can't find
>>>>>>> rules for scope 'inject', using default
>>>>>>> 2012-05-10 22:26:36,434 INFO  crawl.Injector - Injector: Merging
>>>>>>> injected urls into crawl db.
>>>>>>> 2012-05-10 22:26:36,710 WARN  util.NativeCodeLoader - Unable to load
>>>>>>> native-hadoop library for your platform... using builtin-java classes
>>>>>>> where applicable
>>>>>>> 2012-05-10 22:26:37,542 INFO  crawl.Injector - Injector: finished at
>>>>>>> 2012-05-10 22:26:37, elapsed: 00:00:06
>>>>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: starting
>>>>>>> at 2012-05-10 22:26:37
>>>>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: Selecting
>>>>>>> best-scoring urls due for fetch.
>>>>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator:
>>>>>>> filtering: true
>>>>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator:
>>>>>>> normalizing: true
>>>>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: topN: 100
>>>>>>> 2012-05-10 22:26:37,552 INFO  crawl.Generator - Generator: jobtracker
>>>>>>> is 'local', generating exactly one partition.
>>>>>>> 2012-05-10 22:26:37,820 INFO  crawl.FetchScheduleFactory - Using
>>>>>>> FetchSchedule impl: org.apache.nutch.crawl.**DefaultFetchSchedule
>>>>>>> 2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule -
>>>>>>> defaultInterval=2592000
>>>>>>> 2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule -
>>>>>>> maxInterval=7776000
>>>>>>> 2012-05-10 22:26:37,856 INFO  regex.RegexURLNormalizer - can't find
>>>>>>> rules for scope 'partition', using default
>>>>>>> ...
>>>>>>> ...
>>>>>>> INFO: [] webapp=/solr path=/update
>>>>>>>
>>>>>>>
>>>>>>> params={waitSearcher=true&**waitFlush=true&wt=javabin&**
>>>>>>> commit=true&version
>>>>>>> =2}
>>>>>>>
>>>>>>>
>>>>>>> status=0 QTime=221
>>>>>>> 2012-05-10 22:36:26,336 INFO  solr.SolrIndexer - SolrIndexer:
>>>>>>> finished at 2012-05-10 22:36:26, elapsed: 00:00:05
>>>>>>> 2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates -
>>>>>>> SolrDeleteDuplicates: starting at 2012-05-10 22:36:26
>>>>>>> 2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates -
>>>>>>> SolrDeleteDuplicates: Solr url: http://localhost:8983/solr/
>>>>>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>>>>>>> INFO: [] webapp=/solr path=/select
>>>>>>> params={fl=id&wt=javabin&q=id:**[*+TO+*]&rows=1&version=2} hits=220
>>>>>>> status=0 QTime=74
>>>>>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>>>>>>> INFO: [] webapp=/solr path=/select
>>>>>>> params={fl=id&wt=javabin&q=id:**[*+TO+*]&rows=1&version=2} hits=220
>>>>>>> status=0 QTime=0
>>>>>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>>>>>>> INFO: [] webapp=/solr path=/select
>>>>>>>
>>>>>>>
>>>>>>> params={fl=id,boost,tstamp,**digest&start=0&q=id:[*+TO+*]&**
>>>>>>> wt=javabin&rows
>>>>>>> =220&version=2}
>>>>>>>
>>>>>>>
>>>>>>> hits=220 status=0 QTime=9
>>>>>>> 2012-05-10 22:36:27,656 WARN  mapred.LocalJobRunner - job_local_0020
>>>>>>> java.lang.NullPointerException
>>>>>>>
>>>>>>>     at org.apache.hadoop.io.Text.**encode(Text.java:388)
>>>>>>>     at org.apache.hadoop.io.Text.set(**Text.java:178)
>>>>>>>     at
>>>>>>>
>>>>>>> org.apache.nutch.indexer.solr.**SolrDeleteDuplicates$**
>>>>>>> SolrInputFormat$1.ne
>>>>>>> xt(SolrDeleteDuplicates.java:**270)>>>>
>>>>>>>     at
>>>>>>>
>>>>>>> org.apache.nutch.indexer.solr.**SolrDeleteDuplicates$**
>>>>>>> SolrInputFormat$1.ne
>>>>>>> xt(SolrDeleteDuplicates.java:**241)>>>>
>>>>>>>     at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.**MapTask$TrackedRecordReader.**
>>>>>>> moveToNext(MapTask
>>>>>>> .java:192)>>>>
>>>>>>>     at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.**MapTask$TrackedRecordReader.**
>>>>>>> next(MapTask.java:
>>>>>>> 176)
>>>>>>>
>>>>>>>     at org.apache.hadoop.mapred.**MapRunner.run(MapRunner.java:**48)
>>>>>>>     at org.apache.hadoop.mapred.**MapTask.runOldMapper(MapTask.**
>>>>>>> java:358)
>>>>>>>     at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:307)
>>>>>>>     at
>>>>>>>
>>>>>>> org.apache.hadoop.mapred.**LocalJobRunner$Job.run(**
>>>>>>> LocalJobRunner.java:177
>>>>>>> )
>>>>>>>
>>>>>>>  I issued the commands
>>>>>>>>>
>>>>>>>>> bin/nutch crawl urls -dir crawl -depth 3 -topN 5
>>>>>>>>>
>>>>>>>>> and
>>>>>>>>>
>>>>>>>>> bin/nutch solrindex http://127.0.0.1:8983/solr/ crawldb -linkdb
>>>>>>>>> crawldb/linkdb crawldb/segments/*
>>>>>>>>>
>>>>>>>>> separately, after which I got no errors. When I browsed to
>>>>>>>>> http://localhost:8983/solr/**admin<http://localhost:8983/solr/admin>and attempted a search, I got the
>>>>>>>>> error
>>>>>>>>>
>>>>>>>>>    HTTP ERROR 400
>>>>>>>>>
>>>>>>>>> Problem accessing /solr/select. Reason:
>>>>>>>>>     undefined field text
>>>>>>>>>
>>>>>>>> But this is a Solr thing, you have no field named text. Resolve
>>>>>>>> this in Solr or on the Solr mailing list.
>>>>>>>>
>>>>>>>>  ------------------------------**------------------------------**
>>>>>>>>> ---------
>>>>>>>>> ---
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> /Powered by Jetty://
>>>>>>>>>
>>>>>>>>> /What am I doing wrong?
>>>>>>>>>
>>>>>>>>> Regards,/
>>>>>>>>> /
>>>>>>>>>
>>>>>>>> Regards,
>>>>>>>
>>>>>>


-- 
Jean-François Gingras

Re: HTTP error 400

Posted by Tolga <to...@ozses.net>.

I'm still confused. You mean to use 
http://wiki.apache.org/nutch/NutchTutorial#A3.2_Using_Individual_Commands_for_Whole-Web_Crawling 
?

On 5/15/12 2:05 PM, Markus Jelsma wrote:
> Please follow the step-by-step tutorial, it's explained there:
> http://wiki.apache.org/nutch/NutchTutorial
>
> On Tuesday 15 May 2012 13:40:26 Tolga wrote:
>> I'm a little confused. How can I not use the crawl command and execute
>> the separate crawl cycle commands at the same time?
>>
>> Regards,
>>
>> On 5/11/12 9:40 AM, Markus Jelsma wrote:
>>> Ah, that means don't use the crawl command and do a little shell
>>> scripting to execute the separte crawl cycle commands, see the nutch
>>> wiki for examples. And don't do solrdedup. Search the Solr wiki for
>>> deduplication.
>>>
>>> cheers
>>>
>>> On Fri, 11 May 2012 07:39:36 +0300, Tolga<to...@ozses.net>  wrote:
>>>> Hi,
>>>>
>>>> How do I exactly "omit solrdedup and use Solr's internal
>>>> deduplication" instead.? I don't even know what any of that means :D
>>>> I've just used bin/nutch crawl urls -solr http://localhost:8983/solr/
>>>> -depth 3 -topN 100 to get the error. I have to use all the steps?
>>>>
>>>> Regards,
>>>>
>>>> On 05/10/2012 11:38 PM, Markus Jelsma wrote:
>>>>> thanks
>>>>>
>>>>> This is a known issue:
>>>>> https://issues.apache.org/jira/browse/NUTCH-1100
>>>>>
>>>>> I have not been able find the bug nor do i know how to reproduce it
>>>>> from scratch. If you have a public site with which we can reproduce
>>>>> it please comment to the Jira ticket. Make sure you use either
>>>>> default config or little, a seed URL and the exact crawl&  dedup
>>>>> steps to reproduce.
>>>>>
>>>>> If you find it we might fix it. In any case we need to replace the
>>>>> dedup command with a more scalable tool which it currently is not.
>>>>>
>>>>> In the mean time you can omit solrdedup and use Solr's internal
>>>>> deduplication instead, it works similar and uses the same signature
>>>>> algorithm as Nutch has. Please consult the Solr wiki page on
>>>>> deduplication.
>>>>>
>>>>> Good luck
>>>>>
>>>>> On Thu, 10 May 2012 22:54:37 +0300, Tolga<to...@ozses.net>  wrote:
>>>>>> Hi Markus,
>>>>>>
>>>>>> On 05/10/2012 09:42 AM, Markus Jelsma wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> On Thu, 10 May 2012 09:10:04 +0300, Tolga<to...@ozses.net>  wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> This will sound like a duplicate, but actually it differs from the
>>>>>>>> other one. Please bear with me. Following
>>>>>>>> http://wiki.apache.org/nutch/NutchTutorial, I first issued the
>>>>>>>> command
>>>>>>>>
>>>>>>>> bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3
>>>>>>>> -topN 5
>>>>>>>>
>>>>>>>> Then when I got the message
>>>>>>>>
>>>>>>>> Exception in thread "main" java.io.IOException: Job failed!
>>>>>>>>
>>>>>>>>      at
>>>>>>>>
>>>>>>>> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
>>>>>>>>
>>>>>>>>      at
>>>>>>>>
>>>>>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDu
>>>>>>>> plicates.java:373)>>>>>>
>>>>>>>>      at
>>>>>>>>
>>>>>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDu
>>>>>>>> plicates.java:353)>>>>>>
>>>>>>>>      at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
>>>>>>>>      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>>>>>      at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
>>>>>>> Please include the relevant part of the log. This can be a known
>>>>>>> issue.
>>>>>> This is an excerpt from hadoop.log:
>>>>>>
>>>>>> 2012-05-10 22:26:30,349 INFO  crawl.Crawl - crawl started in:
>>>>>> crawl-20120510222629
>>>>>> 2012-05-10 22:26:30,350 INFO  crawl.Crawl - rootUrlDir = urls
>>>>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - threads = 10
>>>>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - depth = 3
>>>>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl -
>>>>>> solrUrl=http://localhost:8983/solr/
>>>>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - topN = 100
>>>>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: starting at
>>>>>> 2012-05-10 22:26:30
>>>>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: crawlDb:
>>>>>> crawl-20120510222629/crawldb
>>>>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: urlDir: urls
>>>>>> 2012-05-10 22:26:30,809 INFO  crawl.Injector - Injector: Converting
>>>>>> injected urls to crawl db entries.
>>>>>> 2012-05-10 22:26:34,173 INFO  plugin.PluginRepository - Plugins:
>>>>>> looking in: /root/apache-nutch-1.4-bin/runtime/local/plugins
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Plugin
>>>>>> Auto-activation mode: [true]
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered
>>>>>> Plugins:
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     the nutch
>>>>>> core extension points (nutch-extensionpoints)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic URL
>>>>>> Normalizer (urlnormalizer-basic)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Html
>>>>>> Parse Plug-in (parse-html)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic
>>>>>> Indexing Filter (index-basic)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     HTTP
>>>>>> Framework (lib-http)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -
>>>>>> Pass-through URL Normalizer (urlnormalizer-pass)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
>>>>>> Filter (urlfilter-regex)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Http
>>>>>> Protocol Plug-in (protocol-http)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
>>>>>> Normalizer (urlnormalizer-regex)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Tika
>>>>>> Parser Plug-in (parse-tika)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     OPIC
>>>>>> Scoring Plug-in (scoring-opic)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     CyberNeko
>>>>>> HTML Parser (lib-nekohtml)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Anchor
>>>>>> Indexing Filter (index-anchor)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
>>>>>> Filter Framework (lib-regex-filter)
>>>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered
>>>>>> Extension-Points:
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch URL
>>>>>> Normalizer (org.apache.nutch.net.URLNormalizer)
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>>>> Protocol (org.apache.nutch.protocol.Protocol)
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>>>> Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch URL
>>>>>> Filter (org.apache.nutch.net.URLFilter)
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>>>> Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     HTML
>>>>>> Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>>>> Content Parser (org.apache.nutch.parse.Parser)
>>>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>>>> Scoring (org.apache.nutch.scoring.ScoringFilter)
>>>>>> 2012-05-10 22:26:35,439 INFO  regex.RegexURLNormalizer - can't find
>>>>>> rules for scope 'inject', using default
>>>>>> 2012-05-10 22:26:36,434 INFO  crawl.Injector - Injector: Merging
>>>>>> injected urls into crawl db.
>>>>>> 2012-05-10 22:26:36,710 WARN  util.NativeCodeLoader - Unable to load
>>>>>> native-hadoop library for your platform... using builtin-java classes
>>>>>> where applicable
>>>>>> 2012-05-10 22:26:37,542 INFO  crawl.Injector - Injector: finished at
>>>>>> 2012-05-10 22:26:37, elapsed: 00:00:06
>>>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: starting
>>>>>> at 2012-05-10 22:26:37
>>>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: Selecting
>>>>>> best-scoring urls due for fetch.
>>>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator:
>>>>>> filtering: true
>>>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator:
>>>>>> normalizing: true
>>>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: topN: 100
>>>>>> 2012-05-10 22:26:37,552 INFO  crawl.Generator - Generator: jobtracker
>>>>>> is 'local', generating exactly one partition.
>>>>>> 2012-05-10 22:26:37,820 INFO  crawl.FetchScheduleFactory - Using
>>>>>> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
>>>>>> 2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule -
>>>>>> defaultInterval=2592000
>>>>>> 2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule -
>>>>>> maxInterval=7776000
>>>>>> 2012-05-10 22:26:37,856 INFO  regex.RegexURLNormalizer - can't find
>>>>>> rules for scope 'partition', using default
>>>>>> ...
>>>>>> ...
>>>>>> INFO: [] webapp=/solr path=/update
>>>>>>
>>>>>>
>>>>>> params={waitSearcher=true&waitFlush=true&wt=javabin&commit=true&version
>>>>>> =2}
>>>>>>
>>>>>>
>>>>>> status=0 QTime=221
>>>>>> 2012-05-10 22:36:26,336 INFO  solr.SolrIndexer - SolrIndexer:
>>>>>> finished at 2012-05-10 22:36:26, elapsed: 00:00:05
>>>>>> 2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates -
>>>>>> SolrDeleteDuplicates: starting at 2012-05-10 22:36:26
>>>>>> 2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates -
>>>>>> SolrDeleteDuplicates: Solr url: http://localhost:8983/solr/
>>>>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>>>>>> INFO: [] webapp=/solr path=/select
>>>>>> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=220
>>>>>> status=0 QTime=74
>>>>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>>>>>> INFO: [] webapp=/solr path=/select
>>>>>> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=220
>>>>>> status=0 QTime=0
>>>>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>>>>>> INFO: [] webapp=/solr path=/select
>>>>>>
>>>>>>
>>>>>> params={fl=id,boost,tstamp,digest&start=0&q=id:[*+TO+*]&wt=javabin&rows
>>>>>> =220&version=2}
>>>>>>
>>>>>>
>>>>>> hits=220 status=0 QTime=9
>>>>>> 2012-05-10 22:36:27,656 WARN  mapred.LocalJobRunner - job_local_0020
>>>>>> java.lang.NullPointerException
>>>>>>
>>>>>>      at org.apache.hadoop.io.Text.encode(Text.java:388)
>>>>>>      at org.apache.hadoop.io.Text.set(Text.java:178)
>>>>>>      at
>>>>>>
>>>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.ne
>>>>>> xt(SolrDeleteDuplicates.java:270)>>>>
>>>>>>      at
>>>>>>
>>>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.ne
>>>>>> xt(SolrDeleteDuplicates.java:241)>>>>
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask
>>>>>> .java:192)>>>>
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:
>>>>>> 176)
>>>>>>
>>>>>>      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
>>>>>>      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>>>>>>      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177
>>>>>> )
>>>>>>
>>>>>>>> I issued the commands
>>>>>>>>
>>>>>>>> bin/nutch crawl urls -dir crawl -depth 3 -topN 5
>>>>>>>>
>>>>>>>> and
>>>>>>>>
>>>>>>>> bin/nutch solrindex http://127.0.0.1:8983/solr/ crawldb -linkdb
>>>>>>>> crawldb/linkdb crawldb/segments/*
>>>>>>>>
>>>>>>>> separately, after which I got no errors. When I browsed to
>>>>>>>> http://localhost:8983/solr/admin and attempted a search, I got the
>>>>>>>> error
>>>>>>>>
>>>>>>>>     HTTP ERROR 400
>>>>>>>>
>>>>>>>> Problem accessing /solr/select. Reason:
>>>>>>>>      undefined field text
>>>>>>> But this is a Solr thing, you have no field named text. Resolve
>>>>>>> this in Solr or on the Solr mailing list.
>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> ---
>>>>>>>>
>>>>>>>>
>>>>>>>> /Powered by Jetty://
>>>>>>>>
>>>>>>>> /What am I doing wrong?
>>>>>>>>
>>>>>>>> Regards,/
>>>>>>>> /
>>>>>> Regards,

Re: HTTP error 400

Posted by Markus Jelsma <ma...@openindex.io>.

Please follow the step-by-step tutorial, it's explained there:
http://wiki.apache.org/nutch/NutchTutorial

On Tuesday 15 May 2012 13:40:26 Tolga wrote:
> I'm a little confused. How can I not use the crawl command and execute
> the separate crawl cycle commands at the same time?
> 
> Regards,
> 
> On 5/11/12 9:40 AM, Markus Jelsma wrote:
> > Ah, that means don't use the crawl command and do a little shell
> > scripting to execute the separte crawl cycle commands, see the nutch
> > wiki for examples. And don't do solrdedup. Search the Solr wiki for
> > deduplication.
> > 
> > cheers
> > 
> > On Fri, 11 May 2012 07:39:36 +0300, Tolga <to...@ozses.net> wrote:
> >> Hi,
> >> 
> >> How do I exactly "omit solrdedup and use Solr's internal
> >> deduplication" instead.? I don't even know what any of that means :D
> >> I've just used bin/nutch crawl urls -solr http://localhost:8983/solr/
> >> -depth 3 -topN 100 to get the error. I have to use all the steps?
> >> 
> >> Regards,
> >> 
> >> On 05/10/2012 11:38 PM, Markus Jelsma wrote:
> >>> thanks
> >>> 
> >>> This is a known issue:
> >>> https://issues.apache.org/jira/browse/NUTCH-1100
> >>> 
> >>> I have not been able find the bug nor do i know how to reproduce it
> >>> from scratch. If you have a public site with which we can reproduce
> >>> it please comment to the Jira ticket. Make sure you use either
> >>> default config or little, a seed URL and the exact crawl & dedup
> >>> steps to reproduce.
> >>> 
> >>> If you find it we might fix it. In any case we need to replace the
> >>> dedup command with a more scalable tool which it currently is not.
> >>> 
> >>> In the mean time you can omit solrdedup and use Solr's internal
> >>> deduplication instead, it works similar and uses the same signature
> >>> algorithm as Nutch has. Please consult the Solr wiki page on
> >>> deduplication.
> >>> 
> >>> Good luck
> >>> 
> >>> On Thu, 10 May 2012 22:54:37 +0300, Tolga <to...@ozses.net> wrote:
> >>>> Hi Markus,
> >>>> 
> >>>> On 05/10/2012 09:42 AM, Markus Jelsma wrote:
> >>>>> Hi,
> >>>>> 
> >>>>> On Thu, 10 May 2012 09:10:04 +0300, Tolga <to...@ozses.net> wrote:
> >>>>>> Hi,
> >>>>>> 
> >>>>>> This will sound like a duplicate, but actually it differs from the
> >>>>>> other one. Please bear with me. Following
> >>>>>> http://wiki.apache.org/nutch/NutchTutorial, I first issued the
> >>>>>> command
> >>>>>> 
> >>>>>> bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3
> >>>>>> -topN 5
> >>>>>> 
> >>>>>> Then when I got the message
> >>>>>> 
> >>>>>> Exception in thread "main" java.io.IOException: Job failed!
> >>>>>> 
> >>>>>>     at
> >>>>>> 
> >>>>>> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> >>>>>> 
> >>>>>>     at
> >>>>>> 
> >>>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDu
> >>>>>> plicates.java:373)>>>>>> 
> >>>>>>     at
> >>>>>> 
> >>>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDu
> >>>>>> plicates.java:353)>>>>>> 
> >>>>>>     at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
> >>>>>>     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >>>>>>     at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
> >>>>> 
> >>>>> Please include the relevant part of the log. This can be a known
> >>>>> issue.
> >>>> 
> >>>> This is an excerpt from hadoop.log:
> >>>> 
> >>>> 2012-05-10 22:26:30,349 INFO  crawl.Crawl - crawl started in:
> >>>> crawl-20120510222629
> >>>> 2012-05-10 22:26:30,350 INFO  crawl.Crawl - rootUrlDir = urls
> >>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - threads = 10
> >>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - depth = 3
> >>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl -
> >>>> solrUrl=http://localhost:8983/solr/
> >>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - topN = 100
> >>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: starting at
> >>>> 2012-05-10 22:26:30
> >>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: crawlDb:
> >>>> crawl-20120510222629/crawldb
> >>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: urlDir: urls
> >>>> 2012-05-10 22:26:30,809 INFO  crawl.Injector - Injector: Converting
> >>>> injected urls to crawl db entries.
> >>>> 2012-05-10 22:26:34,173 INFO  plugin.PluginRepository - Plugins:
> >>>> looking in: /root/apache-nutch-1.4-bin/runtime/local/plugins
> >>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Plugin
> >>>> Auto-activation mode: [true]
> >>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered
> >>>> Plugins:
> >>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     the nutch
> >>>> core extension points (nutch-extensionpoints)
> >>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic URL
> >>>> Normalizer (urlnormalizer-basic)
> >>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Html
> >>>> Parse Plug-in (parse-html)
> >>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic
> >>>> Indexing Filter (index-basic)
> >>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     HTTP
> >>>> Framework (lib-http)
> >>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -
> >>>> Pass-through URL Normalizer (urlnormalizer-pass)
> >>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
> >>>> Filter (urlfilter-regex)
> >>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Http
> >>>> Protocol Plug-in (protocol-http)
> >>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
> >>>> Normalizer (urlnormalizer-regex)
> >>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Tika
> >>>> Parser Plug-in (parse-tika)
> >>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     OPIC
> >>>> Scoring Plug-in (scoring-opic)
> >>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     CyberNeko
> >>>> HTML Parser (lib-nekohtml)
> >>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Anchor
> >>>> Indexing Filter (index-anchor)
> >>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
> >>>> Filter Framework (lib-regex-filter)
> >>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered
> >>>> Extension-Points:
> >>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch URL
> >>>> Normalizer (org.apache.nutch.net.URLNormalizer)
> >>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
> >>>> Protocol (org.apache.nutch.protocol.Protocol)
> >>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
> >>>> Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
> >>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch URL
> >>>> Filter (org.apache.nutch.net.URLFilter)
> >>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
> >>>> Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
> >>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     HTML
> >>>> Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
> >>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
> >>>> Content Parser (org.apache.nutch.parse.Parser)
> >>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
> >>>> Scoring (org.apache.nutch.scoring.ScoringFilter)
> >>>> 2012-05-10 22:26:35,439 INFO  regex.RegexURLNormalizer - can't find
> >>>> rules for scope 'inject', using default
> >>>> 2012-05-10 22:26:36,434 INFO  crawl.Injector - Injector: Merging
> >>>> injected urls into crawl db.
> >>>> 2012-05-10 22:26:36,710 WARN  util.NativeCodeLoader - Unable to load
> >>>> native-hadoop library for your platform... using builtin-java classes
> >>>> where applicable
> >>>> 2012-05-10 22:26:37,542 INFO  crawl.Injector - Injector: finished at
> >>>> 2012-05-10 22:26:37, elapsed: 00:00:06
> >>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: starting
> >>>> at 2012-05-10 22:26:37
> >>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: Selecting
> >>>> best-scoring urls due for fetch.
> >>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator:
> >>>> filtering: true
> >>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator:
> >>>> normalizing: true
> >>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: topN: 100
> >>>> 2012-05-10 22:26:37,552 INFO  crawl.Generator - Generator: jobtracker
> >>>> is 'local', generating exactly one partition.
> >>>> 2012-05-10 22:26:37,820 INFO  crawl.FetchScheduleFactory - Using
> >>>> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
> >>>> 2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule -
> >>>> defaultInterval=2592000
> >>>> 2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule -
> >>>> maxInterval=7776000
> >>>> 2012-05-10 22:26:37,856 INFO  regex.RegexURLNormalizer - can't find
> >>>> rules for scope 'partition', using default
> >>>> ...
> >>>> ...
> >>>> INFO: [] webapp=/solr path=/update
> >>>> 
> >>>> 
> >>>> params={waitSearcher=true&waitFlush=true&wt=javabin&commit=true&version
> >>>> =2}
> >>>> 
> >>>> 
> >>>> status=0 QTime=221
> >>>> 2012-05-10 22:36:26,336 INFO  solr.SolrIndexer - SolrIndexer:
> >>>> finished at 2012-05-10 22:36:26, elapsed: 00:00:05
> >>>> 2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates -
> >>>> SolrDeleteDuplicates: starting at 2012-05-10 22:36:26
> >>>> 2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates -
> >>>> SolrDeleteDuplicates: Solr url: http://localhost:8983/solr/
> >>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
> >>>> INFO: [] webapp=/solr path=/select
> >>>> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=220
> >>>> status=0 QTime=74
> >>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
> >>>> INFO: [] webapp=/solr path=/select
> >>>> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=220
> >>>> status=0 QTime=0
> >>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
> >>>> INFO: [] webapp=/solr path=/select
> >>>> 
> >>>> 
> >>>> params={fl=id,boost,tstamp,digest&start=0&q=id:[*+TO+*]&wt=javabin&rows
> >>>> =220&version=2}
> >>>> 
> >>>> 
> >>>> hits=220 status=0 QTime=9
> >>>> 2012-05-10 22:36:27,656 WARN  mapred.LocalJobRunner - job_local_0020
> >>>> java.lang.NullPointerException
> >>>> 
> >>>>     at org.apache.hadoop.io.Text.encode(Text.java:388)
> >>>>     at org.apache.hadoop.io.Text.set(Text.java:178)
> >>>>     at
> >>>> 
> >>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.ne
> >>>> xt(SolrDeleteDuplicates.java:270)>>>> 
> >>>>     at
> >>>> 
> >>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.ne
> >>>> xt(SolrDeleteDuplicates.java:241)>>>> 
> >>>>     at
> >>>> 
> >>>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask
> >>>> .java:192)>>>> 
> >>>>     at
> >>>> 
> >>>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:
> >>>> 176)
> >>>> 
> >>>>     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> >>>>     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> >>>>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> >>>>     at
> >>>> 
> >>>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177
> >>>> )
> >>>> 
> >>>>>> I issued the commands
> >>>>>> 
> >>>>>> bin/nutch crawl urls -dir crawl -depth 3 -topN 5
> >>>>>> 
> >>>>>> and
> >>>>>> 
> >>>>>> bin/nutch solrindex http://127.0.0.1:8983/solr/ crawldb -linkdb
> >>>>>> crawldb/linkdb crawldb/segments/*
> >>>>>> 
> >>>>>> separately, after which I got no errors. When I browsed to
> >>>>>> http://localhost:8983/solr/admin and attempted a search, I got the
> >>>>>> error
> >>>>>> 
> >>>>>>    HTTP ERROR 400
> >>>>>> 
> >>>>>> Problem accessing /solr/select. Reason:
> >>>>>>     undefined field text
> >>>>> 
> >>>>> But this is a Solr thing, you have no field named text. Resolve
> >>>>> this in Solr or on the Solr mailing list.
> >>>>> 
> >>>>>> ---------------------------------------------------------------------
> >>>>>> ---
> >>>>>> 
> >>>>>> 
> >>>>>> /Powered by Jetty://
> >>>>>> 
> >>>>>> /What am I doing wrong?
> >>>>>> 
> >>>>>> Regards,/
> >>>>>> /
> >>>> 
> >>>> Regards,
-- 
Markus Jelsma - CTO - Openindex

Re: HTTP error 400

Posted by Tolga <to...@ozses.net>.

I'm a little confused. How can I not use the crawl command and execute 
the separate crawl cycle commands at the same time?

Regards,

On 5/11/12 9:40 AM, Markus Jelsma wrote:
> Ah, that means don't use the crawl command and do a little shell 
> scripting to execute the separte crawl cycle commands, see the nutch 
> wiki for examples. And don't do solrdedup. Search the Solr wiki for 
> deduplication.
>
> cheers
>
> On Fri, 11 May 2012 07:39:36 +0300, Tolga <to...@ozses.net> wrote:
>> Hi,
>>
>> How do I exactly "omit solrdedup and use Solr's internal
>> deduplication" instead.? I don't even know what any of that means :D
>> I've just used bin/nutch crawl urls -solr http://localhost:8983/solr/
>> -depth 3 -topN 100 to get the error. I have to use all the steps?
>>
>> Regards,
>>
>> On 05/10/2012 11:38 PM, Markus Jelsma wrote:
>>> thanks
>>>
>>> This is a known issue:
>>> https://issues.apache.org/jira/browse/NUTCH-1100
>>>
>>> I have not been able find the bug nor do i know how to reproduce it 
>>> from scratch. If you have a public site with which we can reproduce 
>>> it please comment to the Jira ticket. Make sure you use either 
>>> default config or little, a seed URL and the exact crawl & dedup 
>>> steps to reproduce.
>>>
>>> If you find it we might fix it. In any case we need to replace the 
>>> dedup command with a more scalable tool which it currently is not.
>>>
>>> In the mean time you can omit solrdedup and use Solr's internal 
>>> deduplication instead, it works similar and uses the same signature 
>>> algorithm as Nutch has. Please consult the Solr wiki page on 
>>> deduplication.
>>>
>>> Good luck
>>>
>>>
>>> On Thu, 10 May 2012 22:54:37 +0300, Tolga <to...@ozses.net> wrote:
>>>> Hi Markus,
>>>>
>>>> On 05/10/2012 09:42 AM, Markus Jelsma wrote:
>>>>> Hi,
>>>>>
>>>>> On Thu, 10 May 2012 09:10:04 +0300, Tolga <to...@ozses.net> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> This will sound like a duplicate, but actually it differs from the
>>>>>> other one. Please bear with me. Following
>>>>>> http://wiki.apache.org/nutch/NutchTutorial, I first issued the 
>>>>>> command
>>>>>>
>>>>>> bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3 
>>>>>> -topN 5
>>>>>>
>>>>>> Then when I got the message
>>>>>>
>>>>>> Exception in thread "main" java.io.IOException: Job failed!
>>>>>>     at 
>>>>>> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
>>>>>>     at
>>>>>>
>>>>>>
>>>>>>
>>>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:373) 
>>>>>>
>>>>>>
>>>>>>
>>>>>>     at
>>>>>>
>>>>>>
>>>>>>
>>>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:353) 
>>>>>>
>>>>>>
>>>>>>
>>>>>>     at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
>>>>>>     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>>>     at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
>>>>>
>>>>> Please include the relevant part of the log. This can be a known 
>>>>> issue.
>>>>
>>>> This is an excerpt from hadoop.log:
>>>>
>>>> 2012-05-10 22:26:30,349 INFO  crawl.Crawl - crawl started in:
>>>> crawl-20120510222629
>>>> 2012-05-10 22:26:30,350 INFO  crawl.Crawl - rootUrlDir = urls
>>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - threads = 10
>>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - depth = 3
>>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl -
>>>> solrUrl=http://localhost:8983/solr/
>>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - topN = 100
>>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: starting at
>>>> 2012-05-10 22:26:30
>>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: crawlDb:
>>>> crawl-20120510222629/crawldb
>>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: urlDir: urls
>>>> 2012-05-10 22:26:30,809 INFO  crawl.Injector - Injector: Converting
>>>> injected urls to crawl db entries.
>>>> 2012-05-10 22:26:34,173 INFO  plugin.PluginRepository - Plugins:
>>>> looking in: /root/apache-nutch-1.4-bin/runtime/local/plugins
>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Plugin
>>>> Auto-activation mode: [true]
>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered 
>>>> Plugins:
>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     the nutch
>>>> core extension points (nutch-extensionpoints)
>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic URL
>>>> Normalizer (urlnormalizer-basic)
>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Html
>>>> Parse Plug-in (parse-html)
>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic
>>>> Indexing Filter (index-basic)
>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     HTTP
>>>> Framework (lib-http)
>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -
>>>> Pass-through URL Normalizer (urlnormalizer-pass)
>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
>>>> Filter (urlfilter-regex)
>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Http
>>>> Protocol Plug-in (protocol-http)
>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
>>>> Normalizer (urlnormalizer-regex)
>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Tika
>>>> Parser Plug-in (parse-tika)
>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     OPIC
>>>> Scoring Plug-in (scoring-opic)
>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     CyberNeko
>>>> HTML Parser (lib-nekohtml)
>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Anchor
>>>> Indexing Filter (index-anchor)
>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
>>>> Filter Framework (lib-regex-filter)
>>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered
>>>> Extension-Points:
>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch URL
>>>> Normalizer (org.apache.nutch.net.URLNormalizer)
>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>> Protocol (org.apache.nutch.protocol.Protocol)
>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>> Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch URL
>>>> Filter (org.apache.nutch.net.URLFilter)
>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>> Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     HTML
>>>> Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>> Content Parser (org.apache.nutch.parse.Parser)
>>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>>> Scoring (org.apache.nutch.scoring.ScoringFilter)
>>>> 2012-05-10 22:26:35,439 INFO  regex.RegexURLNormalizer - can't find
>>>> rules for scope 'inject', using default
>>>> 2012-05-10 22:26:36,434 INFO  crawl.Injector - Injector: Merging
>>>> injected urls into crawl db.
>>>> 2012-05-10 22:26:36,710 WARN  util.NativeCodeLoader - Unable to load
>>>> native-hadoop library for your platform... using builtin-java classes
>>>> where applicable
>>>> 2012-05-10 22:26:37,542 INFO  crawl.Injector - Injector: finished at
>>>> 2012-05-10 22:26:37, elapsed: 00:00:06
>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: starting
>>>> at 2012-05-10 22:26:37
>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: Selecting
>>>> best-scoring urls due for fetch.
>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: 
>>>> filtering: true
>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: 
>>>> normalizing: true
>>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: topN: 100
>>>> 2012-05-10 22:26:37,552 INFO  crawl.Generator - Generator: jobtracker
>>>> is 'local', generating exactly one partition.
>>>> 2012-05-10 22:26:37,820 INFO  crawl.FetchScheduleFactory - Using
>>>> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
>>>> 2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule -
>>>> defaultInterval=2592000
>>>> 2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule -
>>>> maxInterval=7776000
>>>> 2012-05-10 22:26:37,856 INFO  regex.RegexURLNormalizer - can't find
>>>> rules for scope 'partition', using default
>>>> ...
>>>> ...
>>>> INFO: [] webapp=/solr path=/update
>>>>
>>>>
>>>> params={waitSearcher=true&waitFlush=true&wt=javabin&commit=true&version=2} 
>>>>
>>>>
>>>> status=0 QTime=221
>>>> 2012-05-10 22:36:26,336 INFO  solr.SolrIndexer - SolrIndexer:
>>>> finished at 2012-05-10 22:36:26, elapsed: 00:00:05
>>>> 2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates -
>>>> SolrDeleteDuplicates: starting at 2012-05-10 22:36:26
>>>> 2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates -
>>>> SolrDeleteDuplicates: Solr url: http://localhost:8983/solr/
>>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>>>> INFO: [] webapp=/solr path=/select
>>>> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=220
>>>> status=0 QTime=74
>>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>>>> INFO: [] webapp=/solr path=/select
>>>> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=220
>>>> status=0 QTime=0
>>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>>>> INFO: [] webapp=/solr path=/select
>>>>
>>>>
>>>> params={fl=id,boost,tstamp,digest&start=0&q=id:[*+TO+*]&wt=javabin&rows=220&version=2} 
>>>>
>>>>
>>>> hits=220 status=0 QTime=9
>>>> 2012-05-10 22:36:27,656 WARN  mapred.LocalJobRunner - job_local_0020
>>>> java.lang.NullPointerException
>>>>     at org.apache.hadoop.io.Text.encode(Text.java:388)
>>>>     at org.apache.hadoop.io.Text.set(Text.java:178)
>>>>     at
>>>>
>>>>
>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.next(SolrDeleteDuplicates.java:270) 
>>>>
>>>>
>>>>     at
>>>>
>>>>
>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.next(SolrDeleteDuplicates.java:241) 
>>>>
>>>>
>>>>     at
>>>>
>>>>
>>>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192) 
>>>>
>>>>
>>>>     at
>>>>
>>>>
>>>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176) 
>>>>
>>>>
>>>>     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
>>>>     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>>>>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>>     at
>>>>
>>>>
>>>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) 
>>>>
>>>>>
>>>>>>
>>>>>> I issued the commands
>>>>>>
>>>>>> bin/nutch crawl urls -dir crawl -depth 3 -topN 5
>>>>>>
>>>>>> and
>>>>>>
>>>>>> bin/nutch solrindex http://127.0.0.1:8983/solr/ crawldb -linkdb
>>>>>> crawldb/linkdb crawldb/segments/*
>>>>>>
>>>>>> separately, after which I got no errors. When I browsed to
>>>>>> http://localhost:8983/solr/admin and attempted a search, I got the
>>>>>> error
>>>>>>
>>>>>>
>>>>>>    HTTP ERROR 400
>>>>>>
>>>>>> Problem accessing /solr/select. Reason:
>>>>>>
>>>>>>     undefined field text
>>>>>
>>>>> But this is a Solr thing, you have no field named text. Resolve 
>>>>> this in Solr or on the Solr mailing list.
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------ 
>>>>>>
>>>>>>
>>>>>> /Powered by Jetty://
>>>>>>
>>>>>> /What am I doing wrong?
>>>>>>
>>>>>> Regards,/
>>>>>> /
>>>>>
>>>> Regards,
>>>
>

Re: HTTP error 400

Posted by Markus Jelsma <ma...@openindex.io>.

 Ah, that means don't use the crawl command and do a little shell 
 scripting to execute the separte crawl cycle commands, see the nutch 
 wiki for examples. And don't do solrdedup. Search the Solr wiki for 
 deduplication.

 cheers

 On Fri, 11 May 2012 07:39:36 +0300, Tolga <to...@ozses.net> wrote:
> Hi,
>
> How do I exactly "omit solrdedup and use Solr's internal
> deduplication" instead.? I don't even know what any of that means :D
> I've just used bin/nutch crawl urls -solr http://localhost:8983/solr/
> -depth 3 -topN 100 to get the error. I have to use all the steps?
>
> Regards,
>
> On 05/10/2012 11:38 PM, Markus Jelsma wrote:
>> thanks
>>
>> This is a known issue:
>> https://issues.apache.org/jira/browse/NUTCH-1100
>>
>> I have not been able find the bug nor do i know how to reproduce it 
>> from scratch. If you have a public site with which we can reproduce it 
>> please comment to the Jira ticket. Make sure you use either default 
>> config or little, a seed URL and the exact crawl & dedup steps to 
>> reproduce.
>>
>> If you find it we might fix it. In any case we need to replace the 
>> dedup command with a more scalable tool which it currently is not.
>>
>> In the mean time you can omit solrdedup and use Solr's internal 
>> deduplication instead, it works similar and uses the same signature 
>> algorithm as Nutch has. Please consult the Solr wiki page on 
>> deduplication.
>>
>> Good luck
>>
>>
>> On Thu, 10 May 2012 22:54:37 +0300, Tolga <to...@ozses.net> wrote:
>>> Hi Markus,
>>>
>>> On 05/10/2012 09:42 AM, Markus Jelsma wrote:
>>>> Hi,
>>>>
>>>> On Thu, 10 May 2012 09:10:04 +0300, Tolga <to...@ozses.net> wrote:
>>>>> Hi,
>>>>>
>>>>> This will sound like a duplicate, but actually it differs from 
>>>>> the
>>>>> other one. Please bear with me. Following
>>>>> http://wiki.apache.org/nutch/NutchTutorial, I first issued the 
>>>>> command
>>>>>
>>>>> bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3 
>>>>> -topN 5
>>>>>
>>>>> Then when I got the message
>>>>>
>>>>> Exception in thread "main" java.io.IOException: Job failed!
>>>>>     at 
>>>>> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:373)
>>>>>
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:353)
>>>>>
>>>>>
>>>>>     at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
>>>>>     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>>     at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
>>>>
>>>> Please include the relevant part of the log. This can be a known 
>>>> issue.
>>>
>>> This is an excerpt from hadoop.log:
>>>
>>> 2012-05-10 22:26:30,349 INFO  crawl.Crawl - crawl started in:
>>> crawl-20120510222629
>>> 2012-05-10 22:26:30,350 INFO  crawl.Crawl - rootUrlDir = urls
>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - threads = 10
>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - depth = 3
>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl -
>>> solrUrl=http://localhost:8983/solr/
>>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - topN = 100
>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: starting 
>>> at
>>> 2012-05-10 22:26:30
>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: crawlDb:
>>> crawl-20120510222629/crawldb
>>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: urlDir: 
>>> urls
>>> 2012-05-10 22:26:30,809 INFO  crawl.Injector - Injector: Converting
>>> injected urls to crawl db entries.
>>> 2012-05-10 22:26:34,173 INFO  plugin.PluginRepository - Plugins:
>>> looking in: /root/apache-nutch-1.4-bin/runtime/local/plugins
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Plugin
>>> Auto-activation mode: [true]
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered 
>>> Plugins:
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     the 
>>> nutch
>>> core extension points (nutch-extensionpoints)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic 
>>> URL
>>> Normalizer (urlnormalizer-basic)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Html
>>> Parse Plug-in (parse-html)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic
>>> Indexing Filter (index-basic)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     HTTP
>>> Framework (lib-http)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -
>>> Pass-through URL Normalizer (urlnormalizer-pass)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex 
>>> URL
>>> Filter (urlfilter-regex)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Http
>>> Protocol Plug-in (protocol-http)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex 
>>> URL
>>> Normalizer (urlnormalizer-regex)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Tika
>>> Parser Plug-in (parse-tika)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     OPIC
>>> Scoring Plug-in (scoring-opic)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     
>>> CyberNeko
>>> HTML Parser (lib-nekohtml)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Anchor
>>> Indexing Filter (index-anchor)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex 
>>> URL
>>> Filter Framework (lib-regex-filter)
>>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered
>>> Extension-Points:
>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch 
>>> URL
>>> Normalizer (org.apache.nutch.net.URLNormalizer)
>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>> Protocol (org.apache.nutch.protocol.Protocol)
>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>> Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch 
>>> URL
>>> Filter (org.apache.nutch.net.URLFilter)
>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>> Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     HTML
>>> Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>> Content Parser (org.apache.nutch.parse.Parser)
>>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>>> Scoring (org.apache.nutch.scoring.ScoringFilter)
>>> 2012-05-10 22:26:35,439 INFO  regex.RegexURLNormalizer - can't find
>>> rules for scope 'inject', using default
>>> 2012-05-10 22:26:36,434 INFO  crawl.Injector - Injector: Merging
>>> injected urls into crawl db.
>>> 2012-05-10 22:26:36,710 WARN  util.NativeCodeLoader - Unable to 
>>> load
>>> native-hadoop library for your platform... using builtin-java 
>>> classes
>>> where applicable
>>> 2012-05-10 22:26:37,542 INFO  crawl.Injector - Injector: finished 
>>> at
>>> 2012-05-10 22:26:37, elapsed: 00:00:06
>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: starting
>>> at 2012-05-10 22:26:37
>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: 
>>> Selecting
>>> best-scoring urls due for fetch.
>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: 
>>> filtering: true
>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: 
>>> normalizing: true
>>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: topN: 
>>> 100
>>> 2012-05-10 22:26:37,552 INFO  crawl.Generator - Generator: 
>>> jobtracker
>>> is 'local', generating exactly one partition.
>>> 2012-05-10 22:26:37,820 INFO  crawl.FetchScheduleFactory - Using
>>> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
>>> 2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule -
>>> defaultInterval=2592000
>>> 2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule -
>>> maxInterval=7776000
>>> 2012-05-10 22:26:37,856 INFO  regex.RegexURLNormalizer - can't find
>>> rules for scope 'partition', using default
>>> ...
>>> ...
>>> INFO: [] webapp=/solr path=/update
>>>
>>> 
>>> params={waitSearcher=true&waitFlush=true&wt=javabin&commit=true&version=2}
>>>
>>> status=0 QTime=221
>>> 2012-05-10 22:36:26,336 INFO  solr.SolrIndexer - SolrIndexer:
>>> finished at 2012-05-10 22:36:26, elapsed: 00:00:05
>>> 2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates -
>>> SolrDeleteDuplicates: starting at 2012-05-10 22:36:26
>>> 2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates -
>>> SolrDeleteDuplicates: Solr url: http://localhost:8983/solr/
>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>>> INFO: [] webapp=/solr path=/select
>>> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=220
>>> status=0 QTime=74
>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>>> INFO: [] webapp=/solr path=/select
>>> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=220
>>> status=0 QTime=0
>>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>>> INFO: [] webapp=/solr path=/select
>>>
>>> 
>>> params={fl=id,boost,tstamp,digest&start=0&q=id:[*+TO+*]&wt=javabin&rows=220&version=2}
>>>
>>> hits=220 status=0 QTime=9
>>> 2012-05-10 22:36:27,656 WARN  mapred.LocalJobRunner - 
>>> job_local_0020
>>> java.lang.NullPointerException
>>>     at org.apache.hadoop.io.Text.encode(Text.java:388)
>>>     at org.apache.hadoop.io.Text.set(Text.java:178)
>>>     at
>>>
>>> 
>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.next(SolrDeleteDuplicates.java:270)
>>>
>>>     at
>>>
>>> 
>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.next(SolrDeleteDuplicates.java:241)
>>>
>>>     at
>>>
>>> 
>>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
>>>
>>>     at
>>>
>>> 
>>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
>>>
>>>     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
>>>     at 
>>> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>>>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>>     at
>>>
>>> 
>>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
>>>>
>>>>>
>>>>> I issued the commands
>>>>>
>>>>> bin/nutch crawl urls -dir crawl -depth 3 -topN 5
>>>>>
>>>>> and
>>>>>
>>>>> bin/nutch solrindex http://127.0.0.1:8983/solr/ crawldb -linkdb
>>>>> crawldb/linkdb crawldb/segments/*
>>>>>
>>>>> separately, after which I got no errors. When I browsed to
>>>>> http://localhost:8983/solr/admin and attempted a search, I got 
>>>>> the
>>>>> error
>>>>>
>>>>>
>>>>>    HTTP ERROR 400
>>>>>
>>>>> Problem accessing /solr/select. Reason:
>>>>>
>>>>>     undefined field text
>>>>
>>>> But this is a Solr thing, you have no field named text. Resolve 
>>>> this in Solr or on the Solr mailing list.
>>>>
>>>>>
>>>>>
>>>>>
>>>>> 
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> /Powered by Jetty://
>>>>>
>>>>> /What am I doing wrong?
>>>>>
>>>>> Regards,/
>>>>> /
>>>>
>>> Regards,
>>

-- 
 Markus Jelsma - CTO - Openindex

Re: HTTP error 400

Posted by Tolga <to...@ozses.net>.

Hi,

How do I exactly "omit solrdedup and use Solr's internal deduplication" 
instead.? I don't even know what any of that means :D I've just used 
bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3 -topN 
100 to get the error. I have to use all the steps?

Regards,

On 05/10/2012 11:38 PM, Markus Jelsma wrote:
> thanks
>
> This is a known issue:
> https://issues.apache.org/jira/browse/NUTCH-1100
>
> I have not been able find the bug nor do i know how to reproduce it 
> from scratch. If you have a public site with which we can reproduce it 
> please comment to the Jira ticket. Make sure you use either default 
> config or little, a seed URL and the exact crawl & dedup steps to 
> reproduce.
>
> If you find it we might fix it. In any case we need to replace the 
> dedup command with a more scalable tool which it currently is not.
>
> In the mean time you can omit solrdedup and use Solr's internal 
> deduplication instead, it works similar and uses the same signature 
> algorithm as Nutch has. Please consult the Solr wiki page on 
> deduplication.
>
> Good luck
>
>
> On Thu, 10 May 2012 22:54:37 +0300, Tolga <to...@ozses.net> wrote:
>> Hi Markus,
>>
>> On 05/10/2012 09:42 AM, Markus Jelsma wrote:
>>> Hi,
>>>
>>> On Thu, 10 May 2012 09:10:04 +0300, Tolga <to...@ozses.net> wrote:
>>>> Hi,
>>>>
>>>> This will sound like a duplicate, but actually it differs from the
>>>> other one. Please bear with me. Following
>>>> http://wiki.apache.org/nutch/NutchTutorial, I first issued the command
>>>>
>>>> bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3 
>>>> -topN 5
>>>>
>>>> Then when I got the message
>>>>
>>>> Exception in thread "main" java.io.IOException: Job failed!
>>>>     at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
>>>>     at
>>>>
>>>>
>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:373) 
>>>>
>>>>
>>>>     at
>>>>
>>>>
>>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:353) 
>>>>
>>>>
>>>>     at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
>>>>     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>     at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
>>>
>>> Please include the relevant part of the log. This can be a known issue.
>>
>> This is an excerpt from hadoop.log:
>>
>> 2012-05-10 22:26:30,349 INFO  crawl.Crawl - crawl started in:
>> crawl-20120510222629
>> 2012-05-10 22:26:30,350 INFO  crawl.Crawl - rootUrlDir = urls
>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - threads = 10
>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - depth = 3
>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl -
>> solrUrl=http://localhost:8983/solr/
>> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - topN = 100
>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: starting at
>> 2012-05-10 22:26:30
>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: crawlDb:
>> crawl-20120510222629/crawldb
>> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: urlDir: urls
>> 2012-05-10 22:26:30,809 INFO  crawl.Injector - Injector: Converting
>> injected urls to crawl db entries.
>> 2012-05-10 22:26:34,173 INFO  plugin.PluginRepository - Plugins:
>> looking in: /root/apache-nutch-1.4-bin/runtime/local/plugins
>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Plugin
>> Auto-activation mode: [true]
>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered 
>> Plugins:
>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     the nutch
>> core extension points (nutch-extensionpoints)
>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic URL
>> Normalizer (urlnormalizer-basic)
>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Html
>> Parse Plug-in (parse-html)
>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic
>> Indexing Filter (index-basic)
>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     HTTP
>> Framework (lib-http)
>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -
>> Pass-through URL Normalizer (urlnormalizer-pass)
>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
>> Filter (urlfilter-regex)
>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Http
>> Protocol Plug-in (protocol-http)
>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
>> Normalizer (urlnormalizer-regex)
>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Tika
>> Parser Plug-in (parse-tika)
>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     OPIC
>> Scoring Plug-in (scoring-opic)
>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     CyberNeko
>> HTML Parser (lib-nekohtml)
>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Anchor
>> Indexing Filter (index-anchor)
>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
>> Filter Framework (lib-regex-filter)
>> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered
>> Extension-Points:
>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch URL
>> Normalizer (org.apache.nutch.net.URLNormalizer)
>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>> Protocol (org.apache.nutch.protocol.Protocol)
>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>> Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch URL
>> Filter (org.apache.nutch.net.URLFilter)
>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>> Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     HTML
>> Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>> Content Parser (org.apache.nutch.parse.Parser)
>> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
>> Scoring (org.apache.nutch.scoring.ScoringFilter)
>> 2012-05-10 22:26:35,439 INFO  regex.RegexURLNormalizer - can't find
>> rules for scope 'inject', using default
>> 2012-05-10 22:26:36,434 INFO  crawl.Injector - Injector: Merging
>> injected urls into crawl db.
>> 2012-05-10 22:26:36,710 WARN  util.NativeCodeLoader - Unable to load
>> native-hadoop library for your platform... using builtin-java classes
>> where applicable
>> 2012-05-10 22:26:37,542 INFO  crawl.Injector - Injector: finished at
>> 2012-05-10 22:26:37, elapsed: 00:00:06
>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: starting
>> at 2012-05-10 22:26:37
>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: Selecting
>> best-scoring urls due for fetch.
>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: filtering: 
>> true
>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: 
>> normalizing: true
>> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: topN: 100
>> 2012-05-10 22:26:37,552 INFO  crawl.Generator - Generator: jobtracker
>> is 'local', generating exactly one partition.
>> 2012-05-10 22:26:37,820 INFO  crawl.FetchScheduleFactory - Using
>> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
>> 2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule -
>> defaultInterval=2592000
>> 2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule -
>> maxInterval=7776000
>> 2012-05-10 22:26:37,856 INFO  regex.RegexURLNormalizer - can't find
>> rules for scope 'partition', using default
>> ...
>> ...
>> INFO: [] webapp=/solr path=/update
>>
>> params={waitSearcher=true&waitFlush=true&wt=javabin&commit=true&version=2} 
>>
>> status=0 QTime=221
>> 2012-05-10 22:36:26,336 INFO  solr.SolrIndexer - SolrIndexer:
>> finished at 2012-05-10 22:36:26, elapsed: 00:00:05
>> 2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates -
>> SolrDeleteDuplicates: starting at 2012-05-10 22:36:26
>> 2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates -
>> SolrDeleteDuplicates: Solr url: http://localhost:8983/solr/
>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>> INFO: [] webapp=/solr path=/select
>> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=220
>> status=0 QTime=74
>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>> INFO: [] webapp=/solr path=/select
>> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=220
>> status=0 QTime=0
>> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
>> INFO: [] webapp=/solr path=/select
>>
>> params={fl=id,boost,tstamp,digest&start=0&q=id:[*+TO+*]&wt=javabin&rows=220&version=2} 
>>
>> hits=220 status=0 QTime=9
>> 2012-05-10 22:36:27,656 WARN  mapred.LocalJobRunner - job_local_0020
>> java.lang.NullPointerException
>>     at org.apache.hadoop.io.Text.encode(Text.java:388)
>>     at org.apache.hadoop.io.Text.set(Text.java:178)
>>     at
>>
>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.next(SolrDeleteDuplicates.java:270) 
>>
>>     at
>>
>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.next(SolrDeleteDuplicates.java:241) 
>>
>>     at
>>
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192) 
>>
>>     at
>>
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176) 
>>
>>     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
>>     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>     at
>>
>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
>>>
>>>>
>>>> I issued the commands
>>>>
>>>> bin/nutch crawl urls -dir crawl -depth 3 -topN 5
>>>>
>>>> and
>>>>
>>>> bin/nutch solrindex http://127.0.0.1:8983/solr/ crawldb -linkdb
>>>> crawldb/linkdb crawldb/segments/*
>>>>
>>>> separately, after which I got no errors. When I browsed to
>>>> http://localhost:8983/solr/admin and attempted a search, I got the
>>>> error
>>>>
>>>>
>>>>    HTTP ERROR 400
>>>>
>>>> Problem accessing /solr/select. Reason:
>>>>
>>>>     undefined field text
>>>
>>> But this is a Solr thing, you have no field named text. Resolve this 
>>> in Solr or on the Solr mailing list.
>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------ 
>>>>
>>>> /Powered by Jetty://
>>>>
>>>> /What am I doing wrong?
>>>>
>>>> Regards,/
>>>> /
>>>
>> Regards,
>

Re: HTTP error 400

Posted by Markus Jelsma <ma...@openindex.io>.

 thanks

 This is a known issue:
 https://issues.apache.org/jira/browse/NUTCH-1100

 I have not been able find the bug nor do i know how to reproduce it 
 from scratch. If you have a public site with which we can reproduce it 
 please comment to the Jira ticket. Make sure you use either default 
 config or little, a seed URL and the exact crawl & dedup steps to 
 reproduce.

 If you find it we might fix it. In any case we need to replace the 
 dedup command with a more scalable tool which it currently is not.

 In the mean time you can omit solrdedup and use Solr's internal 
 deduplication instead, it works similar and uses the same signature 
 algorithm as Nutch has. Please consult the Solr wiki page on 
 deduplication.

 Good luck


 On Thu, 10 May 2012 22:54:37 +0300, Tolga <to...@ozses.net> wrote:
> Hi Markus,
>
> On 05/10/2012 09:42 AM, Markus Jelsma wrote:
>> Hi,
>>
>> On Thu, 10 May 2012 09:10:04 +0300, Tolga <to...@ozses.net> wrote:
>>> Hi,
>>>
>>> This will sound like a duplicate, but actually it differs from the
>>> other one. Please bear with me. Following
>>> http://wiki.apache.org/nutch/NutchTutorial, I first issued the 
>>> command
>>>
>>> bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3 
>>> -topN 5
>>>
>>> Then when I got the message
>>>
>>> Exception in thread "main" java.io.IOException: Job failed!
>>>     at 
>>> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
>>>     at
>>>
>>> 
>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:373)
>>>
>>>     at
>>>
>>> 
>>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:353)
>>>
>>>     at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
>>>     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>     at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
>>
>> Please include the relevant part of the log. This can be a known 
>> issue.
>
> This is an excerpt from hadoop.log:
>
> 2012-05-10 22:26:30,349 INFO  crawl.Crawl - crawl started in:
> crawl-20120510222629
> 2012-05-10 22:26:30,350 INFO  crawl.Crawl - rootUrlDir = urls
> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - threads = 10
> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - depth = 3
> 2012-05-10 22:26:30,351 INFO  crawl.Crawl -
> solrUrl=http://localhost:8983/solr/
> 2012-05-10 22:26:30,351 INFO  crawl.Crawl - topN = 100
> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: starting at
> 2012-05-10 22:26:30
> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: crawlDb:
> crawl-20120510222629/crawldb
> 2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: urlDir: urls
> 2012-05-10 22:26:30,809 INFO  crawl.Injector - Injector: Converting
> injected urls to crawl db entries.
> 2012-05-10 22:26:34,173 INFO  plugin.PluginRepository - Plugins:
> looking in: /root/apache-nutch-1.4-bin/runtime/local/plugins
> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Plugin
> Auto-activation mode: [true]
> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered 
> Plugins:
> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     the nutch
> core extension points (nutch-extensionpoints)
> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic URL
> Normalizer (urlnormalizer-basic)
> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Html
> Parse Plug-in (parse-html)
> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic
> Indexing Filter (index-basic)
> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     HTTP
> Framework (lib-http)
> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -
> Pass-through URL Normalizer (urlnormalizer-pass)
> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
> Filter (urlfilter-regex)
> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Http
> Protocol Plug-in (protocol-http)
> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
> Normalizer (urlnormalizer-regex)
> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Tika
> Parser Plug-in (parse-tika)
> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     OPIC
> Scoring Plug-in (scoring-opic)
> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     CyberNeko
> HTML Parser (lib-nekohtml)
> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Anchor
> Indexing Filter (index-anchor)
> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL
> Filter Framework (lib-regex-filter)
> 2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered
> Extension-Points:
> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch URL
> Normalizer (org.apache.nutch.net.URLNormalizer)
> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
> Protocol (org.apache.nutch.protocol.Protocol)
> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
> Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch URL
> Filter (org.apache.nutch.net.URLFilter)
> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
> Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     HTML
> Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
> Content Parser (org.apache.nutch.parse.Parser)
> 2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch
> Scoring (org.apache.nutch.scoring.ScoringFilter)
> 2012-05-10 22:26:35,439 INFO  regex.RegexURLNormalizer - can't find
> rules for scope 'inject', using default
> 2012-05-10 22:26:36,434 INFO  crawl.Injector - Injector: Merging
> injected urls into crawl db.
> 2012-05-10 22:26:36,710 WARN  util.NativeCodeLoader - Unable to load
> native-hadoop library for your platform... using builtin-java classes
> where applicable
> 2012-05-10 22:26:37,542 INFO  crawl.Injector - Injector: finished at
> 2012-05-10 22:26:37, elapsed: 00:00:06
> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: starting
> at 2012-05-10 22:26:37
> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: Selecting
> best-scoring urls due for fetch.
> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: filtering: 
> true
> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: 
> normalizing: true
> 2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: topN: 100
> 2012-05-10 22:26:37,552 INFO  crawl.Generator - Generator: jobtracker
> is 'local', generating exactly one partition.
> 2012-05-10 22:26:37,820 INFO  crawl.FetchScheduleFactory - Using
> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
> 2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule -
> defaultInterval=2592000
> 2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule -
> maxInterval=7776000
> 2012-05-10 22:26:37,856 INFO  regex.RegexURLNormalizer - can't find
> rules for scope 'partition', using default
> ...
> ...
> INFO: [] webapp=/solr path=/update
> 
> params={waitSearcher=true&waitFlush=true&wt=javabin&commit=true&version=2}
> status=0 QTime=221
> 2012-05-10 22:36:26,336 INFO  solr.SolrIndexer - SolrIndexer:
> finished at 2012-05-10 22:36:26, elapsed: 00:00:05
> 2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates -
> SolrDeleteDuplicates: starting at 2012-05-10 22:36:26
> 2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates -
> SolrDeleteDuplicates: Solr url: http://localhost:8983/solr/
> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/select
> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=220
> status=0 QTime=74
> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/select
> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=220
> status=0 QTime=0
> May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/select
> 
> params={fl=id,boost,tstamp,digest&start=0&q=id:[*+TO+*]&wt=javabin&rows=220&version=2}
> hits=220 status=0 QTime=9
> 2012-05-10 22:36:27,656 WARN  mapred.LocalJobRunner - job_local_0020
> java.lang.NullPointerException
>     at org.apache.hadoop.io.Text.encode(Text.java:388)
>     at org.apache.hadoop.io.Text.set(Text.java:178)
>     at
> 
> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.next(SolrDeleteDuplicates.java:270)
>     at
> 
> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.next(SolrDeleteDuplicates.java:241)
>     at
> 
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
>     at
> 
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
>     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
>     at 
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>     at
> 
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
>>
>>>
>>> I issued the commands
>>>
>>> bin/nutch crawl urls -dir crawl -depth 3 -topN 5
>>>
>>> and
>>>
>>> bin/nutch solrindex http://127.0.0.1:8983/solr/ crawldb -linkdb
>>> crawldb/linkdb crawldb/segments/*
>>>
>>> separately, after which I got no errors. When I browsed to
>>> http://localhost:8983/solr/admin and attempted a search, I got the
>>> error
>>>
>>>
>>>    HTTP ERROR 400
>>>
>>> Problem accessing /solr/select. Reason:
>>>
>>>     undefined field text
>>
>> But this is a Solr thing, you have no field named text. Resolve this 
>> in Solr or on the Solr mailing list.
>>
>>>
>>>
>>> 
>>> ------------------------------------------------------------------------
>>> /Powered by Jetty://
>>>
>>> /What am I doing wrong?
>>>
>>> Regards,/
>>> /
>>
> Regards,

-- 
 Markus Jelsma - CTO - Openindex

Re: HTTP error 400

Posted by Tolga <to...@ozses.net>.

Hi Markus,

On 05/10/2012 09:42 AM, Markus Jelsma wrote:
> Hi,
>
> On Thu, 10 May 2012 09:10:04 +0300, Tolga <to...@ozses.net> wrote:
>> Hi,
>>
>> This will sound like a duplicate, but actually it differs from the
>> other one. Please bear with me. Following
>> http://wiki.apache.org/nutch/NutchTutorial, I first issued the command
>>
>> bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3 -topN 5
>>
>> Then when I got the message
>>
>> Exception in thread "main" java.io.IOException: Job failed!
>>     at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
>>     at
>>
>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:373) 
>>
>>     at
>>
>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:353) 
>>
>>     at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
>>     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>     at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
>
> Please include the relevant part of the log. This can be a known issue.

This is an excerpt from hadoop.log:

2012-05-10 22:26:30,349 INFO  crawl.Crawl - crawl started in: 
crawl-20120510222629
2012-05-10 22:26:30,350 INFO  crawl.Crawl - rootUrlDir = urls
2012-05-10 22:26:30,351 INFO  crawl.Crawl - threads = 10
2012-05-10 22:26:30,351 INFO  crawl.Crawl - depth = 3
2012-05-10 22:26:30,351 INFO  crawl.Crawl - 
solrUrl=http://localhost:8983/solr/
2012-05-10 22:26:30,351 INFO  crawl.Crawl - topN = 100
2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: starting at 
2012-05-10 22:26:30
2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: crawlDb: 
crawl-20120510222629/crawldb
2012-05-10 22:26:30,750 INFO  crawl.Injector - Injector: urlDir: urls
2012-05-10 22:26:30,809 INFO  crawl.Injector - Injector: Converting 
injected urls to crawl db entries.
2012-05-10 22:26:34,173 INFO  plugin.PluginRepository - Plugins: looking 
in: /root/apache-nutch-1.4-bin/runtime/local/plugins
2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Plugin 
Auto-activation mode: [true]
2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered Plugins:
2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     the nutch 
core extension points (nutch-extensionpoints)
2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic URL 
Normalizer (urlnormalizer-basic)
2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Html Parse 
Plug-in (parse-html)
2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Basic 
Indexing Filter (index-basic)
2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     HTTP 
Framework (lib-http)
2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Pass-through 
URL Normalizer (urlnormalizer-pass)
2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL 
Filter (urlfilter-regex)
2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Http 
Protocol Plug-in (protocol-http)
2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL 
Normalizer (urlnormalizer-regex)
2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Tika Parser 
Plug-in (parse-tika)
2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     OPIC Scoring 
Plug-in (scoring-opic)
2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     CyberNeko 
HTML Parser (lib-nekohtml)
2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Anchor 
Indexing Filter (index-anchor)
2012-05-10 22:26:34,962 INFO  plugin.PluginRepository -     Regex URL 
Filter Framework (lib-regex-filter)
2012-05-10 22:26:34,962 INFO  plugin.PluginRepository - Registered 
Extension-Points:
2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch URL 
Normalizer (org.apache.nutch.net.URLNormalizer)
2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch 
Protocol (org.apache.nutch.protocol.Protocol)
2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch 
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch URL 
Filter (org.apache.nutch.net.URLFilter)
2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch 
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     HTML Parse 
Filter (org.apache.nutch.parse.HtmlParseFilter)
2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch 
Content Parser (org.apache.nutch.parse.Parser)
2012-05-10 22:26:34,963 INFO  plugin.PluginRepository -     Nutch 
Scoring (org.apache.nutch.scoring.ScoringFilter)
2012-05-10 22:26:35,439 INFO  regex.RegexURLNormalizer - can't find 
rules for scope 'inject', using default
2012-05-10 22:26:36,434 INFO  crawl.Injector - Injector: Merging 
injected urls into crawl db.
2012-05-10 22:26:36,710 WARN  util.NativeCodeLoader - Unable to load 
native-hadoop library for your platform... using builtin-java classes 
where applicable
2012-05-10 22:26:37,542 INFO  crawl.Injector - Injector: finished at 
2012-05-10 22:26:37, elapsed: 00:00:06
2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: starting at 
2012-05-10 22:26:37
2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: Selecting 
best-scoring urls due for fetch.
2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: filtering: true
2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: normalizing: true
2012-05-10 22:26:37,551 INFO  crawl.Generator - Generator: topN: 100
2012-05-10 22:26:37,552 INFO  crawl.Generator - Generator: jobtracker is 
'local', generating exactly one partition.
2012-05-10 22:26:37,820 INFO  crawl.FetchScheduleFactory - Using 
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule - 
defaultInterval=2592000
2012-05-10 22:26:37,820 INFO  crawl.AbstractFetchSchedule - 
maxInterval=7776000
2012-05-10 22:26:37,856 INFO  regex.RegexURLNormalizer - can't find 
rules for scope 'partition', using default
...
...
INFO: [] webapp=/solr path=/update 
params={waitSearcher=true&waitFlush=true&wt=javabin&commit=true&version=2} 
status=0 QTime=221
2012-05-10 22:36:26,336 INFO  solr.SolrIndexer - SolrIndexer: finished 
at 2012-05-10 22:36:26, elapsed: 00:00:05
2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates - 
SolrDeleteDuplicates: starting at 2012-05-10 22:36:26
2012-05-10 22:36:26,339 INFO  solr.SolrDeleteDuplicates - 
SolrDeleteDuplicates: Solr url: http://localhost:8983/solr/
May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select 
params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=220 
status=0 QTime=74
May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select 
params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=220 
status=0 QTime=0
May 10, 2012 10:36:27 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select 
params={fl=id,boost,tstamp,digest&start=0&q=id:[*+TO+*]&wt=javabin&rows=220&version=2} 
hits=220 status=0 QTime=9
2012-05-10 22:36:27,656 WARN  mapred.LocalJobRunner - job_local_0020
java.lang.NullPointerException
     at org.apache.hadoop.io.Text.encode(Text.java:388)
     at org.apache.hadoop.io.Text.set(Text.java:178)
     at 
org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.next(SolrDeleteDuplicates.java:270)
     at 
org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.next(SolrDeleteDuplicates.java:241)
     at 
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
     at 
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
     at 
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
>
>>
>> I issued the commands
>>
>> bin/nutch crawl urls -dir crawl -depth 3 -topN 5
>>
>> and
>>
>> bin/nutch solrindex http://127.0.0.1:8983/solr/ crawldb -linkdb
>> crawldb/linkdb crawldb/segments/*
>>
>> separately, after which I got no errors. When I browsed to
>> http://localhost:8983/solr/admin and attempted a search, I got the
>> error
>>
>>
>>    HTTP ERROR 400
>>
>> Problem accessing /solr/select. Reason:
>>
>>     undefined field text
>
> But this is a Solr thing, you have no field named text. Resolve this 
> in Solr or on the Solr mailing list.
>
>>
>>
>> ------------------------------------------------------------------------
>> /Powered by Jetty://
>>
>> /What am I doing wrong?
>>
>> Regards,/
>> /
>
Regards,

Re: HTTP error 400

Posted by Markus Jelsma <ma...@openindex.io>.

 Hi,

 On Thu, 10 May 2012 09:10:04 +0300, Tolga <to...@ozses.net> wrote:
> Hi,
>
> This will sound like a duplicate, but actually it differs from the
> other one. Please bear with me. Following
> http://wiki.apache.org/nutch/NutchTutorial, I first issued the 
> command
>
> bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3 -topN 
> 5
>
> Then when I got the message
>
> Exception in thread "main" java.io.IOException: Job failed!
>     at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
>     at
> 
> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:373)
>     at
> 
> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:353)
>     at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
>     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>     at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)

 Please include the relevant part of the log. This can be a known issue.

>
> I issued the commands
>
> bin/nutch crawl urls -dir crawl -depth 3 -topN 5
>
> and
>
> bin/nutch solrindex http://127.0.0.1:8983/solr/ crawldb -linkdb
> crawldb/linkdb crawldb/segments/*
>
> separately, after which I got no errors. When I browsed to
> http://localhost:8983/solr/admin and attempted a search, I got the
> error
>
>
>    HTTP ERROR 400
>
> Problem accessing /solr/select. Reason:
>
>     undefined field text

 But this is a Solr thing, you have no field named text. Resolve this in 
 Solr or on the Solr mailing list.

>
> 
> ------------------------------------------------------------------------
> /Powered by Jetty://
>
> /What am I doing wrong?
>
> Regards,/
> /

-- 
 Markus Jelsma - CTO - Openindex