You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by John john <ze...@yahoo.fr> on 2006/06/21 04:16:15 UTC
Do nutch allow an advanced search?
Hello
I'm new in the nutch world and i'm wondering whether it's possible to search with date range? or specify a date and then nutch retrieves pages updated after this date?
thanks
---------------------------------
Yahoo! Mail réinvente le mail ! Découvrez le nouveau Yahoo! Mail et son interface révolutionnaire.
Re: Do nutch allow an advanced search?
Posted by Scott McCammon <mc...@onid.orst.edu>.
Stefan Neufeind wrote:
> requiring a keyword/phrase has been mentioned at several places before
> already. Is there a technical background for it, or could that
> limitation maybe be removed (and should we file a JIRA for that)?
>
Hi Stefan,
In org.apache.nutch.searcher.LuceneQueryOptimizer, the decision was made
to convert some range queries into Lucene range filters in order to
optimize search speed. Filters of course require query results to
operate upon, so they cannot be used alone. It looks like this also
avoids a TooManyClauses exception in Lucene which can occur when it
expands range queries.
http://wiki.apache.org/jakarta-lucene/LuceneFAQ#head-06fafb5d19e786a50fb3dfb8821a6af9f37aa831
I'm happy with the compromise.
Cheers,
Scott
Re: NEWBIE help: java.lang.IllegalAccessError
Posted by Mike Blackstock <mi...@hedgx.com>.
Mike Blackstock wrote:
>
>
>
> Exception in thread "main" java.lang.IllegalAccessError: tried to
> access field org.apache.lucene.index.IndexWriter.mergeFactor from
> class org.apache.nutch.indexer.IndexSegment
I'm using Nutch 7.2; I spent the better part of yesterday searching the
net for possible
answers before subscribing here, but no luck.
Cheers,
Mike
Re: ERROR when recrawling... can ANYONE help?
Posted by Honda-Search Administrator <ad...@honda-search.com>.
20060619230003 Does exist, but it does not have an index directory...
Here are the subdirectories it has:
content/ fetcher/ fetchlist/ parse_data/ parse_text/
This also seems to be the case for 8 other segment directories.
The rest of the segment directories (excluding the 9 above) have this file
structure:
content/ fetchlist/ index.done parse_text/ fetcher/ index/
parse_data/
What would cause these segments to act this way? Is there a sure way to fix
it? Can I prevent it from happening again?
Matt
----- Original Message -----
From: "TDLN" <di...@gmail.com>
To: <nu...@lucene.apache.org>; "Honda-Search Administrator"
<ad...@honda-search.com>
Sent: Friday, June 23, 2006 11:28 AM
Subject: Re: ERROR when recrawling... can ANYONE help?
> Does /home/honda/nutch-0.7.2/crawl/segments/20060619230003/index exist at
> all?
>
> Can you confirm that all segments contain index directory?
>
> Rgrds,. Thomas
>
>
> On 6/23/06, Honda-Search Administrator <ad...@honda-search.com> wrote:
>> To recrawl I use the command:
>>
>> /home/honda/nutch-0.7.2/recrawl.sh /home/honda/nutch-0.7.2/crawl 1 2
>>
>> "crawl" is the name of my database directory.
>>
>> The script "recrawl.sh" is the standard one that comes in the package.
>> I'm
>> pretty sure it's the same for everyone, but I've included a link to the
>> recrawl.sh script I'm using:
>>
>> http://www.honda-search.com/script.html
>>
>> As you can see I'm crawling with a depth of 1, which is intentional. I
>> only
>> desire to recrawl the specific pages injected each night. I'm wondering
>> if
>> the 'adddays' parameter is messing me up.
>>
>> Matt
>>
>> ----- Original Message -----
>> From: "TDLN" <di...@gmail.com>
>> To: <nu...@lucene.apache.org>; "Honda-Search Administrator"
>> <ad...@honda-search.com>
>> Sent: Friday, June 23, 2006 10:46 AM
>> Subject: Re: ERROR when recrawling... can ANYONE help?
>>
>>
>> > Please specify what exact sequence of commands you are using.
>> >
>> > For incremental crawling best to follow the "whole web" style process
>> > as outlined in the tutorial. The one stop crawl command cannot be used
>> > effectively for that.
>> >
>> > HTH Thomas
>> >
>>
>>
>
>
Re: ERROR when recrawling... can ANYONE help?
Posted by TDLN <di...@gmail.com>.
Does /home/honda/nutch-0.7.2/crawl/segments/20060619230003/index exist at all?
Can you confirm that all segments contain index directory?
Rgrds,. Thomas
On 6/23/06, Honda-Search Administrator <ad...@honda-search.com> wrote:
> To recrawl I use the command:
>
> /home/honda/nutch-0.7.2/recrawl.sh /home/honda/nutch-0.7.2/crawl 1 2
>
> "crawl" is the name of my database directory.
>
> The script "recrawl.sh" is the standard one that comes in the package. I'm
> pretty sure it's the same for everyone, but I've included a link to the
> recrawl.sh script I'm using:
>
> http://www.honda-search.com/script.html
>
> As you can see I'm crawling with a depth of 1, which is intentional. I only
> desire to recrawl the specific pages injected each night. I'm wondering if
> the 'adddays' parameter is messing me up.
>
> Matt
>
> ----- Original Message -----
> From: "TDLN" <di...@gmail.com>
> To: <nu...@lucene.apache.org>; "Honda-Search Administrator"
> <ad...@honda-search.com>
> Sent: Friday, June 23, 2006 10:46 AM
> Subject: Re: ERROR when recrawling... can ANYONE help?
>
>
> > Please specify what exact sequence of commands you are using.
> >
> > For incremental crawling best to follow the "whole web" style process
> > as outlined in the tutorial. The one stop crawl command cannot be used
> > effectively for that.
> >
> > HTH Thomas
> >
>
>
Re: ERROR when recrawling... can ANYONE help?
Posted by Honda-Search Administrator <ad...@honda-search.com>.
To recrawl I use the command:
/home/honda/nutch-0.7.2/recrawl.sh /home/honda/nutch-0.7.2/crawl 1 2
"crawl" is the name of my database directory.
The script "recrawl.sh" is the standard one that comes in the package. I'm
pretty sure it's the same for everyone, but I've included a link to the
recrawl.sh script I'm using:
http://www.honda-search.com/script.html
As you can see I'm crawling with a depth of 1, which is intentional. I only
desire to recrawl the specific pages injected each night. I'm wondering if
the 'adddays' parameter is messing me up.
Matt
----- Original Message -----
From: "TDLN" <di...@gmail.com>
To: <nu...@lucene.apache.org>; "Honda-Search Administrator"
<ad...@honda-search.com>
Sent: Friday, June 23, 2006 10:46 AM
Subject: Re: ERROR when recrawling... can ANYONE help?
> Please specify what exact sequence of commands you are using.
>
> For incremental crawling best to follow the "whole web" style process
> as outlined in the tutorial. The one stop crawl command cannot be used
> effectively for that.
>
> HTH Thomas
>
Re: ERROR when recrawling... can ANYONE help?
Posted by TDLN <di...@gmail.com>.
Please specify what exact sequence of commands you are using.
For incremental crawling best to follow the "whole web" style process
as outlined in the tutorial. The one stop crawl command cannot be used
effectively for that.
HTH Thomas
On 6/23/06, Honda-Search Administrator <ad...@honda-search.com> wrote:
> I'm hoping that my emails actually reach other people, as they've been
> ignored so far.
>
> I just ran a recrawl today to crawl a few injected URLs that I have. At the
> end of the recrawl I received the following error:
>
> 060623 122916 merging segment indexes to:
> /home/honda/nutch-0.7.2/crawl/index
> Exception in thread "main" java.io.IOException:
> /home/honda/nutch-0.7.2/crawl/segments/20060619230003/index not a directory
> at org.apache.lucene.store.FSDirectory.init(FSDirectory.java:180)
> at
> org.apache.lucene.store.FSDirectory.getDirectory(FSDirectory.java:141)
> at org.apache.nutch.indexer.IndexMerger.merge(IndexMerger.java:80)
> at org.apache.nutch.indexer.IndexMerger.main(IndexMerger.java:160)
>
> Of course all of the crawled segments are not in the index.
>
> Can ANYONE tellme how to fix this? I'm getting a bit discouraged with Nutch
> due to the large number of errors I keep receiving during crawls. I do not
> want to have to recrawl my entire sitelist AGAIN just to fix this.
>
> Anyone?
>
> Matt
>
>
ERROR when recrawling... can ANYONE help?
Posted by Honda-Search Administrator <ad...@honda-search.com>.
I'm hoping that my emails actually reach other people, as they've been
ignored so far.
I just ran a recrawl today to crawl a few injected URLs that I have. At the
end of the recrawl I received the following error:
060623 122916 merging segment indexes to:
/home/honda/nutch-0.7.2/crawl/index
Exception in thread "main" java.io.IOException:
/home/honda/nutch-0.7.2/crawl/segments/20060619230003/index not a directory
at org.apache.lucene.store.FSDirectory.init(FSDirectory.java:180)
at
org.apache.lucene.store.FSDirectory.getDirectory(FSDirectory.java:141)
at org.apache.nutch.indexer.IndexMerger.merge(IndexMerger.java:80)
at org.apache.nutch.indexer.IndexMerger.main(IndexMerger.java:160)
Of course all of the crawled segments are not in the index.
Can ANYONE tellme how to fix this? I'm getting a bit discouraged with Nutch
due to the large number of errors I keep receiving during crawls. I do not
want to have to recrawl my entire sitelist AGAIN just to fix this.
Anyone?
Matt
Re: NEWBIE help: java.lang.IllegalAccessError
Posted by Mike Blackstock <mi...@hedgx.com>.
> I installed Nutch the past days, and I kept getting blank results
> pages at
> http://localhost:8080/search.jsp... Tomcat was properly installed, I
> think,
> so I looked further and I found that everytime I run a crawl, I get
> this toward the end:
>
>
> 060621 125637 indexing segment: crawl.test/segments/20060620163405
> Exception in thread "main" java.lang.IllegalAccessError: tried to
> access field org.apache.lucene.index.IndexWriter.mergeFactor from
> class org.apache.nutch.indexer.IndexSegment
> at
> org.apache.nutch.indexer.IndexSegment.indexPages(IndexSegment.java:102)
> at
> org.apache.nutch.indexer.IndexSegment.main(IndexSegment.java:263)
>
In case the above happens to anyone else, don't be an idiot like I was and
have newer versions of lucene-core lying around; I had installed
lucene-core-2.0.0.jar
before installing nutch, which comes with lucene-core-1.9.1.jar, and
2.0.0 was being
used rather than 1.9.1 - some class definitions in the newer version are
different
I guess.
Cheers,
Mike
NEWBIE help: java.lang.IllegalAccessError
Posted by Mike Blackstock <mi...@hedgx.com>.
Hi folks.
I installed Nutch the past days, and I kept getting blank results pages at
http://localhost:8080/search.jsp... Tomcat was properly installed, I think,
so I looked further and I found that everytime I run a crawl, I get
this toward the end:
060621 125637 indexing segment: crawl.test/segments/20060620163405
Exception in thread "main" java.lang.IllegalAccessError: tried to access
field org.apache.lucene.index.IndexWriter.mergeFactor from class
org.apache.nutch.indexer.IndexSegment
at
org.apache.nutch.indexer.IndexSegment.indexPages(IndexSegment.java:102)
at org.apache.nutch.indexer.IndexSegment.main(IndexSegment.java:263)
It's crawling ok, but not indexing the segment. I copied the
lucene-core-2.0.0.jar
file to the current directory from which I'm running bin/nutch (just in
case it
wasn't finding the lucene.index.IndexWriter stuff), but no luck.
Anyone seen this before or have any ideas? I hope so.
Cheers and thanks,
Mike
Re: Do nutch allow an advanced search?
Posted by Stefan Neufeind <ap...@stefan-neufeind.de>.
Scott McCammon wrote:
> The index-more plugin indexes each document's last modified date and is
> searchable via a range like: "date:20060521-20060621" Note that a date
> search does not work by itself. At least one keyword or phrase is required.
Hi Scott,
requiring a keyword/phrase has been mentioned at several places before
already. Is there a technical background for it, or could that
limitation maybe be removed (and should we file a JIRA for that)?
Regards,
Stefan
> John john wrote:
>> Hello
>>
>> I'm new in the nutch world and i'm wondering whether it's possible to
>> search with date range? or specify a date and then nutch retrieves
>> pages updated after this date?
>>
>> thanks
Re: Do nutch allow an advanced search?
Posted by Scott McCammon <mc...@onid.orst.edu>.
The index-more plugin indexes each document's last modified date and is
searchable via a range like: "date:20060521-20060621" Note that a date
search does not work by itself. At least one keyword or phrase is required.
Scott
John john wrote:
> Hello
>
> I'm new in the nutch world and i'm wondering whether it's possible to search with date range? or specify a date and then nutch retrieves pages updated after this date?
>
> thanks
>
Re: problem with skiped urls
Posted by Jayant Kumar Gandhi <ja...@gmail.com>.
You can also stop nutch from crawling those pages by modifying the
robots.txt if you have set Nutch to respect those rules, by default it
will respect such rules. If you haven't modified the setting
http.robots.agents in nutch-default.xml/nutch-site.xml , the following
robots.txt rule should work:-
User-agent: NutchCVS
Disallow: /abteilung/pvs/dokus/
Cheers,
Jayant
On 6/21/06, Stefan Neufeind <ap...@stefan-neufeind.de> wrote:
> david.wojciechowski@uniklinik-freiburg.de wrote:
> > hi,
> > i'm trying to run nutch in our clinicum center and i have a little problem.
> > we have a few intranet servers and i want that nutch skip a few
> > direcotries.
> > for example:
> >
> > http://sapdoku.ukl.uni-freiburg.de/abteilung/pvs/dokus/
> >
> > i wrote this urls in the crawl-urlfilter.txt. for example:
> >
> > -^http://([a-z0-9]*\.)*sapdoku.ukl.uni-freiburg.de/abteilung/pvs/dokus
> >
> > but nothing happens. nutch don't skip this urls. and i don't know why...
> >
> > :( kann anyone help me?
> >
> > i'm cwaling with this command:
> >
> > bin/nutch crawl urls -dir crawl060621 -depth 15 &> crawl060621.log &
> >
> > i'm using the release 0.7.1
>
> Hi David,
>
> do you have regex-urlfilter in your crawler-site-configfile or
> nutch-site-configfile? I suspect that the plugin might not yet be
> loaded. Also, do you have another "allow all URLs"-line above the one
> you mentioned, maybe?
> I don't think the ([a-z0-9]*\.)* should lead to problems (it is * and
> not +, so I guess that should be fine). But if your URL does not have
> anything in front of sapdoku, maybe try dropping that part.
>
>
> Good luck,
> Stefan
>
--
www.jkg.in | http://www.jkg.in/contact-me/
Jayant Kr. Gandhi | +91-9871412929
M.Tech. Computer Tech. Class of 2007,
D-38, Aravali Hostel, IIT Delhi,
Hauz Khas, Delhi-110016
Re: problem with skiped urls
Posted by Stefan Neufeind <ap...@stefan-neufeind.de>.
david.wojciechowski@uniklinik-freiburg.de wrote:
> hi,
> i'm trying to run nutch in our clinicum center and i have a little problem.
> we have a few intranet servers and i want that nutch skip a few
> direcotries.
> for example:
>
> http://sapdoku.ukl.uni-freiburg.de/abteilung/pvs/dokus/
>
> i wrote this urls in the crawl-urlfilter.txt. for example:
>
> -^http://([a-z0-9]*\.)*sapdoku.ukl.uni-freiburg.de/abteilung/pvs/dokus
>
> but nothing happens. nutch don't skip this urls. and i don't know why...
>
> :( kann anyone help me?
>
> i'm cwaling with this command:
>
> bin/nutch crawl urls -dir crawl060621 -depth 15 &> crawl060621.log &
>
> i'm using the release 0.7.1
Hi David,
do you have regex-urlfilter in your crawler-site-configfile or
nutch-site-configfile? I suspect that the plugin might not yet be
loaded. Also, do you have another "allow all URLs"-line above the one
you mentioned, maybe?
I don't think the ([a-z0-9]*\.)* should lead to problems (it is * and
not +, so I guess that should be fine). But if your URL does not have
anything in front of sapdoku, maybe try dropping that part.
Good luck,
Stefan
problem with skiped urls
Posted by da...@uniklinik-freiburg.de.
hi,
i'm trying to run nutch in our clinicum center and i have a little problem.
we have a few intranet servers and i want that nutch skip a few
direcotries.
for example:
http://sapdoku.ukl.uni-freiburg.de/abteilung/pvs/dokus/
i wrote this urls in the crawl-urlfilter.txt. for example:
-^http://([a-z0-9]*\.)*sapdoku.ukl.uni-freiburg.de/abteilung/pvs/dokus
but nothing happens. nutch don't skip this urls. and i don't know why...
:( kann anyone help me?
i'm cwaling with this command:
bin/nutch crawl urls -dir crawl060621 -depth 15 &> crawl060621.log &
i'm using the release 0.7.1
greets david
==========================================================
David Wojciechowski
Universitätsklinikum Freiburg
Klinikrechenzentrum
Agnesenstrasse 6-8
D-79106 Freiburg
Telefon : 0761 / 270 - 1842
Fax: 0761 / 270 - 2276
E-Mail : david.wojciechowski@uniklinik-freiburg.de
==========================================================