You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by "Lukas, Ray" <Ra...@idearc.com> on 2009/03/06 13:56:40 UTC
Can not get Nutch query to work.. Can you help..
I am not able to make any nutch query work. I know it is something
simple. Could someone take a look at what I am doing..
Here is the code I am using, it is pretty simple:
NutchBean bean = new NutchBean(conf);
Query query = Query.parse("title:credit", conf);
Hits hits = bean.search(query, 10);
System.out.println("hits.getLength()=>" +
hits.getLength());
The configuration is the exact same configuration I am using to create
the indexes. The very same object. Pointing Luke at these indexes and
issuing the above search yields plenty of hits. Yet this yields no hits.
The other factor in the set up in nutch-site.xml. I added the following
which points me to the root directory of my newly created indexes.
<property>
<name>searcher.dir</name>
<value>outputDir</value>
<description>
Text Removed from this email, remains in code
</description>
</property>
Query returns zero hits. Tried several things, no luck. Can you help me
out?
ray
On Mar 3, 2009, at 7:14 PM, alxsss@aim.com wrote:
>
> Hi,
>
> I will need to index all links in domains then. What do you think a
> linux box like yours with DSL connection is OK to index the domains
> I have?
>
> Why only segments? I thought we need to merge all sub folders under
> crawl folder. What did you use for merging them?
>
> Thanks.
> A.
>
>
>
>
>
>
>
> -----Original Message-----
> From: John Martyniak <jo...@beforedawn.com>
> To: nutch-user@lucene.apache.org
> Sent: Tue, 3 Mar 2009 3:21 pm
> Subject: Re: what is needed to index for about 10000 domains
>
>
>
>
>
>
>
>
>
> Well the way that nutch works is that you would inject your list of
> domains into the DB, and that would be the starting point. Since
> nutch uses a crawler it would grab those pages, and determine if
> there are any links on those pages, and then add them to the DB. So
> the next time that you generated your urls to fetch, it would take
> your original list, plus the ones that it found to generate the new
> segment.?
> ?
>
> If you wanted to limit it to only pages contained on your 10000
> domains, you could use the regex-urlfilter.txt file in the conf
> directory to limit it to your list. But you would have to create a
> regular expression for each one.?
> ?
>
> I am not familiar with the merge script on the wiki, but have merged
> segments before and it did work. But that was on Linux, don't think
> that should make a difference though.?
> ?
>
> -John?
> ?
>
> ?
>
> On Mar 3, 2009, at 5:10 PM, alxsss@aim.com wrote:?
> ?
>
>> ?
>
>> Hi,?
>
>> ?
>
>> Thanks for the reply. I have list? of those domains only. I am not
>> > sure how many pages they have. Is a DSL? connection sufficient to
>> > run nutch in my case. Did you run nutch for all of your pages at
>> > once or separately for a given subset of them. Btw, yesterday I >
>> tried to use merge shell script that we have on wiki. It gave a lot
>> > of errors. I run it on cygwin though.?
>
>> ?
>
>> Thanks.?
>
>> A.?
>
>> ?
>
>> ?
>
>> ?
>
>> ?
>
>> ?
>
>> ?
>
>> ?
>
>> -----Original Message-----?
>
>> From: John Martyniak <jo...@beforedawn.com>?
>
>> To: nutch-user@lucene.apache.org?
>
>> Sent: Tue, 3 Mar 2009 1:44 pm?
>
>> Subject: Re: what is needed to index for about 10000 domains?
>
>> ?
>
>> ?
>
>> ?
>
>> ?
>
>> ?
>
>> ?
>
>> ?
>
>> ?
>
>> ?
>
>> I think that in order to answer that questions, it is necessary to
>> > know how many total pages are being indexed.??
>
>> ??
>
>> ?
>
>> I currently have ~3.5 million pages indexed, and the segment >
>> directories are around 45GB, The response time is relatively fast.??
>
>> ??
>
>> ?
>
>> In the test site it is running on a dual processor Dell 1850 with >
>> 3GB of RAM.??
>
>> ??
>
>> ?
>
>> -John??
>
>> ??
>
>> ?
>
>> On Mar 3, 2009, at 3:44 PM, alxsss@aim.com wrote:??
>
>> ??
>
>> ?
>
>>> Hello,??
>
>> ?
>
>>> ??
>
>> ?
>
>>> I use nutch-0.9 and need to index about 10000? domains.? I want to
>>> >> > know? minimum requirements to hardware and memory.??
>
>> ?
>
>>> ??
>
>> ?
>
>>> Thanks in advance.??
>
>> ?
>
>>> Alex.??
>
>> ??
>
>> ?
>
>> ?
>
>> ?
>
>> ?
>
>> ?
> ?
>
>
>
>
>
RE: Can not get Nutch query to work.. Can you help..
Posted by "Lukas, Ray" <Ra...@idearc.com>.
WOW.. Okay.. Let me work on that.. You are a clever guy.. How you
figured that out is.. Okay.. Let me try that.. Meetings most of the
day.. So will do best I can.. Aren't the plug-in jars used to build the
indexes and do the crawling.. That does work so I assumed (probably
incorrectly) that I was all set.. So this might not be the case then..
Ah.. Well you know what they say about assuming things..
Oh the S versus s .. WORD.. I typed this up in word and so.. It auto
corrected that and made it upper case.. Sorry.. Sorry.. Problem when
you are always rushing. Okay.. Let me work on an ANT script..
Andrzej,.. Hey man thanks really.. I appreciate you takin the time to
look at this with me..
ray
-----Original Message-----
From: Andrzej Bialecki [mailto:ab@getopt.org]
Sent: Friday, March 06, 2009 10:20 AM
To: nutch-user@lucene.apache.org
Subject: Re: Can not get Nutch query to work.. Can you help..
Lukas, Ray wrote:
> Thanks man for helping out on this.. Thanks.. Okay
> Okay.. so Windows is okay.. I do not have much say in what we use
here..
> so. Which is fine.. I am happy.
>
> I have the following directories, directly under my
> C:\EclipseWorkspaces\nutchTest\outputDir directory which nutch
created..
>
>
> crawldb
> index
> indexes
> linkdb
> Segments
Ok ... except the "Segments" is with a lowercase "s" in front, right?
Lowercase letters do differ from uppercase letters (a first-grader truth
that Windows explorer is blissfully unaware of).
>
> I know that these are magic names and the config files states that
nutch
> will look for and use the index directory under this magically named
> tree so.. I am fine with that....
> So I think that I am good..
>
> so can I say "C:\EclipseWorkspaces\nutchTest\outputDir" as a
> searcher.dir value? I am going to try that..
>
>
> Pointing Luke at that index directory works great.
> Now.. I am not running on Tomcat.. I am just running from within
Eclipse
> as a regular old java project. This is not going to have a web
interface
> so.. no need for a web server..
Ah, running from Eclipse ... :| this is a bit tricky. Usually Eclipse
can't find the plugin jars, so Nutch tools don't work from Eclipse. The
quick & dirty hack that I use to make it work is to make an Ant build
(to populate the build/ directory), and then add build/ to Build Path as
one of the class folders.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Can not get Nutch query to work.. Can you help..
Posted by Andrzej Bialecki <ab...@getopt.org>.
Lukas, Ray wrote:
> Thanks man for helping out on this.. Thanks.. Okay
> Okay.. so Windows is okay.. I do not have much say in what we use here..
> so. Which is fine.. I am happy.
>
> I have the following directories, directly under my
> C:\EclipseWorkspaces\nutchTest\outputDir directory which nutch created..
>
>
> crawldb
> index
> indexes
> linkdb
> Segments
Ok ... except the "Segments" is with a lowercase "s" in front, right?
Lowercase letters do differ from uppercase letters (a first-grader truth
that Windows explorer is blissfully unaware of).
>
> I know that these are magic names and the config files states that nutch
> will look for and use the index directory under this magically named
> tree so.. I am fine with that....
> So I think that I am good..
>
> so can I say "C:\EclipseWorkspaces\nutchTest\outputDir" as a
> searcher.dir value? I am going to try that..
>
>
> Pointing Luke at that index directory works great.
> Now.. I am not running on Tomcat.. I am just running from within Eclipse
> as a regular old java project. This is not going to have a web interface
> so.. no need for a web server..
Ah, running from Eclipse ... :| this is a bit tricky. Usually Eclipse
can't find the plugin jars, so Nutch tools don't work from Eclipse. The
quick & dirty hack that I use to make it work is to make an Ant build
(to populate the build/ directory), and then add build/ to Build Path as
one of the class folders.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
RE: Can not get Nutch query to work.. Can you help..
Posted by "Lukas, Ray" <Ra...@idearc.com>.
Thanks man for helping out on this.. Thanks.. Okay
Okay.. so Windows is okay.. I do not have much say in what we use here..
so. Which is fine.. I am happy.
I have the following directories, directly under my
C:\EclipseWorkspaces\nutchTest\outputDir directory which nutch created..
crawldb
index
indexes
linkdb
Segments
I know that these are magic names and the config files states that nutch
will look for and use the index directory under this magically named
tree so.. I am fine with that....
So I think that I am good..
so can I say "C:\EclipseWorkspaces\nutchTest\outputDir" as a
searcher.dir value? I am going to try that..
Pointing Luke at that index directory works great.
Now.. I am not running on Tomcat.. I am just running from within Eclipse
as a regular old java project. This is not going to have a web interface
so.. no need for a web server..
I am looking to output that search directory using a get method off of
the config..
-----Original Message-----
From: Andrzej Bialecki [mailto:ab@getopt.org]
Sent: Friday, March 06, 2009 9:26 AM
To: nutch-user@lucene.apache.org
Subject: Re: Can not get Nutch query to work.. Can you help..
Lukas, Ray wrote:
> Okay.. I did try that.. Good idea..
> I moved the index to C:\ and tried
> <value>\outputDir</value> and
> <value>c:\outputDir</value>
> Did not help.. Basically I am running out of guesses you see..
> Maybe I don't know how to do this on evil MS Windows.. Maybe this does
> not work on evil MS Windows? Is that the problem? All the examples I
> have seen are unix based... Is this my problem.. Windows..
Ray,
Nutch works just fine on Windows, even if it's not the best environment
for it .. Please make sure that in your outputDir you have subdirs named
exactly like this:
linkdb
indexes
(or index, if you merged indexes using bin/nutch merge command)
segments
which contain the stuff that you crawled & parsed & indexed.
These are "magic" names that the searcher tries to use - especially if
it can't find either "indexes" or "index" things won't work. Check the
Tomcat/Jetty logs - NutchBean should print an info there that says what
directory it tries to use for indexes.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Can not get Nutch query to work.. Can you help..
Posted by Andrzej Bialecki <ab...@getopt.org>.
Lukas, Ray wrote:
> Okay.. I did try that.. Good idea..
> I moved the index to C:\ and tried
> <value>\outputDir</value> and
> <value>c:\outputDir</value>
> Did not help.. Basically I am running out of guesses you see..
> Maybe I don't know how to do this on evil MS Windows.. Maybe this does
> not work on evil MS Windows? Is that the problem? All the examples I
> have seen are unix based... Is this my problem.. Windows..
Ray,
Nutch works just fine on Windows, even if it's not the best environment
for it .. Please make sure that in your outputDir you have subdirs named
exactly like this:
linkdb
indexes
(or index, if you merged indexes using bin/nutch merge command)
segments
which contain the stuff that you crawled & parsed & indexed.
These are "magic" names that the searcher tries to use - especially if
it can't find either "indexes" or "index" things won't work. Check the
Tomcat/Jetty logs - NutchBean should print an info there that says what
directory it tries to use for indexes.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
RE: Can not get Nutch query to work.. Can you help..
Posted by "Lukas, Ray" <Ra...@idearc.com>.
Okay.. I did try that.. Good idea..
I moved the index to C:\ and tried
<value>\outputDir</value> and
<value>c:\outputDir</value>
Did not help.. Basically I am running out of guesses you see..
Maybe I don't know how to do this on evil MS Windows.. Maybe this does
not work on evil MS Windows? Is that the problem? All the examples I
have seen are unix based... Is this my problem.. Windows..
ray
-----Original Message-----
From: Alexander Aristov [mailto:alexander.aristov@gmail.com]
Sent: Friday, March 06, 2009 8:44 AM
To: nutch-user@lucene.apache.org
Subject: Re: Can not get Nutch query to work.. Can you help..
To begin specify full path to the nutch index.
2009/3/6 Lukas, Ray <Ra...@idearc.com>
>
> I am not able to make any nutch query work. I know it is something
> simple. Could someone take a look at what I am doing..
>
> Here is the code I am using, it is pretty simple:
>
>
> NutchBean bean = new NutchBean(conf);
> Query query = Query.parse("title:credit", conf);
> Hits hits = bean.search(query, 10);
> System.out.println("hits.getLength()=>" +
> hits.getLength());
>
> The configuration is the exact same configuration I am using to create
> the indexes. The very same object. Pointing Luke at these indexes and
> issuing the above search yields plenty of hits. Yet this yields no
hits.
>
> The other factor in the set up in nutch-site.xml. I added the
following
> which points me to the root directory of my newly created indexes.
>
> <property>
> <name>searcher.dir</name>
> <value>outputDir</value>
> <description>
> Text Removed from this email, remains in code
> </description>
> </property>
>
> Query returns zero hits. Tried several things, no luck. Can you help
me
> out?
>
> ray
>
> On Mar 3, 2009, at 7:14 PM, alxsss@aim.com wrote:
>
Re: Can not get Nutch query to work.. Can you help..
Posted by Alexander Aristov <al...@gmail.com>.
To begin specify full path to the nutch index.
2009/3/6 Lukas, Ray <Ra...@idearc.com>
>
> I am not able to make any nutch query work. I know it is something
> simple. Could someone take a look at what I am doing..
>
> Here is the code I am using, it is pretty simple:
>
>
> NutchBean bean = new NutchBean(conf);
> Query query = Query.parse("title:credit", conf);
> Hits hits = bean.search(query, 10);
> System.out.println("hits.getLength()=>" +
> hits.getLength());
>
> The configuration is the exact same configuration I am using to create
> the indexes. The very same object. Pointing Luke at these indexes and
> issuing the above search yields plenty of hits. Yet this yields no hits.
>
> The other factor in the set up in nutch-site.xml. I added the following
> which points me to the root directory of my newly created indexes.
>
> <property>
> <name>searcher.dir</name>
> <value>outputDir</value>
> <description>
> Text Removed from this email, remains in code
> </description>
> </property>
>
> Query returns zero hits. Tried several things, no luck. Can you help me
> out?
>
> ray
>
> On Mar 3, 2009, at 7:14 PM, alxsss@aim.com wrote:
>
> >
> > Hi,
> >
> > I will need to index all links in domains then. What do you think a
> > linux box like yours with DSL connection is OK to index the domains
> > I have?
> >
> > Why only segments? I thought we need to merge all sub folders under
> > crawl folder. What did you use for merging them?
> >
> > Thanks.
> > A.
> >
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: John Martyniak <jo...@beforedawn.com>
> > To: nutch-user@lucene.apache.org
> > Sent: Tue, 3 Mar 2009 3:21 pm
> > Subject: Re: what is needed to index for about 10000 domains
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Well the way that nutch works is that you would inject your list of
> > domains into the DB, and that would be the starting point. Since
> > nutch uses a crawler it would grab those pages, and determine if
> > there are any links on those pages, and then add them to the DB. So
> > the next time that you generated your urls to fetch, it would take
> > your original list, plus the ones that it found to generate the new
> > segment.?
> > ?
> >
> > If you wanted to limit it to only pages contained on your 10000
> > domains, you could use the regex-urlfilter.txt file in the conf
> > directory to limit it to your list. But you would have to create a
> > regular expression for each one.?
> > ?
> >
> > I am not familiar with the merge script on the wiki, but have merged
> > segments before and it did work. But that was on Linux, don't think
> > that should make a difference though.?
> > ?
> >
> > -John?
> > ?
> >
> > ?
> >
> > On Mar 3, 2009, at 5:10 PM, alxsss@aim.com wrote:?
> > ?
> >
> >> ?
> >
> >> Hi,?
> >
> >> ?
> >
> >> Thanks for the reply. I have list? of those domains only. I am not
> >> > sure how many pages they have. Is a DSL? connection sufficient to
> >> > run nutch in my case. Did you run nutch for all of your pages at
> >> > once or separately for a given subset of them. Btw, yesterday I >
> >> tried to use merge shell script that we have on wiki. It gave a lot
> >> > of errors. I run it on cygwin though.?
> >
> >> ?
> >
> >> Thanks.?
> >
> >> A.?
> >
> >> ?
> >
> >> ?
> >
> >> ?
> >
> >> ?
> >
> >> ?
> >
> >> ?
> >
> >> ?
> >
> >> -----Original Message-----?
> >
> >> From: John Martyniak <jo...@beforedawn.com>?
> >
> >> To: nutch-user@lucene.apache.org?
> >
> >> Sent: Tue, 3 Mar 2009 1:44 pm?
> >
> >> Subject: Re: what is needed to index for about 10000 domains?
> >
> >> ?
> >
> >> ?
> >
> >> ?
> >
> >> ?
> >
> >> ?
> >
> >> ?
> >
> >> ?
> >
> >> ?
> >
> >> ?
> >
> >> I think that in order to answer that questions, it is necessary to
> >> > know how many total pages are being indexed.??
> >
> >> ??
> >
> >> ?
> >
> >> I currently have ~3.5 million pages indexed, and the segment >
> >> directories are around 45GB, The response time is relatively fast.??
> >
> >> ??
> >
> >> ?
> >
> >> In the test site it is running on a dual processor Dell 1850 with >
> >> 3GB of RAM.??
> >
> >> ??
> >
> >> ?
> >
> >> -John??
> >
> >> ??
> >
> >> ?
> >
> >> On Mar 3, 2009, at 3:44 PM, alxsss@aim.com wrote:??
> >
> >> ??
> >
> >> ?
> >
> >>> Hello,??
> >
> >> ?
> >
> >>> ??
> >
> >> ?
> >
> >>> I use nutch-0.9 and need to index about 10000? domains.? I want to
> >>> >> > know? minimum requirements to hardware and memory.??
> >
> >> ?
> >
> >>> ??
> >
> >> ?
> >
> >>> Thanks in advance.??
> >
> >> ?
> >
> >>> Alex.??
> >
> >> ??
> >
> >> ?
> >
> >> ?
> >
> >> ?
> >
> >> ?
> >
> >> ?
> > ?
> >
> >
> >
> >
> >
>
>
--
Best Regards
Alexander Aristov