You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by liat oren <or...@gmail.com> on 2009/03/05 14:40:13 UTC
IndexSearcher
Hi,
I would like to do a search that will return documents that contain a given
word.
For example, I created the following index:
IndexWriter writer = new IndexWriter("C:/TryIndex", new StandardAnalyzer());
Document doc = new Document();
doc.add(new Field(WordIndex.FIELD_WORLDS, "111 222 333", Field.Store.YES,
Field.Index.UN_TOKENIZED));
writer.addDocument(doc);
doc = new Document();
doc.add(new Field(WordIndex.FIELD_WORLDS, "111", Field.Store.YES,
Field.Index.UN_TOKENIZED));
writer.addDocument(doc);
doc = new Document();
doc.add(new Field(WordIndex.FIELD_WORLDS, "222 333", Field.Store.YES,
Field.Index.UN_TOKENIZED));
writer.addDocument(doc);
writer.optimize();
writer.close();
now I want to get all the documents that contain the word "222".
I tried to run the following code but it doesn;t return any doc
IndexSearcher searcher = new IndexSearcher(indexPath);
// // TermQuery mapQuery = new TermQuery(new Term(FIELD_WORLDS,
worldNum)); - this one also didn't word
Analyzer analyzer = new StandardAnalyzer();
QueryParser parser = new QueryParser(FIELD_WORLDS, analyzer);
Query query = parser.parse(worldNum);
Hits mapHits = searcher.search(query);
Thanks a lot,
Liat
Re: IndexSearcher
Posted by Erick Erickson <er...@gmail.com>.
What Shashi said.....
On Sun, Mar 8, 2009 at 10:22 AM, Shashi Kant <sk...@sloan.mit.edu> wrote:
> Liat, i think what Erick suggested was to use the TOKENIZED setting instead
> of UN_TOKENIZED. For example your code should read something like:
>
> Document doc = new Document();
> doc.add(new Field(WordIndex.FIELD_WORLDS, "111 222 333", Field.Store.YES, *
> Field.Index.TOKENIZED*));
>
> Unless I am missing something , you do not need to modify Luke's code-base.
> You could however use it to open your index.
>
> Shashi
>
>
>
> On Sun, Mar 8, 2009 at 10:09 AM, Erick Erickson <erickerickson@gmail.com
> >wrote:
>
> > In Luke, you can select from among most of the analyzers provided by
> > Lucene, there's a drop-down on the search tab. And I'm sure you can
> > add your own custom ones, but I confess I've never tried.
> >
> > Best
> > Erick
> >
> > On Sun, Mar 8, 2009 at 4:14 AM, liat oren <or...@gmail.com> wrote:
> >
> > > Ok, thanks.
> > >
> > > I will have to edit the code of Luke in order to add another analyzer,
> > > right?
> > >
> > > If I need to query a long text - for example to search for articles
> that
> > > are
> > > textually close in their content, I need to parse to the query the
> text
> > of
> > > the article. Then I get the error that it is too long. Is there any way
> > to
> > > solve it?
> > >
> > > Thanks a lot,
> > > Liat
> > >
> > > 2009/3/5 Erick Erickson <er...@gmail.com>
> > >
> > > > I think your root problem is that you're indexing UN_TOKENIZED, which
> > > > means that the tokens you're adding to your index are NOT run through
> > > > the analyzer.
> > > >
> > > > So your terms are exactly "111", "222 333" and "111 222 333", none of
> > > which
> > > > match "222". I expect you wanted your tokens to be "111", "222", and
> > > "333",
> > > > each appearing twice in your index.
> > > >
> > > > Try indexing them tokenized. Although note that I don't remember what
> > > > StandardAnalyzer does with numbers. WhitespaceAnalyzer does the
> > > > more intuitive thing, but beware that it doesn't fold case. But it
> > might
> > > be
> > > > an easier place for you to start until you get more comfortable with
> > what
> > > > various analyzers do.
> > > >
> > > > Also, I *strongly* advise that you get a copy of Luke. It is a
> > wonderful
> > > > tool
> > > > that allows you to examine your index, analyze queries, test queries,
> > > etc.
> > > >
> > > > But be aware that the site that maintains Luke was having problems
> > > > yesterday,
> > > > look over the user list messages from yesterday if you have problems.
> > > >
> > > > Best
> > > > Erick
> > > >
> > > > On Thu, Mar 5, 2009 at 8:40 AM, liat oren <or...@gmail.com>
> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I would like to do a search that will return documents that contain
> a
> > > > given
> > > > > word.
> > > > > For example, I created the following index:
> > > > >
> > > > > IndexWriter writer = new IndexWriter("C:/TryIndex", new
> > > > > StandardAnalyzer());
> > > > > Document doc = new Document();
> > > > > doc.add(new Field(WordIndex.FIELD_WORLDS, "111 222 333",
> > > > Field.Store.YES,
> > > > > Field.Index.UN_TOKENIZED));
> > > > > writer.addDocument(doc);
> > > > > doc = new Document();
> > > > > doc.add(new Field(WordIndex.FIELD_WORLDS, "111", Field.Store.YES,
> > > > > Field.Index.UN_TOKENIZED));
> > > > > writer.addDocument(doc);
> > > > > doc = new Document();
> > > > > doc.add(new Field(WordIndex.FIELD_WORLDS, "222 333",
> > Field.Store.YES,
> > > > > Field.Index.UN_TOKENIZED));
> > > > > writer.addDocument(doc);
> > > > > writer.optimize();
> > > > > writer.close();
> > > > >
> > > > > now I want to get all the documents that contain the word "222".
> > > > >
> > > > > I tried to run the following code but it doesn;t return any doc
> > > > >
> > > > > IndexSearcher searcher = new IndexSearcher(indexPath);
> > > > >
> > > > > // // TermQuery mapQuery = new TermQuery(new Term(FIELD_WORLDS,
> > > > > worldNum)); - this one also didn't word
> > > > > Analyzer analyzer = new StandardAnalyzer();
> > > > > QueryParser parser = new QueryParser(FIELD_WORLDS, analyzer);
> > > > > Query query = parser.parse(worldNum);
> > > > > Hits mapHits = searcher.search(query);
> > > > >
> > > > >
> > > > > Thanks a lot,
> > > > > Liat
> > > > >
> > > >
> > >
> >
>
Re: IndexSearcher
Posted by Shashi Kant <sk...@sloan.mit.edu>.
Liat, i think what Erick suggested was to use the TOKENIZED setting instead
of UN_TOKENIZED. For example your code should read something like:
Document doc = new Document();
doc.add(new Field(WordIndex.FIELD_WORLDS, "111 222 333", Field.Store.YES, *
Field.Index.TOKENIZED*));
Unless I am missing something , you do not need to modify Luke's code-base.
You could however use it to open your index.
Shashi
On Sun, Mar 8, 2009 at 10:09 AM, Erick Erickson <er...@gmail.com>wrote:
> In Luke, you can select from among most of the analyzers provided by
> Lucene, there's a drop-down on the search tab. And I'm sure you can
> add your own custom ones, but I confess I've never tried.
>
> Best
> Erick
>
> On Sun, Mar 8, 2009 at 4:14 AM, liat oren <or...@gmail.com> wrote:
>
> > Ok, thanks.
> >
> > I will have to edit the code of Luke in order to add another analyzer,
> > right?
> >
> > If I need to query a long text - for example to search for articles that
> > are
> > textually close in their content, I need to parse to the query the text
> of
> > the article. Then I get the error that it is too long. Is there any way
> to
> > solve it?
> >
> > Thanks a lot,
> > Liat
> >
> > 2009/3/5 Erick Erickson <er...@gmail.com>
> >
> > > I think your root problem is that you're indexing UN_TOKENIZED, which
> > > means that the tokens you're adding to your index are NOT run through
> > > the analyzer.
> > >
> > > So your terms are exactly "111", "222 333" and "111 222 333", none of
> > which
> > > match "222". I expect you wanted your tokens to be "111", "222", and
> > "333",
> > > each appearing twice in your index.
> > >
> > > Try indexing them tokenized. Although note that I don't remember what
> > > StandardAnalyzer does with numbers. WhitespaceAnalyzer does the
> > > more intuitive thing, but beware that it doesn't fold case. But it
> might
> > be
> > > an easier place for you to start until you get more comfortable with
> what
> > > various analyzers do.
> > >
> > > Also, I *strongly* advise that you get a copy of Luke. It is a
> wonderful
> > > tool
> > > that allows you to examine your index, analyze queries, test queries,
> > etc.
> > >
> > > But be aware that the site that maintains Luke was having problems
> > > yesterday,
> > > look over the user list messages from yesterday if you have problems.
> > >
> > > Best
> > > Erick
> > >
> > > On Thu, Mar 5, 2009 at 8:40 AM, liat oren <or...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I would like to do a search that will return documents that contain a
> > > given
> > > > word.
> > > > For example, I created the following index:
> > > >
> > > > IndexWriter writer = new IndexWriter("C:/TryIndex", new
> > > > StandardAnalyzer());
> > > > Document doc = new Document();
> > > > doc.add(new Field(WordIndex.FIELD_WORLDS, "111 222 333",
> > > Field.Store.YES,
> > > > Field.Index.UN_TOKENIZED));
> > > > writer.addDocument(doc);
> > > > doc = new Document();
> > > > doc.add(new Field(WordIndex.FIELD_WORLDS, "111", Field.Store.YES,
> > > > Field.Index.UN_TOKENIZED));
> > > > writer.addDocument(doc);
> > > > doc = new Document();
> > > > doc.add(new Field(WordIndex.FIELD_WORLDS, "222 333",
> Field.Store.YES,
> > > > Field.Index.UN_TOKENIZED));
> > > > writer.addDocument(doc);
> > > > writer.optimize();
> > > > writer.close();
> > > >
> > > > now I want to get all the documents that contain the word "222".
> > > >
> > > > I tried to run the following code but it doesn;t return any doc
> > > >
> > > > IndexSearcher searcher = new IndexSearcher(indexPath);
> > > >
> > > > // // TermQuery mapQuery = new TermQuery(new Term(FIELD_WORLDS,
> > > > worldNum)); - this one also didn't word
> > > > Analyzer analyzer = new StandardAnalyzer();
> > > > QueryParser parser = new QueryParser(FIELD_WORLDS, analyzer);
> > > > Query query = parser.parse(worldNum);
> > > > Hits mapHits = searcher.search(query);
> > > >
> > > >
> > > > Thanks a lot,
> > > > Liat
> > > >
> > >
> >
>
Re: IndexSearcher
Posted by Erick Erickson <er...@gmail.com>.
In Luke, you can select from among most of the analyzers provided by
Lucene, there's a drop-down on the search tab. And I'm sure you can
add your own custom ones, but I confess I've never tried.
Best
Erick
On Sun, Mar 8, 2009 at 4:14 AM, liat oren <or...@gmail.com> wrote:
> Ok, thanks.
>
> I will have to edit the code of Luke in order to add another analyzer,
> right?
>
> If I need to query a long text - for example to search for articles that
> are
> textually close in their content, I need to parse to the query the text of
> the article. Then I get the error that it is too long. Is there any way to
> solve it?
>
> Thanks a lot,
> Liat
>
> 2009/3/5 Erick Erickson <er...@gmail.com>
>
> > I think your root problem is that you're indexing UN_TOKENIZED, which
> > means that the tokens you're adding to your index are NOT run through
> > the analyzer.
> >
> > So your terms are exactly "111", "222 333" and "111 222 333", none of
> which
> > match "222". I expect you wanted your tokens to be "111", "222", and
> "333",
> > each appearing twice in your index.
> >
> > Try indexing them tokenized. Although note that I don't remember what
> > StandardAnalyzer does with numbers. WhitespaceAnalyzer does the
> > more intuitive thing, but beware that it doesn't fold case. But it might
> be
> > an easier place for you to start until you get more comfortable with what
> > various analyzers do.
> >
> > Also, I *strongly* advise that you get a copy of Luke. It is a wonderful
> > tool
> > that allows you to examine your index, analyze queries, test queries,
> etc.
> >
> > But be aware that the site that maintains Luke was having problems
> > yesterday,
> > look over the user list messages from yesterday if you have problems.
> >
> > Best
> > Erick
> >
> > On Thu, Mar 5, 2009 at 8:40 AM, liat oren <or...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I would like to do a search that will return documents that contain a
> > given
> > > word.
> > > For example, I created the following index:
> > >
> > > IndexWriter writer = new IndexWriter("C:/TryIndex", new
> > > StandardAnalyzer());
> > > Document doc = new Document();
> > > doc.add(new Field(WordIndex.FIELD_WORLDS, "111 222 333",
> > Field.Store.YES,
> > > Field.Index.UN_TOKENIZED));
> > > writer.addDocument(doc);
> > > doc = new Document();
> > > doc.add(new Field(WordIndex.FIELD_WORLDS, "111", Field.Store.YES,
> > > Field.Index.UN_TOKENIZED));
> > > writer.addDocument(doc);
> > > doc = new Document();
> > > doc.add(new Field(WordIndex.FIELD_WORLDS, "222 333", Field.Store.YES,
> > > Field.Index.UN_TOKENIZED));
> > > writer.addDocument(doc);
> > > writer.optimize();
> > > writer.close();
> > >
> > > now I want to get all the documents that contain the word "222".
> > >
> > > I tried to run the following code but it doesn;t return any doc
> > >
> > > IndexSearcher searcher = new IndexSearcher(indexPath);
> > >
> > > // // TermQuery mapQuery = new TermQuery(new Term(FIELD_WORLDS,
> > > worldNum)); - this one also didn't word
> > > Analyzer analyzer = new StandardAnalyzer();
> > > QueryParser parser = new QueryParser(FIELD_WORLDS, analyzer);
> > > Query query = parser.parse(worldNum);
> > > Hits mapHits = searcher.search(query);
> > >
> > >
> > > Thanks a lot,
> > > Liat
> > >
> >
>
RE: Hadopp Config Exception in Nutch
Posted by "Lukas, Ray" <Ra...@idearc.com>.
Okay.. I am switching over right now.. RIGHT NOW!
-----Original Message-----
From: Bartosz Gadzimski [mailto:bartek--g@o2.pl]
Sent: Tuesday, March 10, 2009 8:29 AM
To: nutch-user@lucene.apache.org
Subject: Re: Hadopp Config Exception in Nutch
Hi,
I can use it from cygwin, from eclipse or anyway you wan't it. Crawl,
nutchBean also.
You should try nutch trunk or even rc
http://people.apache.org/~siren/nutch-1.0/rc1/nutch-1.0.tar.gz
It's to much difference to write here, it's just 10 times better than
0.9
Lukas, Ray pisze:
> Oh rats.. Sorry.. Early morning here.. Forgot.. Yes, version.. You
need
> to know that.. I am running Nutch 0.9.. I have gone through the
> tutorial.. I have not done as well as you have..
>
> Yep set up that search.dir setting.. I am going to go through
everything
> from scratch again I think.. I am missing something..
> So you have search from your java app working then I take it..
Hummm...
> Very interesting.. Hum.. Would sure like to spend 15 minutes with
you..
> Ha.. Ah.. Okay.. So there is hope for me.. Good good.. I am really
> close.. Really close.. I can taste it..
> Thanks man.. I am going back throguh that tutorial.. Must have missed
> something..
>
> -----Original Message-----
> From: Bartosz Gadzimski [mailto:bartek--g@o2.pl]
> Sent: Tuesday, March 10, 2009 8:02 AM
> To: nutch-user@lucene.apache.org
> Subject: Re: Hadopp Config Exception in Nutch
>
> Hi,
>
> Which version of nutch are you using?
>
> You have wiki tutorial on running nutch in eclipse (it's important to
> add conf dir to classpath and move it to top of loading libs)
> http://wiki.apache.org/nutch/RunNutchInEclipse0.9
>
> I've installed nutch rc in eclipse on windows just 2 hours ago and
it's
> working great.
>
> Hope you set searcher.dir on you nutch-site.xml properly (full path to
> your crawl dir)
>
> Thanks,
> Bartosz
>
> Lukas, Ray pisze:
>
>> Has anyone seen this.. Do you know the solution.. I will start
looking
>> through the hadopp code but if someone has fixed this already I would
>> appreciate knowing.. Thanks guys..
>>
>> Fri Mar 6 14:48:40 2009 DEBUG main java.io.IOException: config()
>> at
>> org.apache.hadoop.conf.Configuration.<init>(Configuration.java:93)
>> at
>>
>>
>
org.apache.nutch.util.NutchConfiguration.create(NutchConfiguration.java:
>
>> 51)
>> at IndexCreator.<init>(IndexCreator.java:66)
>> at Driver.createIndex(Driver.java:17)
>> at Driver.main(Driver.java:10)
>>
>> and
>>
>> Fri Mar 6 14:48:40 2009 DEBUG main java.io.IOException:
config(config)
>> at
>> org.apache.hadoop.conf.Configuration.<init>(Configuration.java:102)
>> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:77)
>> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:88)
>> at org.apache.nutch.util.NutchJob.<init>(NutchJob.java:27)
>> at IndexCreator.<init>(IndexCreator.java:68)
>> at Driver.createIndex(Driver.java:17)
>> at Driver.main(Driver.java:10)
>>
>> Ray
>>
>> Some background.. I am working in Eclipse, on windows. Put the jar
>>
> files
>
>> in lib directory from the nutch distribution, added the plug-in
>> directory and nutch seems to generate indexes which Luke can open
even
>> though this happens.. But NutchBean search always fails to find
>> anything, (returns zero hit objects) even though Luke finds and
>>
> displays
>
>> the indexes without a problem.. I am thinking index dir is okay but
>>
> the
>
>> other things are not, so Nutch Bean is something that might be
relying
>> on those "other things" and so I am seeking to resolve this..
>> Set up log4J correctly and now I am finally seeing these. Thought I
>> would ask if anyone else has run across this... Sorry I am new to
>>
> Nutch,
>
>> but I don't want to stay that way..
>>
>>
>>
>
>
>
Re: Hadopp Config Exception in Nutch
Posted by Bartosz Gadzimski <ba...@o2.pl>.
Hi,
I can use it from cygwin, from eclipse or anyway you wan't it. Crawl,
nutchBean also.
You should try nutch trunk or even rc
http://people.apache.org/~siren/nutch-1.0/rc1/nutch-1.0.tar.gz
It's to much difference to write here, it's just 10 times better than 0.9
Lukas, Ray pisze:
> Oh rats.. Sorry.. Early morning here.. Forgot.. Yes, version.. You need
> to know that.. I am running Nutch 0.9.. I have gone through the
> tutorial.. I have not done as well as you have..
>
> Yep set up that search.dir setting.. I am going to go through everything
> from scratch again I think.. I am missing something..
> So you have search from your java app working then I take it.. Hummm...
> Very interesting.. Hum.. Would sure like to spend 15 minutes with you..
> Ha.. Ah.. Okay.. So there is hope for me.. Good good.. I am really
> close.. Really close.. I can taste it..
> Thanks man.. I am going back throguh that tutorial.. Must have missed
> something..
>
> -----Original Message-----
> From: Bartosz Gadzimski [mailto:bartek--g@o2.pl]
> Sent: Tuesday, March 10, 2009 8:02 AM
> To: nutch-user@lucene.apache.org
> Subject: Re: Hadopp Config Exception in Nutch
>
> Hi,
>
> Which version of nutch are you using?
>
> You have wiki tutorial on running nutch in eclipse (it's important to
> add conf dir to classpath and move it to top of loading libs)
> http://wiki.apache.org/nutch/RunNutchInEclipse0.9
>
> I've installed nutch rc in eclipse on windows just 2 hours ago and it's
> working great.
>
> Hope you set searcher.dir on you nutch-site.xml properly (full path to
> your crawl dir)
>
> Thanks,
> Bartosz
>
> Lukas, Ray pisze:
>
>> Has anyone seen this.. Do you know the solution.. I will start looking
>> through the hadopp code but if someone has fixed this already I would
>> appreciate knowing.. Thanks guys..
>>
>> Fri Mar 6 14:48:40 2009 DEBUG main java.io.IOException: config()
>> at
>> org.apache.hadoop.conf.Configuration.<init>(Configuration.java:93)
>> at
>>
>>
> org.apache.nutch.util.NutchConfiguration.create(NutchConfiguration.java:
>
>> 51)
>> at IndexCreator.<init>(IndexCreator.java:66)
>> at Driver.createIndex(Driver.java:17)
>> at Driver.main(Driver.java:10)
>>
>> and
>>
>> Fri Mar 6 14:48:40 2009 DEBUG main java.io.IOException: config(config)
>> at
>> org.apache.hadoop.conf.Configuration.<init>(Configuration.java:102)
>> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:77)
>> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:88)
>> at org.apache.nutch.util.NutchJob.<init>(NutchJob.java:27)
>> at IndexCreator.<init>(IndexCreator.java:68)
>> at Driver.createIndex(Driver.java:17)
>> at Driver.main(Driver.java:10)
>>
>> Ray
>>
>> Some background.. I am working in Eclipse, on windows. Put the jar
>>
> files
>
>> in lib directory from the nutch distribution, added the plug-in
>> directory and nutch seems to generate indexes which Luke can open even
>> though this happens.. But NutchBean search always fails to find
>> anything, (returns zero hit objects) even though Luke finds and
>>
> displays
>
>> the indexes without a problem.. I am thinking index dir is okay but
>>
> the
>
>> other things are not, so Nutch Bean is something that might be relying
>> on those "other things" and so I am seeking to resolve this..
>> Set up log4J correctly and now I am finally seeing these. Thought I
>> would ask if anyone else has run across this... Sorry I am new to
>>
> Nutch,
>
>> but I don't want to stay that way..
>>
>>
>>
>
>
>
RE: Hadopp Config Exception in Nutch
Posted by "Lukas, Ray" <Ra...@idearc.com>.
Oh rats.. Sorry.. Early morning here.. Forgot.. Yes, version.. You need
to know that.. I am running Nutch 0.9.. I have gone through the
tutorial.. I have not done as well as you have..
Yep set up that search.dir setting.. I am going to go through everything
from scratch again I think.. I am missing something..
So you have search from your java app working then I take it.. Hummm...
Very interesting.. Hum.. Would sure like to spend 15 minutes with you..
Ha.. Ah.. Okay.. So there is hope for me.. Good good.. I am really
close.. Really close.. I can taste it..
Thanks man.. I am going back throguh that tutorial.. Must have missed
something..
-----Original Message-----
From: Bartosz Gadzimski [mailto:bartek--g@o2.pl]
Sent: Tuesday, March 10, 2009 8:02 AM
To: nutch-user@lucene.apache.org
Subject: Re: Hadopp Config Exception in Nutch
Hi,
Which version of nutch are you using?
You have wiki tutorial on running nutch in eclipse (it's important to
add conf dir to classpath and move it to top of loading libs)
http://wiki.apache.org/nutch/RunNutchInEclipse0.9
I've installed nutch rc in eclipse on windows just 2 hours ago and it's
working great.
Hope you set searcher.dir on you nutch-site.xml properly (full path to
your crawl dir)
Thanks,
Bartosz
Lukas, Ray pisze:
> Has anyone seen this.. Do you know the solution.. I will start looking
> through the hadopp code but if someone has fixed this already I would
> appreciate knowing.. Thanks guys..
>
> Fri Mar 6 14:48:40 2009 DEBUG main java.io.IOException: config()
> at
> org.apache.hadoop.conf.Configuration.<init>(Configuration.java:93)
> at
>
org.apache.nutch.util.NutchConfiguration.create(NutchConfiguration.java:
> 51)
> at IndexCreator.<init>(IndexCreator.java:66)
> at Driver.createIndex(Driver.java:17)
> at Driver.main(Driver.java:10)
>
> and
>
> Fri Mar 6 14:48:40 2009 DEBUG main java.io.IOException: config(config)
> at
> org.apache.hadoop.conf.Configuration.<init>(Configuration.java:102)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:77)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:88)
> at org.apache.nutch.util.NutchJob.<init>(NutchJob.java:27)
> at IndexCreator.<init>(IndexCreator.java:68)
> at Driver.createIndex(Driver.java:17)
> at Driver.main(Driver.java:10)
>
> Ray
>
> Some background.. I am working in Eclipse, on windows. Put the jar
files
> in lib directory from the nutch distribution, added the plug-in
> directory and nutch seems to generate indexes which Luke can open even
> though this happens.. But NutchBean search always fails to find
> anything, (returns zero hit objects) even though Luke finds and
displays
> the indexes without a problem.. I am thinking index dir is okay but
the
> other things are not, so Nutch Bean is something that might be relying
> on those "other things" and so I am seeking to resolve this..
> Set up log4J correctly and now I am finally seeing these. Thought I
> would ask if anyone else has run across this... Sorry I am new to
Nutch,
> but I don't want to stay that way..
>
>
Re: Hadopp Config Exception in Nutch
Posted by Bartosz Gadzimski <ba...@o2.pl>.
Hi,
Which version of nutch are you using?
You have wiki tutorial on running nutch in eclipse (it's important to
add conf dir to classpath and move it to top of loading libs)
http://wiki.apache.org/nutch/RunNutchInEclipse0.9
I've installed nutch rc in eclipse on windows just 2 hours ago and it's
working great.
Hope you set searcher.dir on you nutch-site.xml properly (full path to
your crawl dir)
Thanks,
Bartosz
Lukas, Ray pisze:
> Has anyone seen this.. Do you know the solution.. I will start looking
> through the hadopp code but if someone has fixed this already I would
> appreciate knowing.. Thanks guys..
>
> Fri Mar 6 14:48:40 2009 DEBUG main java.io.IOException: config()
> at
> org.apache.hadoop.conf.Configuration.<init>(Configuration.java:93)
> at
> org.apache.nutch.util.NutchConfiguration.create(NutchConfiguration.java:
> 51)
> at IndexCreator.<init>(IndexCreator.java:66)
> at Driver.createIndex(Driver.java:17)
> at Driver.main(Driver.java:10)
>
> and
>
> Fri Mar 6 14:48:40 2009 DEBUG main java.io.IOException: config(config)
> at
> org.apache.hadoop.conf.Configuration.<init>(Configuration.java:102)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:77)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:88)
> at org.apache.nutch.util.NutchJob.<init>(NutchJob.java:27)
> at IndexCreator.<init>(IndexCreator.java:68)
> at Driver.createIndex(Driver.java:17)
> at Driver.main(Driver.java:10)
>
> Ray
>
> Some background.. I am working in Eclipse, on windows. Put the jar files
> in lib directory from the nutch distribution, added the plug-in
> directory and nutch seems to generate indexes which Luke can open even
> though this happens.. But NutchBean search always fails to find
> anything, (returns zero hit objects) even though Luke finds and displays
> the indexes without a problem.. I am thinking index dir is okay but the
> other things are not, so Nutch Bean is something that might be relying
> on those "other things" and so I am seeking to resolve this..
> Set up log4J correctly and now I am finally seeing these. Thought I
> would ask if anyone else has run across this... Sorry I am new to Nutch,
> but I don't want to stay that way..
>
>
Hadopp Config Exception in Nutch
Posted by "Lukas, Ray" <Ra...@idearc.com>.
Has anyone seen this.. Do you know the solution.. I will start looking
through the hadopp code but if someone has fixed this already I would
appreciate knowing.. Thanks guys..
Fri Mar 6 14:48:40 2009 DEBUG main java.io.IOException: config()
at
org.apache.hadoop.conf.Configuration.<init>(Configuration.java:93)
at
org.apache.nutch.util.NutchConfiguration.create(NutchConfiguration.java:
51)
at IndexCreator.<init>(IndexCreator.java:66)
at Driver.createIndex(Driver.java:17)
at Driver.main(Driver.java:10)
and
Fri Mar 6 14:48:40 2009 DEBUG main java.io.IOException: config(config)
at
org.apache.hadoop.conf.Configuration.<init>(Configuration.java:102)
at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:77)
at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:88)
at org.apache.nutch.util.NutchJob.<init>(NutchJob.java:27)
at IndexCreator.<init>(IndexCreator.java:68)
at Driver.createIndex(Driver.java:17)
at Driver.main(Driver.java:10)
Ray
Some background.. I am working in Eclipse, on windows. Put the jar files
in lib directory from the nutch distribution, added the plug-in
directory and nutch seems to generate indexes which Luke can open even
though this happens.. But NutchBean search always fails to find
anything, (returns zero hit objects) even though Luke finds and displays
the indexes without a problem.. I am thinking index dir is okay but the
other things are not, so Nutch Bean is something that might be relying
on those "other things" and so I am seeking to resolve this..
Set up log4J correctly and now I am finally seeing these. Thought I
would ask if anyone else has run across this... Sorry I am new to Nutch,
but I don't want to stay that way..
Re: IndexSearcher
Posted by liat oren <or...@gmail.com>.
Thanks!
2009/3/9 Andrzej Bialecki <ab...@getopt.org>
> liat oren wrote:
>
>> Yes, I changed it to TOKENIZED and its working now, Thanks!
>>
>> About Luke, what do you mean by saying that the analyzer is in the
>> classpath?
>> It exists in a package in my computer - it also has its filter and other
>> classes. How can it be used in Luke?
>>
>
> You need to add this jar to the classpath when you start Luke.
>
>
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: IndexSearcher
Posted by Andrzej Bialecki <ab...@getopt.org>.
liat oren wrote:
> Yes, I changed it to TOKENIZED and its working now, Thanks!
>
> About Luke, what do you mean by saying that the analyzer is in the
> classpath?
> It exists in a package in my computer - it also has its filter and other
> classes. How can it be used in Luke?
You need to add this jar to the classpath when you start Luke.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexSearcher
Posted by liat oren <or...@gmail.com>.
Yes, I changed it to TOKENIZED and its working now, Thanks!
About Luke, what do you mean by saying that the analyzer is in the
classpath?
It exists in a package in my computer - it also has its filter and other
classes. How can it be used in Luke?
2009/3/8 Andrzej Bialecki <ab...@getopt.org>
> liat oren wrote:
>
>> Ok, thanks.
>>
>> I will have to edit the code of Luke in order to add another analyzer,
>> right?
>>
>
> No - if your analyzer is already on the classpath, then it's enough to type
> in the fully qualified class name in the drop down box (it's editable).
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: IndexSearcher
Posted by Andrzej Bialecki <ab...@getopt.org>.
liat oren wrote:
> Ok, thanks.
>
> I will have to edit the code of Luke in order to add another analyzer,
> right?
No - if your analyzer is already on the classpath, then it's enough to
type in the fully qualified class name in the drop down box (it's editable).
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexSearcher
Posted by liat oren <or...@gmail.com>.
Ok, thanks.
I will have to edit the code of Luke in order to add another analyzer,
right?
If I need to query a long text - for example to search for articles that are
textually close in their content, I need to parse to the query the text of
the article. Then I get the error that it is too long. Is there any way to
solve it?
Thanks a lot,
Liat
2009/3/5 Erick Erickson <er...@gmail.com>
> I think your root problem is that you're indexing UN_TOKENIZED, which
> means that the tokens you're adding to your index are NOT run through
> the analyzer.
>
> So your terms are exactly "111", "222 333" and "111 222 333", none of which
> match "222". I expect you wanted your tokens to be "111", "222", and "333",
> each appearing twice in your index.
>
> Try indexing them tokenized. Although note that I don't remember what
> StandardAnalyzer does with numbers. WhitespaceAnalyzer does the
> more intuitive thing, but beware that it doesn't fold case. But it might be
> an easier place for you to start until you get more comfortable with what
> various analyzers do.
>
> Also, I *strongly* advise that you get a copy of Luke. It is a wonderful
> tool
> that allows you to examine your index, analyze queries, test queries, etc.
>
> But be aware that the site that maintains Luke was having problems
> yesterday,
> look over the user list messages from yesterday if you have problems.
>
> Best
> Erick
>
> On Thu, Mar 5, 2009 at 8:40 AM, liat oren <or...@gmail.com> wrote:
>
> > Hi,
> >
> > I would like to do a search that will return documents that contain a
> given
> > word.
> > For example, I created the following index:
> >
> > IndexWriter writer = new IndexWriter("C:/TryIndex", new
> > StandardAnalyzer());
> > Document doc = new Document();
> > doc.add(new Field(WordIndex.FIELD_WORLDS, "111 222 333",
> Field.Store.YES,
> > Field.Index.UN_TOKENIZED));
> > writer.addDocument(doc);
> > doc = new Document();
> > doc.add(new Field(WordIndex.FIELD_WORLDS, "111", Field.Store.YES,
> > Field.Index.UN_TOKENIZED));
> > writer.addDocument(doc);
> > doc = new Document();
> > doc.add(new Field(WordIndex.FIELD_WORLDS, "222 333", Field.Store.YES,
> > Field.Index.UN_TOKENIZED));
> > writer.addDocument(doc);
> > writer.optimize();
> > writer.close();
> >
> > now I want to get all the documents that contain the word "222".
> >
> > I tried to run the following code but it doesn;t return any doc
> >
> > IndexSearcher searcher = new IndexSearcher(indexPath);
> >
> > // // TermQuery mapQuery = new TermQuery(new Term(FIELD_WORLDS,
> > worldNum)); - this one also didn't word
> > Analyzer analyzer = new StandardAnalyzer();
> > QueryParser parser = new QueryParser(FIELD_WORLDS, analyzer);
> > Query query = parser.parse(worldNum);
> > Hits mapHits = searcher.search(query);
> >
> >
> > Thanks a lot,
> > Liat
> >
>
Re: IndexSearcher
Posted by Erick Erickson <er...@gmail.com>.
I think your root problem is that you're indexing UN_TOKENIZED, which
means that the tokens you're adding to your index are NOT run through
the analyzer.
So your terms are exactly "111", "222 333" and "111 222 333", none of which
match "222". I expect you wanted your tokens to be "111", "222", and "333",
each appearing twice in your index.
Try indexing them tokenized. Although note that I don't remember what
StandardAnalyzer does with numbers. WhitespaceAnalyzer does the
more intuitive thing, but beware that it doesn't fold case. But it might be
an easier place for you to start until you get more comfortable with what
various analyzers do.
Also, I *strongly* advise that you get a copy of Luke. It is a wonderful
tool
that allows you to examine your index, analyze queries, test queries, etc.
But be aware that the site that maintains Luke was having problems
yesterday,
look over the user list messages from yesterday if you have problems.
Best
Erick
On Thu, Mar 5, 2009 at 8:40 AM, liat oren <or...@gmail.com> wrote:
> Hi,
>
> I would like to do a search that will return documents that contain a given
> word.
> For example, I created the following index:
>
> IndexWriter writer = new IndexWriter("C:/TryIndex", new
> StandardAnalyzer());
> Document doc = new Document();
> doc.add(new Field(WordIndex.FIELD_WORLDS, "111 222 333", Field.Store.YES,
> Field.Index.UN_TOKENIZED));
> writer.addDocument(doc);
> doc = new Document();
> doc.add(new Field(WordIndex.FIELD_WORLDS, "111", Field.Store.YES,
> Field.Index.UN_TOKENIZED));
> writer.addDocument(doc);
> doc = new Document();
> doc.add(new Field(WordIndex.FIELD_WORLDS, "222 333", Field.Store.YES,
> Field.Index.UN_TOKENIZED));
> writer.addDocument(doc);
> writer.optimize();
> writer.close();
>
> now I want to get all the documents that contain the word "222".
>
> I tried to run the following code but it doesn;t return any doc
>
> IndexSearcher searcher = new IndexSearcher(indexPath);
>
> // // TermQuery mapQuery = new TermQuery(new Term(FIELD_WORLDS,
> worldNum)); - this one also didn't word
> Analyzer analyzer = new StandardAnalyzer();
> QueryParser parser = new QueryParser(FIELD_WORLDS, analyzer);
> Query query = parser.parse(worldNum);
> Hits mapHits = searcher.search(query);
>
>
> Thanks a lot,
> Liat
>