You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by "axel.reymonet" <ax...@actia.fr> on 2007/04/30 11:22:51 UTC
Spanquery problem
Hello,
I am having some issues with the SpanQuery functionality. As a matter of
fact, I index a single french file containing for instance "climatisation
automatique" (which means automatic air-conditioning) with the classical
FrenchAnalyzer, and when I search in this index with SpanQuery, I have the
following situation :
- I have 1 span for the "climatisation" request
- I have 1 span for the "automatique" request
- I have 0 span for the "climatisation automatique" request
Maybe I am doing something wrong, but I cannot spot my mistake. I have given
the problematic portion of code below. Does anyone have an idea ? Thanks in
advance,
Axel Reymonet
My code is as follows :
- FOR INDEXING
public void fileProcess(String filepath)
{
File index_dir = new File(this.currentProjectPath+"index");
if (index_dir.exists())
{
String[] files = index_dir.list();
for (String f:files)
{
File file = new File (this.currentProjectPath+"index/"+f);
file.delete();
}
index_dir.delete();
}
File toBeIndexed = new File(filepath);
if (!toBeIndexed.exists() || !toBeIndexed.canRead())
{
System.out.println("Document directory '"
+toBeIndexed.getAbsolutePath()+ "' does not exist or is not readable, please
check the path");
System.exit(1);
}
try {
IndexWriter writer = new IndexWriter(index_dir, new
org.apache.lucene.analysis.fr.FrenchAnalyzer(),true);
writer.addDocument(FileDocument.Document(toBeIndexed));
writer.close();
} catch (IOException e) {System.out.println(" caught a " + e.getClass() +"\n
with message: " + e.getMessage());}
}
-----------------------------------------
- FOR SEARCHING
public void testSF(String searched,String indexPath)
{
try{
IndexReader reader = IndexReader.open(indexPath);
Analyzer analyzer = new org.apache.lucene.analysis.fr.FrenchAnalyzer();
TokenStream requestStream = analyzer.tokenStream("contents",new
StringReader(searched));
HashSet<SpanTermQuery> qSet = new HashSet<SpanTermQuery>();
Token currentToken = requestStream.next();
while (currentToken!=null)
{
qSet.add(new SpanTermQuery(new
Term("contents",currentToken.termText())));
currentToken = requestStream.next();
}
SpanQuery[] sQ = new SpanQuery [qSet.size()];
int k = 0;
for (SpanTermQuery stq:qSet)
{
sQ[k]=stq;
k++;
}
SpanNearQuery snQ;
snQ = new SpanNearQuery(sQ,0,true);
Spans spans = snQ.getSpans(reader);
int resultsCpt = 0;
while (spans.next())
resultsCpt++;
System.out.println("Number of results: "+resultsCpt);
}
catch(IOException ioe){ioe.printStackTrace();}
}
--------------------------------------------
- OUTPUT
Query: climatisation
Number of results: 1
Query: automatique
Number of results: 1
Query: climatisation automatique
Number of results: 0
Re: How to index a lot of fields (without FileNotFoundException: Too many open files)
Posted by Paul Elschot <pa...@xs4all.nl>.
On Tuesday 01 May 2007 00:22, pbm-rico@gmx.net wrote:
> I thought about using ulimit, but it does not scale. In the scenario that
the app has to support, client applications could create hundreds of
thousands of unique properties, which would result in this many indexable
fields.
>
> Based on previous answers, the way out of this problem while still being
scalable would be to use 2.1 where there is only 1 norm file.
>
> However, it does not look like upgrading is an option, so I wonder if my
current approach of mapping a property that a client app creates to one field
name is workable at all. Maybe I have to introduce some sort of mapping of
client properties to a fixed number of indexable fields.
>
> ...or modify the 1.4.3 code myself to get rid of norm files.
Do you need the idf computations of lucene for each of these client fields
separately?
When not, another way is to move the names of the client fields into the
term value, and use a single special field for these.
When you do that, you'll also have to move the client field names
in the queries from the field name to the term. This can easily be done by
overriding one of the methods in QueryParser.
Regards,
Paul Elschot
>
> -Rico
>
> -------- Original-Nachricht --------
> Datum: Mon, 30 Apr 2007 15:08:14 -0700
> Von: "Mike Klaas" <mi...@gmail.com>
> An: java-user@lucene.apache.org
> Betreff: Re: Re: How to index a lot of fields (without
FileNotFoundException: Too many open files)
>
> > On 4/30/07, pbm-rico@gmx.net <pb...@gmx.net> wrote:
> > > Thanks for you reply.
> > >
> > > We are still using Lucene v1.4.3 and I'm not sure if upgrading is an
> > option. Is there another way of disabling length normalization/document
boosts
> > to get rid of those files?
> >
> > Why not raise the limit of open files on your system? (man ulimit)
> >
> > -Mike
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
>
> --
> "Feel free" - 10 GB Mailbox, 100 FreeSMS/Monat ...
> Jetzt GMX TopMail testen: http://www.gmx.net/de/go/topmail
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Re: How to index a lot of fields (without FileNotFoundException:
Too many open files)
Posted by Chris Hostetter <ho...@fucit.org>.
: However, it does not look like upgrading is an option, so I wonder if my
: current approach of mapping a property that a client app creates to one
: field name is workable at all. Maybe I have to introduce some sort of
: mapping of client properties to a fixed number of indexable fields.
:
: ...or modify the 1.4.3 code myself to get rid of norm files.
if you can't upgrade past 1.4.3, but you can modify 1.4.3 to change
things (like file formats), why not "modify" your 1.4.3 install to be a
2.1 install?
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Re: How to index a lot of fields (without FileNotFoundException:
Too many open files)
Posted by pb...@gmx.net.
I thought about using ulimit, but it does not scale. In the scenario that the app has to support, client applications could create hundreds of thousands of unique properties, which would result in this many indexable fields.
Based on previous answers, the way out of this problem while still being scalable would be to use 2.1 where there is only 1 norm file.
However, it does not look like upgrading is an option, so I wonder if my current approach of mapping a property that a client app creates to one field name is workable at all. Maybe I have to introduce some sort of mapping of client properties to a fixed number of indexable fields.
...or modify the 1.4.3 code myself to get rid of norm files.
-Rico
-------- Original-Nachricht --------
Datum: Mon, 30 Apr 2007 15:08:14 -0700
Von: "Mike Klaas" <mi...@gmail.com>
An: java-user@lucene.apache.org
Betreff: Re: Re: How to index a lot of fields (without FileNotFoundException: Too many open files)
> On 4/30/07, pbm-rico@gmx.net <pb...@gmx.net> wrote:
> > Thanks for you reply.
> >
> > We are still using Lucene v1.4.3 and I'm not sure if upgrading is an
> option. Is there another way of disabling length normalization/document boosts
> to get rid of those files?
>
> Why not raise the limit of open files on your system? (man ulimit)
>
> -Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
--
"Feel free" - 10 GB Mailbox, 100 FreeSMS/Monat ...
Jetzt GMX TopMail testen: http://www.gmx.net/de/go/topmail
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Re: How to index a lot of fields (without FileNotFoundException: Too many open files)
Posted by Mike Klaas <mi...@gmail.com>.
On 4/30/07, pbm-rico@gmx.net <pb...@gmx.net> wrote:
> Thanks for you reply.
>
> We are still using Lucene v1.4.3 and I'm not sure if upgrading is an option. Is there another way of disabling length normalization/document boosts to get rid of those files?
Why not raise the limit of open files on your system? (man ulimit)
-Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Re: How to index a lot of fields (without FileNotFoundException:
Too many open files)
Posted by pb...@gmx.net.
Thanks for you reply.
We are still using Lucene v1.4.3 and I'm not sure if upgrading is an option. Is there another way of disabling length normalization/document boosts to get rid of those files?
Thanks,
Rico
: >From what I read in the Lucene docs, these .f<m> files store the
: normalization factor for the corresponding field. What exactly is this
: used for and more importantly, can this be disabled so that the files
: are not created in the first place?
field norms are primarily used for length normalization, but they also
store field and document boosts ... more info can be found in the scoring
and Similarity docs.
if you don't care about these things, there is an omitNorms option on the
Field class that can be used to not only reduce the number of open
files you need, but also make your index smaller on disk, and reduce your
memory usage. (i believe it was added in 1.9)
-Hoss
--
"Feel free" - 10 GB Mailbox, 100 FreeSMS/Monat ...
Jetzt GMX TopMail testen: http://www.gmx.net/de/go/topmail
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Spanquery problem
Posted by Erick Erickson <er...@gmail.com>.
If you only knew how many times I've looked at code I've written and
wondered "What was I thinking" <G>...
Anyway, glad it's working for you
Erick
On 4/30/07, axel.reymonet <ax...@actia.fr> wrote:
>
> Hello,
>
> Thank you for your piece of advice. Indeed, my mistake was to use HashSet
> instead of an ArrayList (for instance). I must have been really distracted
> when I wrote my code, even more when I checked it! Anyway, thank you
> again,
>
> Axel Reymonet
>
> -----Message d'origine-----
> De : Erick Erickson [mailto:erickerickson@gmail.com]
> Envoyé : lundi 30 avril 2007 14:52
> À : java-user@lucene.apache.org
> Objet : Re: Spanquery problem
>
>
> The first thing I'd do is not use a HashSet when you collect your
> SpanTermQuerys since the iteration order is not guaranteed. That is,
> the order when putting them in is not necessarily the same as
> when getting them out. So you may be searching for
> "automatique climatisation" rather then "climatisation automatique".
>
> You can easily test this by looking at snQ.toString() to see what query
> is actually constructed.....
>
> If that isn't it, post the results and we'll have another go at it....
>
> Erick
>
> On 4/30/07, axel.reymonet <ax...@actia.fr> wrote:
> >
> > Hello,
> >
> > I am having some issues with the SpanQuery functionality. As a matter of
> > fact, I index a single french file containing for instance
> "climatisation
> > automatique" (which means automatic air-conditioning) with the classical
> > FrenchAnalyzer, and when I search in this index with SpanQuery, I have
> the
> > following situation :
> > - I have 1 span for the "climatisation" request
> > - I have 1 span for the "automatique" request
> > - I have 0 span for the "climatisation automatique" request
> > Maybe I am doing something wrong, but I cannot spot my mistake. I have
> > given
> > the problematic portion of code below. Does anyone have an idea ? Thanks
> > in
> > advance,
> >
> > Axel Reymonet
> >
> >
> > My code is as follows :
> > - FOR INDEXING
> > public void fileProcess(String filepath)
> > {
> > File index_dir = new File(this.currentProjectPath+"index");
> > if (index_dir.exists())
> > {
> > String[] files = index_dir.list();
> > for (String f:files)
> > {
> > File file = new File (this.currentProjectPath+"index/"+f);
> > file.delete();
> > }
> > index_dir.delete();
> > }
> > File toBeIndexed = new File(filepath);
> > if (!toBeIndexed.exists() || !toBeIndexed.canRead())
> > {
> > System.out.println("Document directory '"
> > +toBeIndexed.getAbsolutePath()+ "' does not exist or is not readable,
> > please
> > check the path");
> > System.exit(1);
> > }
> > try {
> > IndexWriter writer = new IndexWriter(index_dir, new
> > org.apache.lucene.analysis.fr.FrenchAnalyzer(),true);
> > writer.addDocument(FileDocument.Document(toBeIndexed));
> > writer.close();
> > } catch (IOException e) {System.out.println(" caught a " + e.getClass()
> > +"\n
> > with message: " + e.getMessage());}
> > }
> >
> > -----------------------------------------
> > - FOR SEARCHING
> > public void testSF(String searched,String indexPath)
> > {
> > try{
> > IndexReader reader = IndexReader.open(indexPath);
> > Analyzer analyzer = new org.apache.lucene.analysis.fr.FrenchAnalyzer();
> > TokenStream requestStream = analyzer.tokenStream("contents",new
> > StringReader(searched));
> > HashSet<SpanTermQuery> qSet = new HashSet<SpanTermQuery>();
> > Token currentToken = requestStream.next();
> > while (currentToken!=null)
> > {
> > qSet.add(new SpanTermQuery(new
> > Term("contents",currentToken.termText())));
> > currentToken = requestStream.next();
> > }
> > SpanQuery[] sQ = new SpanQuery [qSet.size()];
> > int k = 0;
> > for (SpanTermQuery stq:qSet)
> > {
> > sQ[k]=stq;
> > k++;
> > }
> > SpanNearQuery snQ;
> > snQ = new SpanNearQuery(sQ,0,true);
> > Spans spans = snQ.getSpans(reader);
> > int resultsCpt = 0;
> > while (spans.next())
> > resultsCpt++;
> > System.out.println("Number of results: "+resultsCpt);
> > }
> > catch(IOException ioe){ioe.printStackTrace();}
> > }
> > --------------------------------------------
> > - OUTPUT
> > Query: climatisation
> > Number of results: 1
> > Query: automatique
> > Number of results: 1
> > Query: climatisation automatique
> > Number of results: 0
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: Spanquery problem
Posted by "axel.reymonet" <ax...@actia.fr>.
Hello,
Thank you for your piece of advice. Indeed, my mistake was to use HashSet
instead of an ArrayList (for instance). I must have been really distracted
when I wrote my code, even more when I checked it! Anyway, thank you again,
Axel Reymonet
-----Message d'origine-----
De : Erick Erickson [mailto:erickerickson@gmail.com]
Envoyé : lundi 30 avril 2007 14:52
À : java-user@lucene.apache.org
Objet : Re: Spanquery problem
The first thing I'd do is not use a HashSet when you collect your
SpanTermQuerys since the iteration order is not guaranteed. That is,
the order when putting them in is not necessarily the same as
when getting them out. So you may be searching for
"automatique climatisation" rather then "climatisation automatique".
You can easily test this by looking at snQ.toString() to see what query
is actually constructed.....
If that isn't it, post the results and we'll have another go at it....
Erick
On 4/30/07, axel.reymonet <ax...@actia.fr> wrote:
>
> Hello,
>
> I am having some issues with the SpanQuery functionality. As a matter of
> fact, I index a single french file containing for instance "climatisation
> automatique" (which means automatic air-conditioning) with the classical
> FrenchAnalyzer, and when I search in this index with SpanQuery, I have the
> following situation :
> - I have 1 span for the "climatisation" request
> - I have 1 span for the "automatique" request
> - I have 0 span for the "climatisation automatique" request
> Maybe I am doing something wrong, but I cannot spot my mistake. I have
> given
> the problematic portion of code below. Does anyone have an idea ? Thanks
> in
> advance,
>
> Axel Reymonet
>
>
> My code is as follows :
> - FOR INDEXING
> public void fileProcess(String filepath)
> {
> File index_dir = new File(this.currentProjectPath+"index");
> if (index_dir.exists())
> {
> String[] files = index_dir.list();
> for (String f:files)
> {
> File file = new File (this.currentProjectPath+"index/"+f);
> file.delete();
> }
> index_dir.delete();
> }
> File toBeIndexed = new File(filepath);
> if (!toBeIndexed.exists() || !toBeIndexed.canRead())
> {
> System.out.println("Document directory '"
> +toBeIndexed.getAbsolutePath()+ "' does not exist or is not readable,
> please
> check the path");
> System.exit(1);
> }
> try {
> IndexWriter writer = new IndexWriter(index_dir, new
> org.apache.lucene.analysis.fr.FrenchAnalyzer(),true);
> writer.addDocument(FileDocument.Document(toBeIndexed));
> writer.close();
> } catch (IOException e) {System.out.println(" caught a " + e.getClass()
> +"\n
> with message: " + e.getMessage());}
> }
>
> -----------------------------------------
> - FOR SEARCHING
> public void testSF(String searched,String indexPath)
> {
> try{
> IndexReader reader = IndexReader.open(indexPath);
> Analyzer analyzer = new org.apache.lucene.analysis.fr.FrenchAnalyzer();
> TokenStream requestStream = analyzer.tokenStream("contents",new
> StringReader(searched));
> HashSet<SpanTermQuery> qSet = new HashSet<SpanTermQuery>();
> Token currentToken = requestStream.next();
> while (currentToken!=null)
> {
> qSet.add(new SpanTermQuery(new
> Term("contents",currentToken.termText())));
> currentToken = requestStream.next();
> }
> SpanQuery[] sQ = new SpanQuery [qSet.size()];
> int k = 0;
> for (SpanTermQuery stq:qSet)
> {
> sQ[k]=stq;
> k++;
> }
> SpanNearQuery snQ;
> snQ = new SpanNearQuery(sQ,0,true);
> Spans spans = snQ.getSpans(reader);
> int resultsCpt = 0;
> while (spans.next())
> resultsCpt++;
> System.out.println("Number of results: "+resultsCpt);
> }
> catch(IOException ioe){ioe.printStackTrace();}
> }
> --------------------------------------------
> - OUTPUT
> Query: climatisation
> Number of results: 1
> Query: automatique
> Number of results: 1
> Query: climatisation automatique
> Number of results: 0
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Spanquery problem
Posted by Erick Erickson <er...@gmail.com>.
The first thing I'd do is not use a HashSet when you collect your
SpanTermQuerys since the iteration order is not guaranteed. That is,
the order when putting them in is not necessarily the same as
when getting them out. So you may be searching for
"automatique climatisation" rather then "climatisation automatique".
You can easily test this by looking at snQ.toString() to see what query
is actually constructed.....
If that isn't it, post the results and we'll have another go at it....
Erick
On 4/30/07, axel.reymonet <ax...@actia.fr> wrote:
>
> Hello,
>
> I am having some issues with the SpanQuery functionality. As a matter of
> fact, I index a single french file containing for instance "climatisation
> automatique" (which means automatic air-conditioning) with the classical
> FrenchAnalyzer, and when I search in this index with SpanQuery, I have the
> following situation :
> - I have 1 span for the "climatisation" request
> - I have 1 span for the "automatique" request
> - I have 0 span for the "climatisation automatique" request
> Maybe I am doing something wrong, but I cannot spot my mistake. I have
> given
> the problematic portion of code below. Does anyone have an idea ? Thanks
> in
> advance,
>
> Axel Reymonet
>
>
> My code is as follows :
> - FOR INDEXING
> public void fileProcess(String filepath)
> {
> File index_dir = new File(this.currentProjectPath+"index");
> if (index_dir.exists())
> {
> String[] files = index_dir.list();
> for (String f:files)
> {
> File file = new File (this.currentProjectPath+"index/"+f);
> file.delete();
> }
> index_dir.delete();
> }
> File toBeIndexed = new File(filepath);
> if (!toBeIndexed.exists() || !toBeIndexed.canRead())
> {
> System.out.println("Document directory '"
> +toBeIndexed.getAbsolutePath()+ "' does not exist or is not readable,
> please
> check the path");
> System.exit(1);
> }
> try {
> IndexWriter writer = new IndexWriter(index_dir, new
> org.apache.lucene.analysis.fr.FrenchAnalyzer(),true);
> writer.addDocument(FileDocument.Document(toBeIndexed));
> writer.close();
> } catch (IOException e) {System.out.println(" caught a " + e.getClass()
> +"\n
> with message: " + e.getMessage());}
> }
>
> -----------------------------------------
> - FOR SEARCHING
> public void testSF(String searched,String indexPath)
> {
> try{
> IndexReader reader = IndexReader.open(indexPath);
> Analyzer analyzer = new org.apache.lucene.analysis.fr.FrenchAnalyzer();
> TokenStream requestStream = analyzer.tokenStream("contents",new
> StringReader(searched));
> HashSet<SpanTermQuery> qSet = new HashSet<SpanTermQuery>();
> Token currentToken = requestStream.next();
> while (currentToken!=null)
> {
> qSet.add(new SpanTermQuery(new
> Term("contents",currentToken.termText())));
> currentToken = requestStream.next();
> }
> SpanQuery[] sQ = new SpanQuery [qSet.size()];
> int k = 0;
> for (SpanTermQuery stq:qSet)
> {
> sQ[k]=stq;
> k++;
> }
> SpanNearQuery snQ;
> snQ = new SpanNearQuery(sQ,0,true);
> Spans spans = snQ.getSpans(reader);
> int resultsCpt = 0;
> while (spans.next())
> resultsCpt++;
> System.out.println("Number of results: "+resultsCpt);
> }
> catch(IOException ioe){ioe.printStackTrace();}
> }
> --------------------------------------------
> - OUTPUT
> Query: climatisation
> Number of results: 1
> Query: automatique
> Number of results: 1
> Query: climatisation automatique
> Number of results: 0
>