You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Mike Klaas <mi...@gmail.com> on 2007/05/01 00:08:14 UTC
Re: Re: How to index a lot of fields (without FileNotFoundException: Too many open files)
On 4/30/07, pbm-rico@gmx.net <pb...@gmx.net> wrote:
> Thanks for you reply.
>
> We are still using Lucene v1.4.3 and I'm not sure if upgrading is an option. Is there another way of disabling length normalization/document boosts to get rid of those files?
Why not raise the limit of open files on your system? (man ulimit)
-Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to index a lot of fields (without FileNotFoundException: Too many open files)
Posted by Paul Elschot <pa...@xs4all.nl>.
On Tuesday 01 May 2007 00:22, pbm-rico@gmx.net wrote:
> I thought about using ulimit, but it does not scale. In the scenario that
the app has to support, client applications could create hundreds of
thousands of unique properties, which would result in this many indexable
fields.
>
> Based on previous answers, the way out of this problem while still being
scalable would be to use 2.1 where there is only 1 norm file.
>
> However, it does not look like upgrading is an option, so I wonder if my
current approach of mapping a property that a client app creates to one field
name is workable at all. Maybe I have to introduce some sort of mapping of
client properties to a fixed number of indexable fields.
>
> ...or modify the 1.4.3 code myself to get rid of norm files.
Do you need the idf computations of lucene for each of these client fields
separately?
When not, another way is to move the names of the client fields into the
term value, and use a single special field for these.
When you do that, you'll also have to move the client field names
in the queries from the field name to the term. This can easily be done by
overriding one of the methods in QueryParser.
Regards,
Paul Elschot
>
> -Rico
>
> -------- Original-Nachricht --------
> Datum: Mon, 30 Apr 2007 15:08:14 -0700
> Von: "Mike Klaas" <mi...@gmail.com>
> An: java-user@lucene.apache.org
> Betreff: Re: Re: How to index a lot of fields (without
FileNotFoundException: Too many open files)
>
> > On 4/30/07, pbm-rico@gmx.net <pb...@gmx.net> wrote:
> > > Thanks for you reply.
> > >
> > > We are still using Lucene v1.4.3 and I'm not sure if upgrading is an
> > option. Is there another way of disabling length normalization/document
boosts
> > to get rid of those files?
> >
> > Why not raise the limit of open files on your system? (man ulimit)
> >
> > -Mike
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
>
> --
> "Feel free" - 10 GB Mailbox, 100 FreeSMS/Monat ...
> Jetzt GMX TopMail testen: http://www.gmx.net/de/go/topmail
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Re: How to index a lot of fields (without FileNotFoundException:
Too many open files)
Posted by Chris Hostetter <ho...@fucit.org>.
: However, it does not look like upgrading is an option, so I wonder if my
: current approach of mapping a property that a client app creates to one
: field name is workable at all. Maybe I have to introduce some sort of
: mapping of client properties to a fixed number of indexable fields.
:
: ...or modify the 1.4.3 code myself to get rid of norm files.
if you can't upgrade past 1.4.3, but you can modify 1.4.3 to change
things (like file formats), why not "modify" your 1.4.3 install to be a
2.1 install?
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Re: How to index a lot of fields (without FileNotFoundException:
Too many open files)
Posted by pb...@gmx.net.
I thought about using ulimit, but it does not scale. In the scenario that the app has to support, client applications could create hundreds of thousands of unique properties, which would result in this many indexable fields.
Based on previous answers, the way out of this problem while still being scalable would be to use 2.1 where there is only 1 norm file.
However, it does not look like upgrading is an option, so I wonder if my current approach of mapping a property that a client app creates to one field name is workable at all. Maybe I have to introduce some sort of mapping of client properties to a fixed number of indexable fields.
...or modify the 1.4.3 code myself to get rid of norm files.
-Rico
-------- Original-Nachricht --------
Datum: Mon, 30 Apr 2007 15:08:14 -0700
Von: "Mike Klaas" <mi...@gmail.com>
An: java-user@lucene.apache.org
Betreff: Re: Re: How to index a lot of fields (without FileNotFoundException: Too many open files)
> On 4/30/07, pbm-rico@gmx.net <pb...@gmx.net> wrote:
> > Thanks for you reply.
> >
> > We are still using Lucene v1.4.3 and I'm not sure if upgrading is an
> option. Is there another way of disabling length normalization/document boosts
> to get rid of those files?
>
> Why not raise the limit of open files on your system? (man ulimit)
>
> -Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
--
"Feel free" - 10 GB Mailbox, 100 FreeSMS/Monat ...
Jetzt GMX TopMail testen: http://www.gmx.net/de/go/topmail
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org