You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Stephane James Vaucher <va...@cirano.qc.ca> on 2004/03/03 21:34:52 UTC
Sys properties Was: java.io.tmpdir as lock dir .... once again
As I've stated in my earlier mail, I like this change. More importantly,
could this become a "standard" way of changing configurations at runtime?
For example, the default merge factor could also be set in this manner.
sv
On Wed, 3 Mar 2004, Michael Duval wrote:
>
> I agree with both the property name change and also making it static.
>
> Mike
>
> Doug Cutting wrote:
>
> > Michael Duval wrote:
> > > I've hacked the code for the time being by updating FSDirectory and
> >
> >> replaced all System.getProperty("java.io.tmpdir")
> >> calls with a call to a new method "getLockDir()". This method
> >> checks for a "lucene.lockdir" prop before the
> >> "java.io.tmpdir" prop giving the end user a bit more flexibility in
> >> where locks are stored.
> >
> >
> > In general, I support this change.
> >
> >> Here is the method:
> >>
> >> /** Allow flexible locking directories - Michael R. Duval 3/02/04 */
> >> private String getLockDir() {
> >> String lockDir;
> >>
> >> if ((lockDir = System.getProperty("lucene.lockdir")) == null)
> >> return System.getProperty("java.io.tmpdir");
> >> else
> >> return lockDir;
> >> }
> >
> >
> > In particular, I have some quibbles. The property should be named
> > something like "org.apache.lucene.lockdir", not just "lucene.lockdir".
> > And there's no reason to look it up each time: it can just be a static.
> >
> > private static final String LOCK_DIR =
> > System.getProperty("org.apache.lucene.lockdir",
> > System.getProperty("java.io.tmpdir"));
> >
> > Doug
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Indexing HTML
Posted by Chandan Tamrakar <ch...@ccnep.com.np>.
How do I index a HTM document which may have any encoding like
EUC,SJIS,Western European or UTF 8. Can I parse and extract the html into
string and than convert into Text file in UNICODE ?
Is this an appropiate way to index HTML files ? Can anyone suggest me a
simple parser other than a parser found in demo of lucene ?
Also how do i find the "encoding " of files ? Whenever there are ANSI text
files containing japanese characters i am not able to convert into UTF-16
lucene is indexing properly when I convert into SJIS
thnks
chandan
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Getting offset information of hits
Posted by Otis Gospodnetic <ot...@yahoo.com>.
See highlighter link on the Contributions page.
Otis
--- Rick Noel <rn...@rcn.com> wrote:
> I'm new to Lucene and am trying to implement excerpts and
> highlighting. I am
> trying to figure out how I can get offset information from the Hits
> class. TermPositions doesn't appear to get me the offset information
> that
> the Token class provides. Is there a way of getting this information
> at
> search time?
>
> -Rick
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Getting offset information of hits
Posted by Rick Noel <rn...@rcn.com>.
I'm new to Lucene and am trying to implement excerpts and highlighting. I am
trying to figure out how I can get offset information from the Hits
class. TermPositions doesn't appear to get me the offset information that
the Token class provides. Is there a way of getting this information at
search time?
-Rick
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Sys properties Was: java.io.tmpdir as lock dir .... once again
Posted by Stephane James Vaucher <va...@cirano.qc.ca>.
Thanks, I was going to submit a patch for this feature this week-end.
sv
On Thu, 18 Mar 2004, Otis Gospodnetic wrote:
> I added support for all items listed below, except commit/write lock
> file name. I don't see why one would want to change that, considering
> those files are still limited to the index directory.
>
> Otis
>
> --- Stephane James Vaucher <va...@cirano.qc.ca> wrote:
> > How about (looking big rather than small):
> >
> > - MaxClause from BooleanQuery (I know there has been discussions on
> > the dev list, but I haven't been following it)
> > - default commit_lock_name
> > - default commit_lock_timeout
> > - default maxFieldLength
> > - default maxMergeDocs
> > - default mergeFactor
> > - default minMergeDocs
> > - default write_lock_name
> > - default write_lock_timeout
> >
> > I'm currently configuring parts of my app using sys properties,
> > particularly the mergeFactor because my prod system has 2GB of RAM
> > and is
> > windows based and my dev machine has 256MB and is linux. If no one
> > takes a
> > crack at this, I'll see what I can do in 2 weeks, after my vacations.
> >
> > Cheers,
> > sv
> >
> > On Wed, 3 Mar 2004, Doug Cutting wrote:
> >
> > > Stephane James Vaucher wrote:
> > > > As I've stated in my earlier mail, I like this change. More
> > importantly,
> > > > could this become a "standard" way of changing configurations at
> > runtime?
> > > > For example, the default merge factor could also be set in this
> > manner.
> > >
> > > Sure, that's reasonable, so this would be something like:
> > >
> > > private static final int DEFAULT_MERGE_FACTOR =
> > >
> > >
> >
> Integer.parseInt(System.getProperty("org.apache.lucene.mergeFactor","10"));
> > >
> > > In IndexWriter.java.
> > >
> > > What other candidates are there for this treatment?
> > >
> > > Doug
> > >
> > >
> > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > > For additional commands, e-mail:
> > lucene-user-help@jakarta.apache.org
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Sys properties Was: java.io.tmpdir as lock dir .... once again
Posted by Otis Gospodnetic <ot...@yahoo.com>.
I added support for all items listed below, except commit/write lock
file name. I don't see why one would want to change that, considering
those files are still limited to the index directory.
Otis
--- Stephane James Vaucher <va...@cirano.qc.ca> wrote:
> How about (looking big rather than small):
>
> - MaxClause from BooleanQuery (I know there has been discussions on
> the dev list, but I haven't been following it)
> - default commit_lock_name
> - default commit_lock_timeout
> - default maxFieldLength
> - default maxMergeDocs
> - default mergeFactor
> - default minMergeDocs
> - default write_lock_name
> - default write_lock_timeout
>
> I'm currently configuring parts of my app using sys properties,
> particularly the mergeFactor because my prod system has 2GB of RAM
> and is
> windows based and my dev machine has 256MB and is linux. If no one
> takes a
> crack at this, I'll see what I can do in 2 weeks, after my vacations.
>
> Cheers,
> sv
>
> On Wed, 3 Mar 2004, Doug Cutting wrote:
>
> > Stephane James Vaucher wrote:
> > > As I've stated in my earlier mail, I like this change. More
> importantly,
> > > could this become a "standard" way of changing configurations at
> runtime?
> > > For example, the default merge factor could also be set in this
> manner.
> >
> > Sure, that's reasonable, so this would be something like:
> >
> > private static final int DEFAULT_MERGE_FACTOR =
> >
> >
>
Integer.parseInt(System.getProperty("org.apache.lucene.mergeFactor","10"));
> >
> > In IndexWriter.java.
> >
> > What other candidates are there for this treatment?
> >
> > Doug
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Sys properties Was: java.io.tmpdir as lock dir .... once again
Posted by Stephane James Vaucher <va...@cirano.qc.ca>.
How about (looking big rather than small):
- MaxClause from BooleanQuery (I know there has been discussions on
the dev list, but I haven't been following it)
- default commit_lock_name
- default commit_lock_timeout
- default maxFieldLength
- default maxMergeDocs
- default mergeFactor
- default minMergeDocs
- default write_lock_name
- default write_lock_timeout
I'm currently configuring parts of my app using sys properties,
particularly the mergeFactor because my prod system has 2GB of RAM and is
windows based and my dev machine has 256MB and is linux. If no one takes a
crack at this, I'll see what I can do in 2 weeks, after my vacations.
Cheers,
sv
On Wed, 3 Mar 2004, Doug Cutting wrote:
> Stephane James Vaucher wrote:
> > As I've stated in my earlier mail, I like this change. More importantly,
> > could this become a "standard" way of changing configurations at runtime?
> > For example, the default merge factor could also be set in this manner.
>
> Sure, that's reasonable, so this would be something like:
>
> private static final int DEFAULT_MERGE_FACTOR =
>
> Integer.parseInt(System.getProperty("org.apache.lucene.mergeFactor","10"));
>
> In IndexWriter.java.
>
> What other candidates are there for this treatment?
>
> Doug
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
RE: Sys properties Was: java.io.tmpdir as lock dir .... once again
Posted by hui <hu...@triplehop.com>.
Hi,
I did the testing on searching/indexing larger documents for the two index
formats. The indexing application indices the 66k documents incrementally up
to 10 times. At the same time, the search program wakes every 20 minutes to
open the index and do the search.
For the indexing, when indexing on the 9th 66k document, the compound index
case need 500 seconds more but it could be the machine is running other
schedule jobs. (I run the non-compound index case immediately after compound
index case; the first 66k for non-compound index also needs 300-400 seconds
more). So the index speed difference is within 10%.
There is no too much difference on the search itself. But for the compound
index case, the opening index reaches around 1 second twice. I saw this
happened for non-compound index format before, but it looks like compound
index format has higher frequency on this issue.
Regards,
Hui
Indexing time:
non-compound index
2:1709393 total milliseconds index total docs:66100
3:1322806 total milliseconds index total docs:66100
4:1387652 total milliseconds index total docs:66100
5:1371401 total milliseconds index total docs:66100
6:1378230 total milliseconds index total docs:66100
7:1375276 total milliseconds index total docs:66100
8:1411543 total milliseconds index total docs:66100
9:1533842 total milliseconds index total docs:66100
10:1722148 total milliseconds index total docs:66100
11:1893729 total milliseconds index total docs:66100
compound index
2:1437950 total milliseconds index total docs:66100
3:1406980 total milliseconds index total docs:66100
4:1427058 total milliseconds index total docs:66100
5:1462169 total milliseconds index total docs:66100
6:1461356 total milliseconds index total docs:66100
7:1657923 total milliseconds index total docs:66100
8:1901755 total milliseconds index total docs:66100
9:2041649 total milliseconds index total docs:66100
search time:
non-compound index
0 open index time:235
5551 total within(ms) 187
7326 total within(ms) 31
8698 total within(ms) 47
4179 total within(ms) 16
198 total within(ms) 47
1 open index time:63
8242 total within(ms) 15
10918 total within(ms) 0
13022 total within(ms) 16
6138 total within(ms) 15
332 total within(ms) 15
2 open index time:47
12194 total within(ms) 15
16210 total within(ms) 0
19305 total within(ms) 125
9099 total within(ms) 15
485 total within(ms) 31
3 open index time:47
16461 total within(ms) 16
21924 total within(ms) 0
26107 total within(ms) 31
12278 total within(ms) 15
655 total within(ms) 16
4 open index time:47
19857 total within(ms) 16
26560 total within(ms) 15
31583 total within(ms) 31
14834 total within(ms) 15
784 total within(ms) 15
5 open index time:78
23357 total within(ms) 16
31234 total within(ms) 16
37169 total within(ms) 31
17422 total within(ms) 32
895 total within(ms) 47
6 open index time:47
27208 total within(ms) 78
36371 total within(ms) 31
43276 total within(ms) 32
20303 total within(ms) 31
1054 total within(ms) 31
7 open index time:31
30073 total within(ms) 16
40270 total within(ms) 15
47977 total within(ms) 31
22366 total within(ms) 32
1202 total within(ms) 31
8 open index time:63
34133 total within(ms) 31
45709 total within(ms) 32
54424 total within(ms) 31
25418 total within(ms) 31
1354 total within(ms) 62
9 open index time:62
37845 total within(ms) 47
50582 total within(ms) 16
60236 total within(ms) 47
28191 total within(ms) 31
1505 total within(ms) 47
10 open index time:46
41416 total within(ms) 32
55456 total within(ms) 31
65980 total within(ms) 109
30892 total within(ms) 47
1641 total within(ms) 47
11 open index time:31
44121 total within(ms) 78
58992 total within(ms) 31
70242 total within(ms) 62
32871 total within(ms) 31
1729 total within(ms) 47
12 open index time:47
46718 total within(ms) 31
62473 total within(ms) 15
74415 total within(ms) 47
34776 total within(ms) 31
1862 total within(ms) 47
13 open index time:0
47025 total within(ms) 32
63008 total within(ms) 31
75031 total within(ms) 47
35002 total within(ms) 31
1870 total within(ms) 31
14 open index time:0
47025 total within(ms) 31
63008 total within(ms) 31
75031 total within(ms) 47
35002 total within(ms) 31
1870 total within(ms) 32
compound index
0 open index time:203
7876 total within(ms) 125
10326 total within(ms) 0
12317 total within(ms) 32
5885 total within(ms) 16
315 total within(ms) 31
1 open index time:62
10562 total within(ms) 16
14127 total within(ms) 0
16793 total within(ms) 16
7896 total within(ms) 15
385 total within(ms) 15
2 open index time:1016
14466 total within(ms) 15
19304 total within(ms) 0
22944 total within(ms) 31
10826 total within(ms) 15
546 total within(ms) 16
3 open index time:15
17123 total within(ms) 16
22947 total within(ms) 15
27326 total within(ms) 15
12744 total within(ms) 125
680 total within(ms) 31
4 open index time:63
21148 total within(ms) 15
28245 total within(ms) 15
33659 total within(ms) 31
15734 total within(ms) 31
842 total within(ms) 63
5 open index time:47
25019 total within(ms) 16
33398 total within(ms) 16
39772 total within(ms) 32
18645 total within(ms) 32
995 total within(ms) 47
6 open index time:47
28367 total within(ms) 15
37970 total within(ms) 31
45169 total within(ms) 46
21168 total within(ms) 31
1112 total within(ms) 31
7 open index time:31
31424 total within(ms) 31
42002 total within(ms) 16
49994 total within(ms) 31
23432 total within(ms) 32
1223 total within(ms) 47
8 open index time:46
33895 total within(ms) 32
45292 total within(ms) 47
53957 total within(ms) 47
25230 total within(ms) 32
1352 total within(ms) 47
9 open index time:63
37320 total within(ms) 31
49922 total within(ms) 15
59412 total within(ms) 47
27830 total within(ms) 31
1474 total within(ms) 62
10 open index time:984
38475 total within(ms) 16
51552 total within(ms) 16
61389 total within(ms) 47
28638 total within(ms) 46
1530 total within(ms) 157
-----Original Message-----
From: Doug Cutting [mailto:cutting@apache.org]
Sent: Monday, March 08, 2004 1:16 PM
To: Lucene Users List
Subject: Re: Sys properties Was: java.io.tmpdir as lock dir .... once again
hui wrote:
> Index time:
> compound format is 89 seconds slower.
>
> compound format:
> 1389507 total milliseconds
> non-compound format:
> 1300534 total milliseconds
>
> The index size is 85m with 4 fields only. The files are stored in the
index.
> The compound format has only 3 files and the other has 13 files.
Thanks for performing this benchmark!
It looks like the compound format is around 7% slower when indexing. To
my thinking that's acceptable, given the dramatic reduction in file
handles. If folks really need maximal indexing performance, then they
can explicitly disable the compound format.
Would anyone object to making compound format the default for Lucene
1.4? This is an incompatible change, but I don't think it should break
applications.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: compound format as default in 1.4?
Posted by Scott ganyo <sc...@ganyo.com>.
+1. I agree with this. Give the safest option to the general masses,
let the "expert" users choose other options based on their level of
experience.
(BTW: It seems that accessing this compound file format as a memory
mapped file using the NIO library would be a natural fit for improving
Lucene's memory footprint as well...)
Scott
On Mar 8, 2004, at 3:25 PM, Doug Cutting wrote:
> [ I moved this discussion to the developer list.]
>
> My metric here is the rate of complaint.
>
> I'm tired of hearing about "too many file handles" problems. Ususally
> it is caused by folks opening a new searcher for each query, and the
> garbage collector not collecting and closing the old ones fast enough,
> so it signals other problems with the application, but it is still
> annoying, and could be largely quashed.
>
> By some definition, anything which causes so many repeated complaints
> is a bug, and should be fixed. Even if it's really not a bug. It
> pains users of Lucene. It annoys developers of Lucene.
>
> Think of it like mergeFactor, etc.: the default setting may not be the
> absolute fastest, but it is one that is likely to run well in most
> configurations and cause the least confusion.
>
> Doug
>
> Terry Steichen wrote:
>> I tend to agree (but with the same uncertainty as to why I feel that
>> way).
>> Regards,
>> Terry
>> ----- Original Message ----- From: "Otis Gospodnetic"
>> <ot...@yahoo.com>
>> To: "Lucene Users List" <lu...@jakarta.apache.org>
>> Sent: Monday, March 08, 2004 2:34 PM
>> Subject: Re: Sys properties Was: java.io.tmpdir as lock dir .... once
>> again
>>> I can't explain why, but I feel like the old index format should stay
>>> by default. I feel like I'd rather a (slightly) faster index, and
>>> switch to the compound one when/IF I encounter problems, than have a
>>> safer, but slower index, and never realize that there is a faster
>>> option available.
>>>
>>> Weak argument, I know, but some instinct in me thinks that the
>>> current
>>> mode should remain.
>>>
>>> Otis
>>>
>>>
>>> --- Doug Cutting <cu...@apache.org> wrote:
>>>
>>>> hui wrote:
>>>>
>>>>> Index time: compound format is 89 seconds slower.
>>>>>
>>>>> compound format:
>>>>> 1389507 total milliseconds
>>>>> non-compound format:
>>>>> 1300534 total milliseconds
>>>>>
>>>>> The index size is 85m with 4 fields only. The files are stored in
>>>>
>>>> the index.
>>>>
>>>>> The compound format has only 3 files and the other has 13 files.
>>>>
>>>> Thanks for performing this benchmark!
>>>>
>>>> It looks like the compound format is around 7% slower when
>>>> indexing. To my thinking that's acceptable, given the dramatic
>>>> reduction in file handles. If folks really need maximal indexing
>>>> performance, then
>>>> they can explicitly disable the compound format.
>>>>
>>>> Would anyone object to making compound format the default for
>>>> Lucene 1.4? This is an incompatible change, but I don't think it
>>>> should
>>>> break applications.
>>>>
>>>> Doug
>>>>
>>>> --------------------------------------------------------------------
>>>> -
>>>> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>>>> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>>> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>
Re: compound format as default in 1.4?
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
Just to weigh in with my opinion... the compound file format proves
fine in my use of Lucene and I use it 'by default' already. So I'm +1
on making it the default behavior.
Erik
On Mar 8, 2004, at 3:25 PM, Doug Cutting wrote:
> [ I moved this discussion to the developer list.]
>
> My metric here is the rate of complaint.
>
> I'm tired of hearing about "too many file handles" problems. Ususally
> it is caused by folks opening a new searcher for each query, and the
> garbage collector not collecting and closing the old ones fast enough,
> so it signals other problems with the application, but it is still
> annoying, and could be largely quashed.
>
> By some definition, anything which causes so many repeated complaints
> is a bug, and should be fixed. Even if it's really not a bug. It
> pains users of Lucene. It annoys developers of Lucene.
>
> Think of it like mergeFactor, etc.: the default setting may not be the
> absolute fastest, but it is one that is likely to run well in most
> configurations and cause the least confusion.
>
> Doug
>
> Terry Steichen wrote:
>> I tend to agree (but with the same uncertainty as to why I feel that
>> way).
>> Regards,
>> Terry
>> ----- Original Message ----- From: "Otis Gospodnetic"
>> <ot...@yahoo.com>
>> To: "Lucene Users List" <lu...@jakarta.apache.org>
>> Sent: Monday, March 08, 2004 2:34 PM
>> Subject: Re: Sys properties Was: java.io.tmpdir as lock dir .... once
>> again
>>> I can't explain why, but I feel like the old index format should stay
>>> by default. I feel like I'd rather a (slightly) faster index, and
>>> switch to the compound one when/IF I encounter problems, than have a
>>> safer, but slower index, and never realize that there is a faster
>>> option available.
>>>
>>> Weak argument, I know, but some instinct in me thinks that the
>>> current
>>> mode should remain.
>>>
>>> Otis
>>>
>>>
>>> --- Doug Cutting <cu...@apache.org> wrote:
>>>
>>>> hui wrote:
>>>>
>>>>> Index time: compound format is 89 seconds slower.
>>>>>
>>>>> compound format:
>>>>> 1389507 total milliseconds
>>>>> non-compound format:
>>>>> 1300534 total milliseconds
>>>>>
>>>>> The index size is 85m with 4 fields only. The files are stored in
>>>>
>>>> the index.
>>>>
>>>>> The compound format has only 3 files and the other has 13 files.
>>>>
>>>> Thanks for performing this benchmark!
>>>>
>>>> It looks like the compound format is around 7% slower when
>>>> indexing. To my thinking that's acceptable, given the dramatic
>>>> reduction in file handles. If folks really need maximal indexing
>>>> performance, then
>>>> they can explicitly disable the compound format.
>>>>
>>>> Would anyone object to making compound format the default for
>>>> Lucene 1.4? This is an incompatible change, but I don't think it
>>>> should
>>>> break applications.
>>>>
>>>> Doug
>>>>
>>>> --------------------------------------------------------------------
>>>> -
>>>> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>>>> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>>> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: compound format as default in 1.4?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Eh, fine then! ;)
I am using the compound format in my apps, too.
Otis
--- Doug Cutting <cu...@apache.org> wrote:
> [ I moved this discussion to the developer list.]
>
> My metric here is the rate of complaint.
>
> I'm tired of hearing about "too many file handles" problems.
> Ususally
> it is caused by folks opening a new searcher for each query, and the
> garbage collector not collecting and closing the old ones fast
> enough,
> so it signals other problems with the application, but it is still
> annoying, and could be largely quashed.
>
> By some definition, anything which causes so many repeated complaints
> is
> a bug, and should be fixed. Even if it's really not a bug. It pains
>
> users of Lucene. It annoys developers of Lucene.
>
> Think of it like mergeFactor, etc.: the default setting may not be
> the
> absolute fastest, but it is one that is likely to run well in most
> configurations and cause the least confusion.
>
> Doug
>
> Terry Steichen wrote:
> > I tend to agree (but with the same uncertainty as to why I feel
> that way).
> >
> > Regards,
> >
> > Terry
> > ----- Original Message -----
> > From: "Otis Gospodnetic" <ot...@yahoo.com>
> > To: "Lucene Users List" <lu...@jakarta.apache.org>
> > Sent: Monday, March 08, 2004 2:34 PM
> > Subject: Re: Sys properties Was: java.io.tmpdir as lock dir ....
> once again
> >
> >
> >
> >>I can't explain why, but I feel like the old index format should
> stay
> >>by default. I feel like I'd rather a (slightly) faster index, and
> >>switch to the compound one when/IF I encounter problems, than have
> a
> >>safer, but slower index, and never realize that there is a faster
> >>option available.
> >>
> >>Weak argument, I know, but some instinct in me thinks that the
> current
> >>mode should remain.
> >>
> >>Otis
> >>
> >>
> >>--- Doug Cutting <cu...@apache.org> wrote:
> >>
> >>>hui wrote:
> >>>
> >>>>Index time:
> >>>>compound format is 89 seconds slower.
> >>>>
> >>>>compound format:
> >>>>1389507 total milliseconds
> >>>>non-compound format:
> >>>>1300534 total milliseconds
> >>>>
> >>>>The index size is 85m with 4 fields only. The files are stored in
> >>>
> >>>the index.
> >>>
> >>>>The compound format has only 3 files and the other has 13 files.
> >>>
> >>>Thanks for performing this benchmark!
> >>>
> >>>It looks like the compound format is around 7% slower when
> indexing.
> >>>To
> >>>my thinking that's acceptable, given the dramatic reduction in
> file
> >>>handles. If folks really need maximal indexing performance, then
> >>>they
> >>>can explicitly disable the compound format.
> >>>
> >>>Would anyone object to making compound format the default for
> Lucene
> >>>1.4? This is an incompatible change, but I don't think it should
> >>>break
> >>>applications.
> >>>
> >>>Doug
> >>>
>
>>>---------------------------------------------------------------------
> >>>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> >>>For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> >>>
> >>
> >>
>
>>---------------------------------------------------------------------
> >>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> >>For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> >>
> >>
> >
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
compound format as default in 1.4?
Posted by Doug Cutting <cu...@apache.org>.
[ I moved this discussion to the developer list.]
My metric here is the rate of complaint.
I'm tired of hearing about "too many file handles" problems. Ususally
it is caused by folks opening a new searcher for each query, and the
garbage collector not collecting and closing the old ones fast enough,
so it signals other problems with the application, but it is still
annoying, and could be largely quashed.
By some definition, anything which causes so many repeated complaints is
a bug, and should be fixed. Even if it's really not a bug. It pains
users of Lucene. It annoys developers of Lucene.
Think of it like mergeFactor, etc.: the default setting may not be the
absolute fastest, but it is one that is likely to run well in most
configurations and cause the least confusion.
Doug
Terry Steichen wrote:
> I tend to agree (but with the same uncertainty as to why I feel that way).
>
> Regards,
>
> Terry
> ----- Original Message -----
> From: "Otis Gospodnetic" <ot...@yahoo.com>
> To: "Lucene Users List" <lu...@jakarta.apache.org>
> Sent: Monday, March 08, 2004 2:34 PM
> Subject: Re: Sys properties Was: java.io.tmpdir as lock dir .... once again
>
>
>
>>I can't explain why, but I feel like the old index format should stay
>>by default. I feel like I'd rather a (slightly) faster index, and
>>switch to the compound one when/IF I encounter problems, than have a
>>safer, but slower index, and never realize that there is a faster
>>option available.
>>
>>Weak argument, I know, but some instinct in me thinks that the current
>>mode should remain.
>>
>>Otis
>>
>>
>>--- Doug Cutting <cu...@apache.org> wrote:
>>
>>>hui wrote:
>>>
>>>>Index time:
>>>>compound format is 89 seconds slower.
>>>>
>>>>compound format:
>>>>1389507 total milliseconds
>>>>non-compound format:
>>>>1300534 total milliseconds
>>>>
>>>>The index size is 85m with 4 fields only. The files are stored in
>>>
>>>the index.
>>>
>>>>The compound format has only 3 files and the other has 13 files.
>>>
>>>Thanks for performing this benchmark!
>>>
>>>It looks like the compound format is around 7% slower when indexing.
>>>To
>>>my thinking that's acceptable, given the dramatic reduction in file
>>>handles. If folks really need maximal indexing performance, then
>>>they
>>>can explicitly disable the compound format.
>>>
>>>Would anyone object to making compound format the default for Lucene
>>>1.4? This is an incompatible change, but I don't think it should
>>>break
>>>applications.
>>>
>>>Doug
>>>
>>>---------------------------------------------------------------------
>>>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>>>For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>>>
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>>For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: Sys properties Was: java.io.tmpdir as lock dir .... once again
Posted by Terry Steichen <te...@net-frame.com>.
I tend to agree (but with the same uncertainty as to why I feel that way).
Regards,
Terry
----- Original Message -----
From: "Otis Gospodnetic" <ot...@yahoo.com>
To: "Lucene Users List" <lu...@jakarta.apache.org>
Sent: Monday, March 08, 2004 2:34 PM
Subject: Re: Sys properties Was: java.io.tmpdir as lock dir .... once again
> I can't explain why, but I feel like the old index format should stay
> by default. I feel like I'd rather a (slightly) faster index, and
> switch to the compound one when/IF I encounter problems, than have a
> safer, but slower index, and never realize that there is a faster
> option available.
>
> Weak argument, I know, but some instinct in me thinks that the current
> mode should remain.
>
> Otis
>
>
> --- Doug Cutting <cu...@apache.org> wrote:
> > hui wrote:
> > > Index time:
> > > compound format is 89 seconds slower.
> > >
> > > compound format:
> > > 1389507 total milliseconds
> > > non-compound format:
> > > 1300534 total milliseconds
> > >
> > > The index size is 85m with 4 fields only. The files are stored in
> > the index.
> > > The compound format has only 3 files and the other has 13 files.
> >
> > Thanks for performing this benchmark!
> >
> > It looks like the compound format is around 7% slower when indexing.
> > To
> > my thinking that's acceptable, given the dramatic reduction in file
> > handles. If folks really need maximal indexing performance, then
> > they
> > can explicitly disable the compound format.
> >
> > Would anyone object to making compound format the default for Lucene
> > 1.4? This is an incompatible change, but I don't think it should
> > break
> > applications.
> >
> > Doug
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Sys properties Was: java.io.tmpdir as lock dir .... once again
Posted by Otis Gospodnetic <ot...@yahoo.com>.
I can't explain why, but I feel like the old index format should stay
by default. I feel like I'd rather a (slightly) faster index, and
switch to the compound one when/IF I encounter problems, than have a
safer, but slower index, and never realize that there is a faster
option available.
Weak argument, I know, but some instinct in me thinks that the current
mode should remain.
Otis
--- Doug Cutting <cu...@apache.org> wrote:
> hui wrote:
> > Index time:
> > compound format is 89 seconds slower.
> >
> > compound format:
> > 1389507 total milliseconds
> > non-compound format:
> > 1300534 total milliseconds
> >
> > The index size is 85m with 4 fields only. The files are stored in
> the index.
> > The compound format has only 3 files and the other has 13 files.
>
> Thanks for performing this benchmark!
>
> It looks like the compound format is around 7% slower when indexing.
> To
> my thinking that's acceptable, given the dramatic reduction in file
> handles. If folks really need maximal indexing performance, then
> they
> can explicitly disable the compound format.
>
> Would anyone object to making compound format the default for Lucene
> 1.4? This is an incompatible change, but I don't think it should
> break
> applications.
>
> Doug
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Sys properties Was: java.io.tmpdir as lock dir .... once again
Posted by Doug Cutting <cu...@apache.org>.
hui wrote:
> Index time:
> compound format is 89 seconds slower.
>
> compound format:
> 1389507 total milliseconds
> non-compound format:
> 1300534 total milliseconds
>
> The index size is 85m with 4 fields only. The files are stored in the index.
> The compound format has only 3 files and the other has 13 files.
Thanks for performing this benchmark!
It looks like the compound format is around 7% slower when indexing. To
my thinking that's acceptable, given the dramatic reduction in file
handles. If folks really need maximal indexing performance, then they
can explicitly disable the compound format.
Would anyone object to making compound format the default for Lucene
1.4? This is an incompatible change, but I don't think it should break
applications.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
RE: Sys properties Was: java.io.tmpdir as lock dir .... once again
Posted by hui <hu...@triplehop.com>.
Thank you, the converting option from Luke is really helpful for migrate
existing user index.
Regards,
Hui
-----Original Message-----
From: Andrzej Bialecki [mailto:ab@getopt.org]
Sent: Monday, March 08, 2004 10:57 AM
To: Lucene Users List
Subject: Re: Sys properties Was: java.io.tmpdir as lock dir .... once again
hui wrote:
>
>
>
> Hi,
>
> Here is the indexing performance testing result for the two index formats.
A shameless plug: you can use Luke (http://www.getopt.org/luke) to
convert the same index between compound/non-compound formats. Which
could be useful to rule out any possible differences in the
indexing/inserting process between the runs. Luke provides you also with
a simple time measurement for query execution. Just FYI.
--
Best regards,
Andrzej Bialecki
-------------------------------------------------
Software Architect, System Integration Specialist
CEN/ISSS EC Workshop, ECIMF project chair
EU FP6 E-Commerce Expert/Evaluator
-------------------------------------------------
FreeBSD developer (http://www.freebsd.org)
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Sys properties Was: java.io.tmpdir as lock dir .... once again
Posted by Andrzej Bialecki <ab...@getopt.org>.
hui wrote:
>
>
>
> Hi,
>
> Here is the indexing performance testing result for the two index formats.
A shameless plug: you can use Luke (http://www.getopt.org/luke) to
convert the same index between compound/non-compound formats. Which
could be useful to rule out any possible differences in the
indexing/inserting process between the runs. Luke provides you also with
a simple time measurement for query execution. Just FYI.
--
Best regards,
Andrzej Bialecki
-------------------------------------------------
Software Architect, System Integration Specialist
CEN/ISSS EC Workshop, ECIMF project chair
EU FP6 E-Commerce Expert/Evaluator
-------------------------------------------------
FreeBSD developer (http://www.freebsd.org)
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
RE: Sys properties Was: java.io.tmpdir as lock dir .... once again
Posted by hui <hu...@triplehop.com>.
Hi,
Here is the indexing performance testing result for the two index formats.
1000 megahertz Intel Pentium III (2 installed)
32 kilobyte primary memory cache
256 kilobyte secondary memory cache
SCSI Hard drive 145.45 GB
RAm 3G
Windows 2000 Advanced Server, Service Pack 2
JDK 140
JVM memory 512m
Indexed files: local 66100 local text files around 400m
Index time:
compound format is 89 seconds slower.
compound format:
1389507 total milliseconds
non-compound format:
1300534 total milliseconds
The index size is 85m with 4 fields only. The files are stored in the index.
The compound format has only 3 files and the other has 13 files.
Search Time (with only top 10 retrieved, no indexing at the same time, only
one thread search, indices are optimized and opened)
Do not see too much constant difference for the simple situation.
compound format:
Query: iraq
4275 total within(ms) 110
Query: war
5728 total within(ms) 0
Query: iraq AND war
3182 total within(ms) 16
non-compound format:
Query: war
5728 total within(ms) 125
Query: iraq war
6821 total within(ms) 31
Query: iraq AND war
3182 total within(ms) 0
-----Original Message-----
From: Doug Cutting [mailto:cutting@apache.org]
Sent: Thursday, March 04, 2004 11:54 AM
To: Lucene Users List
Subject: Re: Sys properties Was: java.io.tmpdir as lock dir .... once again
hui wrote:
> Not yet. For the compound file format, when the files get bigger, if I add
> few new files frequently, the bigger files has to be updated. Will that
> affect lot on the search and produce heavier disk I/O compared with the
> traditional index format? It seems OS cache makes quite difference when
the
> files not changed differently.
The compound format slows indexing performance slightly, but should not
affect search performance much. It radically reduces the number of file
handles used when searching, by a factor of eight or more, depending on
how many indexed fields you have.
Perhaps the compound format should be the default format in 1.4. Can
folks provide any benchmarks for how it affects performance?
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Sys properties Was: java.io.tmpdir as lock dir .... once again
Posted by Doug Cutting <cu...@apache.org>.
hui wrote:
> Not yet. For the compound file format, when the files get bigger, if I add
> few new files frequently, the bigger files has to be updated. Will that
> affect lot on the search and produce heavier disk I/O compared with the
> traditional index format? It seems OS cache makes quite difference when the
> files not changed differently.
The compound format slows indexing performance slightly, but should not
affect search performance much. It radically reduces the number of file
handles used when searching, by a factor of eight or more, depending on
how many indexed fields you have.
Perhaps the compound format should be the default format in 1.4. Can
folks provide any benchmarks for how it affects performance?
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
RE: Sys properties Was: java.io.tmpdir as lock dir .... once again
Posted by hui <hu...@triplehop.com>.
Not yet. For the compound file format, when the files get bigger, if I add
few new files frequently, the bigger files has to be updated. Will that
affect lot on the search and produce heavier disk I/O compared with the
traditional index format? It seems OS cache makes quite difference when the
files not changed differently.
Regards,
Hui
-----Original Message-----
From: Erik Hatcher [mailto:erik@ehatchersolutions.com]
Sent: Wednesday, March 03, 2004 9:21 PM
To: Lucene Users List
Subject: Re: Sys properties Was: java.io.tmpdir as lock dir .... once again
On Mar 3, 2004, at 4:25 PM, hui wrote:
> Anoterh similar issue. If we could have a parameter to control the max
> number of the files within the index, that is going to avoid the
> problem of
> running of the file handler issue.
> When the file number within one index reaches the limit, optimization
> is
> going to be called.
> Right now, if the file number within one index out of the limit of your
> window system, you lost the index.
> Thank you for the consideration.
Have you tried using the compound file format introduced in 1.3?
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Sys properties Was: java.io.tmpdir as lock dir .... once again
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Mar 3, 2004, at 4:25 PM, hui wrote:
> Anoterh similar issue. If we could have a parameter to control the max
> number of the files within the index, that is going to avoid the
> problem of
> running of the file handler issue.
> When the file number within one index reaches the limit, optimization
> is
> going to be called.
> Right now, if the file number within one index out of the limit of your
> window system, you lost the index.
> Thank you for the consideration.
Have you tried using the compound file format introduced in 1.3?
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
RE: Sys properties Was: java.io.tmpdir as lock dir .... once again
Posted by hui <hu...@triplehop.com>.
Anoterh similar issue. If we could have a parameter to control the max
number of the files within the index, that is going to avoid the problem of
running of the file handler issue.
When the file number within one index reaches the limit, optimization is
going to be called.
Right now, if the file number within one index out of the limit of your
window system, you lost the index.
Thank you for the consideration.
Regards,
hui
-----Original Message-----
From: Doug Cutting [mailto:cutting@apache.org]
Sent: Wednesday, March 03, 2004 3:46 PM
To: Lucene Users List
Subject: Re: Sys properties Was: java.io.tmpdir as lock dir .... once again
Stephane James Vaucher wrote:
> As I've stated in my earlier mail, I like this change. More importantly,
> could this become a "standard" way of changing configurations at runtime?
> For example, the default merge factor could also be set in this manner.
Sure, that's reasonable, so this would be something like:
private static final int DEFAULT_MERGE_FACTOR =
Integer.parseInt(System.getProperty("org.apache.lucene.mergeFactor","10"));
In IndexWriter.java.
What other candidates are there for this treatment?
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Sys properties Was: java.io.tmpdir as lock dir .... once again
Posted by Doug Cutting <cu...@apache.org>.
Stephane James Vaucher wrote:
> As I've stated in my earlier mail, I like this change. More importantly,
> could this become a "standard" way of changing configurations at runtime?
> For example, the default merge factor could also be set in this manner.
Sure, that's reasonable, so this would be something like:
private static final int DEFAULT_MERGE_FACTOR =
Integer.parseInt(System.getProperty("org.apache.lucene.mergeFactor","10"));
In IndexWriter.java.
What other candidates are there for this treatment?
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org