You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Doron Cohen <DO...@il.ibm.com> on 2006/12/15 08:31:04 UTC
potential indexing perormance improvement for compound index - cut IO -
have more files though
Hi,
I would like to propose and get feedback on a potential indexing
performance improvement for the case that compound file is used (this is
the default).
In compound segment mode, each merge operation is ended by writing a
compound file. To be more precise, the merge result is first written to
directory as a non-compound segment file, and then it is 'converted' into a
compound segment file. This conversion involves reading the entire (non
compound) segment, and writing it as a compound segment file. This means
that compound mode indexing does twice as much indexing writing comparing
to non-compound mode. (and there's also the reading of the non-compound
segment).
The reason for this two steps process in writing compound segment files is
that per-segment files cannot be written sequentially, one by one - several
files are created together, written interleaved.
But I think that there is an intermediate state - between
one-compound-segment-file and non-compound-many-files.
To my understanding, at merge time, the following apply:
- .fnm - field infos - independent of other files.
- .fdx .fdt - store fields - interleaved with each other, independent of
other files.
- .tis .tii .frq .prx - dictionary and postings - interleaved with each
other, independent of other files.
- .tvx .tvd .tvf - term vectors - interleaved with each other, independent
of other files.
- .fN - norms - all these files written sequentially, independent of other
files.
Therefore, a "semi compound" segment file can be defined, that would be
made of 4 files (instead of 1):
- File 0: .fdx .tis .tvx
- File 1: .fdt .tii .tvd
- File 2: .frq .tvf
- File 3: .fnm .prx .fN
A merge should be able to write this segment representation at once, - no
need to read and write again.
Few questions:
(1) is this correct at all, or have I overlooked something?
(2) what performance gain would that buy?
(3) is it reasonable to have 4 files per segment comparing to 1 file per
segment?
For (2), the indexing performance of non compound is an upper bound. I
compared indexing speeds of compound and non compound, using the Reuters
input set. Tried with stored+vectors, and without stored fields:
round vect stor cmpnd runCnt recsPerRun rec/s elapsedSec
0 true true true 1 21578 150.2 143.69
- 1 true true false - - 1 - - 21578 - - 178.9 - - 120.58
2 false false true 1 21578 164.7 131.03
- 3 false false false - - 1 - - 21578 - - 184.3 - - 117.07
This is a 19% speed-up with stored+vectors, and 12% speed-up with no stored
fields.
As a side comment, it says something on IO vs. CPU in Lucene indexing, that
cutting 1/2 (I think) the file output speeds-up by less than 20%.
But anyhow, this is not a negligible difference, and for real large
indexes, and busy systems, when the just written non-compound segment is
not in the system caches, it might have more effect. Possibly, search
performance during indexing would be improved by less indexing IO. Also,
delay for addDocument call that triggers a merge should become smaller.
Thanks for your comments, also (but not only) on (1) an (3) above.
Doron
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: potential indexing perormance improvement for compound index - cut IO -
have more files though
Posted by Doron Cohen <DO...@il.ibm.com>.
"Mike Klaas" <mi...@gmail.com> wrote:
>
> My main comment is that the benefits of this change can be achieved by
> using the non-compound index format. For people that care about the
> difference in performance, it isn't difficult to configure your system
> to mitigate the problems of the non-compound format, and they probably
> have already done so.
>
> It would help the people who are file-descriptor conscious, but it
> also increases lucene's fd footprint by a factor of four.
That's right - people worried about indexing performance can easily apply
setUseCompound(false).
My guess though is that most people just keep the default setting.
Large systems that maintain many indexes, would be worried about the number
of file descriptors and would use compound format. But it is not clear to
me what would be the preference in such systems - four times the file
descriptors, or twice as much the IO? If such a third choice is supported
- "semmi compound" - how many systems would {be able to / choose to} use
it? Depending on the specific system maybe.
I verified the IO factor, by counting bytes read in
FSIndexInput.readInternal(byte[],int,int) and written in
FSIndexOutput.flushBuffer(byte[],int):
round vect stor cmpnd runCnt recsPerRun rec/s elapsedSec write
read
0 true true true 1 100000 153.4 651.74 2 GB
1.9 GB
- 1 true true false - - 1 - - 100000 169.5 - - 589.82 - 1 GB
0.9 GB
2 false false true 1 100000 151.4 660.41 2 GB
1.9 GB
- 3 false false false - - 1 - - 100000 168.0 - - 595.37 - 1 GB
0.9 GB
Indeed, there is a factor of two for both read bytes and written bytes.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: potential indexing perormance improvement for compound index - cut IO - have more files though
Posted by Mike Klaas <mi...@gmail.com>.
On 12/14/06, Doron Cohen <DO...@il.ibm.com> wrote:
> But anyhow, this is not a negligible difference, and for real large
> indexes, and busy systems, when the just written non-compound segment is
> not in the system caches, it might have more effect. Possibly, search
> performance during indexing would be improved by less indexing IO. Also,
> delay for addDocument call that triggers a merge should become smaller.
>
> Thanks for your comments, also (but not only) on (1) an (3) above.
My main comment is that the benefits of this change can be achieved by
using the non-compound index format. For people that care about the
difference in performance, it isn't difficult to configure your system
to mitigate the problems of the non-compound format, and they probably
have already done so.
It would help the people who are file-descriptor conscious, but it
also increases lucene's fd footprint by a factor of four.
-Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: potential indexing perormance improvement for compound index - cut IO -
have more files though
Posted by Doron Cohen <DO...@il.ibm.com>.
Doron Cohen wrote:
> Doug Cutting wrote:
> > > Therefore, a "semi compound" segment file can be defined, that would
be
> > > made of 4 files (instead of 1):
> > > - File 0: .fdx .tis .tvx
> > > - File 1: .fdt .tii .tvd
> > > - File 2: .frq .tvf
> > > - File 3: .fnm .prx .fN
> >
> > I think this is a promising direction. Perhaps instead of adding a
> > third index format, we can significantly improve the non-compound
format
> > without too much effort. For example, simply writing all the norms
into
> > a single file could have a large impact on total file handles and would
> > be a rather simple change. We could start with that, then see if there
> > are further incremental improvements to be had.
>
> We can start with that - at least it would set the number of segment
files
> to a fixed number - 11 - currently it depends on the number of fields
with
> norms.
Okay, started with this step - see issue 756
http://issues.apache.org/jira/browse/LUCENE-756
>
> One advantage of keeping the a plain non-compound format is educational /
> debugging - it is often helpful to actually see the files being created
on
> disk. (Although, just concatenating all norms to a single file is simple
> enough in that regard.)
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: potential indexing perormance improvement for compound index - cut IO -
have more files though
Posted by Doron Cohen <DO...@il.ibm.com>.
Doug Cutting wrote:
> > Therefore, a "semi compound" segment file can be defined, that would be
> > made of 4 files (instead of 1):
> > - File 0: .fdx .tis .tvx
> > - File 1: .fdt .tii .tvd
> > - File 2: .frq .tvf
> > - File 3: .fnm .prx .fN
>
> I think this is a promising direction. Perhaps instead of adding a
> third index format, we can significantly improve the non-compound format
> without too much effort. For example, simply writing all the norms into
> a single file could have a large impact on total file handles and would
> be a rather simple change. We could start with that, then see if there
> are further incremental improvements to be had.
We can start with that - at least it would set the number of segment files
to a fixed number - 11 - currently it depends on the number of fields with
norms.
One advantage of keeping the a plain non-compound format is educational /
debugging - it is often helpful to actually see the files being created on
disk. (Although, just concatenating all norms to a single file is simple
enough in that regard.)
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: potential indexing perormance improvement for compound index
- cut IO - have more files though
Posted by Doug Cutting <cu...@apache.org>.
Doug Cutting wrote:
> I'm not yet convinced that the costs of this mid-point justify its
> benefits.
That was too negative. Let me try a more positive angle.
Doron Cohen wrote:
> Therefore, a "semi compound" segment file can be defined, that would be
> made of 4 files (instead of 1):
> - File 0: .fdx .tis .tvx
> - File 1: .fdt .tii .tvd
> - File 2: .frq .tvf
> - File 3: .fnm .prx .fN
I think this is a promising direction. Perhaps instead of adding a
third index format, we can significantly improve the non-compound format
without too much effort. For example, simply writing all the norms into
a single file could have a large impact on total file handles and would
be a rather simple change. We could start with that, then see if there
are further incremental improvements to be had.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org