You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Anton Zenkov <az...@crimsonhexagon.com> on 2015/06/03 22:00:19 UTC
scanning whole index stored fields while using best compression mode
Hello,
I ran into a problem while trying to make a utility which loads all
documents in the index one by one. Loading was super slow. Orders of
magnitude slower then it is supposed to be. After some debugging and
looking at the code I figured that the culprit was the index compression
which I set to BEST_COMPRESSION since index size is critical for me. Every
time lucene loads a document it decompresses the whole 64K block which this
document is in my documents are very small and I have ~ 10^9 documents per
index which makes scanning the index practically impossible.
There was an addition recently made to the CompressingStoredFieldsReader to
eagerly decompress all docs in the block and cache results but the is only
available during merging.
I was able to get access to the merging reader by doing this:
DirectoryReader reader = DirectoryReader.open(directory);
for (LeafReaderContext leafReaderContext : reader.leaves()) {
StoredFieldsReader fieldsReader =
SlowCodecReaderWrapper.wrap(leafReaderContext.reader()).getFieldsReader().getMergeInstance();
for (int i = 0; i < leafReader.maxDoc(); i++) {
DocumentStoredFieldVisitor visitor = new DocumentStoredFieldVisitor();
fieldsReader.visitDocument(i, visitor);
visitor.getDocument();
}
}
I was wondering if there is better way of doing this and if there are plans
to make access to the faster document loading through some API. Should I
try to come up with a patch for this?
Thanks!
Anton
Re: scanning whole index stored fields while using best compression mode
Posted by Robert Muir <rc...@gmail.com>.
On Wed, Jun 3, 2015 at 4:59 PM, Anton Zenkov <az...@crimsonhexagon.com> wrote:
> Reindexing. If I want to add new fields or change existing fields in the
> index I need to go through all documents of the index.
>
if your reindexing process needs all the docs, i dont think i can
really recommend a better way. a merge reader is the correct thing,
might be slower to initialize but really geared at doing just what you
are doing. in general across the codec api, merge readers will avoid
polluting internal caches (maybe one day OS cache too...), they expose
checkIntegrity() method and other things that seem appropriate for a
process like this.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: scanning whole index stored fields while using best compression mode
Posted by Anton Zenkov <az...@crimsonhexagon.com>.
Reindexing. If I want to add new fields or change existing fields in the
index I need to go through all documents of the index.
On Wed, Jun 3, 2015 at 4:46 PM, Robert Muir <rc...@gmail.com> wrote:
> On Wed, Jun 3, 2015 at 4:00 PM, Anton Zenkov <az...@crimsonhexagon.com>
> wrote:
>
> >
> > for (int i = 0; i < leafReader.maxDoc(); i++) {
> > DocumentStoredFieldVisitor visitor = new
> DocumentStoredFieldVisitor();
> > fieldsReader.visitDocument(i, visitor);
> > visitor.getDocument();
> >
> > }
> >
> > }
> >
> > I was wondering if there is better way of doing this and if there are
> plans
> > to make access to the faster document loading through some API. Should I
> > try to come up with a patch for this?
> >
> > Thanks!
> > Anton
>
> I agree its slow, but what process other than merging really needs to
> loop through all documents and read their stored fields? For merging,
> having a 64KB buffer around doesn't impact users, because there is
> just one thread and its short-lived. Keeping around 64KB ram
> per-segment per-thread in general seems too heavy IMO.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Anton Zenkov | Senior Software Developer
Crimson Hexagon
155 Seaport Boulevard
Boston, MA 02210
@zenant | @crimsonhexagon | O: (617) 547-1072 | Email:
azenkov@crimsonhexagon.com
Re: scanning whole index stored fields while using best compression mode
Posted by Robert Muir <rc...@gmail.com>.
On Wed, Jun 3, 2015 at 4:00 PM, Anton Zenkov <az...@crimsonhexagon.com> wrote:
>
> for (int i = 0; i < leafReader.maxDoc(); i++) {
> DocumentStoredFieldVisitor visitor = new DocumentStoredFieldVisitor();
> fieldsReader.visitDocument(i, visitor);
> visitor.getDocument();
>
> }
>
> }
>
> I was wondering if there is better way of doing this and if there are plans
> to make access to the faster document loading through some API. Should I
> try to come up with a patch for this?
>
> Thanks!
> Anton
I agree its slow, but what process other than merging really needs to
loop through all documents and read their stored fields? For merging,
having a 64KB buffer around doesn't impact users, because there is
just one thread and its short-lived. Keeping around 64KB ram
per-segment per-thread in general seems too heavy IMO.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org