You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Jason Rutherglen <ja...@gmail.com> on 2009/04/02 01:05:21 UTC
Future projects
Now that LUCENE-1516 is close to being committed perhaps we can
figure out the priority of other issues:
1. Searchable IndexWriter RAM buffer
2. Finish up benchmarking and perhaps implement passing
filters to the SegmentReader level
3. Deleting by doc id using IndexWriter
With 1) I'm interested in how we will lock a section of the
bytes for use by a given reader? We would not actually lock
them, but we need to set aside the bytes such that for example
if the postings grows, TermDocs iteration does not progress to
beyond it's limits. Are there any modifications that are needed
of the RAM buffer format? How would the term table be stored? We
would not be using the current hash method?
Re: Future projects
Posted by John Wang <jo...@gmail.com>.
Michael:
I love your suggestion on 3)!
This really opens doors for flexible indexing.
-John
On Thu, Apr 2, 2009 at 1:40 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> On Wed, Apr 1, 2009 at 7:05 PM, Jason Rutherglen
> <ja...@gmail.com> wrote:
> > Now that LUCENE-1516 is close to being committed perhaps we can
> > figure out the priority of other issues:
> >
> > 1. Searchable IndexWriter RAM buffer
>
> I think first priority is to get a good assessment of the performance
> of the current implementation (from LUCENE-1516).
>
> My initial tests are very promising: with a writer updating (replacing
> random docs) at 50 docs/second on a full (3.2 M) Wikipedia index, I
> was able to get reopen the reader once per second and do a large (>
> 500K results) search that sorts by date. The reopen time was
> typically ~40 msec, and search time typically ~35 msec (though there
> were random spikes up to ~340 msec). Though, these results were on an
> SSD (Intel X25M 160 GB).
>
> We need more datapoints of the current approach, but this looks likely
> to be good enough for starters. And since we can get it into 2.9,
> hopefully it'll get some early usage and people will report back to
> help us assess whether further performance improvements are necessary.
>
> If they do turn out to be necessary, I think before your step 1, we
> should write small segments into a RAMDirectory instead of the "real"
> directory. That's simpler than truly searching IndexWriter's
> in-memory postings data.
>
> > 2. Finish up benchmarking and perhaps implement passing
> > filters to the SegmentReader level
>
> What is "passing filters to the SegmentReader level"? EG as of
> LUCENE-1483, we now ask a Filter for it's DocIdSet once per
> SegmentReader.
>
> > 3. Deleting by doc id using IndexWriter
>
> We need a clean approach for the "docIDs suddenly shift when merge is
> committed" problem for this...
>
> Thinking more on this... I think one possible solution may be to
> somehow expose IndexWriter's internal docID remapping code.
> IndexWriter does delete by docID internally, and whenever a merge is
> committed we stop-the-world (sync on IW) and go remap those docIDs.
> If we somehow allowed user to register a callback that we could call
> when this remapping occurs, then user's code could carry the docIDs
> without them becoming stale. Or maybe we could make a class
> "PendingDocIDs", which you'd ask the reader to give you, that holds
> docIDs and remaps them after each merge. The problem is, IW
> internally always logically switches to the current reader for any
> further docID deletion, but the user's code may continue to use an old
> reader. So simply exposing this remapping won't fix it... we'd need
> to somehow track the genealogy (quite a bit more complex).
>
> > With 1) I'm interested in how we will lock a section of the
> > bytes for use by a given reader? We would not actually lock
> > them, but we need to set aside the bytes such that for example
> > if the postings grows, TermDocs iteration does not progress to
> > beyond it's limits. Are there any modifications that are needed
> > of the RAM buffer format? How would the term table be stored? We
> > would not be using the current hash method?
>
> I think the realtime reader'd just store the maxDocID it's allowed to
> search, and we would likely keep using the RAM format now used.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: Future projects
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Tue, Apr 7, 2009 at 7:05 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
> > I think we should keep it simple, unless we discover real perf problems
> with the current approach.
>
> Simple is good, however the indexing performance will lag because we're back
> to the indexing speed of pre ram buffer? (i.e. merging segments using a
> ramdirectory).
Only if you open a new reader after each added document.
Then, your indexing throughput will be low. But, that's expected (you
don't use NRT to maximize indexing throughput... you use it to
minimize index-to-search delay).
I wouldn't expect this to be a problem in practice, though. Because
if you find the indexing throughput is too low for your app (ie, you
need NRT and a relative fast indexing throughput), then don't open the
NRT reader after every single doc. EG, open it every 100 ms instead,
and you'll increase your indexing throughput.
>> need to do a merge sort (across the N thread states)
>
> I'm confused about why a merge sort is required?
The docIDs in each thread's ram buffer do not "concatenate" like
MultiReader. Instead, they "interleave". One thread has docIDs 0, 4,
6. Another has 1, 5, 7. Another has 2, 3, 9. Etc. So a TermDocs
must "zip" them back together when iterating.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Future projects
Posted by Jason Rutherglen <ja...@gmail.com>.
> I think we should keep it simple, unless we discover real perf problems
with the current approach.
Simple is good, however the indexing performance will lag because we're back
to the indexing speed of pre ram buffer? (i.e. merging segments using a
ramdirectory).
> need to do a merge sort (across the N thread states)
I'm confused about why a merge sort is required?
On Tue, Apr 7, 2009 at 1:45 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> On Mon, Apr 6, 2009 at 6:43 PM, Jason Rutherglen
> <ja...@gmail.com> wrote:
> >> The realtime reader would have to have sub-readers per thread,
> > and an aggregate reader that "joins" them by interleaving the
> > docIDs
> >
> > Nice (i.e. nice and complex)!
>
> Right, this is why I like the current [simple] near real-time
> approach. I think we should keep it simple, unless we discover real
> perf problems with the current approach.
>
> > Not knowing too much about the
> > internals, how would the interleaving work? Does each subreader
> > have a "start" ala Multi*Reader? Or are the doc ids incremented
> > from a synced place such that no two readers have the same doc
> > id?
>
> The docID must be woven/interleaved together (unlike MultiReader,
> where they are concatenated). DW ensures that a given docID is used
> by only 1 thread. So you'd need to do a merge sort (across the N
> thread states) on reading the postings for a given term. Probably
> we'd then suggest for best searching performance to use a single
> thread for indexing when NRT search will be used.
>
> >> BTW there are benefits to not reusing the RAM buffer, outside
> > of faster near real-time search
> >
> > Not reusing the RAM buffer means not reusing the pooled byte
> > arrays after a flush or something else?
>
> Pooled byte, char and int arrays, *PerThread, *PerField classes, norms,
> etc.
>
> > SSDs are cool, I can't see management approving of those quite
> > yet, are there many places piloting Lucene on SSDs that you're
> > aware of?
>
> Yes they are still somewhat expensive, though the gain in productivity
> is sizable, and prices have been coming down...
>
> EG I have a zillion Lucene source code checkouts, and it used to be
> whenever I switch back to one and do an "svn up" or "svn diff", it's a
> good 30 seconds of disk heads grinding away before anything really
> happened. Now it's a second or two. VMWare/Parellels also become
> much more responsive. Not hearing disk heads grinding is somewhat
> disconcerting at first, though.
>
> At least several people on java-user have posted benchmarks with SSDs.
>
> SSDs are clearly the future and I think we need to think more about
> what their adoption means for our prioritization of Lucene's ongoing
> improvements. EG I think it means the CPU cost of searching, single
> search concurrency (using more than one thread on one search), become
> important, because once your index is on an SSD Lucene will spend far
> less time waiting for IO to complete even on "normal" installations
> that don't cache the entire index in RAM. I think we especially need
> to figure out how to leverage concurrency in the IO system (but alas
> we don't have an async IO API from Java... we'd have to "emulate" it
> using threads).
>
> > From what you've said so far, this is how I understand realtime
> > ram buffer readers could work:
> >
> > There'd be a IndexWriter.getRAMReader method that gathers all
> > the ram buffers from the various threads, marks a doc id as the
> > last one for the overall RAMBufferMultiReader. A new set of
> > classes, RAMBufferTermEnum, RAMBufferTermDocs,
> > RAMBufferTermPositions would be implemented that can read from
> > the ram buffer.
>
> Right, but we shouldn't start work on this until we see a reason to.
> And even once that reason appears, we should next do the intermediate
> optimization of using RAMDir for newly flushed segments.
>
> > I don't think the current field cache API would like growing
> > arrays? Something hopefully LUCENE-831 will support.
>
> I'm thinking for LUCENE-831 we should make field-cache segment
> centric, which would then play well w/ NRT.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: Future projects
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Mon, Apr 6, 2009 at 6:43 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
>> The realtime reader would have to have sub-readers per thread,
> and an aggregate reader that "joins" them by interleaving the
> docIDs
>
> Nice (i.e. nice and complex)!
Right, this is why I like the current [simple] near real-time
approach. I think we should keep it simple, unless we discover real
perf problems with the current approach.
> Not knowing too much about the
> internals, how would the interleaving work? Does each subreader
> have a "start" ala Multi*Reader? Or are the doc ids incremented
> from a synced place such that no two readers have the same doc
> id?
The docID must be woven/interleaved together (unlike MultiReader,
where they are concatenated). DW ensures that a given docID is used
by only 1 thread. So you'd need to do a merge sort (across the N
thread states) on reading the postings for a given term. Probably
we'd then suggest for best searching performance to use a single
thread for indexing when NRT search will be used.
>> BTW there are benefits to not reusing the RAM buffer, outside
> of faster near real-time search
>
> Not reusing the RAM buffer means not reusing the pooled byte
> arrays after a flush or something else?
Pooled byte, char and int arrays, *PerThread, *PerField classes, norms, etc.
> SSDs are cool, I can't see management approving of those quite
> yet, are there many places piloting Lucene on SSDs that you're
> aware of?
Yes they are still somewhat expensive, though the gain in productivity
is sizable, and prices have been coming down...
EG I have a zillion Lucene source code checkouts, and it used to be
whenever I switch back to one and do an "svn up" or "svn diff", it's a
good 30 seconds of disk heads grinding away before anything really
happened. Now it's a second or two. VMWare/Parellels also become
much more responsive. Not hearing disk heads grinding is somewhat
disconcerting at first, though.
At least several people on java-user have posted benchmarks with SSDs.
SSDs are clearly the future and I think we need to think more about
what their adoption means for our prioritization of Lucene's ongoing
improvements. EG I think it means the CPU cost of searching, single
search concurrency (using more than one thread on one search), become
important, because once your index is on an SSD Lucene will spend far
less time waiting for IO to complete even on "normal" installations
that don't cache the entire index in RAM. I think we especially need
to figure out how to leverage concurrency in the IO system (but alas
we don't have an async IO API from Java... we'd have to "emulate" it
using threads).
> From what you've said so far, this is how I understand realtime
> ram buffer readers could work:
>
> There'd be a IndexWriter.getRAMReader method that gathers all
> the ram buffers from the various threads, marks a doc id as the
> last one for the overall RAMBufferMultiReader. A new set of
> classes, RAMBufferTermEnum, RAMBufferTermDocs,
> RAMBufferTermPositions would be implemented that can read from
> the ram buffer.
Right, but we shouldn't start work on this until we see a reason to.
And even once that reason appears, we should next do the intermediate
optimization of using RAMDir for newly flushed segments.
> I don't think the current field cache API would like growing
> arrays? Something hopefully LUCENE-831 will support.
I'm thinking for LUCENE-831 we should make field-cache segment
centric, which would then play well w/ NRT.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Future projects
Posted by Jason Rutherglen <ja...@gmail.com>.
> The realtime reader would have to have sub-readers per thread,
and an aggregate reader that "joins" them by interleaving the
docIDs
Nice (i.e. nice and complex)! Not knowing too much about the
internals, how would the interleaving work? Does each subreader
have a "start" ala Multi*Reader? Or are the doc ids incremented
from a synced place such that no two readers have the same doc
id?
> BTW there are benefits to not reusing the RAM buffer, outside
of faster near real-time search
Not reusing the RAM buffer means not reusing the pooled byte
arrays after a flush or something else?
> thus allowing add/deletes in other threads to run. Currently
they are all blocked ("stop the world") during flush
SSDs are cool, I can't see management approving of those quite
yet, are there many places piloting Lucene on SSDs that you're
aware of?
>From what you've said so far, this is how I understand realtime
ram buffer readers could work:
There'd be a IndexWriter.getRAMReader method that gathers all
the ram buffers from the various threads, marks a doc id as the
last one for the overall RAMBufferMultiReader. A new set of
classes, RAMBufferTermEnum, RAMBufferTermDocs,
RAMBufferTermPositions would be implemented that can read from
the ram buffer.
I don't think the current field cache API would like growing
arrays? Something hopefully LUCENE-831 will support.
On Sat, Apr 4, 2009 at 4:46 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> On Fri, Apr 3, 2009 at 8:01 PM, Jason Rutherglen
> <ja...@gmail.com> wrote:
> > I looked at the IndexWriter code in regards to creating a realtime
> reader,
> > with the many flexible indexing classes I'm unsure of how one would get a
> > frozenish IndexInput of the byte slices, given the byte slices are
> attached
> > to different threads?
>
> The realtime reader would have to have sub-readers per thread, and an
> aggregate reader that "joins" them by interleaving the docIDs. When
> flushing we create such a beast, but, it's not general purpose (ie it
> does not implement IndexReader API; it only implements enough to write
> the postings).
>
> BTW there are benefits to not reusing the RAM buffer, outside of
> faster near real-time search: it would allow flushing to be done in
> the BG. Ie, flush could start, and we'd immediately switch to a new
> RAM buffer, thus allowing add/deletes in other threads to run.
> Currently they are all blocked ("stop the world") during flush, though
> it's not clear on a fast IO device (SSD) how big a deal this "stop the
> world" really is to indexing throughput.
>
> But still it's a complex change.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: Future projects
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Fri, Apr 3, 2009 at 8:01 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
> I looked at the IndexWriter code in regards to creating a realtime reader,
> with the many flexible indexing classes I'm unsure of how one would get a
> frozenish IndexInput of the byte slices, given the byte slices are attached
> to different threads?
The realtime reader would have to have sub-readers per thread, and an
aggregate reader that "joins" them by interleaving the docIDs. When
flushing we create such a beast, but, it's not general purpose (ie it
does not implement IndexReader API; it only implements enough to write
the postings).
BTW there are benefits to not reusing the RAM buffer, outside of
faster near real-time search: it would allow flushing to be done in
the BG. Ie, flush could start, and we'd immediately switch to a new
RAM buffer, thus allowing add/deletes in other threads to run.
Currently they are all blocked ("stop the world") during flush, though
it's not clear on a fast IO device (SSD) how big a deal this "stop the
world" really is to indexing throughput.
But still it's a complex change.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Future projects
Posted by Jason Rutherglen <ja...@gmail.com>.
I looked at the IndexWriter code in regards to creating a realtime reader,
with the many flexible indexing classes I'm unsure of how one would get a
frozenish IndexInput of the byte slices, given the byte slices are attached
to different threads?
On Fri, Apr 3, 2009 at 2:42 PM, Jason Rutherglen <jason.rutherglen@gmail.com
> wrote:
> > I think the realtime reader'd just store the maxDocID it's allowed to
> search, and we would likely keep using the RAM format now used.
>
> Sounds pretty good. Are there any other gotchas in the design?
>
>
>
> On Thu, Apr 2, 2009 at 1:40 AM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> On Wed, Apr 1, 2009 at 7:05 PM, Jason Rutherglen
>> <ja...@gmail.com> wrote:
>> > Now that LUCENE-1516 is close to being committed perhaps we can
>> > figure out the priority of other issues:
>> >
>> > 1. Searchable IndexWriter RAM buffer
>>
>> I think first priority is to get a good assessment of the performance
>> of the current implementation (from LUCENE-1516).
>>
>> My initial tests are very promising: with a writer updating (replacing
>> random docs) at 50 docs/second on a full (3.2 M) Wikipedia index, I
>> was able to get reopen the reader once per second and do a large (>
>> 500K results) search that sorts by date. The reopen time was
>> typically ~40 msec, and search time typically ~35 msec (though there
>> were random spikes up to ~340 msec). Though, these results were on an
>> SSD (Intel X25M 160 GB).
>>
>> We need more datapoints of the current approach, but this looks likely
>> to be good enough for starters. And since we can get it into 2.9,
>> hopefully it'll get some early usage and people will report back to
>> help us assess whether further performance improvements are necessary.
>>
>> If they do turn out to be necessary, I think before your step 1, we
>> should write small segments into a RAMDirectory instead of the "real"
>> directory. That's simpler than truly searching IndexWriter's
>> in-memory postings data.
>>
>> > 2. Finish up benchmarking and perhaps implement passing
>> > filters to the SegmentReader level
>>
>> What is "passing filters to the SegmentReader level"? EG as of
>> LUCENE-1483, we now ask a Filter for it's DocIdSet once per
>> SegmentReader.
>>
>> > 3. Deleting by doc id using IndexWriter
>>
>> We need a clean approach for the "docIDs suddenly shift when merge is
>> committed" problem for this...
>>
>> Thinking more on this... I think one possible solution may be to
>> somehow expose IndexWriter's internal docID remapping code.
>> IndexWriter does delete by docID internally, and whenever a merge is
>> committed we stop-the-world (sync on IW) and go remap those docIDs.
>> If we somehow allowed user to register a callback that we could call
>> when this remapping occurs, then user's code could carry the docIDs
>> without them becoming stale. Or maybe we could make a class
>> "PendingDocIDs", which you'd ask the reader to give you, that holds
>> docIDs and remaps them after each merge. The problem is, IW
>> internally always logically switches to the current reader for any
>> further docID deletion, but the user's code may continue to use an old
>> reader. So simply exposing this remapping won't fix it... we'd need
>> to somehow track the genealogy (quite a bit more complex).
>>
>> > With 1) I'm interested in how we will lock a section of the
>> > bytes for use by a given reader? We would not actually lock
>> > them, but we need to set aside the bytes such that for example
>> > if the postings grows, TermDocs iteration does not progress to
>> > beyond it's limits. Are there any modifications that are needed
>> > of the RAM buffer format? How would the term table be stored? We
>> > would not be using the current hash method?
>>
>> I think the realtime reader'd just store the maxDocID it's allowed to
>> search, and we would likely keep using the RAM format now used.
>>
>> Mike
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>
Re: Future projects
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Fri, Apr 3, 2009 at 5:42 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
>> I think the realtime reader'd just store the maxDocID it's allowed to
>> search, and we would likely keep using the RAM format now used.
>
> Sounds pretty good. Are there any other gotchas in the design?
Yes: the flushing process becomes challenging. When we flush, we must
forcefully cutover any open readers searching that RAM buffer, to the
now-on-disk segment. Such swap-out is tricky because there could be
[many threads of] searches in-flight, iterating through postings, etc.
Which means the RAM buffer would have to become an independent entity
that is not re-used after flushing, but instead sticks around until GC
determines all outstanding readers have switched to the on-disk
segment.
I would rather not go here unless it's clear the current near
real-time performance is too limiting. But my simplistic test the
current performance looks good.
And, if the performance does turn out to be lacking, the next step
(before searching IW's ram buffer) is to flush the new tiny segments
through a RAMDir, first.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Future projects
Posted by Jason Rutherglen <ja...@gmail.com>.
> I think the realtime reader'd just store the maxDocID it's allowed to
search, and we would likely keep using the RAM format now used.
Sounds pretty good. Are there any other gotchas in the design?
On Thu, Apr 2, 2009 at 1:40 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> On Wed, Apr 1, 2009 at 7:05 PM, Jason Rutherglen
> <ja...@gmail.com> wrote:
> > Now that LUCENE-1516 is close to being committed perhaps we can
> > figure out the priority of other issues:
> >
> > 1. Searchable IndexWriter RAM buffer
>
> I think first priority is to get a good assessment of the performance
> of the current implementation (from LUCENE-1516).
>
> My initial tests are very promising: with a writer updating (replacing
> random docs) at 50 docs/second on a full (3.2 M) Wikipedia index, I
> was able to get reopen the reader once per second and do a large (>
> 500K results) search that sorts by date. The reopen time was
> typically ~40 msec, and search time typically ~35 msec (though there
> were random spikes up to ~340 msec). Though, these results were on an
> SSD (Intel X25M 160 GB).
>
> We need more datapoints of the current approach, but this looks likely
> to be good enough for starters. And since we can get it into 2.9,
> hopefully it'll get some early usage and people will report back to
> help us assess whether further performance improvements are necessary.
>
> If they do turn out to be necessary, I think before your step 1, we
> should write small segments into a RAMDirectory instead of the "real"
> directory. That's simpler than truly searching IndexWriter's
> in-memory postings data.
>
> > 2. Finish up benchmarking and perhaps implement passing
> > filters to the SegmentReader level
>
> What is "passing filters to the SegmentReader level"? EG as of
> LUCENE-1483, we now ask a Filter for it's DocIdSet once per
> SegmentReader.
>
> > 3. Deleting by doc id using IndexWriter
>
> We need a clean approach for the "docIDs suddenly shift when merge is
> committed" problem for this...
>
> Thinking more on this... I think one possible solution may be to
> somehow expose IndexWriter's internal docID remapping code.
> IndexWriter does delete by docID internally, and whenever a merge is
> committed we stop-the-world (sync on IW) and go remap those docIDs.
> If we somehow allowed user to register a callback that we could call
> when this remapping occurs, then user's code could carry the docIDs
> without them becoming stale. Or maybe we could make a class
> "PendingDocIDs", which you'd ask the reader to give you, that holds
> docIDs and remaps them after each merge. The problem is, IW
> internally always logically switches to the current reader for any
> further docID deletion, but the user's code may continue to use an old
> reader. So simply exposing this remapping won't fix it... we'd need
> to somehow track the genealogy (quite a bit more complex).
>
> > With 1) I'm interested in how we will lock a section of the
> > bytes for use by a given reader? We would not actually lock
> > them, but we need to set aside the bytes such that for example
> > if the postings grows, TermDocs iteration does not progress to
> > beyond it's limits. Are there any modifications that are needed
> > of the RAM buffer format? How would the term table be stored? We
> > would not be using the current hash method?
>
> I think the realtime reader'd just store the maxDocID it's allowed to
> search, and we would likely keep using the RAM format now used.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: Future projects
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Thu, Apr 2, 2009 at 2:29 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
>> What is "passing filters to the SegmentReader level"? EG as of
> LUCENE-1483, we now ask a Filter for it's DocIdSet once per
> SegmentReader.
>
> The patch I was thinking of is LUCENE-1536. I wasn't sure what
> the next steps are for it, i.e. the JumpScorer,
> Scorer.skipToButNotNext, or simply implementing a commitable
> version of LUCENE-1536?
Ahh OK. We should pursue this one -- many filters are cached, or
would otherwise be able to expose random-access API. For such
filters, it'd also make sense to pre-multiply the deleted docs, to
save doing that multiply for every query that uses the filter. We'd
need some sort of caching / segment wrapper class to manage that,
maybe?
But we should first do the Filter/Query unification, and Filter as
clause on BooleanQuery, and then re-assess the performance difference.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Future projects
Posted by Jason Rutherglen <ja...@gmail.com>.
> What is "passing filters to the SegmentReader level"? EG as of
LUCENE-1483, we now ask a Filter for it's DocIdSet once per
SegmentReader.
The patch I was thinking of is LUCENE-1536. I wasn't sure what
the next steps are for it, i.e. the JumpScorer,
Scorer.skipToButNotNext, or simply implementing a commitable
version of LUCENE-1536?
On Thu, Apr 2, 2009 at 1:40 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> On Wed, Apr 1, 2009 at 7:05 PM, Jason Rutherglen
> <ja...@gmail.com> wrote:
> > Now that LUCENE-1516 is close to being committed perhaps we can
> > figure out the priority of other issues:
> >
> > 1. Searchable IndexWriter RAM buffer
>
> I think first priority is to get a good assessment of the performance
> of the current implementation (from LUCENE-1516).
>
> My initial tests are very promising: with a writer updating (replacing
> random docs) at 50 docs/second on a full (3.2 M) Wikipedia index, I
> was able to get reopen the reader once per second and do a large (>
> 500K results) search that sorts by date. The reopen time was
> typically ~40 msec, and search time typically ~35 msec (though there
> were random spikes up to ~340 msec). Though, these results were on an
> SSD (Intel X25M 160 GB).
>
> We need more datapoints of the current approach, but this looks likely
> to be good enough for starters. And since we can get it into 2.9,
> hopefully it'll get some early usage and people will report back to
> help us assess whether further performance improvements are necessary.
>
> If they do turn out to be necessary, I think before your step 1, we
> should write small segments into a RAMDirectory instead of the "real"
> directory. That's simpler than truly searching IndexWriter's
> in-memory postings data.
>
> > 2. Finish up benchmarking and perhaps implement passing
> > filters to the SegmentReader level
>
> What is "passing filters to the SegmentReader level"? EG as of
> LUCENE-1483, we now ask a Filter for it's DocIdSet once per
> SegmentReader.
>
> > 3. Deleting by doc id using IndexWriter
>
> We need a clean approach for the "docIDs suddenly shift when merge is
> committed" problem for this...
>
> Thinking more on this... I think one possible solution may be to
> somehow expose IndexWriter's internal docID remapping code.
> IndexWriter does delete by docID internally, and whenever a merge is
> committed we stop-the-world (sync on IW) and go remap those docIDs.
> If we somehow allowed user to register a callback that we could call
> when this remapping occurs, then user's code could carry the docIDs
> without them becoming stale. Or maybe we could make a class
> "PendingDocIDs", which you'd ask the reader to give you, that holds
> docIDs and remaps them after each merge. The problem is, IW
> internally always logically switches to the current reader for any
> further docID deletion, but the user's code may continue to use an old
> reader. So simply exposing this remapping won't fix it... we'd need
> to somehow track the genealogy (quite a bit more complex).
>
> > With 1) I'm interested in how we will lock a section of the
> > bytes for use by a given reader? We would not actually lock
> > them, but we need to set aside the bytes such that for example
> > if the postings grows, TermDocs iteration does not progress to
> > beyond it's limits. Are there any modifications that are needed
> > of the RAM buffer format? How would the term table be stored? We
> > would not be using the current hash method?
>
> I think the realtime reader'd just store the maxDocID it's allowed to
> search, and we would likely keep using the RAM format now used.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: Future projects
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Thu, Apr 2, 2009 at 6:55 PM, John Wang <jo...@gmail.com> wrote:
> Just to clarify, Approach 1 and approach 2 are both currently performing ok
> currently for us.
OK that's very good to know.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Future projects
Posted by John Wang <jo...@gmail.com>.
Just to clarify, Approach 1 and approach 2 are both currently performing ok
currently for us.
-John
On Thu, Apr 2, 2009 at 2:41 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> On Thu, Apr 2, 2009 at 4:43 PM, Jason Rutherglen
> <ja...@gmail.com> wrote:
> >> What does Bobo use the cached bitsets for?
> >
> > Bobo is a faceting engine that uses custom field caches and sometimes
> cached
> > bitsets rather than relying exclusively on bitsets to calculate facets.
> It
> > is useful where many facets (50+) need to be calculated and bitset
> caching,
> > loading and intersection would be too costly. Instead it iterates over
> in
> > memory custom field caches while hit collecting. Because we're also
> doing
> > realtime search, making the loading more efficient via the in memory
> field
> > cache merging is interesting.
>
> OK.
>
> Does it operate at the segment level? Seems like that'd give you good
> enough realtime performance (though merging in RAM will definitely be
> faster).
>
> > True, we do the in memory merging with deleted docs, norms would be good
> as
> > well.
>
> Yes, and eventually column stride fields.
>
> > As a first step how should we expose the segments a segment has
> > originated from?
>
> I'm not sure; it's quite messy. Each segment must track what other
> segment it got merged to, and must hold a copy of its deletes as of
> the time it was merged. And each segment must know what other
> segments it got merged with.
>
> Is this really a serious problem in your realtime search? Eg, from
> John's numbers in using payloads to read in the docID -> UID mapping,
> it seems like you could make a Query that when given a reader would go
> and do the "Approach 2" to perform the deletes (if indeed you are
> needing to delete thousands of docs with each update). What sort of
> docs/sec rates are you needing to handle?
>
> > I would like to get this implemented for 2.9 as a building
> > block that perhaps we can write other things on.
>
> I think that's optimistic. It's still at the
> hairy-can't-see-a-clean-way-to-do-it phase still. Plus I'd like to
> understand that all other options have been exhausted first.
>
> Especially once we have column stride fields and they are merged in
> RAM, you'll be handed a reader pre-warmed and you can then jump
> through those arrays to find docs to delete.
>
> > Column stride fields still
> > requires some encoding and merging field caches in ram would presumably
> be
> > faster?
>
> Yes, potentially much faster. There's no sense in writing through to
> disk until commit is called.
>
> >> Ie we only have to renumber from gen X to X+1, then from X+1 to X+2
> (where
> >> each "generation" is a renumbering event).
> >
> > Couldn't each SegmentReader keep a docmap and the names of the segments
> it
> > originated from. However the name is not enough of a unique key as
> there's
> > the deleted docs that change? It seems like we need a unique id for each
> > segment reader, where the id is assigned to cloned readers (which
> normally
> > have the same segment name as the original SR). The ID could be a stamp
> > (perhaps only given to readonlyreaders?). That way the
> > SegmentReader.getMergedFrom method does not need to return the actual
> > readers, but a docmap and the parent readers IDs? It would be assumed
> the
> > user would be holding the readers somewhere? Perhaps all this can be
> > achieved with a callback in IW, and all this logic could be kept somewhat
> > internal to Lucene?
>
> The docMap is a costly way to store it, since it consumes 32 bits per
> doc (vs storing a copy of the deleted docs).
>
> But, then docMap gives you random-access on the map.
>
> What if prior to merging, or committing merged deletes, there were a
> callback to force the app to materialize any privately buffered
> deletes? And then the app is not allowed to use those readers for
> further deletes? Still kinda messy.
>
> I think I need to understand better why delete by Query isn't viable
> in your situation...
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: Future projects
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Fri, Apr 3, 2009 at 3:16 PM, John Wang <jo...@gmail.com> wrote:
> By default bobo DOES use a flavor of the field cache data structure with
> some addition information for performance. (e.g. minDocid,maxDocid,freq per
> term)
> Bobo is architected as a platform where clients can write their own
> "FacetHandlers" in which each FacetHandler manages its own view of memory
> structure, and thus can be more complicated that field cache.
> At LinkedIn, we write FacetHandlers for geo lat/lon filtering and social
> network faceting.
OK, neat. Hopefully with LUCENE-831, LUCENE-1231 we can properly take
this usage of field cache into account.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Future projects
Posted by John Wang <jo...@gmail.com>.
By default bobo DOES use a flavor of the field cache data structure with
some addition information for performance. (e.g. minDocid,maxDocid,freq per
term)
Bobo is architected as a platform where clients can write their own
"FacetHandlers" in which each FacetHandler manages its own view of memory
structure, and thus can be more complicated that field cache.
At LinkedIn, we write FacetHandlers for geo lat/lon filtering and social
network faceting.
-John
On Fri, Apr 3, 2009 at 3:35 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> On Thu, Apr 2, 2009 at 5:56 PM, Jason Rutherglen
> <ja...@gmail.com> wrote:
> >> I think I need to understand better why delete by Query isn't
> > viable in your situation...
> >
> > The delete by query is a separate problem which I haven't fully
> > explored yet.
>
> Oh, I had thought we were tugging on this thread in order to explore
> delete-by-docID in the writer. OK.
>
> > Tracking the segment genealogy is really an
> > interim step for merging field caches before column stride
> > fields gets implemented.
>
> I see -- meaning in Bobo you'd like to manage your own memory resident
> field caches, and merge them whenever IW has merged a segment? Seems
> like you don't need genealogy for that.
>
> > Actually CSF cannot be used with Bobo's
> > field caches anyways which means we'd need a way to find out
> > about the segment parents.
>
> CSF isn't really designed yet. How come it can't be used with Bobo's
> field caches? We can try to accommodate Bobo's field cache needs when
> designing CSF.
>
> >> Does it operate at the segment level? Seems like that'd give
> > you good enough realtime performance (though merging in RAM will
> > definitely be faster).
> >
> > We need to see how Bobo integrates with LUCENE-1483.
>
> Lucene's internal field cache usage is now entirely at the segment
> level (ie, Lucene core should never request full field cache array at
> the MultiSegmentReader level). I think Bobo must have to do the same,
> if it handles near realtime updates, to get adequate performance.
>
> Though... since we have LUCENE-831 (rework API Lucene exposes for
> accessing arrays-of-atomic-types-per-segment) and LUCENE-1231 (CSF = a
> more efficient impl (than uninversion) of the API we expose in
> LUCENE-831) on deck, we should try to understand Bobo's needs.
>
> EG how come Bobo made its own field cache impl? Just because
> uninversion is too slow?
>
> > It seems like we've been talking about CSF for 2 years and there
> > isn't a patch for it? If I had more time I'd take a look. What
> > is the status of it?
>
> I think Michael is looking into it? I'd really like to get it into
> 2.9. We should do it in conjunction with 831 since they are so tied.
>
> > I'll write a patch that implements a callback for the segment
> > merging such that the user can decide what information they want
> > to record about the merged SRs (I'm pretty sure there isn't a
> > way to do this with MergePolicy?)
>
> Actually I think you can do this w/ a simple MergeScheduler wrapper or
> by subclassing CMS. I'll put a comment on the issue.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: Future projects
Posted by Jason Rutherglen <ja...@gmail.com>.
Hey Michael,
You're in San Jose? Feel free to come by one of these days on our pizza
days.
Also, can you post what you have of LUCENE-1231? I got a lot more familiar
with IndexWriter internals with LUCENE-1516 and could to a good whack at
getting LUCENE-1231 integrated.
Cheers!
Jason
On Sun, Apr 12, 2009 at 3:28 PM, Michael Busch <bu...@gmail.com> wrote:
> On 4/4/09 4:42 AM, Michael McCandless wrote:
>
> As I recently mentioned on 1231 I'm looking into changing the Document and
> Field APIs. I've some rough prototype. I think we should also try to get it
> in before 2.9? On the other hand I don't want to block the 2.9 release with
> too much stuff.
>
>
> That'd be great -- I'd say post the rough prototype and let's iterate?
>
>
>
>
> OK. I'll attach it as a new Jira issue. It's not really integrated into
> anything (like DocumentsWriter), but I wrote some demo classes to show how I
> intend to use it.
>
> -Michael
>
Re: Future projects
Posted by Michael Busch <bu...@gmail.com>.
On 4/4/09 4:42 AM, Michael McCandless wrote:
>> As I recently mentioned on 1231 I'm looking into changing the Document and
>> Field APIs. I've some rough prototype. I think we should also try to get it
>> in before 2.9? On the other hand I don't want to block the 2.9 release with
>> too much stuff.
>>
>
> That'd be great -- I'd say post the rough prototype and let's iterate?
>
>
OK. I'll attach it as a new Jira issue. It's not really integrated into
anything (like DocumentsWriter), but I wrote some demo classes to show
how I intend to use it.
-Michael
Re: Future projects
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Fri, Apr 3, 2009 at 7:11 PM, Michael Busch <bu...@gmail.com> wrote:
> Yeah me too. I think eventually we want this to be a Codec, but we probably
> don't want to wait until all the flexible indexing work is done.
> So maybe we should just not worry too much about a perfectly integrated API
> yet and release it as experimental API with 2.9 and 3.0, just like we did
> with the initial payloads implementation. Then when we hopefully get
> flexible indexing and codecs into 3.1 we can rework CSF and integrate it as
> a Codec.
+1
LUCENE-1458 shouldn't block 831/1231.
> As I recently mentioned on 1231 I'm looking into changing the Document and
> Field APIs. I've some rough prototype. I think we should also try to get it
> in before 2.9? On the other hand I don't want to block the 2.9 release with
> too much stuff.
That'd be great -- I'd say post the rough prototype and let's iterate?
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Future projects
Posted by Michael Busch <bu...@gmail.com>.
On 4/3/09 3:35 AM, Michael McCandless wrote:
>> It seems like we've been talking about CSF for 2 years and there
>> isn't a patch for it? If I had more time I'd take a look. What
>> is the status of it?
>>
>
> I think Michael is looking into it? I'd really like to get it into
> 2.9. We should do it in conjunction with 831 since they are so tied.
>
Yeah me too. I think eventually we want this to be a Codec, but we
probably don't want to wait until all the flexible indexing work is done.
So maybe we should just not worry too much about a perfectly integrated
API yet and release it as experimental API with 2.9 and 3.0, just like
we did with the initial payloads implementation. Then when we hopefully
get flexible indexing and codecs into 3.1 we can rework CSF and
integrate it as a Codec.
As I recently mentioned on 1231 I'm looking into changing the Document
and Field APIs. I've some rough prototype. I think we should also try to
get it in before 2.9? On the other hand I don't want to block the 2.9
release with too much stuff.
-Michael
Re: Future projects
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Fri, Apr 3, 2009 at 5:32 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
>> meaning in Bobo you'd like to manage your own memory resident
> field caches, and merge them whenever IW has merged a segment?
> Seems like you don't need genealogy for that.
>
> Agreed, there is no need for full genealogy.
OK
>> CSF isn't really designed yet. How come it can't be used with
> Bobo's field caches?
>
> I guess CSF should be able to support it, makes sense. As long
> as the container is flexible with the encoding (I need to look
> into this more on the Bobo side).
Well as CSF unfolds let's take Bobo's usage into account.
>> Lucene's internal field cache usage is now entirely at the
> segment level (ie, Lucene core should never request full field
> cache array at the MultiSegmentReader level). I think Bobo must
> have to do the same, if it handles near realtime updates, to get
> adequate performance.
>
> Bobo needs to migrate to this model, I don't think we've done
> that yet.
Hmm OK. That's means reopen is very costly?
>> EG how come Bobo made its own field cache impl? Just because
> uninversion is too slow?
>
> It could be integrated once LUCENE-831 is completed. I think the
> current model of a weak reference and the inability to unload if
> needed is a concern. I don't think it's because of uninversion.
Ahh OK. We know about that one (for LUCENE-831): the app should have
control over the caching policy.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Future projects
Posted by Jason Rutherglen <ja...@gmail.com>.
> meaning in Bobo you'd like to manage your own memory resident
field caches, and merge them whenever IW has merged a segment?
Seems like you don't need genealogy for that.
Agreed, there is no need for full genealogy.
> CSF isn't really designed yet. How come it can't be used with
Bobo's field caches?
I guess CSF should be able to support it, makes sense. As long
as the container is flexible with the encoding (I need to look
into this more on the Bobo side).
> Lucene's internal field cache usage is now entirely at the
segment level (ie, Lucene core should never request full field
cache array at the MultiSegmentReader level). I think Bobo must
have to do the same, if it handles near realtime updates, to get
adequate performance.
Bobo needs to migrate to this model, I don't think we've done
that yet.
> EG how come Bobo made its own field cache impl? Just because
uninversion is too slow?
It could be integrated once LUCENE-831 is completed. I think the
current model of a weak reference and the inability to unload if
needed is a concern. I don't think it's because of uninversion.
On Fri, Apr 3, 2009 at 3:35 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> On Thu, Apr 2, 2009 at 5:56 PM, Jason Rutherglen
> <ja...@gmail.com> wrote:
> >> I think I need to understand better why delete by Query isn't
> > viable in your situation...
> >
> > The delete by query is a separate problem which I haven't fully
> > explored yet.
>
> Oh, I had thought we were tugging on this thread in order to explore
> delete-by-docID in the writer. OK.
>
> > Tracking the segment genealogy is really an
> > interim step for merging field caches before column stride
> > fields gets implemented.
>
> I see -- meaning in Bobo you'd like to manage your own memory resident
> field caches, and merge them whenever IW has merged a segment? Seems
> like you don't need genealogy for that.
>
> > Actually CSF cannot be used with Bobo's
> > field caches anyways which means we'd need a way to find out
> > about the segment parents.
>
> CSF isn't really designed yet. How come it can't be used with Bobo's
> field caches? We can try to accommodate Bobo's field cache needs when
> designing CSF.
>
> >> Does it operate at the segment level? Seems like that'd give
> > you good enough realtime performance (though merging in RAM will
> > definitely be faster).
> >
> > We need to see how Bobo integrates with LUCENE-1483.
>
> Lucene's internal field cache usage is now entirely at the segment
> level (ie, Lucene core should never request full field cache array at
> the MultiSegmentReader level). I think Bobo must have to do the same,
> if it handles near realtime updates, to get adequate performance.
>
> Though... since we have LUCENE-831 (rework API Lucene exposes for
> accessing arrays-of-atomic-types-per-segment) and LUCENE-1231 (CSF = a
> more efficient impl (than uninversion) of the API we expose in
> LUCENE-831) on deck, we should try to understand Bobo's needs.
>
> EG how come Bobo made its own field cache impl? Just because
> uninversion is too slow?
>
> > It seems like we've been talking about CSF for 2 years and there
> > isn't a patch for it? If I had more time I'd take a look. What
> > is the status of it?
>
> I think Michael is looking into it? I'd really like to get it into
> 2.9. We should do it in conjunction with 831 since they are so tied.
>
> > I'll write a patch that implements a callback for the segment
> > merging such that the user can decide what information they want
> > to record about the merged SRs (I'm pretty sure there isn't a
> > way to do this with MergePolicy?)
>
> Actually I think you can do this w/ a simple MergeScheduler wrapper or
> by subclassing CMS. I'll put a comment on the issue.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: Future projects
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Thu, Apr 2, 2009 at 5:56 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
>> I think I need to understand better why delete by Query isn't
> viable in your situation...
>
> The delete by query is a separate problem which I haven't fully
> explored yet.
Oh, I had thought we were tugging on this thread in order to explore
delete-by-docID in the writer. OK.
> Tracking the segment genealogy is really an
> interim step for merging field caches before column stride
> fields gets implemented.
I see -- meaning in Bobo you'd like to manage your own memory resident
field caches, and merge them whenever IW has merged a segment? Seems
like you don't need genealogy for that.
> Actually CSF cannot be used with Bobo's
> field caches anyways which means we'd need a way to find out
> about the segment parents.
CSF isn't really designed yet. How come it can't be used with Bobo's
field caches? We can try to accommodate Bobo's field cache needs when
designing CSF.
>> Does it operate at the segment level? Seems like that'd give
> you good enough realtime performance (though merging in RAM will
> definitely be faster).
>
> We need to see how Bobo integrates with LUCENE-1483.
Lucene's internal field cache usage is now entirely at the segment
level (ie, Lucene core should never request full field cache array at
the MultiSegmentReader level). I think Bobo must have to do the same,
if it handles near realtime updates, to get adequate performance.
Though... since we have LUCENE-831 (rework API Lucene exposes for
accessing arrays-of-atomic-types-per-segment) and LUCENE-1231 (CSF = a
more efficient impl (than uninversion) of the API we expose in
LUCENE-831) on deck, we should try to understand Bobo's needs.
EG how come Bobo made its own field cache impl? Just because
uninversion is too slow?
> It seems like we've been talking about CSF for 2 years and there
> isn't a patch for it? If I had more time I'd take a look. What
> is the status of it?
I think Michael is looking into it? I'd really like to get it into
2.9. We should do it in conjunction with 831 since they are so tied.
> I'll write a patch that implements a callback for the segment
> merging such that the user can decide what information they want
> to record about the merged SRs (I'm pretty sure there isn't a
> way to do this with MergePolicy?)
Actually I think you can do this w/ a simple MergeScheduler wrapper or
by subclassing CMS. I'll put a comment on the issue.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Future projects
Posted by Jason Rutherglen <ja...@gmail.com>.
> I think I need to understand better why delete by Query isn't
viable in your situation...
The delete by query is a separate problem which I haven't fully
explored yet. Tracking the segment genealogy is really an
interim step for merging field caches before column stride
fields gets implemented. Actually CSF cannot be used with Bobo's
field caches anyways which means we'd need a way to find out
about the segment parents.
> Does it operate at the segment level? Seems like that'd give
you good enough realtime performance (though merging in RAM will
definitely be faster).
We need to see how Bobo integrates with LUCENE-1483.
It seems like we've been talking about CSF for 2 years and there
isn't a patch for it? If I had more time I'd take a look. What
is the status of it?
I'll write a patch that implements a callback for the segment
merging such that the user can decide what information they want
to record about the merged SRs (I'm pretty sure there isn't a
way to do this with MergePolicy?)
On Thu, Apr 2, 2009 at 2:41 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> On Thu, Apr 2, 2009 at 4:43 PM, Jason Rutherglen
> <ja...@gmail.com> wrote:
> >> What does Bobo use the cached bitsets for?
> >
> > Bobo is a faceting engine that uses custom field caches and sometimes
> cached
> > bitsets rather than relying exclusively on bitsets to calculate facets.
> It
> > is useful where many facets (50+) need to be calculated and bitset
> caching,
> > loading and intersection would be too costly. Instead it iterates over
> in
> > memory custom field caches while hit collecting. Because we're also
> doing
> > realtime search, making the loading more efficient via the in memory
> field
> > cache merging is interesting.
>
> OK.
>
> Does it operate at the segment level? Seems like that'd give you good
> enough realtime performance (though merging in RAM will definitely be
> faster).
>
> > True, we do the in memory merging with deleted docs, norms would be good
> as
> > well.
>
> Yes, and eventually column stride fields.
>
> > As a first step how should we expose the segments a segment has
> > originated from?
>
> I'm not sure; it's quite messy. Each segment must track what other
> segment it got merged to, and must hold a copy of its deletes as of
> the time it was merged. And each segment must know what other
> segments it got merged with.
>
> Is this really a serious problem in your realtime search? Eg, from
> John's numbers in using payloads to read in the docID -> UID mapping,
> it seems like you could make a Query that when given a reader would go
> and do the "Approach 2" to perform the deletes (if indeed you are
> needing to delete thousands of docs with each update). What sort of
> docs/sec rates are you needing to handle?
>
> > I would like to get this implemented for 2.9 as a building
> > block that perhaps we can write other things on.
>
> I think that's optimistic. It's still at the
> hairy-can't-see-a-clean-way-to-do-it phase still. Plus I'd like to
> understand that all other options have been exhausted first.
>
> Especially once we have column stride fields and they are merged in
> RAM, you'll be handed a reader pre-warmed and you can then jump
> through those arrays to find docs to delete.
>
> > Column stride fields still
> > requires some encoding and merging field caches in ram would presumably
> be
> > faster?
>
> Yes, potentially much faster. There's no sense in writing through to
> disk until commit is called.
>
> >> Ie we only have to renumber from gen X to X+1, then from X+1 to X+2
> (where
> >> each "generation" is a renumbering event).
> >
> > Couldn't each SegmentReader keep a docmap and the names of the segments
> it
> > originated from. However the name is not enough of a unique key as
> there's
> > the deleted docs that change? It seems like we need a unique id for each
> > segment reader, where the id is assigned to cloned readers (which
> normally
> > have the same segment name as the original SR). The ID could be a stamp
> > (perhaps only given to readonlyreaders?). That way the
> > SegmentReader.getMergedFrom method does not need to return the actual
> > readers, but a docmap and the parent readers IDs? It would be assumed
> the
> > user would be holding the readers somewhere? Perhaps all this can be
> > achieved with a callback in IW, and all this logic could be kept somewhat
> > internal to Lucene?
>
> The docMap is a costly way to store it, since it consumes 32 bits per
> doc (vs storing a copy of the deleted docs).
>
> But, then docMap gives you random-access on the map.
>
> What if prior to merging, or committing merged deletes, there were a
> callback to force the app to materialize any privately buffered
> deletes? And then the app is not allowed to use those readers for
> further deletes? Still kinda messy.
>
> I think I need to understand better why delete by Query isn't viable
> in your situation...
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: Future projects
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Thu, Apr 2, 2009 at 4:43 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
>> What does Bobo use the cached bitsets for?
>
> Bobo is a faceting engine that uses custom field caches and sometimes cached
> bitsets rather than relying exclusively on bitsets to calculate facets. It
> is useful where many facets (50+) need to be calculated and bitset caching,
> loading and intersection would be too costly. Instead it iterates over in
> memory custom field caches while hit collecting. Because we're also doing
> realtime search, making the loading more efficient via the in memory field
> cache merging is interesting.
OK.
Does it operate at the segment level? Seems like that'd give you good
enough realtime performance (though merging in RAM will definitely be
faster).
> True, we do the in memory merging with deleted docs, norms would be good as
> well.
Yes, and eventually column stride fields.
> As a first step how should we expose the segments a segment has
> originated from?
I'm not sure; it's quite messy. Each segment must track what other
segment it got merged to, and must hold a copy of its deletes as of
the time it was merged. And each segment must know what other
segments it got merged with.
Is this really a serious problem in your realtime search? Eg, from
John's numbers in using payloads to read in the docID -> UID mapping,
it seems like you could make a Query that when given a reader would go
and do the "Approach 2" to perform the deletes (if indeed you are
needing to delete thousands of docs with each update). What sort of
docs/sec rates are you needing to handle?
> I would like to get this implemented for 2.9 as a building
> block that perhaps we can write other things on.
I think that's optimistic. It's still at the
hairy-can't-see-a-clean-way-to-do-it phase still. Plus I'd like to
understand that all other options have been exhausted first.
Especially once we have column stride fields and they are merged in
RAM, you'll be handed a reader pre-warmed and you can then jump
through those arrays to find docs to delete.
> Column stride fields still
> requires some encoding and merging field caches in ram would presumably be
> faster?
Yes, potentially much faster. There's no sense in writing through to
disk until commit is called.
>> Ie we only have to renumber from gen X to X+1, then from X+1 to X+2 (where
>> each "generation" is a renumbering event).
>
> Couldn't each SegmentReader keep a docmap and the names of the segments it
> originated from. However the name is not enough of a unique key as there's
> the deleted docs that change? It seems like we need a unique id for each
> segment reader, where the id is assigned to cloned readers (which normally
> have the same segment name as the original SR). The ID could be a stamp
> (perhaps only given to readonlyreaders?). That way the
> SegmentReader.getMergedFrom method does not need to return the actual
> readers, but a docmap and the parent readers IDs? It would be assumed the
> user would be holding the readers somewhere? Perhaps all this can be
> achieved with a callback in IW, and all this logic could be kept somewhat
> internal to Lucene?
The docMap is a costly way to store it, since it consumes 32 bits per
doc (vs storing a copy of the deleted docs).
But, then docMap gives you random-access on the map.
What if prior to merging, or committing merged deletes, there were a
callback to force the app to materialize any privately buffered
deletes? And then the app is not allowed to use those readers for
further deletes? Still kinda messy.
I think I need to understand better why delete by Query isn't viable
in your situation...
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Future projects
Posted by Jason Rutherglen <ja...@gmail.com>.
> What does Bobo use the cached bitsets for?
Bobo is a faceting engine that uses custom field caches and sometimes cached
bitsets rather than relying exclusively on bitsets to calculate facets. It
is useful where many facets (50+) need to be calculated and bitset caching,
loading and intersection would be too costly. Instead it iterates over in
memory custom field caches while hit collecting. Because we're also doing
realtime search, making the loading more efficient via the in memory field
cache merging is interesting.
True, we do the in memory merging with deleted docs, norms would be good as
well. As a first step how should we expose the segments a segment has
originated from? I would like to get this implemented for 2.9 as a building
block that perhaps we can write other things on. Column stride fields still
requires some encoding and merging field caches in ram would presumably be
faster?
> Ie we only have to renumber from gen X to X+1, then from X+1 to X+2 (where
each "generation" is a renumbering event).
Couldn't each SegmentReader keep a docmap and the names of the segments it
originated from. However the name is not enough of a unique key as there's
the deleted docs that change? It seems like we need a unique id for each
segment reader, where the id is assigned to cloned readers (which normally
have the same segment name as the original SR). The ID could be a stamp
(perhaps only given to readonlyreaders?). That way the
SegmentReader.getMergedFrom method does not need to return the actual
readers, but a docmap and the parent readers IDs? It would be assumed the
user would be holding the readers somewhere? Perhaps all this can be
achieved with a callback in IW, and all this logic could be kept somewhat
internal to Lucene?
On Thu, Apr 2, 2009 at 12:59 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> On Thu, Apr 2, 2009 at 2:07 PM, Jason Rutherglen
> <ja...@gmail.com> wrote:
> > I'm interested in merging cached bitsets and field caches. While this
> may
> > be something related to LUCENE-831, in Bobo there are custom field caches
> > which we want to merge in RAM (rather than reload from the reader using
> > termenum + termdocs). This could somehow lead to delete by doc id.
>
> What does Bobo use the cached bitsets for?
>
> Merging FieldCache in RAM is also interesting for near-realtime
> search, once we have column stride fields. Ie, they should behave
> like deleted docs: there's no reason to go through disk when merging
> them -- just carry them straight to the merged reader. Only on commit
> do they need to go to disk. Hmm in fact we could do this today, too,
> eg with norms as a future optimization if needed. And that
> optimization applies to flushing as well (ie, when flushing a new
> segment, since we know we will open a reader, we could NOT flush the
> norms, and instead put them into the reader, and only on eventual
> commit, flush to disk).
>
> > Tracking the genealogy of segments is something we can provide as a
> callback
> > from IndexWriter? Or could we add a method to IndexCommit or
> SegmentReader
> > that returns the segments it originated from?
>
> Well.... the problem with my idea (callback from IW when docs shift)
> is internally IW always uses the latest reader to get any new docIDs.
>
> Ie we only have to renumber from gen X to X+1, then from X+1 to X+2
> (where each "generation" is a renumbering event).
>
> But if you have a reader, perhaps oldish by now, we'd need to give you
> a way to map across N generations of docID shifts (which'd require the
> genealogy tracking).
>
> Alas I think it will quickly get hairy.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: Future projects
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Thu, Apr 2, 2009 at 2:07 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
> I'm interested in merging cached bitsets and field caches. While this may
> be something related to LUCENE-831, in Bobo there are custom field caches
> which we want to merge in RAM (rather than reload from the reader using
> termenum + termdocs). This could somehow lead to delete by doc id.
What does Bobo use the cached bitsets for?
Merging FieldCache in RAM is also interesting for near-realtime
search, once we have column stride fields. Ie, they should behave
like deleted docs: there's no reason to go through disk when merging
them -- just carry them straight to the merged reader. Only on commit
do they need to go to disk. Hmm in fact we could do this today, too,
eg with norms as a future optimization if needed. And that
optimization applies to flushing as well (ie, when flushing a new
segment, since we know we will open a reader, we could NOT flush the
norms, and instead put them into the reader, and only on eventual
commit, flush to disk).
> Tracking the genealogy of segments is something we can provide as a callback
> from IndexWriter? Or could we add a method to IndexCommit or SegmentReader
> that returns the segments it originated from?
Well.... the problem with my idea (callback from IW when docs shift)
is internally IW always uses the latest reader to get any new docIDs.
Ie we only have to renumber from gen X to X+1, then from X+1 to X+2
(where each "generation" is a renumbering event).
But if you have a reader, perhaps oldish by now, we'd need to give you
a way to map across N generations of docID shifts (which'd require the
genealogy tracking).
Alas I think it will quickly get hairy.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Future projects
Posted by Jason Rutherglen <ja...@gmail.com>.
I'm interested in merging cached bitsets and field caches. While this may
be something related to LUCENE-831, in Bobo there are custom field caches
which we want to merge in RAM (rather than reload from the reader using
termenum + termdocs). This could somehow lead to delete by doc id.
Tracking the genealogy of segments is something we can provide as a callback
from IndexWriter? Or could we add a method to IndexCommit or SegmentReader
that returns the segments it originated from?
On Thu, Apr 2, 2009 at 1:40 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> On Wed, Apr 1, 2009 at 7:05 PM, Jason Rutherglen
> <ja...@gmail.com> wrote:
> > Now that LUCENE-1516 is close to being committed perhaps we can
> > figure out the priority of other issues:
> >
> > 1. Searchable IndexWriter RAM buffer
>
> I think first priority is to get a good assessment of the performance
> of the current implementation (from LUCENE-1516).
>
> My initial tests are very promising: with a writer updating (replacing
> random docs) at 50 docs/second on a full (3.2 M) Wikipedia index, I
> was able to get reopen the reader once per second and do a large (>
> 500K results) search that sorts by date. The reopen time was
> typically ~40 msec, and search time typically ~35 msec (though there
> were random spikes up to ~340 msec). Though, these results were on an
> SSD (Intel X25M 160 GB).
>
> We need more datapoints of the current approach, but this looks likely
> to be good enough for starters. And since we can get it into 2.9,
> hopefully it'll get some early usage and people will report back to
> help us assess whether further performance improvements are necessary.
>
> If they do turn out to be necessary, I think before your step 1, we
> should write small segments into a RAMDirectory instead of the "real"
> directory. That's simpler than truly searching IndexWriter's
> in-memory postings data.
>
> > 2. Finish up benchmarking and perhaps implement passing
> > filters to the SegmentReader level
>
> What is "passing filters to the SegmentReader level"? EG as of
> LUCENE-1483, we now ask a Filter for it's DocIdSet once per
> SegmentReader.
>
> > 3. Deleting by doc id using IndexWriter
>
> We need a clean approach for the "docIDs suddenly shift when merge is
> committed" problem for this...
>
> Thinking more on this... I think one possible solution may be to
> somehow expose IndexWriter's internal docID remapping code.
> IndexWriter does delete by docID internally, and whenever a merge is
> committed we stop-the-world (sync on IW) and go remap those docIDs.
> If we somehow allowed user to register a callback that we could call
> when this remapping occurs, then user's code could carry the docIDs
> without them becoming stale. Or maybe we could make a class
> "PendingDocIDs", which you'd ask the reader to give you, that holds
> docIDs and remaps them after each merge. The problem is, IW
> internally always logically switches to the current reader for any
> further docID deletion, but the user's code may continue to use an old
> reader. So simply exposing this remapping won't fix it... we'd need
> to somehow track the genealogy (quite a bit more complex).
>
> > With 1) I'm interested in how we will lock a section of the
> > bytes for use by a given reader? We would not actually lock
> > them, but we need to set aside the bytes such that for example
> > if the postings grows, TermDocs iteration does not progress to
> > beyond it's limits. Are there any modifications that are needed
> > of the RAM buffer format? How would the term table be stored? We
> > would not be using the current hash method?
>
> I think the realtime reader'd just store the maxDocID it's allowed to
> search, and we would likely keep using the RAM format now used.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: Future projects
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Wed, Apr 1, 2009 at 7:05 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
> Now that LUCENE-1516 is close to being committed perhaps we can
> figure out the priority of other issues:
>
> 1. Searchable IndexWriter RAM buffer
I think first priority is to get a good assessment of the performance
of the current implementation (from LUCENE-1516).
My initial tests are very promising: with a writer updating (replacing
random docs) at 50 docs/second on a full (3.2 M) Wikipedia index, I
was able to get reopen the reader once per second and do a large (>
500K results) search that sorts by date. The reopen time was
typically ~40 msec, and search time typically ~35 msec (though there
were random spikes up to ~340 msec). Though, these results were on an
SSD (Intel X25M 160 GB).
We need more datapoints of the current approach, but this looks likely
to be good enough for starters. And since we can get it into 2.9,
hopefully it'll get some early usage and people will report back to
help us assess whether further performance improvements are necessary.
If they do turn out to be necessary, I think before your step 1, we
should write small segments into a RAMDirectory instead of the "real"
directory. That's simpler than truly searching IndexWriter's
in-memory postings data.
> 2. Finish up benchmarking and perhaps implement passing
> filters to the SegmentReader level
What is "passing filters to the SegmentReader level"? EG as of
LUCENE-1483, we now ask a Filter for it's DocIdSet once per
SegmentReader.
> 3. Deleting by doc id using IndexWriter
We need a clean approach for the "docIDs suddenly shift when merge is
committed" problem for this...
Thinking more on this... I think one possible solution may be to
somehow expose IndexWriter's internal docID remapping code.
IndexWriter does delete by docID internally, and whenever a merge is
committed we stop-the-world (sync on IW) and go remap those docIDs.
If we somehow allowed user to register a callback that we could call
when this remapping occurs, then user's code could carry the docIDs
without them becoming stale. Or maybe we could make a class
"PendingDocIDs", which you'd ask the reader to give you, that holds
docIDs and remaps them after each merge. The problem is, IW
internally always logically switches to the current reader for any
further docID deletion, but the user's code may continue to use an old
reader. So simply exposing this remapping won't fix it... we'd need
to somehow track the genealogy (quite a bit more complex).
> With 1) I'm interested in how we will lock a section of the
> bytes for use by a given reader? We would not actually lock
> them, but we need to set aside the bytes such that for example
> if the postings grows, TermDocs iteration does not progress to
> beyond it's limits. Are there any modifications that are needed
> of the RAM buffer format? How would the term table be stored? We
> would not be using the current hash method?
I think the realtime reader'd just store the maxDocID it's allowed to
search, and we would likely keep using the RAM format now used.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Future projects
Posted by Michael McCandless <lu...@mikemccandless.com>.
I'm not sure how big a win this'd be, since the OS will cache those in
RAM and the CPU cost there (to pull from OS's cache and reprocess) is
maybe not high.
Optimizing search is interesting, because it's the wicked slow queries
that you need to make faster even when it's at the expense of wicked
fast queries. If you make a wicked fast query 3X slower (eg 1 ms -> 3
ms), it's almost harmless in nearly all apps.
So this makes things like PFOR (and LUCENE-1458, to enable pluggable
codecs for postings) important since it addresses the very large
queries. In fact for very large postings we should do PFOR minus the
exceptions, ie, do a simple Nbit encode, even if it wastes some bits.
Mike
On Thu, Apr 2, 2009 at 1:52 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
> 4) An additional possibly contrib module is caching the results of
> TermQueries. In looking at the TermQuery code would we need to cache the
> entire docs and freqs as arrays which would be a memory hog?
>
> On Wed, Apr 1, 2009 at 4:05 PM, Jason Rutherglen
> <ja...@gmail.com> wrote:
>>
>> Now that LUCENE-1516 is close to being committed perhaps we can
>> figure out the priority of other issues:
>>
>> 1. Searchable IndexWriter RAM buffer
>>
>> 2. Finish up benchmarking and perhaps implement passing
>> filters to the SegmentReader level
>>
>> 3. Deleting by doc id using IndexWriter
>>
>> With 1) I'm interested in how we will lock a section of the
>> bytes for use by a given reader? We would not actually lock
>> them, but we need to set aside the bytes such that for example
>> if the postings grows, TermDocs iteration does not progress to
>> beyond it's limits. Are there any modifications that are needed
>> of the RAM buffer format? How would the term table be stored? We
>> would not be using the current hash method?
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Future projects
Posted by Jason Rutherglen <ja...@gmail.com>.
4) An additional possibly contrib module is caching the results of
TermQueries. In looking at the TermQuery code would we need to cache the
entire docs and freqs as arrays which would be a memory hog?
On Wed, Apr 1, 2009 at 4:05 PM, Jason Rutherglen <jason.rutherglen@gmail.com
> wrote:
> Now that LUCENE-1516 is close to being committed perhaps we can
> figure out the priority of other issues:
>
> 1. Searchable IndexWriter RAM buffer
>
> 2. Finish up benchmarking and perhaps implement passing
> filters to the SegmentReader level
>
> 3. Deleting by doc id using IndexWriter
>
> With 1) I'm interested in how we will lock a section of the
> bytes for use by a given reader? We would not actually lock
> them, but we need to set aside the bytes such that for example
> if the postings grows, TermDocs iteration does not progress to
> beyond it's limits. Are there any modifications that are needed
> of the RAM buffer format? How would the term table be stored? We
> would not be using the current hash method?
>