You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael Busch (JIRA)" <ji...@apache.org> on 2010/03/15 06:41:27 UTC
[jira] Created: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Per thread DocumentsWriters that write their own private segments
-----------------------------------------------------------------
Key: LUCENE-2324
URL: https://issues.apache.org/jira/browse/LUCENE-2324
Project: Lucene - Java
Issue Type: Improvement
Components: Index
Reporter: Michael Busch
Assignee: Michael Busch
Priority: Minor
Fix For: 3.1
See LUCENE-2293 for motivation and more details.
I'm copying here Mike's summary he posted on 2293:
Change the approach for how we buffer in RAM to a more isolated
approach, whereby IW has N fully independent RAM segments
in-process and when a doc needs to be indexed it's added to one of
them. Each segment would also write its own doc stores and
"normal" segment merging (not the inefficient merge we now do on
flush) would merge them. This should be a good simplification in
the chain (eg maybe we can remove the *PerThread classes). The
segments can flush independently, letting us make much better
concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850754#action_12850754 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
{quote}I don't think we should delete in FG - I suspect this'll
give net/net worse performance, due to loss of locality. It also
means you must always keep readers available, which is an
unnecessary cost for non-NRT apps. {quote}
Right, I agree it's best to buffer the deleted terms/queries...
The comments below touch on why this came up as a thought.
{quote}deletesInRAM are those deletes done during the current
segment. deletesFlushed absorbs deletesInRAM on successful
segment flush. We have to segregate the two for proper recovery
if we fail to flush the RAM buffer, eg say you hit a disk full
while flushing a new segment, and then you close your IW
successfully. We have to make sure in that case that
deletesInRAM are discarded. {quote}
Ok, this helps...
The design issue I'm running into is the buffered deletes "num
limit" which seems to be the highest doc id of DW at the time
the delete docs method is called? Then when apply deletes is
called, deletes are made only up to "num limit"? This is to
insure that documents (with for example a del term) that are
added after the delete docs call, are not deleted when apply
deletes is called, unless another call to del docs is made again
(at which point the "num limit" is set to the current DW max doc
id).
If the above is true, in order to not remap deletes on each
merge, would we need to maintain this "num limit" variable per
DW? I don't think the global remap is useful with per thread DWs
because a DW could fail, and we wouldn't know the order in
segment infos it could/would've been placed? Whereas with a
single DW, we know the new segment will be placed last in
segment infos.
We could maintain a global deleted terms/queries queue, but then
we'd also need to maintain a term -> "num limit" map per DW? It
seems a bit redundant, but maybe it's ok?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858591#action_12858591 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
{quote}
But... could we allow an add/updateDocument call to express this
affinity, explicitly? If you index homogenous docs you wouldn't use
it, but, if you index drastically different docs that fall into clear
"categories", expressing the affinity can get you a good gain in
indexing throughput.
This may be the best solution, since then one could pass the affinity
even through a thread pool, and then we would fallback to thread
binding if the document class wasn't declared?
{quote}
I would like this if we then also added an API that can be used to specify the
per-DWPT RAM size. E.g. if someone has such an app where different threads
index docs of different sizes, then the DW that indexes big docs can be given
more memory?
What I'm mainly trying to avoid is synchronization points between the
different DWPTs. For example, currently the same ByteBlockAllocator is shared
between the different threads, so all its methods need to be synchronized.
{quote}
The other DWs would keep indexing That's the beauty of this
approach... a flush of one DW doesn't stop all other DWs from
indexing, unliked today.
And you want to serialize the flushing right? Ie, only one DW flushes
at a time (the others keep indexing).
Hmm I suppose flushing more than one should be allowed (OS/IO have
alot of concurrency, esp since IO goes into write cache)... perhaps
that's the best way to balance index vs flush time? EG we pick one to
flush @ 90%, if we cross 95% we pick another to flush, another at
100%, etc.
{quote}
Oh I don't want to disallow flushing in parallel! I think it makes perfect
sense to allow more than one DW to flush at the same time. If each DWPT has a
private max buffer size, then it can decide on its own when it's time to
flush.
{quote}
Hmm I suppose flushing more than one should be allowed (OS/IO have
alot of concurrency, esp since IO goes into write cache)... perhaps
that's the best way to balance index vs flush time? EG we pick one to
flush @ 90%, if we cross 95% we pick another to flush, another at
100%, etc.
{quote}
If we allow flushing in parallel and also allow specifying the max RAM per
DWPT, then there doesn't even have to be any cross-thread RAM tracking? Each
DWPT could just flush when its own buffer is full?
So let's summarize:
# Expose a ThreadBinder API for controlling number of DWPT instances and
thread affinity of DWPTs explicitly. (We can later decide if we want to also
support such an affinity after a segment was flushed, as Tim is asking for.
But that should IMO not be part of this patch.)
# Also expose an API for specifying the RAM buffer size per DWPT.
# Allow flushing in parallel (multiple DWPTs can flush at the same time). A
DWPT flushes when its buffer is full, independent of what the other DWPTs are
doing.
# The default implementation of the ThreadBinder API assigns threads to DWPT
randomly and gives each DWPT 1/n-th of the overall memory.
How does this sound?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850792#action_12850792 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
{quote}
However, in the apply deletes
method how would we know which doc to stop deleting at? How
would the seq id map to a DW's doc id?
{quote}
We could have a global deletes-map that stores seqID -> DeleteAction. DeleteAction either contains a Term or a Query, and in addition an int "flushCount" (I'll explain in a bit what flushCount is used for.)
Each DocumentsWriterPerThread would have a growing array that contains each seqID that "affected" that DWPT, i.e. the seqIDs of *all* deletes, plus the seqIDs of the adds/updates performed by that particular DWPT. One bit of a seqID in that array can indicate if it's a delete or add/update.
When it's time to flush we sort the array by increasing seqID and then loop a single time through it to find the seqIDs of all DeleteActions. During the loop we count the number of adds/updates to determine the number of docs the DeleteActions affect. After applying the deletes the DWPT makes a synchronized call to the global deletes-map and increments the flushCount int for each applied DeleteAction. If flushCount==numThreadStates (== number of DWPT instances) the corresponding DeleteAction entry can be removed, because it was applied to all DWPT.
I think this should work? Or is there a simpler solution?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859966#action_12859966 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
{quote}We just have to explain this in the javadocs. It's a change that we should also mention in the backwards-compatibility section.{quote}
We're punting eh?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850780#action_12850780 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
{quote}I mentioned the idea of having a sequence ID, that gets
incremented on add, delete, update. What if we had even with
separate DWs a global sequence ID? The sequence ID would tell
you unambiguously which action happened when. The
add/update/delete methods could return the sequenceID that was
assigned to that particular action.{quote}
I think adding the global sequence id makes sense and would be
simple to add (eg, AtomicLong). However, in the apply deletes
method how would we know which doc to stop deleting at? How
would the seq id map to a DW's doc id?
If we have independent buffered deletes per DW-thread, then how
will we keep track of the memory usage? eg, we'll be flushing
DWs independently, but will not want to double count the same
term/query's size for memory tracking? I'm not sure how we'd do
that without having a global deletes manager that individual DWs
interact with (maybe have a ref count per term/query?). The deletes
manager would have the size/ram usage method, not individual
DWs. The DWs would need to keep a hash map of term ->
BufferedDeletes.Num.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850211#action_12850211 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
Mike, lets do option 1. I think the process of making the doc id absolute is simply adding up the previous segments num docs to be the base?
Option 2 would use reader cloning?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Busch updated LUCENE-2324:
----------------------------------
Attachment: lucene-2324-no-pooling.patch
All tests pass but I have to review if with the changes the memory consumption calculation still works correctly. Not sure if the junits test that?
Also haven't done any performance testing yet.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324-no-pooling.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846110#action_12846110 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
NormsWriterPerField has a growing norm byte array, we'd need a way to read/write lock it...
I think we have concurrency issues in the TermsHash table? Maybe it'd need to be rewritten to use ConcurrentHashMap?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859737#action_12859737 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
{quote}
I think we simply keep track of maxBufferedDocs and
maxBufferedDeleteTerms globally. We can't deprecate because
that'd be a pain for users.
{quote}
The problem with maxBufferedDocs is that you can only enforce it by
"stopping the world", i.e. preventing other DWPTs from adding docs while
one DWPT is flushing. Unless we also introduce the 90%-95% etc flush
triggers for maxBufferedDocs.
The other option would be to have maxBufferedDocs per DWPT. Then
the theoretical total limit of docs in RAM would be
maxNumberThreadStates * maxBufferedDocs.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850056#action_12850056 ]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
Good question...
When we buffer delete Term/Query we record the current docID as of when that delete had arrived (so that interleaved delete/adds are resolved properly). The docID we record is "absolute" (ie, adds in the base flushedDocCount), so that we can decouple when deletes are materialized (moved into the deletedDocs BitVectors) from when new segments are flushed.
I think we have a couple options.
Option 1 is to use a relative (within the current segment) docID when the deleted Term/Query/docID is first buffered, but then make it absolute only when the segment is finally flushed.
Option 2 is to use a relative docID, but do away with the decoupling, ie force deletions to always flush at the same time the segment is flushed.
I think I like option 1 the best -- I suspect the decoupling gains us performance as it allows us to batch up more deletions (doing deletions in batch gets better locality, and also means opening/closing readers left often, in the non-pooling case).
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845400#action_12845400 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
{quote}
Sounds great - let's test it in practice.
{quote}
I have to admit that I need to catch up a bit on the flex branch. I was wondering if it makes sense to make these kinds of experiments (pooling vs. non-pooling) with the flex code? Is it as fast as trunk already, or are there related nocommits left that affect indexing performance? I would think not much of the flex changes should affect the in-memory indexing performance (in TermsHash*).
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850214#action_12850214 ]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
bq. I think the process of making the doc id absolute is simply adding up the previous segments num docs to be the base?
Right.
bq. Option 2 would use reader cloning?
I don't think so -- I think it'd have to pull a SegmentReader for every segment every time we flush a new segment, to resolve the deletions. In the non-pooled case that'd be a newly opened SegmentReader for every segment in the index every time a new segment is flushed.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859942#action_12859942 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
bq. I think the consensus is to have a DWPT maxBufferedDocs flush trigger, not a global one.
Ok, so we'll need to implement dynamic resizing of the maxBufferedDocs per DWPT in the case where the maxNumberThreadStates changes?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859375#action_12859375 ]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
{quote}
I think the reason why we have two different APIs in mind (you: sourceID, I:
expert thread binder API) is that we're having different goals with
them?
{quote}
Yes, I think you're right!
{quote}
You want to make the out-of-the-box indexing performance as good as
possible, and users should have to set a minimum amount of
easy-to-understand parameters (such as buffer size in MB). I think
that's the right thing to do of course. (though that doesn't prevent
us from adding an expert API in addition, as we always have)
{quote}
Right -- simple things should be simple and complex things should be
possible.
{quote}
I'm thinking a lot about real-time indexing and the searchable RAM buffer
these days, so the thread-binder API could help you to have more control over
where your docs will actually end up and which reader will see them. But I
think too that this API would be very "expert" and not many people would use
it.
{quote}
In fact now I want both :)
Ie, make it possible (optional) to declare the sourceID, and IW
optimizes based on this hint.
But, also, letting advanced apps directly control individual DWPTs.
(I think a new experimental Indexer interface can work well here...).
{quote}
bq. We can do this as a separate issue... it's fairly orthogonal.
Yeah I was just thinking the same - I agree.
{quote}
OK I'll open this...
{quote}
bq. Is this sync really so bad? First, we should move all allocators/pools to per-DWPT, so they don't need to be sync'd.
OK cool that we agree on that. I was worried you wanted to have global pools
too, if it's only the single long it's not very complicated, I agree.
{quote}
Yeah let's not do global pools (anymore!)...
bq. Sorry if I'm being annoying :)
No, you're not! You're asking good questions (as usual)!
{quote}
My goal is to have a default indexing chain that isn't slower than the one we
have today, but searchable and that very fast. That's not trivial, but I think
we can do it!
{quote}
This is an awesome goal, and I agree very reachable. Though the
deletes/sequence ID/merging/NRT interaction is going to be fun...
{quote}
I'll implement the global flush trigger and make all pools DWPT-local. The
explicit thread-binder or sourceID APIs we can worry about later, as we agreed
above.
{quote}
OK thanks.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859590#action_12859590 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
{quote}
The deletes aren't entirely in the foreground, only the RAM
buffer deletes. Deletes to existing segments would use the
existing clone and delete mechanism.
{quote}
I think deletes should all be based on the new sequenceIDs. Today the buffered
deletes depend on a value that can change (numDocs changes when segments are
merged). But with sequenceIDs they're absolute: each delete operation will
have a sequenceID assigned, and the ordering of all write operations (add,
update, delete) is unambiguously defined by the sequenceIDs. Remember that
addDocument/updateDocument/deleteDocuments will all return the sequenceID.
This means that we don't have to remap deletes anymore. Also the pooled
SegmentReaders that read the flushed segments can use arrays of sequenceIDs
instead of BitSets? Of course that needs more memory, but even if you add 1M
docs without ever calling IW.commit/close you only need 8MB - I think that's
acceptable. And this size is independent on how many times you call
reopen/clone on the realtime readers, because they can all share the same
deletes array.
We can also modify IW.commit/close to return the latest sequenceID. This would
be very nice for document tracking. E.g. when you hit an aborting exception
after you flushed already some segments, then even though you must discard
everything in the DW's buffer you can still call commit/close to commit
everything that doesn't have to be discarded. IW.commit/close would in that
case return the sequenceID of the latest write operation that was successfully
committed, i.e. that would be visible to an IndexReader. Though we have to be
careful here: multiple segments can have interleaving segmentIDs, so we must
discard every segment that has one or more sequenceIDs greater than the lowest
one in the DW. So we still need the push-deletes logic, that keeps RAM deletes
separate from the flushed ones until flushing was successful.
DW/IW need to keep track of the largest sequenceID that is *safe*, i.e. that
could be committed even if DW hits an aborting exception. Some invariants:
* All deletes with sequenceID smaller or equal to safeSeqID have already been
applied to the deletes arrays of the flushed segments.
* All deletes with sequenceID greater than safeSeqID are in a deletesInRAM
buffer.
* safeSeqID is always smaller than any buffered doc or buffered delete in DW
or DWPT.
* safeSeqID is always equal to the maximum sequenceID of one, and only one,
flushed segment.
When IW.close/commit is called then safeSeqID is returned. If no aborting
exception occurred it equals the highest sequenceID ever assigned (during that
IW "session"). In any case it's always the sequenceID of the latest write
operation an IndexReader will "see" that you open after IW.close/commit
finished.
But this would be nice for apps to track which docs made it successfully into
the index. Apps can then externally keep a log to figure out what they have to
reply in case of exceptions.
Does all this make sense? :) This is very complex stuff, I wouldn't be
surprised if there's something I didn't think about.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Rutherglen updated LUCENE-2324:
-------------------------------------
Attachment: LUCENE-2324.patch
The patch is not committable.
The basics are here. We can worry about max threads later. For now there's a doc writer per thread. I still don't fully understand remap deletes, so it's commented out. There's a flushedDocCount per DW, and a global one in IW.
IWC is accessed directly where possible.
When running TestIndexWriter, the DW resumeAllThreads assertion fails.
If the total ram buffer size is too large, and the current thread's DW is the largest, it's flushed. The next test case can be in regards to the memory management.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845426#action_12845426 ]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
bq. Hmm I think we'd need a separate hash. Otherwise you have to subclass PostingList for the different cases (freq. vs. non-frequent terms) and do instanceof checks? Or with the parallel arrays idea maybe we could encode more information in the dense ID? E.g. use one bit to indicate if that term occurred more than once.
Or 2 sets of parallel arrays (one for the singletons).... or, something.
bq. So in a 32Bit JVM we would safe 4 bytes (pointer) + 8 bytes (header) - 4 bytes (ID) = 8 bytes. For fields with tons of unique terms that might be worth it?
And also the GC cost.
But it seems like specializing singleton fields will be the bigger win.
bq. I was wondering if it makes sense to make these kinds of experiments (pooling vs. non-pooling) with the flex code?
Last I tested (a while back now) indexing perf was the same -- need to
test again w/ recent changes (eg terms index is switching to packed
ints). For pooling vs not I'd just do the experiment on trunk?
And most of this change (changing how postings data is buffered in
RAM) is "above" flex I expect.
But if for some reason you need to start changing index postings
format then you should probably do that on flex.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859489#action_12859489 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
How shall we actually handle flushing by maxBufferedDocs? It'd be the easiest to just make this a per-DWPT flush trigger. Of course that'd be a change in runtime behavior, but I think that's acceptable?
And flushing by maxBufferedDeleteTerms is also tricky, because that needs to be a flush across all DWPTs? With searchable RAM buffers and deletes in the foreground this trigger should actually not be necessary anymore? We would probably apply deletes when the realtime IndexReader is (re)opened?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858819#action_12858819 ]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
I still think this "zero sync'd code" at the cost of perf loss /
exposing per-DWPT details is taking things too far. You're cutting
into the bone...
I don't think we should apps to be setting per-DWPT RAM limits, or,
even expose to apps how IW manages threads (this is an impl. detail).
I think we should keep the approach we have today -- you set the
overall RAM limit and IW internally manages flushing when that
allotted RAM is full.
{quote}
E.g. if someone has such an app where different threads
index docs of different sizes, then the DW that indexes big docs can be given
more memory?
{quote}
Hmm this isn't really fair -- the app in general can't predict how
many docs of each type will come in, how IW allocates RAM for
different kinds of docs (this is an impl detail), etc.
{quote}
What I'm mainly trying to avoid is synchronization points between the
different DWPTs. For example, currently the same ByteBlockAllocator is shared
between the different threads, so all its methods need to be synchronized.
{quote}
I understand the motivation, but...
Is this sync really so bad? First, we should move all
allocators/pools to per-DWPT, so they don't need to be sync'd.
Then, all that needs to be sync'd is the tracking of net RAM used (a
single long), and then the logic to pick the DWPT(s) to flush? So
then each DWPT would allocate its own RAM (unsync'd), track its own
RAM used (unsync'd), and update the total (in tiny sync block) after
the update (add/del) is serviced?
We're still gonna need sync'd code, anyway (global sequence ID,
grabbing a DWPT), right? We can put this "go flush a DWPT" logic in
the same block if we really have to? It feels like we're going to
great lengths (cutting into the bone) to avoid a trivial cost (the
minor complexity of managing flushing based on aggregate RAM used).
{quote}
# Expose a ThreadBinder API for controlling number of DWPT instances and
thread affinity of DWPTs explicitly. (We can later decide if we want to also
support such an affinity after a segment was flushed, as Tim is asking for.
But that should IMO not be part of this patch.)
# Also expose an API for specifying the RAM buffer size per DWPT.
{quote}
I don't think we should expose so much.
I think, instead, we should add an optional method to Document (eg
set/getSourceID or something), that'd reference which "source" this
doc comes from. The app would set it, optionally, as a "hint" to IW.
The source ID should not be publically tied to DWPT -- how IW
optimizes based on this "hint" from the app is really an impl detail.
Yes, today we'll use it for DWPT affinity; tomorrow, who knows. EG,
that source ID need not be less than the max DWPTs.
When source ID isn't provided we'd fallback to the same "best guess"
we have today (same thread = same source ID).
The javadoc would be something like "as a hint to IW, to possibly
improve its indexing performance, if you have docs from difference
sources you should set the source ID on your Document". And
how/whether IW makes use of this information is "under the hood"...
We can do this as a separate issue... it's fairly orthogonal.
bq. Allow flushing in parallel (multiple DWPTs can flush at the same time).
+1
This would be a natural way to protect against too much RAM usage
while flush(es) are happening. Start one flush going, but keep
indexing docs into the other DWTPs... if RAM usage grows too much
beyond your first trigger and before that first flush has finished,
start a 2nd DWPT flushing, etc. This is naturally self-regulating,
since the "mutator" threads are tied up doing the flushing...
{quote}
The DWPT RAM value must be updateable. E.g. when you first start indexing
only one DWPT should be created with the max RAM. Then when multiple threads
are used for adding documents another DWPT should be added and the RAM
value of the already existing one should be reduced, and possibly a flush of that
DWPT needs to be triggered.
{quote}
This isn't great... I mean it's weird that on adding say a 3rd
indexing thread I suddenly see a flush triggered even though I'm
nowehere near the RAM limit. Then, later, if I cut back to using only
2 threads, I still only ever use up to 2/3rd of my RAM buffer. IW's
API really shouldn't have such "surprising" behavior where how many /
which threads come through it so drastically affect it's flushing
behavior.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857381#action_12857381 ]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
{quote}
i would love to be able to explicitly define a "segment" affinity for documents i'm feeding
this would then allow me to say:
all docs from table a has affinity 1
all docs from table b has affinity 2
{quote}
Right, this is exactly what affinity would be good for -- so IW would
try to send "table a" docs their own DW(s) and "table b" docs to their
own DW(s), which should give faster indexing than randomly binding to
DWs.
But:
bq. this would ideally result in indexing documents from each table into a different segment (obviously, i would then need to be able to have segment merging be affinity aware so optimize/merging would only merge segments that share an affinity)
This part I was not proposing :)
The affinity would just be an optimization hint in creating the
initial flushed segments, so IW can speed up indexing.
Probably if you really want to keep the segments segregated like that,
you should in fact index to separate indices?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851017#action_12851017 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
Michael B.: What you're talking about here:
https://issues.apache.org/jira/browse/LUCENE-2324?focusedCommentI
d=12850792page=com.atlassian.jira.plugin.system.issuetabpanels%3A
comment-tabpanel#action_12850792 is a transaction log?
I'm not sure we need that level of complexity just yet? How
would we make the transaction log memory efficient? Are there
other uses you foresee? Maybe there's a simpler solution for the
BufferedDeletes.Num per DW problem that could make use of global
sequence ids? I'd prefer to continue to use the per term/query
max doc id. There aren't performance issues with concurrently
accessing and updating maps, so a global sync lock as the DW map
values are updated should be OK?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851099#action_12851099 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
{quote}You only need one additional int per doc in the DWPTs, and the global map for the delete terms.{quote}
Ok, lets give it a try, it'll be more clear with the prototype.
The clarify, the apply deletes doc id up to will be the flushed doc count saved per term/query per DW, though it won't be saved, it'll be derived from the sequence id int array where the action has been encoded into the seq id int?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850223#action_12850223 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
Currently the doc writer manages the ram buffer size, however
this needs to be implemented across doc writers for this issue
to be complete. IW addDoc returns doFlush from DW. I don't think
doFlush will be useful anymore?
A slightly different memory management needs to be designed.
Right now we allow the user to set the max ram buffer size and
when the doc writer's buffers exceed the ram limit, the buffer
is flushed and the process is complete.
With this issue, the flush logic probably needs to be bumped up
into IW, and flushing becomes a multi-docwriter ram usage
examination. For starters, if the aggregate ram usage of all doc
writers exceeds the IWC defined ram buffer size, we need to
schedule flushing the doc writer with the greatest ram usage? I
wonder if there's something I'm missing here in regards to
synchronization issues with DW?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Tim Smith (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857375#action_12857375 ]
Tim Smith commented on LUCENE-2324:
-----------------------------------
bq. But... could we allow an add/updateDocument call to express this affinity, explicitly?
i would love to be able to explicitly define a "segment" affinity for documents i'm feeding
this would then allow me to say:
all docs from table a has affinity 1
all docs from table b has affinity 2
this would ideally result in indexing documents from each table into a different segment (obviously, i would then need to be able to have segment merging be affinity aware so optimize/merging would only merge segments that share an affinity)
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849808#action_12849808 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
Actually, I just browsed the patch again, I don't think it implements private doc writers as of yet?
I think you're right, we can get this issue completed. LUCENE-2312's path looks clear at this point. Shall I take a whack at it?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324-no-pooling.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845428#action_12845428 ]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
{quote}
bq. Seems ilke it's 8 bytes
Object header is two words, so that's 16bytes for 64bit arch. (probably 12 for 64bit+CompressedOops?)
{quote}
Right, and the pointer'd also be 8 bytes (but compact int stays at 4
bytes) so net/net on 64bit JRE savings would be 16-20 bytes per term.
Another thing we could do if we cutover to parallel arrays is to
switch to packed ints. Many of these fields are horribly wasteful as
ints, eg docFreq or lastPosition.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846102#action_12846102 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
Michael,
For LUCENE-2312, I think the searching isn't going to be an
issue, I've got basic per thread doc writers working (though not
thoroughly tested). I didn't see a great need to rework all the
classes, which even if we did, I'm not sure helps with the byte
array read write issues? I'd prefer to get a proof of concept
more or less working, then refine it from there. I think there's
two main design/implementation issues before we can roll
something out:
1) A new skip list implementation that at specific intervals
writes a new skip (ie, single level). Right now in trunk we have
a multilevel skiplist that requires ahead of time the number of
docs.
2) Figure out the low -> high levels of byte/char/int array
visibility to reader threads. The main challenge here is the
fact that the DW related code that utilizes this is really hard
for me to understand enough to know what can be changed, without
the side effect being bunches of other broken stuff. If there
was a Directory like class abstraction we could simply override
and reimplement, we could do that, and maybe there is one, I'm
not sure yet.
However if reworking the PerThread classes somehow makes the tie
into the IO (eg, the byte array pooling) system abstracted and
easier, then I'm all for it.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845398#action_12845398 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
{quote}
I think we can do even better, ie, that class wastes RAM for the single posting case (intStart, byteStart, lastDocID, docFreq, lastDocCode, lastDocPosition are not needed).
EG we could have a separate class dedicated to the singleton case. When term is first encountered it's enrolled there. We'd probably need a separate hash to store these (though not necessarily?). If it's seen again it's switched to the full posting.
{quote}
Hmm I think we'd need a separate hash. Otherwise you have to subclass PostingList for the different cases (freq. vs. non-frequent terms) and do instanceof checks? Or with the parallel arrays idea maybe we could encode more information in the dense ID? E.g. use one bit to indicate if that term occurred more than once.
{quote}
I mean instead of allocating an instance per unique term, we assign an integer ID (dense, ie, 0, 1, 2...).
And then we have an array for each member now in FreqProxTermsWriter.PostingList, ie int[] docFreqs, int [] lastDocIDs, etc. Then to look up say the lastDocID for a given postingID you just get lastDocIDs[postingID]. If we're worried about oversize allocation overhead, we can make these arrays paged... but that'd slow down each access.
{quote}
Yeah I like that idea. I've done something similar for representing trees - I had a very compact Node class with no data but such a dense ID, and arrays that stored the associated data. Very easy to add another data type with no RAM overhead (you only use the amount of RAM the data needs).
Though, the price you pay is for dereferencing multiple times for each array?
And how much RAM would we safe? The pointer for the PostingList object (4-8 bytes), plus the size of the object header - how much is that in Java?
Seems ilke it's 8 bytes: http://www.codeinstructions.com/2008/12/java-objects-memory-structure.html
So in a 32Bit JVM we would safe 4 bytes (pointer) + 8 bytes (header) - 4 bytes (ID) = 8 bytes. For fields with tons of unique terms that might be worth it?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850857#action_12850857 ]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
Yeah I think we're gonna need the global sequenceID in some form -- my Options 1 or 2 can't work because the interleaving issue (as seen/required by the app) is a global thing.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853594#action_12853594 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
Michael B, I was going to wait for your prototype before continuing... :)
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850760#action_12850760 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
Yes, we would need to buffer terms/queries per DW and also per DW the BufferedDeletes.Num. The docID spaces in two DWs will be completely independent of each other after this change.
One potential problem that we (I think) even today have is the following: If you index with multiple threads, and then call e.g. deleteDocuments(Term) with one of the indexer threads while you keep adding documents with the other threads, it's not clear to the caller when exactly the deleteDocuments(Term) will happen. It depends on the thread scheduling.
Going back to the idea I mentioned here:
https://issues.apache.org/jira/browse/LUCENE-2293?focusedCommentId=12841407&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12841407
I mentioned the idea of having a sequence ID, that gets incremented on add, delete, update. What if we had even with separate DWs a global sequence ID? The sequence ID would tell you unambiguously which action happened when. The add/update/delete methods could return the sequenceID that was assigned to that particular action.
Then we could e.g. track the delete terms globally together with the sequenceID of the corresponding delete call, while we still apply deletes during flush. Since sequenceIDs enforce a strict ordering we can figure out to how many docs per DW we need to apply the delete terms.
Later when we switch to real-time deletes (when the RAM is searchable) we will simply store the sequenceIDs in the deletes int[] array which I mentioned in my comment on LUCENE-2293.
Does this make sense?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850312#action_12850312 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
bq. Not all apps index only 140 character docs from all threads
What a luxury! :)
{quote}
I think for this same reason the ThreadBinder should have affinity, ie, try to schedule the same thread to the same DW, assuming it's free. If it's not free and another DW is free you should use the other one.
{quote}
If you didn't have such an affinity but use a random assignment of DWs to threads, would that balance the RAM usage across DWs without a global RAM management?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857164#action_12857164 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
{quote}
It's for performance. I expect there are apps where a given
thread/pool indexes certain kind of docs, ie, the app threads
themselves have "affinity" for docs with similar term distributions.
In which case, it's best (most RAM efficient) if those docs w/
presumably similar term stats are sent back to the same DW. If you
mix in different term stats into one buffer you get worse RAM
efficiency.
{quote}
I do see your point, but I feel like we shouldn't optimize/make compromises for this use case. Mainly, because I think apps with such an affinity that you describe are very rare? The usual design is a queued ingestion pipeline, where a pool of indexer threads take docs out of a queue and feed them to an IndexWriter, I think? In such a world the threads wouldn't have an affinity for similar docs.
And if a user really has so different docs, maybe the right answer would be to have more than one single index? Even if today an app utilizes the thread affinity, this only results in maybe somewhat faster indexing performance, but the benefits would be lost after flusing/merging.
If we assign docs randomly to available DocumentsWriterPerThreads, then we should on average make good use of the overall memory? Alternatively we could also select the DWPT from the pool of available DWPTs that has the highest amount of free memory?
Having a fully decoupled memory management is compelling I think, mainly because it makes everything so much simpler. A DWPT could decide itself when it's time to flush, and the other ones can keep going independently.
If you do have a global RAM management, how would the flushing work? E.g. when a global flush is triggered because all RAM is consumed, and we pick the DWPT with the highest amount of allocated memory for flushing, what will the other DWPTs do during that flush? Wouldn't we have to pause the other DWPTs to make sure we don't exceed the maxRAMBufferSize?
Of course we could say "always flush when 90% of the overall memory is consumed", but how would we know that the remaining 10% won't fill up during the time the flush takes?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846084#action_12846084 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
Shall we not first try to remove the downstream *PerThread classes and make the DocumentsWriter single-threaded without locking. Then we add a PerThreadDocumentsWriter and DocumentsWriterThreadBinder, which talks to the PerThreadDWs and IW talks to DWTB. We can pick other names :)
When that's done we can think about what kind of locking/synchronization/volatile stuff we need for LUCENE-2312.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849844#action_12849844 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
Michael, I'm working on a patch and will post one (hopefully) shortly.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850290#action_12850290 ]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
bq. But is that a realistic scenario? That the "avg. document size per thread" differ significantly in an application?
I think this could happen, eg if an app uses different threads for indexing different sources of docs. Not all apps index only 140 character docs from all threads ;)
I think for this same reason the ThreadBinder should have affinity, ie, try to schedule the same thread to the same DW, assuming it's free. If it's not free and another DW is free you should use the other one.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Busch updated LUCENE-2324:
----------------------------------
Attachment: (was: lucene-2324-no-pooling.patch)
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859910#action_12859910 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
bq. It'd sort of "match" the current behaviour, in that you get segments flushed to the index with that many docs.
It seems like the consensus is, most users don't use the maxBufferedDocs feature, however for the ones that do, we can stop the world and flush all segments? That way we can focus on the RAM consumed use case?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846220#action_12846220 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
Michael, Agreed, can you outline how you think we should proceed then?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859906#action_12859906 ]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
{quote}
I'm not sure I understand how this would help for ParallelReader?
I think you can't use multi-threaded indexing even today, because you
have no control over the order in which the docs will make it into the
index.
{quote}
Well, to set up indexes for PR today, you have to run IndexWriter in a very degraded state -- flush by doc count, use a single thread, turn off concurrent merging (use SMS), use LogDocMergePolicy.
{quote}
So having maxBufferedDocs per DWPT seems tempting to me. Then you know
that each written segment will have exactly a size of maxBufferedDocs,
so this is much more predictable. And if you index with a single
thread only the behavior is identical to a "global" maxBufferedDocs
flush trigger.
{quote}
Yeah, maybe that'd be sufficient...? It'd sort of "match" the current behaviour, in that you get segments flushed to the index with that many docs.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857373#action_12857373 ]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
bq. The usual design is a queued ingestion pipeline, where a pool of indexer threads take docs out of a queue and feed them to an IndexWriter, I think?
bq. Mainly, because I think apps with such an affinity that you describe are very rare?
Hmm I suspect it's not that rare.... yes one design is a single
indexing queue w/ dedicated thread pool only for indexing, but a push
model is equal valid, where your app already has separate threads (or
thread pools) servicing different content sources, so when a doc
arrives to one of those source-specific threads, it's that thread that
indexes it, rather than handing off to a separately pool.
Lucene is used in a very wide variety of apps -- we shouldn't optimize
the indexer on such hard app specific assumptions.
bq. And if a user really has so different docs, maybe the right answer would be to have more than one single index?
Hmm but the app shouldn't have to resort to this... (it doesn't have
to today).
But... could we allow an add/updateDocument call to express this
affinity, explicitly? If you index homogenous docs you wouldn't use
it, but, if you index drastically different docs that fall into clear
"categories", expressing the affinity can get you a good gain in
indexing throughput.
This may be the best solution, since then one could pass the affinity
even through a thread pool, and then we would fallback to thread
binding if the document class wasn't declared?
I mean this is virtually identical to "having more than one index",
since the DW is like its own index. It just saves some of the
copy-back/merge cost of addIndexes...
bq. Even if today an app utilizes the thread affinity, this only results in maybe somewhat faster indexing performance, but the benefits would be lost after flusing/merging.
Yes this optimization is only about the initial flush, but, it's
potentially sizable. Merging matters less since typically it's not
the bottleneck (happens in the BG, quickly enough).
On the right apps, thread affinity can make a huge difference. EG if
you allow up to 8 thread states, and the threads are indexing content
w/ highly divergent terms (eg, one language per thread, or, docs w/
very different field names), in the worst case you'll be up to 1/8 as
efficient since each term must now be copied in up to 8 places
instead of one. We have a high per-term RAM cost (reduced thanks to
the parallel arrays, but, still high).
bq. If we assign docs randomly to available DocumentsWriterPerThreads, then we should on average make good use of the overall memory?
It really depends on the app -- if the term space is highly thread
dependent (above examples) you an end up flush much more frequently for
a given RAM buffer.
bq. Alternatively we could also select the DWPT from the pool of available DWPTs that has the highest amount of free memory?
Hmm... this would be kinda costly binder? You'd need a pqueue?
Thread affinity (or the explicit affinity) is a single
map/array/member lookup. But it's an interesting idea...
bq. If you do have a global RAM management, how would the flushing work? E.g. when a global flush is triggered because all RAM is consumed, and we pick the DWPT with the highest amount of allocated memory for flushing, what will the other DWPTs do during that flush? Wouldn't we have to pause the other DWPTs to make sure we don't exceed the maxRAMBufferSize?
The other DWs would keep indexing :) That's the beauty of this
approach... a flush of one DW doesn't stop all other DWs from
indexing, unliked today.
And you want to serialize the flushing right? Ie, only one DW flushes
at a time (the others keep indexing).
Hmm I suppose flushing more than one should be allowed (OS/IO have
alot of concurrency, esp since IO goes into write cache)... perhaps
that's the best way to balance index vs flush time? EG we pick one to
flush @ 90%, if we cross 95% we pick another to flush, another at
100%, etc.
bq. Of course we could say "always flush when 90% of the overall memory is consumed", but how would we know that the remaining 10% won't fill up during the time the flush takes?
Regardless of the approach for document -> DW binding, this is an
issue (ie it's non-differentiating here)? Ie the other DWs continue
to consume RAM while one DW is flushing. I think the low/high water
mark is an OK solution here? Or the tiered flushing (I think I like
that better :) ).
bq. Having a fully decoupled memory management is compelling I think, mainly because it makes everything so much simpler. A DWPT could decide itself when it's time to flush, and the other ones can keep going independently.
I'm all for simplifying things, which you've already nicely done here,
but not of it's at the cost of a non-trivial potential indexing perf
loss. We're already taking a perf hit here, since the doc stores
can't be shared... I think that case is justifiable (good
simplification).
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845199#action_12845199 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
Here is an interesting article about allocation/deallocation on modern JVMs:
http://www.ibm.com/developerworks/java/library/j-jtp09275.html
And here is a snippet that mentions how pooling is generally not faster anymore:
----
Allocation in JVMs was not always so fast -- early JVMs indeed had poor allocation and garbage collection performance, which is almost certainly where this myth got started. In the very early days, we saw a lot of "allocation is slow" advice -- because it was, along with everything else in early JVMs -- and performance gurus advocated various tricks to avoid allocation, such as object pooling. (Public service announcement: Object pooling is now a serious performance loss for all but the most heavyweight of objects, and even then it is tricky to get right without introducing concurrency bottlenecks.) However, a lot has happened since the JDK 1.0 days; the introduction of generational collectors in JDK 1.2 has enabled a much simpler approach to allocation, greatly improving performance.
----
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850500#action_12850500 ]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
I don't think we should delete in FG -- I suspect this'll give net/net worse performance, due to loss of locality. It also means you must always keep readers available, which is an unnecessary cost for non-NRT apps.
deletesInRAM are those deletes done during the current segment. deletesFlushed absorbs deletesInRAM on successful segment flush. We have to segregate the two for proper recovery if we fail to flush the RAM buffer, eg say you hit a disk full while flushing a new segment, and then you close your IW successfully. We have to make sure in that case that deletesInRAM are discarded.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850244#action_12850244 ]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
But if 1 thread tends to index lots of biggish docs... don't we want to allow it to use up more than 1/nth?
Ie we don't want to flush unless total RAM usage has hit the limit?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858948#action_12858948 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
I get Michael B's motivation however Mike M's global ram limit
is probably better from a user perspective and it would be a
sync only on a long RAM used variable, so we should be good?
Sounds like we use the current thread affinity system (ie, a
hash map), that when the max threads is reached, new threads get
kind of round robined onto existing DWPTs?
{quote}if RAM usage grows too much beyond your first trigger and
before that first flush has finished, start a 2nd DWPT flushing,
etc. {quote}
What's the definition of the "too much" portion of the above
statement?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850227#action_12850227 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
Following up on the previous comment, if the current thread (the one calling add doc) is also the one that needs to do the flushing, then only the thread attached to the doc writer with the greatest ram usage can/should do the flushing?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Busch updated LUCENE-2324:
----------------------------------
Attachment: lucene-2324.patch
The patch removes all *PerThread classes downstream of DocumentsWriter.
This simplifies a lot of the flushing logic in the different consumers. The patch also removes FreqProxMergeState, because we don't have to interleave posting lists from different threads anymore of course. I really like these simplifications!
There is still a lot to do: The changes in DocumentsWriter and IndexWriter are currently just experimental to make everything compile. Next I will introduce DocumentsWriterPerThread and implement the sequenceID logic (which was discussed here in earlier comments) and the new RAM management. I also want to go through the indexing chain once again - there are probably a few more things to clean up or simplify.
The patch compiles and actually a surprising amount of tests pass. Only multi-threaded tests seem to fail,
which is not very surprising, considering I removed all thread-handling logic from DocumentsWriter. :)
So this patch isn't working yet - just wanted to post my current progress.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Earwin Burrfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845408#action_12845408 ]
Earwin Burrfoot commented on LUCENE-2324:
-----------------------------------------
> Seems ilke it's 8 bytes
Object header is two words, so that's 16bytes for 64bit arch. (probably 12 for 64bit+CompressedOops?)
Also, GC time is (roughly) linear in number of objects on heap, so replacing single huge array of objects with few huge primitive arrays for their fields does miracles to your GC delays.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845261#action_12845261 ]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
Sounds great -- let's test it in practice.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Issue Comment Edited: (LUCENE-2324) Per thread
DocumentsWriters that write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859590#action_12859590 ]
Michael Busch edited comment on LUCENE-2324 at 4/21/10 8:06 PM:
----------------------------------------------------------------
{quote}
The deletes aren't entirely in the foreground, only the RAM
buffer deletes. Deletes to existing segments would use the
existing clone and delete mechanism.
{quote}
I think deletes should all be based on the new sequenceIDs. Today the buffered
deletes depend on a value that can change (numDocs changes when segments are
merged). But with sequenceIDs they're absolute: each delete operation will
have a sequenceID assigned, and the ordering of all write operations (add,
update, delete) is unambiguously defined by the sequenceIDs. Remember that
addDocument/updateDocument/deleteDocuments will all return the sequenceID.
This means that we don't have to remap deletes anymore. Also the pooled
SegmentReaders that read the flushed segments can use arrays of sequenceIDs
instead of BitSets? Of course that needs more memory, but even if you add 1M
docs without ever calling IW.commit/close you only need 8MB - I think that's
acceptable. And this size is independent on how many times you call
reopen/clone on the realtime readers, because they can all share the same
deletes array.
We can also modify IW.commit/close to return the latest sequenceID. This would
be very nice for document tracking. E.g. when you hit an aborting exception
after you flushed already some segments, then even though you must discard
everything in the DW's buffer you can still call commit/close to commit
everything that doesn't have to be discarded. IW.commit/close would in that
case return the sequenceID of the latest write operation that was successfully
committed, i.e. that would be visible to an IndexReader. Though we have to be
careful here: multiple segments can have interleaving sequenceIDs, so we must
discard every segment that has one or more sequenceIDs greater than the lowest
one in the DW. So we still need the push-deletes logic, that keeps RAM deletes
separate from the flushed ones until flushing was successful.
DW/IW need to keep track of the largest sequenceID that is *safe*, i.e. that
could be committed even if DW hits an aborting exception. Some invariants:
* All deletes with sequenceID smaller or equal to safeSeqID have already been
applied to the deletes arrays of the flushed segments.
* All deletes with sequenceID greater than safeSeqID are in a deletesInRAM
buffer.
* safeSeqID is always smaller than any buffered doc or buffered delete in DW
or DWPT.
* safeSeqID is always equal to the maximum sequenceID of one, and only one,
flushed segment.
When IW.close/commit is called then safeSeqID is returned. If no aborting
exception occurred it equals the highest sequenceID ever assigned (during that
IW "session"). In any case it's always the sequenceID of the latest write
operation an IndexReader will "see" that you open after IW.close/commit
finished.
But this would be nice for apps to track which docs made it successfully into
the index. Apps can then externally keep a log to figure out what they have to
reply in case of exceptions.
Does all this make sense? :) This is very complex stuff, I wouldn't be
surprised if there's something I didn't think about.
was (Author: michaelbusch):
{quote}
The deletes aren't entirely in the foreground, only the RAM
buffer deletes. Deletes to existing segments would use the
existing clone and delete mechanism.
{quote}
I think deletes should all be based on the new sequenceIDs. Today the buffered
deletes depend on a value that can change (numDocs changes when segments are
merged). But with sequenceIDs they're absolute: each delete operation will
have a sequenceID assigned, and the ordering of all write operations (add,
update, delete) is unambiguously defined by the sequenceIDs. Remember that
addDocument/updateDocument/deleteDocuments will all return the sequenceID.
This means that we don't have to remap deletes anymore. Also the pooled
SegmentReaders that read the flushed segments can use arrays of sequenceIDs
instead of BitSets? Of course that needs more memory, but even if you add 1M
docs without ever calling IW.commit/close you only need 8MB - I think that's
acceptable. And this size is independent on how many times you call
reopen/clone on the realtime readers, because they can all share the same
deletes array.
We can also modify IW.commit/close to return the latest sequenceID. This would
be very nice for document tracking. E.g. when you hit an aborting exception
after you flushed already some segments, then even though you must discard
everything in the DW's buffer you can still call commit/close to commit
everything that doesn't have to be discarded. IW.commit/close would in that
case return the sequenceID of the latest write operation that was successfully
committed, i.e. that would be visible to an IndexReader. Though we have to be
careful here: multiple segments can have interleaving segmentIDs, so we must
discard every segment that has one or more sequenceIDs greater than the lowest
one in the DW. So we still need the push-deletes logic, that keeps RAM deletes
separate from the flushed ones until flushing was successful.
DW/IW need to keep track of the largest sequenceID that is *safe*, i.e. that
could be committed even if DW hits an aborting exception. Some invariants:
* All deletes with sequenceID smaller or equal to safeSeqID have already been
applied to the deletes arrays of the flushed segments.
* All deletes with sequenceID greater than safeSeqID are in a deletesInRAM
buffer.
* safeSeqID is always smaller than any buffered doc or buffered delete in DW
or DWPT.
* safeSeqID is always equal to the maximum sequenceID of one, and only one,
flushed segment.
When IW.close/commit is called then safeSeqID is returned. If no aborting
exception occurred it equals the highest sequenceID ever assigned (during that
IW "session"). In any case it's always the sequenceID of the latest write
operation an IndexReader will "see" that you open after IW.close/commit
finished.
But this would be nice for apps to track which docs made it successfully into
the index. Apps can then externally keep a log to figure out what they have to
reply in case of exceptions.
Does all this make sense? :) This is very complex stuff, I wouldn't be
surprised if there's something I didn't think about.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853751#action_12853751 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
Sorry, Jason, I got sidetracked with LUCENE-2329 and other things at work. I'll try to write the sequence ID stuff asap. However, there's more we need to do here that is sort of independent of the deleted docs problem. E.g. removing all the downstream perThread classes.
We should work with the flex code from now on, as the flex branch will be merged into trunk soon.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859900#action_12859900 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
{quote}
bq. The problem with maxBufferedDocs is that you can only enforce it by "stopping the world"
Why not just stop the world for maxBufferedDocs/maxBufferedDelDocs?
It should be pretty simple to implement?
Obviously you'll take a perf hit if you flush by these counts, so, you
really should flush by RAM instead... But some apps need this (eg
setting up indices for ParallelReader, until we get SliceWriter
working...).
{quote}
I'm not sure I understand how this would help for ParallelReader?
I think you can't use multi-threaded indexing even today, because you
have no control over the order in which the docs will make it into the
index.
So having maxBufferedDocs per DWPT seems tempting to me. Then you know
that each written segment will have exactly a size of maxBufferedDocs,
so this is much more predictable. And if you index with a single
thread only the behavior is identical to a "global" maxBufferedDocs
flush trigger.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849819#action_12849819 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
Hey Jason,
Disregard my patch here. I just experimented with removal of pooling, but then did LUCENE-2329 instead. TermsHash and TermsHashPerThread are now much simpler, because all the pooling code is gone after 2329 was committed. Should make it a little easier to get this patch done.
Sure it'd be awesome if you could provide a patch here. I can help you, we should just frequently post patches here so that we don't both work on the same areas.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850235#action_12850235 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
The easiest would be if each DocumentsWriterPerThread had a fixed buffer size, then they can flush fully independently and you don't need to manage RAM globally across threads.
Of course then you'd need two config parameters: number of concurrent threads and buffer size per thread.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850292#action_12850292 ]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
bq. I was thinking we were going to do that... having a fixed number of DocumentsWriterPerThread instances, and a ThreadBinder that let's a thread wait if the perthread is not available. You don't need to interleave docIds then?
I think we'd have a fixed max? And then we'd create a new DW when all existing ones are in-use and we're not yet at the max?
Eg if it's a thread pool of size 50 that's indexing, but, the rate of docs is very slow such that in practice only one of these 50 threads is indexing at once, we'd only use one DW. And we'd flush when that DW hits the RAM limit.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Issue Comment Edited: (LUCENE-2324) Per thread
DocumentsWriters that write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858591#action_12858591 ]
Michael Busch edited comment on LUCENE-2324 at 4/19/10 1:38 PM:
----------------------------------------------------------------
{quote}
But... could we allow an add/updateDocument call to express this
affinity, explicitly? If you index homogenous docs you wouldn't use
it, but, if you index drastically different docs that fall into clear
"categories", expressing the affinity can get you a good gain in
indexing throughput.
This may be the best solution, since then one could pass the affinity
even through a thread pool, and then we would fallback to thread
binding if the document class wasn't declared?
{quote}
I would like this if we then also added an API that can be used to specify the
per-DWPT RAM size. E.g. if someone has such an app where different threads
index docs of different sizes, then the DW that indexes big docs can be given
more memory?
What I'm mainly trying to avoid is synchronization points between the
different DWPTs. For example, currently the same ByteBlockAllocator is shared
between the different threads, so all its methods need to be synchronized.
{quote}
The other DWs would keep indexing That's the beauty of this
approach... a flush of one DW doesn't stop all other DWs from
indexing, unliked today.
And you want to serialize the flushing right? Ie, only one DW flushes
at a time (the others keep indexing).
Hmm I suppose flushing more than one should be allowed (OS/IO have
alot of concurrency, esp since IO goes into write cache)... perhaps
that's the best way to balance index vs flush time? EG we pick one to
flush @ 90%, if we cross 95% we pick another to flush, another at
100%, etc.
{quote}
Oh I don't want to disallow flushing in parallel! I think it makes perfect
sense to allow more than one DW to flush at the same time. If each DWPT has a
private max buffer size, then it can decide on its own when it's time to
flush.
{quote}
Hmm I suppose flushing more than one should be allowed (OS/IO have
alot of concurrency, esp since IO goes into write cache)... perhaps
that's the best way to balance index vs flush time? EG we pick one to
flush @ 90%, if we cross 95% we pick another to flush, another at
100%, etc.
{quote}
If we allow flushing in parallel and also allow specifying the max RAM per
DWPT, then there doesn't even have to be any cross-thread RAM tracking? Each
DWPT could just flush when its own buffer is full?
So let's summarize:
# Expose a ThreadBinder API for controlling number of DWPT instances and
thread affinity of DWPTs explicitly. (We can later decide if we want to also
support such an affinity after a segment was flushed, as Tim is asking for.
But that should IMO not be part of this patch.)
# Also expose an API for specifying the RAM buffer size per DWPT.
# Allow flushing in parallel (multiple DWPTs can flush at the same time). A
DWPT flushes when its buffer is full, independent of what the other DWPTs are
doing.
# The default implementation of the ThreadBinder API assigns threads to DWPT
randomly and gives each DWPT 1/n-th of the overall memory.
# The DWPT RAM value must be updateable. E.g. when you first start indexing
only one DWPT should be created with the max RAM. Then when multiple threads
are used for adding documents another DWPT should be added and the RAM
value of the already existing one should be reduced, and possibly a flush of that
DWPT needs to be triggered.
How does this sound?
was (Author: michaelbusch):
{quote}
But... could we allow an add/updateDocument call to express this
affinity, explicitly? If you index homogenous docs you wouldn't use
it, but, if you index drastically different docs that fall into clear
"categories", expressing the affinity can get you a good gain in
indexing throughput.
This may be the best solution, since then one could pass the affinity
even through a thread pool, and then we would fallback to thread
binding if the document class wasn't declared?
{quote}
I would like this if we then also added an API that can be used to specify the
per-DWPT RAM size. E.g. if someone has such an app where different threads
index docs of different sizes, then the DW that indexes big docs can be given
more memory?
What I'm mainly trying to avoid is synchronization points between the
different DWPTs. For example, currently the same ByteBlockAllocator is shared
between the different threads, so all its methods need to be synchronized.
{quote}
The other DWs would keep indexing That's the beauty of this
approach... a flush of one DW doesn't stop all other DWs from
indexing, unliked today.
And you want to serialize the flushing right? Ie, only one DW flushes
at a time (the others keep indexing).
Hmm I suppose flushing more than one should be allowed (OS/IO have
alot of concurrency, esp since IO goes into write cache)... perhaps
that's the best way to balance index vs flush time? EG we pick one to
flush @ 90%, if we cross 95% we pick another to flush, another at
100%, etc.
{quote}
Oh I don't want to disallow flushing in parallel! I think it makes perfect
sense to allow more than one DW to flush at the same time. If each DWPT has a
private max buffer size, then it can decide on its own when it's time to
flush.
{quote}
Hmm I suppose flushing more than one should be allowed (OS/IO have
alot of concurrency, esp since IO goes into write cache)... perhaps
that's the best way to balance index vs flush time? EG we pick one to
flush @ 90%, if we cross 95% we pick another to flush, another at
100%, etc.
{quote}
If we allow flushing in parallel and also allow specifying the max RAM per
DWPT, then there doesn't even have to be any cross-thread RAM tracking? Each
DWPT could just flush when its own buffer is full?
So let's summarize:
# Expose a ThreadBinder API for controlling number of DWPT instances and
thread affinity of DWPTs explicitly. (We can later decide if we want to also
support such an affinity after a segment was flushed, as Tim is asking for.
But that should IMO not be part of this patch.)
# Also expose an API for specifying the RAM buffer size per DWPT.
# Allow flushing in parallel (multiple DWPTs can flush at the same time). A
DWPT flushes when its buffer is full, independent of what the other DWPTs are
doing.
# The default implementation of the ThreadBinder API assigns threads to DWPT
randomly and gives each DWPT 1/n-th of the overall memory.
How does this sound?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857380#action_12857380 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
bq. only one DW flushes at a time (the others keep indexing).
I think it's best to simply flush at 90% for now. We already
exceed the ram buffer size because of over allocation? Perhaps
we can view the ram buffer size as a rough guideline not a hard
and fast limit because, lets face it, we're using Java which is
about as inexact when it comes to RAM consumption as it gets?
Also, hopefully it would move the patch along faster and more
complex algorithms could easily be added later.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859957#action_12859957 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
{quote}This means that we don't have to remap deletes
anymore.{quote}
Good.
{quote}the pooled SegmentReaders that read the flushed segments
can use arrays of sequenceIDs instead of BitSets?{quote}
Ok, that'd be an improvement. In order to understand all the
logic, I'd need to see a prototype, I can't really regurgitate
what I've read thus far. Are there any concurrency issues with
the seq arrays? Like say if a reader is iterating a posting
list? I guess not because we're always incrementing?
{quote}safeSeqID is always smaller than any buffered doc or
buffered delete in DW or DWPT.{quote}
Can you elaborate on this? Wouldn't safeSeqID be greater than
buffered doc or deleted doc due to add
docs and interleaving?
Also can we modify the terminology, perhaps committed seq id is
better? To me that's a little more clear. Maybe it'd be good to
start a wiki on this so we can have a master doc we're all
referring to (kind of a Solr thing that because the projects are
merged we can try for?) ;)
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851142#action_12851142 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
{quote}
The clarify, the apply deletes doc id up to will be the flushed doc count saved per term/query per DW, though it won't be saved, it'll be derived from the sequence id int array where the action has been encoded into the seq id int?
{quote}
Yeah, that's the idea. Let's see if it works :)
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850295#action_12850295 ]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
Yes, doFlushInternal should no longer be sync'd. It should have a small sync'd at the end where it 1) inserts the new SegmentInfo into the in-memory segments, and 2) computes the new base for remapping the deletes. I think it can then release the sync while it does the remapping of buffered deletes?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857112#action_12857112 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
Michael, nice! I guess I should've spent more time removing the PerThread classes, but now we're pretty much there. Indeed the simplification should make things a lot better. I guess I'll wait for the next patch to work on something.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849806#action_12849806 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
Michael, I'm guessing this patch needs to be updated as per LUCENE-2329?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324-no-pooling.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846128#action_12846128 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
I think we all agree that we want to have a single writer thread, multi reader thread model. Only then the thread-safety problems in LUCENE-2312 can be reduced to visibility (no write-locking). So I think making this change first makes most sense. It involves a bit boring refactoring work unfortunately.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850262#action_12850262 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
{quote}
But if 1 thread tends to index lots of biggish docs... don't we want to allow it to use up more than 1/nth?
Ie we don't want to flush unless total RAM usage has hit the limit?
{quote}
Sure that'd be the disadvantage. But is that a realistic scenario? That the "avg. document size per thread" differ significantly in an application?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Issue Comment Edited: (LUCENE-2324) Per thread
DocumentsWriters that write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845398#action_12845398 ]
Michael Busch edited comment on LUCENE-2324 at 3/15/10 4:34 PM:
----------------------------------------------------------------
Reply to Mike's comment on LUCENE-2293: https://issues.apache.org/jira/browse/LUCENE-2293?focusedCommentId=12845263&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12845263
{quote}
I think we can do even better, ie, that class wastes RAM for the single posting case (intStart, byteStart, lastDocID, docFreq, lastDocCode, lastDocPosition are not needed).
EG we could have a separate class dedicated to the singleton case. When term is first encountered it's enrolled there. We'd probably need a separate hash to store these (though not necessarily?). If it's seen again it's switched to the full posting.
{quote}
Hmm I think we'd need a separate hash. Otherwise you have to subclass PostingList for the different cases (freq. vs. non-frequent terms) and do instanceof checks? Or with the parallel arrays idea maybe we could encode more information in the dense ID? E.g. use one bit to indicate if that term occurred more than once.
{quote}
I mean instead of allocating an instance per unique term, we assign an integer ID (dense, ie, 0, 1, 2...).
And then we have an array for each member now in FreqProxTermsWriter.PostingList, ie int[] docFreqs, int [] lastDocIDs, etc. Then to look up say the lastDocID for a given postingID you just get lastDocIDs[postingID]. If we're worried about oversize allocation overhead, we can make these arrays paged... but that'd slow down each access.
{quote}
Yeah I like that idea. I've done something similar for representing trees - I had a very compact Node class with no data but such a dense ID, and arrays that stored the associated data. Very easy to add another data type with no RAM overhead (you only use the amount of RAM the data needs).
Though, the price you pay is for dereferencing multiple times for each array?
And how much RAM would we safe? The pointer for the PostingList object (4-8 bytes), plus the size of the object header - how much is that in Java?
Seems ilke it's 8 bytes: http://www.codeinstructions.com/2008/12/java-objects-memory-structure.html
So in a 32Bit JVM we would safe 4 bytes (pointer) + 8 bytes (header) - 4 bytes (ID) = 8 bytes. For fields with tons of unique terms that might be worth it?
was (Author: michaelbusch):
{quote}
I think we can do even better, ie, that class wastes RAM for the single posting case (intStart, byteStart, lastDocID, docFreq, lastDocCode, lastDocPosition are not needed).
EG we could have a separate class dedicated to the singleton case. When term is first encountered it's enrolled there. We'd probably need a separate hash to store these (though not necessarily?). If it's seen again it's switched to the full posting.
{quote}
Hmm I think we'd need a separate hash. Otherwise you have to subclass PostingList for the different cases (freq. vs. non-frequent terms) and do instanceof checks? Or with the parallel arrays idea maybe we could encode more information in the dense ID? E.g. use one bit to indicate if that term occurred more than once.
{quote}
I mean instead of allocating an instance per unique term, we assign an integer ID (dense, ie, 0, 1, 2...).
And then we have an array for each member now in FreqProxTermsWriter.PostingList, ie int[] docFreqs, int [] lastDocIDs, etc. Then to look up say the lastDocID for a given postingID you just get lastDocIDs[postingID]. If we're worried about oversize allocation overhead, we can make these arrays paged... but that'd slow down each access.
{quote}
Yeah I like that idea. I've done something similar for representing trees - I had a very compact Node class with no data but such a dense ID, and arrays that stored the associated data. Very easy to add another data type with no RAM overhead (you only use the amount of RAM the data needs).
Though, the price you pay is for dereferencing multiple times for each array?
And how much RAM would we safe? The pointer for the PostingList object (4-8 bytes), plus the size of the object header - how much is that in Java?
Seems ilke it's 8 bytes: http://www.codeinstructions.com/2008/12/java-objects-memory-structure.html
So in a 32Bit JVM we would safe 4 bytes (pointer) + 8 bytes (header) - 4 bytes (ID) = 8 bytes. For fields with tons of unique terms that might be worth it?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849965#action_12849965 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
I'm a little confused in the flushedDocCount, remap deletes conversion portions of DocWriter. flushedDocCount is used as a global counter, however when we move to per thread doc writers, it won't be global anymore. Is there a different (easier) way to perform remap deletes?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850864#action_12850864 ]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
bq. Mike, can you explain what the advantages of this kind of thread affinity are? I was always wondering why the DocumentsWriter code currently makes efforts to assign a ThreadState always to the same Thread? Is that being done for performance reasons?
It's for performance. I expect there are apps where a given
thread/pool indexes certain kind of docs, ie, the app threads
themselves have "affinity" for docs with similar term distributions.
In which case, it's best (most RAM efficient) if those docs w/
presumably similar term stats are sent back to the same DW. If you
mix in different term stats into one buffer you get worse RAM
efficiency.
Also, for better RAM efficiency you want *fewer* DWs... because we get
more RAM efficiency the higher the freq of the terms... but of course
you want more DWs for better CPU efficiency whenever that many threads
are running at once.
Net/net CPU efficiency should trump RAM efficiency, I think, so if
there is a conflict we should favor CPU efficiency.
Though, thread affinity doesn't seem that CPU costly to implement?
Lookup the DW your thread first used... if it's free, seize it. If
it's not, fallback to any DW that's free.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850440#action_12850440 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
In regards to remap deletes. I looked at buffered deletes in
depth when implementing LUCENE-2047. I think deletes are fairly
simple, though there's all this logic to manage them because
they're buffered. I'm liking deleting in the foreground and
queuing the deleted doc ids per segment as a way to avoid
remapping absolute doc ids in commit merge. Hopefully this
simplifies things?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859953#action_12859953 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
bq. No, I think the consensus is to have a DWPT maxBufferedDocs flush trigger, not a global one.
I think we should just keep it simple. If you set maxBufferedDocs to 1000, then every thread will flush at 1000 docs.
If you set maxThreadStates to 5, then you could at the same time in theory have 5000 docs in memory.
We just have to explain this in the javadocs. It's a change that we should also mention in the backwards-compatibility section.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849899#action_12849899 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
Awesome!
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859850#action_12859850 ]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
bq. The problem with maxBufferedDocs is that you can only enforce it by "stopping the world"
Why not just stop the world for maxBufferedDocs/maxBufferedDelDocs?
It should be pretty simple to implement?
Obviously you'll take a perf hit if you flush by these counts, so, you
really should flush by RAM instead... But some apps need this (eg
setting up indices for ParallelReader, until we get SliceWriter
working...).
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846037#action_12846037 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
Are there going to be issues with the char array buffers as well (ie, will we need to also flush them for concurrency?)
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859940#action_12859940 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
No, I think the consensus is to have a DWPT maxBufferedDocs flush trigger, not a global one.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859511#action_12859511 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
{quote}flushing by maxBufferedDeleteTerms is also tricky,
because that needs to be a flush across all DWPTs{quote}
I think we simply keep track of maxBufferedDocs and
maxBufferedDeleteTerms globally. We can't deprecate because
that'd be a pain for users.
{quote}With searchable RAM buffers and deletes in the foreground
this trigger should actually not be necessary anymore?{quote}
The deletes aren't entirely in the foreground, only the RAM
buffer deletes. Deletes to existing segments would use the
existing clone and delete mechanism. I asked about foreground
deletes to existing segments before, and we agreed that it's not
a good idea due to locality of terms/postings.
{quote}We would probably apply deletes when the realtime
IndexReader is (re)opened?{quote}
Right, that's how the existing apply deletes basically works.
That'd be the same. Oh ok, I see, perhaps the global deletes
manager could cache the deletes for the existing segments, the
DWPTs can keep track of their deletes separately. We probably
won't apply segment or DWPT deletes in the foreground? We'd
cache them separately, and update ram consumption and unique
term/query counts globally.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858949#action_12858949 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
{quote}
I still think this "zero sync'd code" at the cost of perf loss /
exposing per-DWPT details is taking things too far. You're cutting
into the bone...
{quote}
No worries - I haven't started implementing the RAM management part yet. :)
{quote}
I don't think we should apps to be setting per-DWPT RAM limits, or,
even expose to apps how IW manages threads (this is an impl. detail).
I think we should keep the approach we have today - you set the
overall RAM limit and IW internally manages flushing when that
allotted RAM is full.
{quote}
I think the reason why we have two different APIs in mind (you: sourceID, I:
expert thread binder API) is that we're having different goals with them? You
want to make the out-of-the-box indexing performance as good as possible, and
users should have to set a minimum amount of easy-to-understand parameters
(such as buffer size in MB). I think that's the right thing to do of course.
(though that doesn't prevent us from adding an expert API in addition, as we
always have)
I'm thinking a lot about real-time indexing and the searchable RAM buffer
these days, so the thread-binder API could help you to have more control over
where your docs will actually end up and which reader will see them. But I
think too that this API would be very "expert" and not many people would use
it.
bq. We can do this as a separate issue... it's fairly orthogonal.
Yeah I was just thinking the same - I agree.
{quote}
Is this sync really so bad? First, we should move all
allocators/pools to per-DWPT, so they don't need to be sync'd.
{quote}
OK cool that we agree on that. I was worried you wanted to have global pools
too, if it's only the single long it's not very complicated, I agree.
{quote}
We're still gonna need sync'd code, anyway (global sequence ID,
grabbing a DWPT), right? We can put this "go flush a DWPT" logic in
the same block if we really have to? It feels like we're going to
great lengths (cutting into the bone) to avoid a trivial cost (the
minor complexity of managing flushing based on aggregate RAM used).
{quote}
Sorry if I'm being annoying :) Yeah sure, there will be several sync'd spots.
If we don't share any data structures between threads that hold indexed (and
in the future searchable) data I'm happy.
I haven't spent as much time as you thinking about the current RAM management
yet and the current code that ensures thread safety - still learning some
parts of the code. I do appreciate all your patient feedback!
{quote}
This isn't great... I mean it's weird that on adding say a 3rd
indexing thread I suddenly see a flush triggered even though I'm
nowehere near the RAM limit. Then, later, if I cut back to using only
2 threads, I still only ever use up to 2/3rd of my RAM buffer. IW's
API really shouldn't have such "surprising" behavior where how many /
which threads come through it so drastically affect it's flushing
behavior.
{quote}
Yeah I don't really like that either. Let's not do that. I had first not
thought about that disadvantage, added this point later to the list, and never
really liked it. (and knew you would complain about it :) )
My goal is to have a default indexing chain that isn't slower than the one we
have today, but searchable and that very fast. That's not trivial, but I think
we can do it!
I'll implement the global flush trigger and make all pools DWPT-local. The
explicit thread-binder or sourceID APIs we can worry about later, as we agreed
above.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857124#action_12857124 ]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
This is awesome Michael! Much simpler... nor more FreqProxMergeState, nor the logic to interleave/synchronize writing to the doc stores. I like it!
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851078#action_12851078 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
{quote}
I'm not sure we need that level of complexity just yet? How
would we make the transaction log memory efficient?
{quote}
Is that really so complex? You only need one additional int per doc in the DWPTs, and the global map for the delete terms. You don't need to buffer the actual terms per DWPT. I thought that's quite efficient? But I'm totally open to other ideas.
I can try tonight to code a prototype of this - I don't think it would be very complex actually. But of course there might be complications I haven't thought of.
bq. Are there other uses you foresee?
Not really for the "transaction log" as you called it. I'd remove that log once we switch to deletes in the FG (when the RAM buffer is searchable). But a nice thing would be for add/update/delete to return the seqID, and also the if RAMReader in the future had an API to check up to which seqID it's able to "see". Then it's very clear to user of the API where a given reader is at.
For this to work we have to assign the seqID at the *end* of a call. E.g. when adding a large document, which takes a long time to process, it should get the seqID assigned after the "work" is done and right before the addDocument() call returns.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850766#action_12850766 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
bq. I think for this same reason the ThreadBinder should have affinity
Mike, can you explain what the advantages of this kind of thread affinity are? I was always wondering why the DocumentsWriter code currently makes efforts to assign a ThreadState always to the same Thread? Is that being done for performance reasons?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850265#action_12850265 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
{quote}
I'm not sure how we'd enforce the number of threads? Or we'd
have to re-implement the wait system implemented in DW?
{quote}
I was thinking we were going to do that... having a fixed number of DocumentsWriterPerThread instances, and a ThreadBinder that let's a thread wait if the perthread is not available. You don't need to interleave docIds then?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846112#action_12846112 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
Actually TermsHashField doesn't need to be concurrent, it's only being written to and the terms concurrent skiplist (was a btree) holds the reference to the posting list. So I think we're good there because terms enum never accesses the terms hash. Nice!
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846028#action_12846028 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
Carrying over from LUCENE-2312. I'm proposing we for starters have a byte slice writer, lock, move or copy(?) the bytes from the writable byte pool/writer to a read only byte block pool, unlock. This sounds like a fairly self-contained thing that can be unit tested at a low level.
Mike, can you add a bit as to how this could work? Also, what is the IntBlockPool used for?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846586#action_12846586 ]
Michael Busch commented on LUCENE-2324:
---------------------------------------
bq. Michael, Agreed, can you outline how you think we should proceed then?
Sorry for not responding earlier...
I'm currently working on removing the PostingList object pooling, because it makes TermsHash and TermsHashPerThread much easier. Have written the patch and all tests pass, though I haven't done performance testing yet. Making TermsHash and TermsHashPerThread smaller will also make the patch here easier which will remove them. I'll post the patch soon.
Next steps I think here are to make everything downstream of DocumentsWriter single-threaded (removal of *PerThread) classes. Then we need to write the DocumentsWriterThreadBinder and have to think about how to apply deletes, commits and rollbacks to all DocumentsWriter instances.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850260#action_12850260 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
Also, I think we can remove the sync on doFlushInternal?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Tim Smith (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857385#action_12857385 ]
Tim Smith commented on LUCENE-2324:
-----------------------------------
bq. Probably if you really want to keep the segments segregated like that, you should in fact index to separate indices?
Thats what i'm currently thinking i'll have to do
however it would be ideal if i could either subclass IndexWriter or use IndexWriter directly with this affinity concept (potentially writing my own segment merger that is affinity aware)
that makes it so i can easily use near real time indexing, as only one IndexWriter will be in the mix, as well as make managing deletes and a whole other host of issues with multiple indexes disappear
Also makes it so i can configure memory settings across all "affinity groups" instead of having to dynamically create them, each with their own memory bounds
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850253#action_12850253 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
bq. Of course then you'd need two config parameters: number of
concurrent threads and buffer size per thread.
I'm not sure how we'd enforce the number of threads? Or we'd
have to re-implement the wait system implemented in DW? In
practice, each thread's DW will probably have roughly the same
ram buffer size so if they each have the same max size, that'd
be ok. I don't think we can limit the number of threads though,
because we'd need to then implement interleaving doc ids?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858623#action_12858623 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
Michael B, sounds good. The approach outlined is straightforward and covers the edge cases.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859387#action_12859387 ]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
{quote}
Sounds like we use the current thread affinity system (ie, a
hash map), that when the max threads is reached, new threads get
kind of round robined onto existing DWPTs?
{quote}
Yeah something along those lines... and clearing out all mappings for
a given DWPT when it flushes.
{quote}
bq. if RAM usage grows too much beyond your first trigger and before that first flush has finished, start a 2nd DWPT flushing, etc.
What's the definition of the "too much" portion of the above
statement?
{quote}
We could do something simple, eg, at 90% RAM used, you flush your
first DWPT. At 110% RAM used, you flush all DWPTs. And take linear
steps in between?
EG if I have 5 DWPTs, I'd flush first one at 90%, 2nd at 95%, 3rd at
100%, 4th at 105% and 5th at 110%.
Of course, if flushing is fast, then RAM is quickly freed up, then we
only flush 1 DWPT at a time... we only need these tiers to
self-regulate RAM consumed from ongoing indexing vs time it takes to
do the flush.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2324) Per thread DocumentsWriters that
write their own private segments
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850449#action_12850449 ]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
Mike, can you describe the difference between DW's deletesInRAM and deletesFlushed? I have some idea, however maybe some clarification will help.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org