You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Peter K <pe...@yahoo.de> on 2012/01/12 16:10:47 UTC
ThreadAffinityDocumentsWriterThreadPool
Hi,
questions regarding ThreadAffinityDocumentsWriterThreadPool:
1. Would someone explain to me the
ThreadAffinityDocumentsWriterThreadPool class? E.g. why couldn't I use a
ThreadLocal (I'll get exceptions)? Also what does the TODO mean?
2. When profiling I see that "threadBindings.get" uses a quite a bit of
my CPU - when using a trove collection* indexing gets a bit faster
(~10%). But this test was only for one thread and yes, I know that
TLongObjectHashMap is not thread safe. But probably worth a try in
general to somehow improve the "threadBindings.get" performance?
Regards,
Peter.
*
<http://trove.nla.gov.au/>http://trove.starlight-systems.com/
Re: ThreadAffinityDocumentsWriterThreadPool
Posted by Simon Willnauer <si...@googlemail.com>.
hey peter,
On Mon, Jan 16, 2012 at 8:45 AM, Peter K <pe...@yahoo.de> wrote:
> Thanks Simon for all the explanations so far :) !
no problem!
>
>>> And what happens if two threads accessing the same ThreadState? The
>>> second will try to lock and fail and then get the minimal contended
>>> state (?) You said there is no problem when two threads accessing one
>>> ThreadState, but won't two threads accessing one
>>> DocumentsWriterPerThread make problems (e.g. when using the singleton
>>> object docState) ?
>> DocumentsWriterPerThread should never be accessed by multiple threads
>> at the same time. This is why we lock the ThreadState.
>> The maxNumThreadStates defines the max. concurrency level IW supports.
>> if more threads than threadstates enter the pool extra threads are
>> queued.
>
> queued means blocked?
yes
>
>> the problem with the min contented is that this number could change
>> while you iterate, ie. you see a state with 0 queued thread and select
>> it but while you try to lock it somebody else already locked it so it
>> might not be the min contented anymore.
>
> ok, I understand.
>
> BTW: the TODO in DocumentsWriterPerThreadPool.minContendedThreadState
> ("new Iterator per indexed doc is overkill") is easily fixable and also
> improves my stupid tests ;)
>
> for (int i = 0; i < numThreadStatesActive; i++) {
> final ThreadState state = perThreads[i];
I will revisit this API soonish - that is one option.
simon
>
> Peter.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: ThreadAffinityDocumentsWriterThreadPool
Posted by Peter K <pe...@yahoo.de>.
Thanks Simon for all the explanations so far :) !
>> And what happens if two threads accessing the same ThreadState? The
>> second will try to lock and fail and then get the minimal contended
>> state (?) You said there is no problem when two threads accessing one
>> ThreadState, but won't two threads accessing one
>> DocumentsWriterPerThread make problems (e.g. when using the singleton
>> object docState) ?
> DocumentsWriterPerThread should never be accessed by multiple threads
> at the same time. This is why we lock the ThreadState.
> The maxNumThreadStates defines the max. concurrency level IW supports.
> if more threads than threadstates enter the pool extra threads are
> queued.
queued means blocked?
> the problem with the min contented is that this number could change
> while you iterate, ie. you see a state with 0 queued thread and select
> it but while you try to lock it somebody else already locked it so it
> might not be the min contented anymore.
ok, I understand.
BTW: the TODO in DocumentsWriterPerThreadPool.minContendedThreadState
("new Iterator per indexed doc is overkill") is easily fixable and also
improves my stupid tests ;)
for (int i = 0; i < numThreadStatesActive; i++) {
final ThreadState state = perThreads[i];
Peter.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: ThreadAffinityDocumentsWriterThreadPool
Posted by Simon Willnauer <si...@googlemail.com>.
On Fri, Jan 13, 2012 at 11:52 AM, Peter K <pe...@yahoo.de> wrote:
> Hi Simon,
>
>> ThreadAffinityDocumentsWriterThreadPool tries to select the least
>> contended DocumentsWriterPerThread for the incoming thread and if
>> possible assigns the same DWPT to the thread if the associated
>> (previously selected) DWPT is not contended.
>
> ah, ok and contended or not is done via queue length ...
>
> Some further questions if you allow :)
sure - as may as you like :)
>
> The naming of the collection 'perThreads' in
> DocumentsWriterPerThreadPool does not make sense to me. Why not
> 'availableThreads'?
yeah I think we should just name it "threadStates"
>
> And what happens if two threads accessing the same ThreadState? The
> second will try to lock and fail and then get the minimal contended
> state (?) You said there is no problem when two threads accessing one
> ThreadState, but won't two threads accessing one
> DocumentsWriterPerThread make problems (e.g. when using the singleton
> object docState) ?
DocumentsWriterPerThread should never be accessed by multiple threads
at the same time. This is why we lock the ThreadState.
The maxNumThreadStates defines the max. concurrency level IW supports.
if more threads than threadstates enter the pool extra threads are
queued.
the problem with the min contented is that this number could change
while you iterate, ie. you see a state with 0 queued thread and select
it but while you try to lock it somebody else already locked it so it
might not be the min contented anymore.
>
>>
>> The TODO is there since there could be a selection which is not ideal
>> if two threads are racing but it won't cause any exceptions etc.
>
> Would this problem also occur if more threads than maxThreadStates try
> to feed lucene?
see above - I don't think its a big issue but it would be nice to get
rid of that limitation.
>
>
>> Is that the test you send the other day to
>> the user list like 900k docs in 29 sec or something like that?
>
> yes, still on that thing.
> sorry, its certainly the way you described. I was and I am just curious
> ... digging day by day a bit deeper into lucene :)
sure! :)
simon
>
> Peter.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: ThreadAffinityDocumentsWriterThreadPool
Posted by Peter K <pe...@yahoo.de>.
Hi Simon,
> ThreadAffinityDocumentsWriterThreadPool tries to select the least
> contended DocumentsWriterPerThread for the incoming thread and if
> possible assigns the same DWPT to the thread if the associated
> (previously selected) DWPT is not contended.
ah, ok and contended or not is done via queue length ...
Some further questions if you allow :)
The naming of the collection 'perThreads' in
DocumentsWriterPerThreadPool does not make sense to me. Why not
'availableThreads'?
And what happens if two threads accessing the same ThreadState? The
second will try to lock and fail and then get the minimal contended
state (?) You said there is no problem when two threads accessing one
ThreadState, but won't two threads accessing one
DocumentsWriterPerThread make problems (e.g. when using the singleton
object docState) ?
>
> The TODO is there since there could be a selection which is not ideal
> if two threads are racing but it won't cause any exceptions etc.
Would this problem also occur if more threads than maxThreadStates try
to feed lucene?
> Is that the test you send the other day to
> the user list like 900k docs in 29 sec or something like that?
yes, still on that thing.
sorry, its certainly the way you described. I was and I am just curious
... digging day by day a bit deeper into lucene :)
Peter.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: ThreadAffinityDocumentsWriterThreadPool
Posted by Simon Willnauer <si...@googlemail.com>.
hey peter,
On Thu, Jan 12, 2012 at 4:10 PM, Peter K <pe...@yahoo.de> wrote:
> Hi,
>
> questions regarding ThreadAffinityDocumentsWriterThreadPool:
>
> 1. Would someone explain to me the ThreadAffinityDocumentsWriterThreadPool
> class? E.g. why couldn't I use a ThreadLocal (I'll get exceptions)? Also
> what does the TODO mean?
ThreadAffinityDocumentsWriterThreadPool tries to select the least
contended DocumentsWriterPerThread for the incoming thread and if
possible assigns the same DWPT to the thread if the associated
(previously selected) DWPT is not contended.
The TODO is there since there could be a selection which is not ideal
if two threads are racing but it won't cause any exceptions etc.
does that make sense?
>
> 2. When profiling I see that "threadBindings.get" uses a quite a bit of my
> CPU - when using a trove collection* indexing gets a bit faster (~10%). But
> this test was only for one thread and yes, I know that TLongObjectHashMap is
> not thread safe. But probably worth a try in general to somehow improve the
> "threadBindings.get" performance?
you must be indexing very very small documents with very low number of
unique terms that you see issues with this class. I can't imagine that
the indexing process is dominated by this class. you can just try to
use an impl that doesn't use the threadBinding map at all. that should
work just fine though.
one more thing, if replacing the test gives you 10% indexing perf what
the hack are yo indexing? Is that the test you send the other day to
the user list like 900k docs in 29 sec or something like that?
simon
>
>
> Regards,
> Peter.
>
> *
> http://trove.starlight-systems.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org