You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "jpountz (via GitHub)" <gi...@apache.org> on 2023/03/10 16:44:56 UTC
[GitHub] [lucene] jpountz opened a new pull request, #12199: Reduce contention in DocumentsWriterPerThreadPool.
jpountz opened a new pull request, #12199:
URL: https://github.com/apache/lucene/pull/12199
Obtaining a DWPT and putting it back into the pool is subject to contention. This change reduces contention by using 8 sub pools that are tried sequentially. When applied on top of #12198, this reduces the time to index geonames with 20 threads from ~19s to ~16-17s.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on a diff in pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on code in PR #12199:
URL: https://github.com/apache/lucene/pull/12199#discussion_r1133704928
##########
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThreadPool.java:
##########
@@ -113,15 +113,17 @@ private synchronized DocumentsWriterPerThread newWriter() {
* operation (add/updateDocument).
*/
DocumentsWriterPerThread getAndLock() {
- synchronized (this) {
- ensureOpen();
- DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
- if (dwpt == null) {
- dwpt = newWriter();
- }
- // DWPT is already locked before return by this method:
- return dwpt;
+ ensureOpen();
+ DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
+ if (dwpt == null) {
+ // newWriter() adds the DWPT to the `dwpts` set as a side-effect. However it is not added to
+ // `freeList` at this point, it will be added later on once DocumentsWriter has indexed a
+ // document into this DWPT and then gives it back to the pool by calling
+ // #marksAsFreeAndUnlock.
+ dwpt = newWriter();
Review Comment:
I would change this to `return newWriter();`
For better readability also invert the if statement like that:
```
DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
if (dwpt != null) {
return dwpt;
}
return newWriter();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz commented on pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #12199:
URL: https://github.com/apache/lucene/pull/12199#issuecomment-1473272224
I suspect this change to be the source of the speedup when indexing vectors on https://home.apache.org/~mikemccand/lucenebench/indexing.html, but maybe more because of the introduced affinity between indexing threads and DWPTs than because of reduced contention since contention generally shows up when indexing is fast, which isn't the case with vectors?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz commented on a diff in pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on code in PR #12199:
URL: https://github.com/apache/lucene/pull/12199#discussion_r1133629633
##########
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThreadPool.java:
##########
@@ -113,15 +113,19 @@ private synchronized DocumentsWriterPerThread newWriter() {
* operation (add/updateDocument).
*/
DocumentsWriterPerThread getAndLock() {
- synchronized (this) {
- ensureOpen();
- DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
- if (dwpt == null) {
- dwpt = newWriter();
+ ensureOpen();
+ DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
+ if (dwpt == null) {
Review Comment:
I still have the feeling that you are making incorrect assumptions about what this piece of code is doing (this method itself doesn't publish the object to other threads), but I took this thread as a call to keep things simple, so I removed the retry and added a few more comments about the logic of this pool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz commented on a diff in pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on code in PR #12199:
URL: https://github.com/apache/lucene/pull/12199#discussion_r1133614094
##########
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThreadPool.java:
##########
@@ -130,13 +134,15 @@ private void ensureOpen() {
}
}
+ private synchronized boolean contains(DocumentsWriterPerThread state) {
+ return dwpts.contains(state);
+ }
+
void marksAsFreeAndUnlock(DocumentsWriterPerThread state) {
final long ramBytesUsed = state.ramBytesUsed();
- synchronized (this) {
- assert dwpts.contains(state)
- : "we tried to add a DWPT back to the pool but the pool doesn't know aobut this DWPT";
- freeList.add(state, ramBytesUsed);
- }
+ assert contains(state)
+ : "we tried to add a DWPT back to the pool but the pool doesn't know aobut this DWPT";
Review Comment:
Thanks for catching Uwe, I fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] dweiss commented on a diff in pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on code in PR #12199:
URL: https://github.com/apache/lucene/pull/12199#discussion_r1132698882
##########
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThreadPool.java:
##########
@@ -113,15 +113,19 @@ private synchronized DocumentsWriterPerThread newWriter() {
* operation (add/updateDocument).
*/
DocumentsWriterPerThread getAndLock() {
- synchronized (this) {
- ensureOpen();
- DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
- if (dwpt == null) {
- dwpt = newWriter();
+ ensureOpen();
+ DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
+ if (dwpt == null) {
Review Comment:
It's not about the poll. It's about whether dwpt = newWriter(); can be expanded and reordered by the compiler so that dwpt gets assigned a value before the constructor (or whatever initialization inside newWriter) takes place. Any thread checking dwpt == null outside synchronized could, at least in theory, see such a "partially constructed" object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on a diff in pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on code in PR #12199:
URL: https://github.com/apache/lucene/pull/12199#discussion_r1133704928
##########
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThreadPool.java:
##########
@@ -113,15 +113,17 @@ private synchronized DocumentsWriterPerThread newWriter() {
* operation (add/updateDocument).
*/
DocumentsWriterPerThread getAndLock() {
- synchronized (this) {
- ensureOpen();
- DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
- if (dwpt == null) {
- dwpt = newWriter();
- }
- // DWPT is already locked before return by this method:
- return dwpt;
+ ensureOpen();
+ DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
+ if (dwpt == null) {
+ // newWriter() adds the DWPT to the `dwpts` set as a side-effect. However it is not added to
+ // `freeList` at this point, it will be added later on once DocumentsWriter has indexed a
+ // document into this DWPT and then gives it back to the pool by calling
+ // #marksAsFreeAndUnlock.
+ dwpt = newWriter();
Review Comment:
I would change this to `return newWriter();`
For better readability also invert the if statement like that:
```java
DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
if (dwpt != null) {
return dwpt;
}
return newWriter();
```
(of course add your comments!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on a diff in pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on code in PR #12199:
URL: https://github.com/apache/lucene/pull/12199#discussion_r1133699509
##########
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThreadPool.java:
##########
@@ -113,15 +113,19 @@ private synchronized DocumentsWriterPerThread newWriter() {
* operation (add/updateDocument).
*/
DocumentsWriterPerThread getAndLock() {
- synchronized (this) {
- ensureOpen();
- DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
- if (dwpt == null) {
- dwpt = newWriter();
+ ensureOpen();
+ DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
+ if (dwpt == null) {
Review Comment:
Thanks code looks much better. I was also irritated by the long chain of "tries" with and without synchronized. To me actualy code is easier to read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz merged pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz merged PR #12199:
URL: https://github.com/apache/lucene/pull/12199
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] dweiss commented on a diff in pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on code in PR #12199:
URL: https://github.com/apache/lucene/pull/12199#discussion_r1132628795
##########
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThreadPool.java:
##########
@@ -113,15 +113,19 @@ private synchronized DocumentsWriterPerThread newWriter() {
* operation (add/updateDocument).
*/
DocumentsWriterPerThread getAndLock() {
- synchronized (this) {
- ensureOpen();
- DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
- if (dwpt == null) {
- dwpt = newWriter();
+ ensureOpen();
+ DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
+ if (dwpt == null) {
Review Comment:
For some reason, double locking always makes me cringe. There are lengthy discussions about this idiom all around the web (visibility of partially constructed objects).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #12199:
URL: https://github.com/apache/lucene/pull/12199#issuecomment-1466877603
Thanks for the updates. Sorry for nitpicking, I just prefer code that goes sequentially and uses the pattern exit method as soon as condition mets. This makes it easier to understand.
Nice that there's no double locking anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz commented on a diff in pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on code in PR #12199:
URL: https://github.com/apache/lucene/pull/12199#discussion_r1132647604
##########
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThreadPool.java:
##########
@@ -113,15 +113,19 @@ private synchronized DocumentsWriterPerThread newWriter() {
* operation (add/updateDocument).
*/
DocumentsWriterPerThread getAndLock() {
- synchronized (this) {
- ensureOpen();
- DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
- if (dwpt == null) {
- dwpt = newWriter();
+ ensureOpen();
+ DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
+ if (dwpt == null) {
Review Comment:
I don't think that this is a case of double-checked locking. In general double-checked locking tries to reduce the overhead of acquiring a lock by adding a quick check before the lock. Here the logic is different, it would be legal to remove the call to `poll` under lock, I only added it because there is a chance that a thread had to wait on the lock, so we could check if another DWPT was added to the queue in the meantime in order to save creating a new DWPT. I don't think it's actually important, I could remove the second call to `poll` to make it look less like double-checked locking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] dweiss commented on a diff in pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on code in PR #12199:
URL: https://github.com/apache/lucene/pull/12199#discussion_r1132739002
##########
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThreadPool.java:
##########
@@ -113,15 +113,19 @@ private synchronized DocumentsWriterPerThread newWriter() {
* operation (add/updateDocument).
*/
DocumentsWriterPerThread getAndLock() {
- synchronized (this) {
- ensureOpen();
- DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
- if (dwpt == null) {
- dwpt = newWriter();
+ ensureOpen();
+ DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
+ if (dwpt == null) {
Review Comment:
No need to convince me, @jpountz - I just expressed my reluctance at it because, well, it requires convincing. :) Unless there's really a huge gain, I typically just go with what I understand works (making the field volatile, for example).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on a diff in pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on code in PR #12199:
URL: https://github.com/apache/lucene/pull/12199#discussion_r1133247211
##########
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThreadPool.java:
##########
@@ -130,13 +134,15 @@ private void ensureOpen() {
}
}
+ private synchronized boolean contains(DocumentsWriterPerThread state) {
+ return dwpts.contains(state);
+ }
+
void marksAsFreeAndUnlock(DocumentsWriterPerThread state) {
final long ramBytesUsed = state.ramBytesUsed();
- synchronized (this) {
- assert dwpts.contains(state)
- : "we tried to add a DWPT back to the pool but the pool doesn't know aobut this DWPT";
- freeList.add(state, ramBytesUsed);
- }
+ assert contains(state)
+ : "we tried to add a DWPT back to the pool but the pool doesn't know aobut this DWPT";
Review Comment:
typo :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] dweiss commented on a diff in pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on code in PR #12199:
URL: https://github.com/apache/lucene/pull/12199#discussion_r1132703519
##########
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThreadPool.java:
##########
@@ -113,15 +113,19 @@ private synchronized DocumentsWriterPerThread newWriter() {
* operation (add/updateDocument).
*/
DocumentsWriterPerThread getAndLock() {
- synchronized (this) {
- ensureOpen();
- DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
- if (dwpt == null) {
- dwpt = newWriter();
+ ensureOpen();
+ DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
+ if (dwpt == null) {
Review Comment:
Alex Shipilev had a nice writeup about it - found it here: https://shipilev.net/blog/2014/safe-public-construction/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] dweiss commented on a diff in pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on code in PR #12199:
URL: https://github.com/apache/lucene/pull/12199#discussion_r1133644248
##########
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThreadPool.java:
##########
@@ -113,15 +113,19 @@ private synchronized DocumentsWriterPerThread newWriter() {
* operation (add/updateDocument).
*/
DocumentsWriterPerThread getAndLock() {
- synchronized (this) {
- ensureOpen();
- DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
- if (dwpt == null) {
- dwpt = newWriter();
+ ensureOpen();
+ DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
+ if (dwpt == null) {
Review Comment:
I certainly am! But anyone looking at this code without jumping deep will have the same doubts, I think. Unless there is a convincing (performance) argument that it's worth it, I like the updated version a lot better - it's clear and simple.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on a diff in pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on code in PR #12199:
URL: https://github.com/apache/lucene/pull/12199#discussion_r1133708757
##########
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThreadPool.java:
##########
@@ -113,15 +113,17 @@ private synchronized DocumentsWriterPerThread newWriter() {
* operation (add/updateDocument).
*/
DocumentsWriterPerThread getAndLock() {
- synchronized (this) {
- ensureOpen();
- DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
- if (dwpt == null) {
- dwpt = newWriter();
- }
- // DWPT is already locked before return by this method:
- return dwpt;
+ ensureOpen();
+ DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
+ if (dwpt == null) {
+ // newWriter() adds the DWPT to the `dwpts` set as a side-effect. However it is not added to
+ // `freeList` at this point, it will be added later on once DocumentsWriter has indexed a
+ // document into this DWPT and then gives it back to the pool by calling
+ // #marksAsFreeAndUnlock.
+ dwpt = newWriter();
Review Comment:
Don't need to do this, but looks much more "Java 17 like" - just for demonstation purposes:
```java
return Optional.ofNullable(freeList.poll(DocumentsWriterPerThread::tryLock)).orElseGet(this::newWriter);
```
Of course it brings no comments anymore. Don't do it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on a diff in pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on code in PR #12199:
URL: https://github.com/apache/lucene/pull/12199#discussion_r1133798472
##########
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThreadPool.java:
##########
@@ -113,15 +113,17 @@ private synchronized DocumentsWriterPerThread newWriter() {
* operation (add/updateDocument).
*/
DocumentsWriterPerThread getAndLock() {
- synchronized (this) {
- ensureOpen();
- DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
- if (dwpt == null) {
- dwpt = newWriter();
- }
- // DWPT is already locked before return by this method:
- return dwpt;
+ ensureOpen();
+ DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
+ if (dwpt == null) {
+ // newWriter() adds the DWPT to the `dwpts` set as a side-effect. However it is not added to
+ // `freeList` at this point, it will be added later on once DocumentsWriter has indexed a
+ // document into this DWPT and then gives it back to the pool by calling
+ // #marksAsFreeAndUnlock.
+ dwpt = newWriter();
Review Comment:
It was a joke!
My intention was the first comment, to invert the if logic so use "return on success"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz commented on a diff in pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on code in PR #12199:
URL: https://github.com/apache/lucene/pull/12199#discussion_r1132726733
##########
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThreadPool.java:
##########
@@ -113,15 +113,19 @@ private synchronized DocumentsWriterPerThread newWriter() {
* operation (add/updateDocument).
*/
DocumentsWriterPerThread getAndLock() {
- synchronized (this) {
- ensureOpen();
- DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
- if (dwpt == null) {
- dwpt = newWriter();
+ ensureOpen();
+ DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
+ if (dwpt == null) {
Review Comment:
Thanks for sharing this blog post, I remember reading it in the past, it was a good re-read. I mentioned polls because I thought that they were what made you think that this code is a case of double-checked locking, as there is a first call before the lock and another one under the lock, like the null checks with double-checked locking with singletons. I need to go on weekend, I'll try to post a convincing explanation of why this is not a case of double-checked locking and why it is safe next week. By the way, thanks for looking!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on a diff in pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on code in PR #12199:
URL: https://github.com/apache/lucene/pull/12199#discussion_r1133708757
##########
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThreadPool.java:
##########
@@ -113,15 +113,17 @@ private synchronized DocumentsWriterPerThread newWriter() {
* operation (add/updateDocument).
*/
DocumentsWriterPerThread getAndLock() {
- synchronized (this) {
- ensureOpen();
- DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
- if (dwpt == null) {
- dwpt = newWriter();
- }
- // DWPT is already locked before return by this method:
- return dwpt;
+ ensureOpen();
+ DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
+ if (dwpt == null) {
+ // newWriter() adds the DWPT to the `dwpts` set as a side-effect. However it is not added to
+ // `freeList` at this point, it will be added later on once DocumentsWriter has indexed a
+ // document into this DWPT and then gives it back to the pool by calling
+ // #marksAsFreeAndUnlock.
+ dwpt = newWriter();
Review Comment:
Don't need to do this, but looks much more "Java 17 like" - just for demonstation purposes:
```java
return Optional.ofNullable(freeList.poll(DocumentsWriterPerThread::tryLock)).orElse(this::newWriter);
```
##########
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThreadPool.java:
##########
@@ -113,15 +113,17 @@ private synchronized DocumentsWriterPerThread newWriter() {
* operation (add/updateDocument).
*/
DocumentsWriterPerThread getAndLock() {
- synchronized (this) {
- ensureOpen();
- DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
- if (dwpt == null) {
- dwpt = newWriter();
- }
- // DWPT is already locked before return by this method:
- return dwpt;
+ ensureOpen();
+ DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
+ if (dwpt == null) {
+ // newWriter() adds the DWPT to the `dwpts` set as a side-effect. However it is not added to
+ // `freeList` at this point, it will be added later on once DocumentsWriter has indexed a
+ // document into this DWPT and then gives it back to the pool by calling
+ // #marksAsFreeAndUnlock.
+ dwpt = newWriter();
Review Comment:
I would change this to `return newWriter();`
For better readability also invert the if statement like that:
```java
DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
if (dwpt != null) {
return dwpt;
}
return newWriter();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on a diff in pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on code in PR #12199:
URL: https://github.com/apache/lucene/pull/12199#discussion_r1133708757
##########
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThreadPool.java:
##########
@@ -113,15 +113,17 @@ private synchronized DocumentsWriterPerThread newWriter() {
* operation (add/updateDocument).
*/
DocumentsWriterPerThread getAndLock() {
- synchronized (this) {
- ensureOpen();
- DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
- if (dwpt == null) {
- dwpt = newWriter();
- }
- // DWPT is already locked before return by this method:
- return dwpt;
+ ensureOpen();
+ DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
+ if (dwpt == null) {
+ // newWriter() adds the DWPT to the `dwpts` set as a side-effect. However it is not added to
+ // `freeList` at this point, it will be added later on once DocumentsWriter has indexed a
+ // document into this DWPT and then gives it back to the pool by calling
+ // #marksAsFreeAndUnlock.
+ dwpt = newWriter();
Review Comment:
Don't need to do this, but looks much more "Java 17 like" - just for demonstation purposes:
```java
return Optional.ofNullable(freeList.poll(DocumentsWriterPerThread::tryLock)).orElseGet(this::newWriter);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] dweiss commented on a diff in pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on code in PR #12199:
URL: https://github.com/apache/lucene/pull/12199#discussion_r1133753108
##########
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThreadPool.java:
##########
@@ -113,15 +113,17 @@ private synchronized DocumentsWriterPerThread newWriter() {
* operation (add/updateDocument).
*/
DocumentsWriterPerThread getAndLock() {
- synchronized (this) {
- ensureOpen();
- DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
- if (dwpt == null) {
- dwpt = newWriter();
- }
- // DWPT is already locked before return by this method:
- return dwpt;
+ ensureOpen();
+ DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
+ if (dwpt == null) {
+ // newWriter() adds the DWPT to the `dwpts` set as a side-effect. However it is not added to
+ // `freeList` at this point, it will be added later on once DocumentsWriter has indexed a
+ // document into this DWPT and then gives it back to the pool by calling
+ // #marksAsFreeAndUnlock.
+ dwpt = newWriter();
Review Comment:
Nice, Uwe. Although nobody can ever convince me those optional expressions are easier to read and understand than a plain if... :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] s1monw commented on a diff in pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "s1monw (via GitHub)" <gi...@apache.org>.
s1monw commented on code in PR #12199:
URL: https://github.com/apache/lucene/pull/12199#discussion_r1136706886
##########
lucene/core/src/java/org/apache/lucene/index/ConcurrentApproximatePriorityQueue.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.index;
+
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.function.Predicate;
+
+/**
+ * Concurrent version of {@link ApproximatePriorityQueue}, which trades a bit more of ordering for
+ * better concurrency by maintaining 8 sub {@link ApproximatePriorityQueue}s that are locked
+ * independently.
+ */
+final class ConcurrentApproximatePriorityQueue<T> {
+
+ /** Keeping 8 queues should already help a lot compared to a single one. */
+ static final int CONCURRENCY = 8;
+
+ private static final int MASK = 0x07;
+
+ final Lock[] locks;
+ final ApproximatePriorityQueue<T>[] queues;
+
+ ConcurrentApproximatePriorityQueue() {
+ locks = new Lock[CONCURRENCY];
+ @SuppressWarnings({"rawtypes", "unchecked"})
+ ApproximatePriorityQueue<T>[] queues = new ApproximatePriorityQueue[CONCURRENCY];
+ this.queues = queues;
+ for (int i = 0; i < CONCURRENCY; ++i) {
+ locks[i] = new ReentrantLock();
+ queues[i] = new ApproximatePriorityQueue<>();
+ }
+ }
+
+ void add(T entry, long weight) {
+ // Seed the order in which to look at entries based on the current thread. This helps distribute
+ // entries across queues and gives a bit of thread affinity between entries and threads, which
+ // can't hurt.
+ final int threadHash = Thread.currentThread().hashCode();
+ for (int i = 0; i < CONCURRENCY; ++i) {
+ final int index = (threadHash + i) & MASK;
+ final Lock lock = locks[index];
+ final ApproximatePriorityQueue<T> queue = queues[index];
+ if (lock.tryLock()) {
+ try {
+ queue.add(entry, weight);
+ return;
+ } finally {
+ lock.unlock();
+ }
+ }
+ }
+ final int index = threadHash & MASK;
+ final Lock lock = locks[index];
+ final ApproximatePriorityQueue<T> queue = queues[index];
+ lock.lock();
+ try {
+ queue.add(entry, weight);
+ } finally {
+ lock.unlock();
+ }
+ }
+
+ T poll(Predicate<T> predicate) {
+ final int threadHash = Thread.currentThread().hashCode();
+ for (int i = 0; i < CONCURRENCY; ++i) {
+ final int index = (threadHash + i) & MASK;
+ final Lock lock = locks[index];
+ final ApproximatePriorityQueue<T> queue = queues[index];
+ if (lock.tryLock()) {
+ try {
+ T entry = queue.poll(predicate);
+ if (entry != null) {
+ return entry;
+ }
+ } finally {
+ lock.unlock();
+ }
+ }
+ }
+ for (int i = 0; i < CONCURRENCY; ++i) {
+ final int index = (threadHash + i) & MASK;
+ final Lock lock = locks[index];
+ final ApproximatePriorityQueue<T> queue = queues[index];
+ lock.lock();
+ try {
+ T entry = queue.poll(predicate);
+ if (entry != null) {
+ return entry;
+ }
+ } finally {
+ lock.unlock();
+ }
+ }
+ return null;
+ }
+
+ // Only used for assertions
+ boolean contains(Object o) {
+ for (int i = 0; i < CONCURRENCY; ++i) {
Review Comment:
nitpick can you add a check that assertions are enabled?
##########
lucene/core/src/java/org/apache/lucene/index/ConcurrentApproximatePriorityQueue.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.index;
+
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.function.Predicate;
+
+/**
+ * Concurrent version of {@link ApproximatePriorityQueue}, which trades a bit more of ordering for
+ * better concurrency by maintaining 8 sub {@link ApproximatePriorityQueue}s that are locked
+ * independently.
+ */
+final class ConcurrentApproximatePriorityQueue<T> {
+
+ /** Keeping 8 queues should already help a lot compared to a single one. */
+ static final int CONCURRENCY = 8;
+
+ private static final int MASK = 0x07;
+
+ final Lock[] locks;
+ final ApproximatePriorityQueue<T>[] queues;
+
+ ConcurrentApproximatePriorityQueue() {
+ locks = new Lock[CONCURRENCY];
+ @SuppressWarnings({"rawtypes", "unchecked"})
+ ApproximatePriorityQueue<T>[] queues = new ApproximatePriorityQueue[CONCURRENCY];
+ this.queues = queues;
+ for (int i = 0; i < CONCURRENCY; ++i) {
+ locks[i] = new ReentrantLock();
+ queues[i] = new ApproximatePriorityQueue<>();
+ }
+ }
+
+ void add(T entry, long weight) {
+ // Seed the order in which to look at entries based on the current thread. This helps distribute
+ // entries across queues and gives a bit of thread affinity between entries and threads, which
+ // can't hurt.
+ final int threadHash = Thread.currentThread().hashCode();
+ for (int i = 0; i < CONCURRENCY; ++i) {
+ final int index = (threadHash + i) & MASK;
+ final Lock lock = locks[index];
+ final ApproximatePriorityQueue<T> queue = queues[index];
+ if (lock.tryLock()) {
+ try {
+ queue.add(entry, weight);
+ return;
+ } finally {
+ lock.unlock();
+ }
+ }
+ }
+ final int index = threadHash & MASK;
+ final Lock lock = locks[index];
+ final ApproximatePriorityQueue<T> queue = queues[index];
+ lock.lock();
+ try {
+ queue.add(entry, weight);
+ } finally {
+ lock.unlock();
+ }
+ }
+
+ T poll(Predicate<T> predicate) {
+ final int threadHash = Thread.currentThread().hashCode();
+ for (int i = 0; i < CONCURRENCY; ++i) {
+ final int index = (threadHash + i) & MASK;
+ final Lock lock = locks[index];
+ final ApproximatePriorityQueue<T> queue = queues[index];
+ if (lock.tryLock()) {
+ try {
+ T entry = queue.poll(predicate);
+ if (entry != null) {
+ return entry;
+ }
+ } finally {
+ lock.unlock();
+ }
+ }
+ }
+ for (int i = 0; i < CONCURRENCY; ++i) {
+ final int index = (threadHash + i) & MASK;
+ final Lock lock = locks[index];
+ final ApproximatePriorityQueue<T> queue = queues[index];
+ lock.lock();
+ try {
+ T entry = queue.poll(predicate);
+ if (entry != null) {
+ return entry;
+ }
+ } finally {
+ lock.unlock();
+ }
+ }
+ return null;
+ }
+
+ // Only used for assertions
+ boolean contains(Object o) {
+ for (int i = 0; i < CONCURRENCY; ++i) {
+ locks[i].lock();
Review Comment:
this is really a nit pick but for methods that use a lock I'd prefer to assign the lock to a local var instead of dereferencing it again in the finally block.
##########
lucene/core/src/java/org/apache/lucene/index/ConcurrentApproximatePriorityQueue.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.index;
+
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.function.Predicate;
+
+/**
+ * Concurrent version of {@link ApproximatePriorityQueue}, which trades a bit more of ordering for
+ * better concurrency by maintaining 8 sub {@link ApproximatePriorityQueue}s that are locked
+ * independently.
+ */
+final class ConcurrentApproximatePriorityQueue<T> {
+
+ /** Keeping 8 queues should already help a lot compared to a single one. */
+ static final int CONCURRENCY = 8;
+
+ private static final int MASK = 0x07;
+
+ final Lock[] locks;
+ final ApproximatePriorityQueue<T>[] queues;
+
+ ConcurrentApproximatePriorityQueue() {
+ locks = new Lock[CONCURRENCY];
+ @SuppressWarnings({"rawtypes", "unchecked"})
+ ApproximatePriorityQueue<T>[] queues = new ApproximatePriorityQueue[CONCURRENCY];
+ this.queues = queues;
+ for (int i = 0; i < CONCURRENCY; ++i) {
+ locks[i] = new ReentrantLock();
+ queues[i] = new ApproximatePriorityQueue<>();
+ }
+ }
+
+ void add(T entry, long weight) {
+ // Seed the order in which to look at entries based on the current thread. This helps distribute
+ // entries across queues and gives a bit of thread affinity between entries and threads, which
+ // can't hurt.
+ final int threadHash = Thread.currentThread().hashCode();
+ for (int i = 0; i < CONCURRENCY; ++i) {
+ final int index = (threadHash + i) & MASK;
+ final Lock lock = locks[index];
+ final ApproximatePriorityQueue<T> queue = queues[index];
+ if (lock.tryLock()) {
+ try {
+ queue.add(entry, weight);
+ return;
+ } finally {
+ lock.unlock();
+ }
+ }
+ }
+ final int index = threadHash & MASK;
+ final Lock lock = locks[index];
+ final ApproximatePriorityQueue<T> queue = queues[index];
+ lock.lock();
+ try {
+ queue.add(entry, weight);
+ } finally {
+ lock.unlock();
+ }
+ }
+
+ T poll(Predicate<T> predicate) {
+ final int threadHash = Thread.currentThread().hashCode();
+ for (int i = 0; i < CONCURRENCY; ++i) {
+ final int index = (threadHash + i) & MASK;
+ final Lock lock = locks[index];
+ final ApproximatePriorityQueue<T> queue = queues[index];
+ if (lock.tryLock()) {
+ try {
+ T entry = queue.poll(predicate);
+ if (entry != null) {
+ return entry;
+ }
+ } finally {
+ lock.unlock();
+ }
+ }
+ }
+ for (int i = 0; i < CONCURRENCY; ++i) {
+ final int index = (threadHash + i) & MASK;
+ final Lock lock = locks[index];
+ final ApproximatePriorityQueue<T> queue = queues[index];
+ lock.lock();
+ try {
+ T entry = queue.poll(predicate);
+ if (entry != null) {
+ return entry;
+ }
+ } finally {
+ lock.unlock();
+ }
+ }
+ return null;
+ }
+
+ // Only used for assertions
+ boolean contains(Object o) {
+ for (int i = 0; i < CONCURRENCY; ++i) {
+ locks[i].lock();
+ try {
+ if (queues[i].contains(o)) {
+ return true;
+ }
+ } finally {
+ locks[i].unlock();
+ }
+ }
+ return false;
+ }
+
+ boolean remove(Object o) {
+ for (int i = 0; i < CONCURRENCY; ++i) {
+ locks[i].lock();
Review Comment:
same here, maybe use a local var for the lock. It really looks cleaner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] vsop-479 commented on pull request #12199: Reduce contention in DocumentsWriterPerThreadPool.
Posted by "vsop-479 (via GitHub)" <gi...@apache.org>.
vsop-479 commented on PR #12199:
URL: https://github.com/apache/lucene/pull/12199#issuecomment-1508380027
> DocumentsWriterPerThread getAndLock() {
ensureOpen();
DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock);
if (dwpt != null) {
return dwpt;
}
// newWriter() adds the DWPT to the dwpts set as a side-effect. However it is not added to
// freeList at this point, it will be added later on once DocumentsWriter has indexed a
// document into this DWPT and then gives it back to the pool by calling
// #marksAsFreeAndUnlock.
return newWriter();
}
@jpountz Does this change makes more small segments generated than the synchronized version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org