You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2020/03/05 09:44:57 UTC
[GitHub] [lucene-solr] s1monw opened a new pull request #1319: LUCENE-9164:
process all events before closing gracefully
s1monw opened a new pull request #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319
This is yet another / simpler approach to https://github.com/apache/lucene-solr/pull/1274 to ensure that all event are processed if we are closing the IW gracefully. This also improves the case where we closing due to a tragic event where we don't try to be heroic and just drop all pending events on the floor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on a change in pull request
#1319: LUCENE-9164: process all events before closing gracefully
Posted by GitBox <gi...@apache.org>.
mikemccand commented on a change in pull request #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319#discussion_r389313897
##########
File path: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
##########
@@ -299,7 +300,77 @@ static int getActualMaxDocs() {
final FieldNumbers globalFieldNumberMap;
final DocumentsWriter docWriter;
- private final Queue<Event> eventQueue = new ConcurrentLinkedQueue<>();
+ private final EventQueue eventQueue = new EventQueue(this);
+
+ static final class EventQueue implements Closeable {
+ private volatile boolean closed = false;
Review comment:
No need to i it with `false` ... this is already the default for Java.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] s1monw commented on issue #1319: LUCENE-9164: process
all events before closing gracefully
Posted by GitBox <gi...@apache.org>.
s1monw commented on issue #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319#issuecomment-595136991
thanks for looking @dweiss
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] s1monw commented on a change in pull request #1319:
LUCENE-9164: process all events before closing gracefully
Posted by GitBox <gi...@apache.org>.
s1monw commented on a change in pull request #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319#discussion_r389397721
##########
File path: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
##########
@@ -299,7 +300,77 @@ static int getActualMaxDocs() {
final FieldNumbers globalFieldNumberMap;
final DocumentsWriter docWriter;
- private final Queue<Event> eventQueue = new ConcurrentLinkedQueue<>();
+ private final EventQueue eventQueue = new EventQueue(this);
+
+ static final class EventQueue implements Closeable {
+ private volatile boolean closed = false;
+ private final Semaphore permits = new Semaphore(Integer.MAX_VALUE);
+ private final Queue<Event> queue = new ConcurrentLinkedQueue<>();
+ private final IndexWriter writer;
+
+ EventQueue(IndexWriter writer) {
+ this.writer = writer;
+ }
+
+ private void tryAcquire() {
+ if (permits.tryAcquire() == false) {
+ throw new AlreadyClosedException("queue is closed");
+ }
+ if (closed) {
+ throw new AlreadyClosedException("queue is closed");
Review comment:
I added a test to ensure this works
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] s1monw merged pull request #1319: LUCENE-9164:
process all events before closing gracefully
Posted by GitBox <gi...@apache.org>.
s1monw merged pull request #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dnhatn commented on a change in pull request #1319:
LUCENE-9164: process all events before closing gracefully
Posted by GitBox <gi...@apache.org>.
dnhatn commented on a change in pull request #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319#discussion_r389028289
##########
File path: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
##########
@@ -299,7 +300,76 @@ static int getActualMaxDocs() {
final FieldNumbers globalFieldNumberMap;
final DocumentsWriter docWriter;
- private final Queue<Event> eventQueue = new ConcurrentLinkedQueue<>();
+ private final CloseableQueue eventQueue = new CloseableQueue(this);
+
+ static final class CloseableQueue implements Closeable {
+ private volatile boolean closed = false;
+ private final Semaphore permits = new Semaphore(Integer.MAX_VALUE);
+ private final Queue<Event> queue = new ConcurrentLinkedQueue<>();
+ private final IndexWriter writer;
+
+ CloseableQueue(IndexWriter writer) {
+ this.writer = writer;
+ }
+
+ private void tryAcquire() {
+ if (permits.tryAcquire() == false) {
+ throw new AlreadyClosedException("queue is closed");
+ }
+ if (closed) {
+ throw new AlreadyClosedException("queue is closed");
+ }
+ }
+
+ boolean add(Event event) {
+ tryAcquire();
+ try {
+ return queue.add(event);
+ } finally {
+ permits.release();
+ }
+ }
+
+ void processEvents() throws IOException {
+ tryAcquire();
+ try {
+ processEventsInternal();
+ }finally {
+ permits.release();
+ }
+ }
+ private void processEventsInternal() throws IOException {
Review comment:
nit: add a new line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on a change in pull request
#1319: LUCENE-9164: process all events before closing gracefully
Posted by GitBox <gi...@apache.org>.
mikemccand commented on a change in pull request #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319#discussion_r389387448
##########
File path: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
##########
@@ -299,7 +300,77 @@ static int getActualMaxDocs() {
final FieldNumbers globalFieldNumberMap;
final DocumentsWriter docWriter;
- private final Queue<Event> eventQueue = new ConcurrentLinkedQueue<>();
+ private final EventQueue eventQueue = new EventQueue(this);
+
+ static final class EventQueue implements Closeable {
+ private volatile boolean closed = false;
+ private final Semaphore permits = new Semaphore(Integer.MAX_VALUE);
+ private final Queue<Event> queue = new ConcurrentLinkedQueue<>();
+ private final IndexWriter writer;
+
+ EventQueue(IndexWriter writer) {
+ this.writer = writer;
+ }
+
+ private void tryAcquire() {
+ if (permits.tryAcquire() == false) {
+ throw new AlreadyClosedException("queue is closed");
+ }
+ if (closed) {
+ throw new AlreadyClosedException("queue is closed");
+ }
+ }
+
+ boolean add(Event event) {
+ tryAcquire();
+ try {
+ return queue.add(event);
+ } finally {
+ permits.release();
+ }
+ }
+
+ void processEvents() throws IOException {
+ tryAcquire();
+ try {
+ processEventsInternal();
+ } finally {
+ permits.release();
+ }
+ }
+
+ private void processEventsInternal() throws IOException {
+ assert Integer.MAX_VALUE - permits.availablePermits() > 0 : "must acquire a permit before processing events";
Review comment:
Maybe rewrite to `assert permits.availablePermits() < Integer.MAX_VALUE` for easier readability?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] s1monw commented on a change in pull request #1319:
LUCENE-9164: process all events before closing gracefully
Posted by GitBox <gi...@apache.org>.
s1monw commented on a change in pull request #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319#discussion_r389397123
##########
File path: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
##########
@@ -299,7 +300,77 @@ static int getActualMaxDocs() {
final FieldNumbers globalFieldNumberMap;
final DocumentsWriter docWriter;
- private final Queue<Event> eventQueue = new ConcurrentLinkedQueue<>();
+ private final EventQueue eventQueue = new EventQueue(this);
+
+ static final class EventQueue implements Closeable {
+ private volatile boolean closed = false;
+ private final Semaphore permits = new Semaphore(Integer.MAX_VALUE);
+ private final Queue<Event> queue = new ConcurrentLinkedQueue<>();
+ private final IndexWriter writer;
+
+ EventQueue(IndexWriter writer) {
+ this.writer = writer;
+ }
+
+ private void tryAcquire() {
+ if (permits.tryAcquire() == false) {
+ throw new AlreadyClosedException("queue is closed");
+ }
+ if (closed) {
+ throw new AlreadyClosedException("queue is closed");
+ }
+ }
+
+ boolean add(Event event) {
+ tryAcquire();
+ try {
+ return queue.add(event);
+ } finally {
+ permits.release();
+ }
+ }
+
+ void processEvents() throws IOException {
+ tryAcquire();
+ try {
+ processEventsInternal();
+ } finally {
+ permits.release();
+ }
+ }
+
+ private void processEventsInternal() throws IOException {
+ assert Integer.MAX_VALUE - permits.availablePermits() > 0 : "must acquire a permit before processing events";
Review comment:
but this wouldn't catch the case where we didn't acquire any permit?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] s1monw commented on issue #1319: LUCENE-9164: process
all events before closing gracefully
Posted by GitBox <gi...@apache.org>.
s1monw commented on issue #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319#issuecomment-596929510
@mikemccand are you ok with pushing this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dnhatn commented on issue #1319: LUCENE-9164: process
all events before closing gracefully
Posted by GitBox <gi...@apache.org>.
dnhatn commented on issue #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319#issuecomment-595598568
Thanks, Simon. I will take a look at this tomorrow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dnhatn commented on a change in pull request #1319:
LUCENE-9164: process all events before closing gracefully
Posted by GitBox <gi...@apache.org>.
dnhatn commented on a change in pull request #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319#discussion_r389394156
##########
File path: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
##########
@@ -299,7 +300,77 @@ static int getActualMaxDocs() {
final FieldNumbers globalFieldNumberMap;
final DocumentsWriter docWriter;
- private final Queue<Event> eventQueue = new ConcurrentLinkedQueue<>();
+ private final EventQueue eventQueue = new EventQueue(this);
+
+ static final class EventQueue implements Closeable {
+ private volatile boolean closed = false;
+ private final Semaphore permits = new Semaphore(Integer.MAX_VALUE);
+ private final Queue<Event> queue = new ConcurrentLinkedQueue<>();
+ private final IndexWriter writer;
+
+ EventQueue(IndexWriter writer) {
+ this.writer = writer;
+ }
+
+ private void tryAcquire() {
+ if (permits.tryAcquire() == false) {
+ throw new AlreadyClosedException("queue is closed");
+ }
+ if (closed) {
+ throw new AlreadyClosedException("queue is closed");
Review comment:
Great catch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] s1monw commented on a change in pull request #1319:
LUCENE-9164: process all events before closing gracefully
Posted by GitBox <gi...@apache.org>.
s1monw commented on a change in pull request #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319#discussion_r389397690
##########
File path: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
##########
@@ -299,7 +300,77 @@ static int getActualMaxDocs() {
final FieldNumbers globalFieldNumberMap;
final DocumentsWriter docWriter;
- private final Queue<Event> eventQueue = new ConcurrentLinkedQueue<>();
+ private final EventQueue eventQueue = new EventQueue(this);
+
+ static final class EventQueue implements Closeable {
+ private volatile boolean closed = false;
+ private final Semaphore permits = new Semaphore(Integer.MAX_VALUE);
+ private final Queue<Event> queue = new ConcurrentLinkedQueue<>();
+ private final IndexWriter writer;
+
+ EventQueue(IndexWriter writer) {
+ this.writer = writer;
+ }
+
+ private void tryAcquire() {
+ if (permits.tryAcquire() == false) {
+ throw new AlreadyClosedException("queue is closed");
+ }
+ if (closed) {
+ throw new AlreadyClosedException("queue is closed");
+ }
+ }
+
+ boolean add(Event event) {
+ tryAcquire();
+ try {
+ return queue.add(event);
+ } finally {
+ permits.release();
+ }
+ }
+
+ void processEvents() throws IOException {
+ tryAcquire();
+ try {
+ processEventsInternal();
+ } finally {
+ permits.release();
+ }
+ }
+
+ private void processEventsInternal() throws IOException {
+ assert Integer.MAX_VALUE - permits.availablePermits() > 0 : "must acquire a permit before processing events";
+ Event event;
+ while ((event = queue.poll()) != null) {
+ event.process(writer);
+ }
+ }
+
+ @Override
+ public synchronized void close() throws IOException {
Review comment:
correct
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on a change in pull request
#1319: LUCENE-9164: process all events before closing gracefully
Posted by GitBox <gi...@apache.org>.
mikemccand commented on a change in pull request #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319#discussion_r389313930
##########
File path: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
##########
@@ -299,7 +300,77 @@ static int getActualMaxDocs() {
final FieldNumbers globalFieldNumberMap;
final DocumentsWriter docWriter;
- private final Queue<Event> eventQueue = new ConcurrentLinkedQueue<>();
+ private final EventQueue eventQueue = new EventQueue(this);
+
+ static final class EventQueue implements Closeable {
+ private volatile boolean closed = false;
+ private final Semaphore permits = new Semaphore(Integer.MAX_VALUE);
+ private final Queue<Event> queue = new ConcurrentLinkedQueue<>();
+ private final IndexWriter writer;
+
+ EventQueue(IndexWriter writer) {
+ this.writer = writer;
+ }
+
+ private void tryAcquire() {
+ if (permits.tryAcquire() == false) {
+ throw new AlreadyClosedException("queue is closed");
+ }
+ if (closed) {
+ throw new AlreadyClosedException("queue is closed");
Review comment:
Hmm should we `permits.release` before throwing `AlreadyClosedException`? Otherwise couldn't we maybe deadlock another thread doing `close` when it does `permits.acquire(Integer.MAX_VALUE)`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dnhatn commented on a change in pull request #1319:
LUCENE-9164: process all events before closing gracefully
Posted by GitBox <gi...@apache.org>.
dnhatn commented on a change in pull request #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319#discussion_r389028879
##########
File path: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
##########
@@ -299,7 +300,76 @@ static int getActualMaxDocs() {
final FieldNumbers globalFieldNumberMap;
final DocumentsWriter docWriter;
- private final Queue<Event> eventQueue = new ConcurrentLinkedQueue<>();
+ private final CloseableQueue eventQueue = new CloseableQueue(this);
+
+ static final class CloseableQueue implements Closeable {
+ private volatile boolean closed = false;
+ private final Semaphore permits = new Semaphore(Integer.MAX_VALUE);
+ private final Queue<Event> queue = new ConcurrentLinkedQueue<>();
+ private final IndexWriter writer;
+
+ CloseableQueue(IndexWriter writer) {
+ this.writer = writer;
+ }
+
+ private void tryAcquire() {
+ if (permits.tryAcquire() == false) {
+ throw new AlreadyClosedException("queue is closed");
+ }
+ if (closed) {
+ throw new AlreadyClosedException("queue is closed");
+ }
+ }
+
+ boolean add(Event event) {
+ tryAcquire();
+ try {
+ return queue.add(event);
+ } finally {
+ permits.release();
+ }
+ }
+
+ void processEvents() throws IOException {
+ tryAcquire();
+ try {
+ processEventsInternal();
+ }finally {
Review comment:
nit: space after `{`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on a change in pull request
#1319: LUCENE-9164: process all events before closing gracefully
Posted by GitBox <gi...@apache.org>.
mikemccand commented on a change in pull request #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319#discussion_r389387731
##########
File path: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
##########
@@ -299,7 +300,77 @@ static int getActualMaxDocs() {
final FieldNumbers globalFieldNumberMap;
final DocumentsWriter docWriter;
- private final Queue<Event> eventQueue = new ConcurrentLinkedQueue<>();
+ private final EventQueue eventQueue = new EventQueue(this);
+
+ static final class EventQueue implements Closeable {
+ private volatile boolean closed = false;
+ private final Semaphore permits = new Semaphore(Integer.MAX_VALUE);
+ private final Queue<Event> queue = new ConcurrentLinkedQueue<>();
+ private final IndexWriter writer;
+
+ EventQueue(IndexWriter writer) {
+ this.writer = writer;
+ }
+
+ private void tryAcquire() {
+ if (permits.tryAcquire() == false) {
+ throw new AlreadyClosedException("queue is closed");
+ }
+ if (closed) {
+ throw new AlreadyClosedException("queue is closed");
+ }
+ }
+
+ boolean add(Event event) {
+ tryAcquire();
+ try {
+ return queue.add(event);
+ } finally {
+ permits.release();
+ }
+ }
+
+ void processEvents() throws IOException {
+ tryAcquire();
+ try {
+ processEventsInternal();
+ } finally {
+ permits.release();
+ }
+ }
+
+ private void processEventsInternal() throws IOException {
+ assert Integer.MAX_VALUE - permits.availablePermits() > 0 : "must acquire a permit before processing events";
+ Event event;
+ while ((event = queue.poll()) != null) {
+ event.process(writer);
+ }
+ }
+
+ @Override
+ public synchronized void close() throws IOException {
Review comment:
Why `synchronized` here (no other method is `synchronized`)? Just to prevent two threads from calling `close` at once?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dnhatn commented on a change in pull request #1319:
LUCENE-9164: process all events before closing gracefully
Posted by GitBox <gi...@apache.org>.
dnhatn commented on a change in pull request #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319#discussion_r389028473
##########
File path: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
##########
@@ -299,7 +300,76 @@ static int getActualMaxDocs() {
final FieldNumbers globalFieldNumberMap;
final DocumentsWriter docWriter;
- private final Queue<Event> eventQueue = new ConcurrentLinkedQueue<>();
+ private final CloseableQueue eventQueue = new CloseableQueue(this);
+
+ static final class CloseableQueue implements Closeable {
Review comment:
I am not sure if `EventQueue` is a better name?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1319:
LUCENE-9164: process all events before closing gracefully
Posted by GitBox <gi...@apache.org>.
dweiss commented on a change in pull request #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319#discussion_r388181480
##########
File path: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
##########
@@ -299,7 +300,70 @@ static int getActualMaxDocs() {
final FieldNumbers globalFieldNumberMap;
final DocumentsWriter docWriter;
- private final Queue<Event> eventQueue = new ConcurrentLinkedQueue<>();
+ private final CloseableQueue eventQueue = new CloseableQueue();
Review comment:
Wouldn't it be nicer to make it just Closeable and pass IndexWriter in the constructor (instead of each method)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dnhatn commented on a change in pull request #1319:
LUCENE-9164: process all events before closing gracefully
Posted by GitBox <gi...@apache.org>.
dnhatn commented on a change in pull request #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319#discussion_r389029514
##########
File path: lucene/core/src/test/org/apache/lucene/index/TestIndexWriter.java
##########
@@ -3773,7 +3774,58 @@ public void testRefreshAndRollbackConcurrently() throws Exception {
stopped.set(true);
indexer.join();
refresher.join();
+ if (w.getTragicException() != null) {
+ w.getTragicException().printStackTrace();
Review comment:
I think we don't need to print the stack trace here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on a change in pull request
#1319: LUCENE-9164: process all events before closing gracefully
Posted by GitBox <gi...@apache.org>.
mikemccand commented on a change in pull request #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319#discussion_r389388106
##########
File path: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
##########
@@ -299,7 +300,77 @@ static int getActualMaxDocs() {
final FieldNumbers globalFieldNumberMap;
final DocumentsWriter docWriter;
- private final Queue<Event> eventQueue = new ConcurrentLinkedQueue<>();
+ private final EventQueue eventQueue = new EventQueue(this);
+
+ static final class EventQueue implements Closeable {
+ private volatile boolean closed = false;
+ private final Semaphore permits = new Semaphore(Integer.MAX_VALUE);
+ private final Queue<Event> queue = new ConcurrentLinkedQueue<>();
+ private final IndexWriter writer;
+
+ EventQueue(IndexWriter writer) {
+ this.writer = writer;
+ }
+
+ private void tryAcquire() {
Review comment:
Can we rename this to `acquire`? If it fails, it throws an exception, so upon returning, it always succeeds?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] s1monw commented on issue #1319: LUCENE-9164: process
all events before closing gracefully
Posted by GitBox <gi...@apache.org>.
s1monw commented on issue #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319#issuecomment-596242053
> I wonder/wish if we could simply use synchronized methods instead of a Semaphore with Integer.MAX_VALUE permits, but I guess we want to allow multiple threads to process events concurrently, and to allow adding new events while events are being processed, etc.
this is indeed the reason
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on issue #1319: LUCENE-9164:
process all events before closing gracefully
Posted by GitBox <gi...@apache.org>.
mikemccand commented on issue #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319#issuecomment-595367820
I'll try to review this one soon!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dnhatn commented on a change in pull request #1319:
LUCENE-9164: process all events before closing gracefully
Posted by GitBox <gi...@apache.org>.
dnhatn commented on a change in pull request #1319: LUCENE-9164: process all events before closing gracefully
URL: https://github.com/apache/lucene-solr/pull/1319#discussion_r389028879
##########
File path: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
##########
@@ -299,7 +300,76 @@ static int getActualMaxDocs() {
final FieldNumbers globalFieldNumberMap;
final DocumentsWriter docWriter;
- private final Queue<Event> eventQueue = new ConcurrentLinkedQueue<>();
+ private final CloseableQueue eventQueue = new CloseableQueue(this);
+
+ static final class CloseableQueue implements Closeable {
+ private volatile boolean closed = false;
+ private final Semaphore permits = new Semaphore(Integer.MAX_VALUE);
+ private final Queue<Event> queue = new ConcurrentLinkedQueue<>();
+ private final IndexWriter writer;
+
+ CloseableQueue(IndexWriter writer) {
+ this.writer = writer;
+ }
+
+ private void tryAcquire() {
+ if (permits.tryAcquire() == false) {
+ throw new AlreadyClosedException("queue is closed");
+ }
+ if (closed) {
+ throw new AlreadyClosedException("queue is closed");
+ }
+ }
+
+ boolean add(Event event) {
+ tryAcquire();
+ try {
+ return queue.add(event);
+ } finally {
+ permits.release();
+ }
+ }
+
+ void processEvents() throws IOException {
+ tryAcquire();
+ try {
+ processEventsInternal();
+ }finally {
Review comment:
nit: space after `}`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org