You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/08/26 01:27:49 UTC
[GitHub] [pulsar] Shoothzj opened a new issue #11783: ConcurrentModificationException occurs when failPendingMessages
Shoothzj opened a new issue #11783:
URL: https://github.com/apache/pulsar/issues/11783
### Describe the bug
### To Reproduce
It's hard to reproduce.
### Expected behavior
Keep the list operation safely.
Additional context
It occurs when restarting one of brokers(there were 2 running brokers), first client recevied `DisConnected` event, and then ConcurrentModificationException and some recycled already exception in timer task's OpSendMsg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #11783: ConcurrentModificationException occurs when failPendingMessages
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #11783:
URL: https://github.com/apache/pulsar/issues/11783#issuecomment-914043008
> Is there any progress here? Why not just set the queue to concurrent?
there's progress in #11884, it's currently in draft.
@315157973 I replied in https://github.com/apache/pulsar/pull/11884#issuecomment-911169103 . Does that answer your question?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on issue #11783: ConcurrentModificationException occurs when failPendingMessages
Posted by GitBox <gi...@apache.org>.
315157973 commented on issue #11783:
URL: https://github.com/apache/pulsar/issues/11783#issuecomment-909820390
In my opinion, we can only use interfaces to restrict the use of methods, verbal conventions are not a good way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on issue #11783: ConcurrentModificationException occurs when failPendingMessages
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #11783:
URL: https://github.com/apache/pulsar/issues/11783#issuecomment-909939183
Cannot prevent the user to call `sendAsync` that way.
We have to deal with this case.
Now that you have a reproducer it will be easier to create a test case that reproduces the issue.
good catch !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on issue #11783: ConcurrentModificationException occurs when failPendingMessages
Posted by GitBox <gi...@apache.org>.
315157973 commented on issue #11783:
URL: https://github.com/apache/pulsar/issues/11783#issuecomment-909820390
In my opinion, we can only use interfaces to restrict the use of methods, verbal conventions are not a good way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #11783: ConcurrentModificationException occurs when failPendingMessages
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #11783:
URL: https://github.com/apache/pulsar/issues/11783#issuecomment-909947108
Good catch @Shoothzj !
> I have a question. Should we allow to retry on exceptionally callback or prohibit usage like that ?
I think that this is a bug that needs to be fixed. I took a quick look and I have a relatively simple way to solve it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on issue #11783: ConcurrentModificationException occurs when failPendingMessages
Posted by GitBox <gi...@apache.org>.
315157973 commented on issue #11783:
URL: https://github.com/apache/pulsar/issues/11783#issuecomment-914020824
Is there any progress here? Why not just set the queue to concurrent?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Shoothzj commented on issue #11783: ConcurrentModificationException occurs when failPendingMessages
Posted by GitBox <gi...@apache.org>.
Shoothzj commented on issue #11783:
URL: https://github.com/apache/pulsar/issues/11783#issuecomment-909347986
I just figure out why, because the client code retry to send messages when exception occurred to avoid network "Jitter"(Translate on google). the sample code are like this
```java
String topic = "test";
PulsarClient pulsarClient = PulsarClient.builder()
.serviceUrl("http://127.0.0.1:8080")
.build();
ProducerBuilder<String> producerBuilder = pulsarClient.newProducer(Schema.STRING).enableBatching(true);
Producer<String> producer = producerBuilder.topic(topic).create();
final TypedMessageBuilder<String> stringTypedMessageBuilder = producer.newMessage();
final TypedMessageBuilder<String> value = stringTypedMessageBuilder.key("1").value("2");
final CompletableFuture<MessageId> completableFuture = value.sendAsync();
completableFuture.exceptionally(throwable -> {
producer.sendAsync("xxxx");
return null;
});
```
The `exceptionally` is called from `ProducerImpl.failPendingMessages`'s forEachLoop, and `sendAsync` will add messages to `pendingMessages`, then the error ocurred.
I have a question. Should we allow to retry on `exceptionally` callback or prohibit usage like that ?
@315157973 @eolivelli @merlimat , thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on issue #11783: ConcurrentModificationException occurs when failPendingMessages
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #11783:
URL: https://github.com/apache/pulsar/issues/11783#issuecomment-909939183
Cannot prevent the user to call `sendAsync` that way.
We have to deal with this case.
Now that you have a reproducer it will be easier to create a test case that reproduces the issue.
good catch !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Shoothzj commented on issue #11783: ConcurrentModificationException occurs when failPendingMessages
Posted by GitBox <gi...@apache.org>.
Shoothzj commented on issue #11783:
URL: https://github.com/apache/pulsar/issues/11783#issuecomment-909347986
I just figure out why, because the client code retry to send messages when exception occurred to avoid network "Jitter"(Translate on google). the sample code are like this
```java
String topic = "test";
PulsarClient pulsarClient = PulsarClient.builder()
.serviceUrl("http://127.0.0.1:8080")
.build();
ProducerBuilder<String> producerBuilder = pulsarClient.newProducer(Schema.STRING).enableBatching(true);
Producer<String> producer = producerBuilder.topic(topic).create();
final TypedMessageBuilder<String> stringTypedMessageBuilder = producer.newMessage();
final TypedMessageBuilder<String> value = stringTypedMessageBuilder.key("1").value("2");
final CompletableFuture<MessageId> completableFuture = value.sendAsync();
completableFuture.exceptionally(throwable -> {
producer.sendAsync("xxxx");
return null;
});
```
The `exceptionally` is called from `ProducerImpl.failPendingMessages`'s forEachLoop, and `sendAsync` will add messages to `pendingMessages`, then the error ocurred.
I have a question. Should we allow to retry on `exceptionally` callback or prohibit usage like that ?
@315157973 @eolivelli @merlimat , thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #11783: ConcurrentModificationException occurs when failPendingMessages
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #11783:
URL: https://github.com/apache/pulsar/issues/11783#issuecomment-910468088
I created #11884 as a draft PR to fix the issue. I don't have a way to reproduce the problem in a test case and that's one of the problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli closed issue #11783: ConcurrentModificationException occurs when failPendingMessages
Posted by GitBox <gi...@apache.org>.
eolivelli closed issue #11783:
URL: https://github.com/apache/pulsar/issues/11783
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #11783: ConcurrentModificationException occurs when failPendingMessages
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #11783:
URL: https://github.com/apache/pulsar/issues/11783#issuecomment-909947108
Good catch @Shoothzj !
> I have a question. Should we allow to retry on exceptionally callback or prohibit usage like that ?
I think that this is a bug that needs to be fixed. I took a quick look and I have a relatively simple way to solve it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org