You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/01/04 09:42:55 UTC
[GitHub] [pulsar] BewareMyPower opened a new pull request #9113: Fix NPE when MultiTopicsConsumerImpl receives null value messages
BewareMyPower opened a new pull request #9113:
URL: https://github.com/apache/pulsar/pull/9113
### Motivation
[#6379](https://github.com/apache/pulsar/pull/6379) introduced the feature to handle null value messages, but it only checks the null value in `ConsumerImpl` when `INCOMING_MESSAGES_SIZE_UPDATER` is updated. Therefore, if a partitioned topic with at least 2 partitions was consumed with a null value message, the NPE would be thrown.
### Modifications
- Check the null value message in `MultiTopicsConsumerImpl` as well as `ConsumerImpl`. To reduce repeated code, two protected methods are added to `ConsumerBase` and `INCOMING_MESSAGES_SIZE_UPDATER` becomes private now, the derived consumer classes just use these two methods to update or reset `INCOMING_MESSAGES_SIZE_UPDATER`.
- Add tests for partitioned topics in `NullValueTest`. Since the existed tests rely on the message send order, here we only send messages to a single partition only.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
- *Added tests to `NullValueTest` for partitioned topics*
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a change in pull request #9113: Fix NPE when MultiTopicsConsumerImpl receives null value messages
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #9113:
URL: https://github.com/apache/pulsar/pull/9113#discussion_r555474735
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java
##########
@@ -847,6 +847,15 @@ protected boolean hasPendingBatchReceive() {
return pendingBatchReceives != null && peekNextBatchReceive() != null;
}
+ protected void resetIncomingMessageSize() {
+ INCOMING_MESSAGES_SIZE_UPDATER.set(this, 0);
+ }
+
+ protected void updateIncomingMessageSize(final Message<?> message) {
+ INCOMING_MESSAGES_SIZE_UPDATER.addAndGet(this,
+ (message.getData() != null) ? message.getData().length : 0);
+ }
Review comment:
Hi @lhotari @BewareMyPower, I will push a fix soon. We should decrease the incoming queue size when take messages from the queue and increase the incoming queue size while adding messages to the queue. With #9113, will always increase the incoming queue size.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #9113: Fix NPE when MultiTopicsConsumerImpl receives null value messages
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on pull request #9113:
URL: https://github.com/apache/pulsar/pull/9113#issuecomment-753939601
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie merged pull request #9113: Fix NPE when MultiTopicsConsumerImpl receives null value messages
Posted by GitBox <gi...@apache.org>.
sijie merged pull request #9113:
URL: https://github.com/apache/pulsar/pull/9113
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #9113: Fix NPE when MultiTopicsConsumerImpl receives null value messages
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #9113:
URL: https://github.com/apache/pulsar/pull/9113#discussion_r555021563
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java
##########
@@ -847,6 +847,15 @@ protected boolean hasPendingBatchReceive() {
return pendingBatchReceives != null && peekNextBatchReceive() != null;
}
+ protected void resetIncomingMessageSize() {
+ INCOMING_MESSAGES_SIZE_UPDATER.set(this, 0);
+ }
+
+ protected void updateIncomingMessageSize(final Message<?> message) {
+ INCOMING_MESSAGES_SIZE_UPDATER.addAndGet(this,
+ (message.getData() != null) ? message.getData().length : 0);
+ }
Review comment:
just wondering if `message.getData().length` should be negated? it's negated in the original code that it's replacing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on a change in pull request #9113: Fix NPE when MultiTopicsConsumerImpl receives null value messages
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on a change in pull request #9113:
URL: https://github.com/apache/pulsar/pull/9113#discussion_r555447864
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java
##########
@@ -847,6 +847,15 @@ protected boolean hasPendingBatchReceive() {
return pendingBatchReceives != null && peekNextBatchReceive() != null;
}
+ protected void resetIncomingMessageSize() {
+ INCOMING_MESSAGES_SIZE_UPDATER.set(this, 0);
+ }
+
+ protected void updateIncomingMessageSize(final Message<?> message) {
+ INCOMING_MESSAGES_SIZE_UPDATER.addAndGet(this,
+ (message.getData() != null) ? message.getData().length : 0);
+ }
Review comment:
I think the previous implementation is wrong but there're no tests that expose the bug. The value only affects https://github.com/apache/pulsar/blob/d4c1677be93e5673672018f79da8a213514f01a3/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java#L673-L679
If it's negative, `INCOMING_MESSAGES_SIZE_UPDATER.get(this)` will be always negative and the comparison with `batchReceivePolicy.getMaxNumBytes()` is meaningless. I noticed the change was involved in #4621 , could you also take a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a change in pull request #9113: Fix NPE when MultiTopicsConsumerImpl receives null value messages
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #9113:
URL: https://github.com/apache/pulsar/pull/9113#discussion_r555103554
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java
##########
@@ -847,6 +847,15 @@ protected boolean hasPendingBatchReceive() {
return pendingBatchReceives != null && peekNextBatchReceive() != null;
}
+ protected void resetIncomingMessageSize() {
+ INCOMING_MESSAGES_SIZE_UPDATER.set(this, 0);
+ }
+
+ protected void updateIncomingMessageSize(final Message<?> message) {
+ INCOMING_MESSAGES_SIZE_UPDATER.addAndGet(this,
+ (message.getData() != null) ? message.getData().length : 0);
+ }
Review comment:
@BewareMyPower Could you please take a look @lhotari 's comment?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #9113: Fix NPE when MultiTopicsConsumerImpl receives null value messages
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #9113:
URL: https://github.com/apache/pulsar/pull/9113#issuecomment-755088911
Remove from release 2.6.3 since conflict with #9046
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on pull request #9113: Fix NPE when MultiTopicsConsumerImpl receives null value messages
Posted by GitBox <gi...@apache.org>.
sijie commented on pull request #9113:
URL: https://github.com/apache/pulsar/pull/9113#issuecomment-755101725
@codelipenghui it might worth cherry-picking the change and submit a separate pull request.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #9113: Fix NPE when MultiTopicsConsumerImpl receives null value messages
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #9113:
URL: https://github.com/apache/pulsar/pull/9113#discussion_r555021563
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java
##########
@@ -847,6 +847,15 @@ protected boolean hasPendingBatchReceive() {
return pendingBatchReceives != null && peekNextBatchReceive() != null;
}
+ protected void resetIncomingMessageSize() {
+ INCOMING_MESSAGES_SIZE_UPDATER.set(this, 0);
+ }
+
+ protected void updateIncomingMessageSize(final Message<?> message) {
+ INCOMING_MESSAGES_SIZE_UPDATER.addAndGet(this,
+ (message.getData() != null) ? message.getData().length : 0);
+ }
Review comment:
just wondering if `message.getData().length` this should be negated as it's in the original code that it's replacing?
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java
##########
@@ -847,6 +847,15 @@ protected boolean hasPendingBatchReceive() {
return pendingBatchReceives != null && peekNextBatchReceive() != null;
}
+ protected void resetIncomingMessageSize() {
+ INCOMING_MESSAGES_SIZE_UPDATER.set(this, 0);
+ }
+
+ protected void updateIncomingMessageSize(final Message<?> message) {
+ INCOMING_MESSAGES_SIZE_UPDATER.addAndGet(this,
+ (message.getData() != null) ? message.getData().length : 0);
+ }
Review comment:
just wondering if `message.getData().length` should be negated as it's in the original code that it's replacing?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #9113: Fix NPE when MultiTopicsConsumerImpl receives null value messages
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #9113:
URL: https://github.com/apache/pulsar/pull/9113#discussion_r555021563
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java
##########
@@ -847,6 +847,15 @@ protected boolean hasPendingBatchReceive() {
return pendingBatchReceives != null && peekNextBatchReceive() != null;
}
+ protected void resetIncomingMessageSize() {
+ INCOMING_MESSAGES_SIZE_UPDATER.set(this, 0);
+ }
+
+ protected void updateIncomingMessageSize(final Message<?> message) {
+ INCOMING_MESSAGES_SIZE_UPDATER.addAndGet(this,
+ (message.getData() != null) ? message.getData().length : 0);
+ }
Review comment:
just wondering if `message.getData().length` should be negated as it's negated in the original code that it's replacing?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on a change in pull request #9113: Fix NPE when MultiTopicsConsumerImpl receives null value messages
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on a change in pull request #9113:
URL: https://github.com/apache/pulsar/pull/9113#discussion_r555447864
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java
##########
@@ -847,6 +847,15 @@ protected boolean hasPendingBatchReceive() {
return pendingBatchReceives != null && peekNextBatchReceive() != null;
}
+ protected void resetIncomingMessageSize() {
+ INCOMING_MESSAGES_SIZE_UPDATER.set(this, 0);
+ }
+
+ protected void updateIncomingMessageSize(final Message<?> message) {
+ INCOMING_MESSAGES_SIZE_UPDATER.addAndGet(this,
+ (message.getData() != null) ? message.getData().length : 0);
+ }
Review comment:
I think the previous implementation is wrong but there're no tests that expose the bug. The value only affects https://github.com/apache/pulsar/blob/d4c1677be93e5673672018f79da8a213514f01a3/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java#L673-L679
If it's negative, `INCOMING_MESSAGES_SIZE_UPDATER.get(this)` will be always negative and the comparison with `batchReceivePolicy.getMaxNumBytes()` is meaningless. I noticed the change was involved in #4621 , could you also take a look? @codelipenghui
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org