You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/08/14 16:51:39 UTC
[GitHub] [pulsar] AnonHxy opened a new pull request, #17093: [fix][test][sql]Fix presto sql avro decode error when publish non-batched msgs
AnonHxy opened a new pull request, #17093:
URL: https://github.com/apache/pulsar/pull/17093
### Motivation
Finally I find the bug, which makes `org.apache.pulsar.tests.integration.presto.TestBasicPresto#testForSchema` fail, But I still don't know why other PRs can run it success
* This PR will fix https://github.com/apache/pulsar/issues/16975 OR https://github.com/apache/pulsar/issues/9704
* The root cause is that:
* The `keyByteBuf ` we get from `currentMessage ` is not correct (line545 in code block 1), because we didn't decode the key with Base64 when the message is non-batched published. Explain in details:
* As we know `SingleMessageMetadata` is only set when publish batched messages. So we will use `RawMessageImpl#singleMessageMetadata` to decode the key when messages is batched, or use `RawMessageImpl#msgMetadata` if messages in non-batched, As line174 in code block 2.
* The problem is that `RawMessageImpl#singleMessageMetadata` will never be null (line38 code block 3)
* If we try to decode non-batched message, the line175 in code block 2 will execute and return false.
* Then line166 in code block 2 will execute and decode the key without Base64
code block 1: [PulsarRecordCursor.java]:
https://github.com/apache/pulsar/blob/70020f14c3959f232ae53816022268fc61e0f955/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarRecordCursor.java#L542-L547
code block 2 [step into `currentMessage.getKeyBytes()` in `RawMessageImpl.java`]:
https://github.com/apache/pulsar/blob/70020f14c3959f232ae53816022268fc61e0f955/pulsar-common/src/main/java/org/apache/pulsar/common/api/raw/RawMessageImpl.java#L161-L178
code block 3 [RawMessageImpl.java]
https://github.com/apache/pulsar/blob/70020f14c3959f232ae53816022268fc61e0f955/pulsar-common/src/main/java/org/apache/pulsar/common/api/raw/RawMessageImpl.java#L34-L41
### Modifications
* Add `boolean setSingleMessageMetadata` in `org.apache.pulsar.common.api.raw.RawMessageImpl` in order to check if
`singleMessageMetadata` is set or not
### Verifying this change
- [x] Make sure that the change passes the CI checks.
- `org.apache.pulsar.tests.integration.presto.TestBasicPresto#testForSchema` has covered this change
### Documentation
Check the box below or label this PR directly.
Need to update docs?
- [ ] `doc-required`
(Your PR needs to update docs and you will update later)
- [x] `doc-not-needed`
(Please explain why)
- [ ] `doc`
(Your PR contains doc changes)
- [ ] `doc-complete`
(Docs have been already added)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #17093: [fix][test][sql]Fix presto sql avro decode error when publish non-batched msgs
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #17093:
URL: https://github.com/apache/pulsar/pull/17093#issuecomment-1214499365
@codelipenghui @gaoran10 PTAL~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #17093: [fix][test][sql]Fix presto sql avro decode error when publish non-batched msgs
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #17093:
URL: https://github.com/apache/pulsar/pull/17093#issuecomment-1216413669
@Jason918 @Technoboy- Could you also please help take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #17093: [fix][test][sql]Fix presto sql avro decode error when publish non-batched msgs
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #17093:
URL: https://github.com/apache/pulsar/pull/17093#issuecomment-1216169917
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #17093: [fix][test][sql]Fix presto sql avro decode error when publish non-batched msgs
Posted by GitBox <gi...@apache.org>.
codelipenghui merged PR #17093:
URL: https://github.com/apache/pulsar/pull/17093
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #17093: [fix][test][sql]Fix presto sql avro decode error when publish non-batched msgs
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #17093:
URL: https://github.com/apache/pulsar/pull/17093#issuecomment-1214427326
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #17093: [fix][test][sql]Fix presto sql avro decode error when publish non-batched msgs
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #17093:
URL: https://github.com/apache/pulsar/pull/17093#issuecomment-1214544069
> It seems we can barely hold the ref from my first glance, though
We didn't know weather `RawMessageImpl#singleMessageMetadata` is set value or has been cleared, through `SingleMessageMetadata`'s properties
There is a field `SingleMessageMetadata#_bitField0`, which value of 0 means `SingleMessageMetadata` didn't set any value. But it is private accessible, we can not use this flag @tisonkun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tisonkun commented on pull request #17093: [fix][test][sql]Fix presto sql avro decode error when publish non-batched msgs
Posted by GitBox <gi...@apache.org>.
tisonkun commented on PR #17093:
URL: https://github.com/apache/pulsar/pull/17093#issuecomment-1214548624
@AnonHxy please take a closer look at the patch, it sets `singleMessageMetadata` to `null` when `release`. Then you don't have to modify other code in the file and we can use `singleMessageMetadata == null` checker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #17093: [fix][common]Fix presto sql avro decode error when publish non-batched msgs
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #17093:
URL: https://github.com/apache/pulsar/pull/17093#issuecomment-1221855788
>
The modification in `pulsar-common/src/main/java/org/apache/pulsar/common/api/raw/RawMessageImpl.java` as show below changed the behavior.
Before this modification, the `singleMessageMetadata ` could be null if the method param is null. But after this modification, the `singleMessageMetadata` will never be null, and it uses `copyFrom` to set value. @gaoran10
<img width="1674" alt="image" src="https://user-images.githubusercontent.com/10233437/185846309-567a77bd-a455-44e2-8f5b-859d1d404946.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #17093: [fix][test][sql]Fix presto sql avro decode error when publish non-batched msgs
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #17093:
URL: https://github.com/apache/pulsar/pull/17093#issuecomment-1214906812
> The fix looks good, could you please help add test to avoid the regression?
OK, have add UT, PTAL @codelipenghui
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #17093: [fix][test][sql]Fix presto sql avro decode error when publish non-batched msgs
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #17093:
URL: https://github.com/apache/pulsar/pull/17093#issuecomment-1215057992
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on pull request #17093: [fix][common]Fix presto sql avro decode error when publish non-batched msgs
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on PR #17093:
URL: https://github.com/apache/pulsar/pull/17093#issuecomment-1221817876
More context
Before this [change](https://github.com/apache/pulsar/pull/9046/files#diff-1c4549d484ef1478755a0b8ee495544bc4d2062fcfdba6b2dca7d3ea2d88a939R40) we can use this `singleMessageMetadata != null` to judge the message is batched or not, it seems that after this change, the test should be failed. It's strange.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #17093: [fix][test][sql]Fix presto sql avro decode error when publish non-batched msgs
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #17093:
URL: https://github.com/apache/pulsar/pull/17093#issuecomment-1214634214
>
Now I got that @tisonkun . But I think we have to use deep copy here. The reason is that:
- There is only one `SingleMessageMetadata` instance in the thread, which get from threadlocal(Line163). And the
`processor` maybe process in another thread(Line173).
- If we didn't deep copy the `SingleMessageMetadata`, the `SingleMessageMetadata` could be modified while
`processor` processes the `RawMessageImpl`.
https://github.com/apache/pulsar/blob/70020f14c3959f232ae53816022268fc61e0f955/pulsar-common/src/main/java/org/apache/pulsar/common/api/raw/MessageParser.java#L157-L175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #17093: [fix][test][sql]Fix presto sql avro decode error when publish non-batched msgs
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #17093:
URL: https://github.com/apache/pulsar/pull/17093#issuecomment-1214424211
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org