You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/12/16 14:50:21 UTC
[GitHub] [pulsar] eolivelli opened a new issue #8974: Peeking at compressed messages throws an exception
eolivelli opened a new issue #8974:
URL: https://github.com/apache/pulsar/issues/8974
Earlier in the 2.6.0 branch, the compression library was swapped for https://github.com/airlift/aircompressor.
When messages are published with LZ4 compression and you try to use the Peek API, https://pulsar.apache.org/admin/v2/persistent/{tenant}/{namespace}/{topic}/subscription/{subName}/position/{messagePosition} on the message, the broker throws an exception, causing a 500 return code. This breaks the dashboard.
Here is the exception:
```
14:54:25.006 [pulsar-web-43-1] ERROR org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [superuser] Failed to get message at position 1 from persistent://xxxxxx
java.lang.IllegalArgumentException: Unsupported input ByteBuffer implementation java.nio.HeapByteBufferR
at io.airlift.compress.lz4.Lz4Decompressor.decompress(Lz4Decompressor.java:58) ~[io.airlift-aircompressor-0.16.jar:0.16]
at org.apache.pulsar.common.compression.CompressionCodecLZ4.decode(CompressionCodecLZ4.java:100) ~[org.apache.pulsar-pulsar-common-2.6.1.jar:2.6.1]
at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.generateResponseWithEntry(PersistentTopicsBase.java:1977) ~[org.apache.pulsar-pulsar-broker-2.6.1.jar
:2.6.1]
at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalPeekNthMessage(PersistentTopicsBase.java:1921) ~[org.apache.pulsar-pulsar-broker-2.6.1.jar:2.
6.1]
```
If I recall correctly, this also fails for SNAPPY compression, but works fine for other compression types (ZLIB).
Here is the pull request where this change was introduced: apache/pulsar#5390
I am 100% certain this is the cause of the issue. Reverting this PR resolves the issue.
Some potentially useful references:
See:
https://github.com/airlift/aircompressor/blob/24b4f18dcb8a70d273ba1d1cd4c0715f15909573/src/main/java/io/airlift/compress/lz4/Lz4Decompressor.java#L58
real-logic/agrona#66 (comment)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat commented on issue #8974: Peeking at compressed messages throws an exception
Posted by GitBox <gi...@apache.org>.
merlimat commented on issue #8974:
URL: https://github.com/apache/pulsar/issues/8974#issuecomment-747278231
The problem seems to be the bytebuffer is wrapped and is neither "direct" or it has an array. We should be probably taking a copy of that buffer to solve this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie closed issue #8974: Peeking at compressed messages throws an exception (Readonly buffers not supported by Airlift)
Posted by GitBox <gi...@apache.org>.
sijie closed issue #8974:
URL: https://github.com/apache/pulsar/issues/8974
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jennifer88huang commented on issue #8974: Peeking at compressed messages throws an exception
Posted by GitBox <gi...@apache.org>.
Jennifer88huang commented on issue #8974:
URL: https://github.com/apache/pulsar/issues/8974#issuecomment-747273945
It seems that some PR introduced issue...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on issue #8974: Peeking at compressed messages throws an exception
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #8974:
URL: https://github.com/apache/pulsar/issues/8974#issuecomment-747448008
@merlimat I am working on a fix, thanks for the advice
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on issue #8974: Peeking at compressed messages throws an exception
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #8974:
URL: https://github.com/apache/pulsar/issues/8974#issuecomment-746413843
This is a regression on 2.6 release line
@wolfstudy @codelipenghui please consider this problem before releasing 2.6.3
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org