You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/02/22 03:39:54 UTC
[GitHub] [pulsar] gaoran10 opened a new pull request #9667: [Compression] Fix ByteBuffer allocate error in the AirliftUtils
gaoran10 opened a new pull request #9667:
URL: https://github.com/apache/pulsar/pull/9667
Fixes #9666
### Motivation
The compressed data length may be bigger than the original data length, so we can't use the uncompressed length as the allocated length to initial the ByteBuffer.
### Modifications
Use the capacity of the ByteBuffer instead of the uncompressed length as the allocated length.
### Verifying this change
Update the existing tests.
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (no)
- The public API: (no)
- The schema: (no)
- The default values of configurations: (no)
- The wire protocol: (no)
- The rest endpoints: (no)
- The admin cli options: (no)
- Anything that affects deployment: (no)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #9667: [Compression] Fix ByteBuffer allocate error in the AirliftUtils
Posted by GitBox <gi...@apache.org>.
codelipenghui merged pull request #9667:
URL: https://github.com/apache/pulsar/pull/9667
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #9667: [Compression] Fix ByteBuffer allocate error in the AirliftUtils
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #9667:
URL: https://github.com/apache/pulsar/pull/9667#discussion_r580000333
##########
File path: pulsar-common/src/main/java/org/apache/pulsar/common/compression/AirliftUtils.java
##########
@@ -25,10 +25,10 @@
*/
public abstract class AirliftUtils {
- static ByteBuffer ensureAirliftSupported(ByteBuffer encodedNio, int uncompressedLength) {
+ static ByteBuffer ensureAirliftSupported(ByteBuffer encodedNio) {
if (!encodedNio.isDirect() && !encodedNio.hasArray()) {
// airlift needs a raw ByteArray
- ByteBuffer copy = ByteBuffer.allocate(uncompressedLength);
+ ByteBuffer copy = ByteBuffer.allocate(encodedNio.capacity());
Review comment:
I am not sure I understand the fix here.
'uncompressedLength' may be usually bigger than the size of the buffer passed to this method, because data is compressed.
What about using the 'max' between the two values?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on pull request #9667: [Compression] Fix ByteBuffer allocate error in the AirliftUtils
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on pull request #9667:
URL: https://github.com/apache/pulsar/pull/9667#issuecomment-783218574
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a change in pull request #9667: [Compression] Fix ByteBuffer allocate error in the AirliftUtils
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #9667:
URL: https://github.com/apache/pulsar/pull/9667#discussion_r580004822
##########
File path: pulsar-common/src/main/java/org/apache/pulsar/common/compression/AirliftUtils.java
##########
@@ -25,10 +25,10 @@
*/
public abstract class AirliftUtils {
- static ByteBuffer ensureAirliftSupported(ByteBuffer encodedNio, int uncompressedLength) {
+ static ByteBuffer ensureAirliftSupported(ByteBuffer encodedNio) {
if (!encodedNio.isDirect() && !encodedNio.hasArray()) {
// airlift needs a raw ByteArray
- ByteBuffer copy = ByteBuffer.allocate(uncompressedLength);
+ ByteBuffer copy = ByteBuffer.allocate(encodedNio.capacity());
Review comment:
Yes, using max is better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a change in pull request #9667: [Compression] Fix ByteBuffer allocate error in the AirliftUtils
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on a change in pull request #9667:
URL: https://github.com/apache/pulsar/pull/9667#discussion_r579992634
##########
File path: pulsar-common/src/test/java/org/apache/pulsar/common/compression/CompressorCodecTest.java
##########
@@ -54,10 +72,10 @@
};
}
- @Test(dataProvider = "codec")
- void testCompressDecompress(CompressionType type, String compressedText) throws IOException {
+ @Test(dataProvider = "codecAndText")
+ void testCompressDecompress(CompressionType type, String sourceText) throws IOException {
Review comment:
The text param `compressedText` is not useful for this test, so I change the param from `compressedText` to `sourceText`, the purpose is to test complex text and no-repeated text, the complex text is used for the original test, the no duplicate text is used to verify this fix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a change in pull request #9667: [Compression] Fix ByteBuffer allocate error in the AirliftUtils
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on a change in pull request #9667:
URL: https://github.com/apache/pulsar/pull/9667#discussion_r580014577
##########
File path: pulsar-common/src/main/java/org/apache/pulsar/common/compression/AirliftUtils.java
##########
@@ -25,10 +25,10 @@
*/
public abstract class AirliftUtils {
- static ByteBuffer ensureAirliftSupported(ByteBuffer encodedNio, int uncompressedLength) {
+ static ByteBuffer ensureAirliftSupported(ByteBuffer encodedNio) {
if (!encodedNio.isDirect() && !encodedNio.hasArray()) {
// airlift needs a raw ByteArray
- ByteBuffer copy = ByteBuffer.allocate(uncompressedLength);
+ ByteBuffer copy = ByteBuffer.allocate(encodedNio.capacity());
Review comment:
Currently, if the source text is no repeated, the compressed data length is original data length + 1, I haven't found out the reason, but I think the capacity is enough for the ByteBuffer copy operation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a change in pull request #9667: [Compression] Fix ByteBuffer allocate error in the AirliftUtils
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on a change in pull request #9667:
URL: https://github.com/apache/pulsar/pull/9667#discussion_r579992634
##########
File path: pulsar-common/src/test/java/org/apache/pulsar/common/compression/CompressorCodecTest.java
##########
@@ -54,10 +72,10 @@
};
}
- @Test(dataProvider = "codec")
- void testCompressDecompress(CompressionType type, String compressedText) throws IOException {
+ @Test(dataProvider = "codecAndText")
+ void testCompressDecompress(CompressionType type, String sourceText) throws IOException {
Review comment:
The compressed text param is not useful for this test, so I change the param from `compressedText` to `sourceText`, the purpose is to test complex text and no-repeated text, the complex text is used for the original test, the no duplicate text is used to verify this fix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a change in pull request #9667: [Compression] Fix ByteBuffer allocate error in the AirliftUtils
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on a change in pull request #9667:
URL: https://github.com/apache/pulsar/pull/9667#discussion_r579992634
##########
File path: pulsar-common/src/test/java/org/apache/pulsar/common/compression/CompressorCodecTest.java
##########
@@ -54,10 +72,10 @@
};
}
- @Test(dataProvider = "codec")
- void testCompressDecompress(CompressionType type, String compressedText) throws IOException {
+ @Test(dataProvider = "codecAndText")
+ void testCompressDecompress(CompressionType type, String sourceText) throws IOException {
Review comment:
The text param `compressedText` is not useful for this test, so I change the param from `compressedText` to `sourceText`, the purpose is to test complex text and no-repeated text, the complex text is used for the original test, the no-repeated text is used to verify this fix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jiazhai commented on a change in pull request #9667: [Compression] Fix ByteBuffer allocate error in the AirliftUtils
Posted by GitBox <gi...@apache.org>.
jiazhai commented on a change in pull request #9667:
URL: https://github.com/apache/pulsar/pull/9667#discussion_r579976176
##########
File path: pulsar-common/src/test/java/org/apache/pulsar/common/compression/CompressorCodecTest.java
##########
@@ -54,10 +72,10 @@
};
}
- @Test(dataProvider = "codec")
- void testCompressDecompress(CompressionType type, String compressedText) throws IOException {
+ @Test(dataProvider = "codecAndText")
+ void testCompressDecompress(CompressionType type, String sourceText) throws IOException {
Review comment:
@gaoran10 Thanks for the fix. Is there any test that could reproduce the original failure without this fix? If not, I would suggest we add a test to cover this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org