You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/02/16 06:01:00 UTC
[jira] [Commented] (PHOENIX-4902) Snappy compression benefit is
lost when generate hash cache RPC
[ https://issues.apache.org/jira/browse/PHOENIX-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17285051#comment-17285051 ]
ASF GitHub Bot commented on PHOENIX-4902:
-----------------------------------------
stoty commented on pull request #349:
URL: https://github.com/apache/phoenix/pull/349#issuecomment-779603366
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 5m 52s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 17m 1s | master passed |
| +0 | hbaserecompile | 24m 12s | HBase recompiled. |
| +1 :green_heart: | compile | 1m 15s | master passed |
| +1 :green_heart: | checkstyle | 0m 34s | master passed |
| +1 :green_heart: | javadoc | 0m 52s | master passed |
| +0 :ok: | spotbugs | 3m 43s | phoenix-core in master has 959 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 10m 13s | the patch passed |
| +0 | hbaserecompile | 21m 18s | HBase recompiled. |
| +1 :green_heart: | compile | 1m 15s | the patch passed |
| +1 :green_heart: | javac | 1m 15s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 38s | phoenix-core: The patch generated 0 new + 101 unchanged - 2 fixed = 101 total (was 103) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 58s | the patch passed |
| +1 :green_heart: | spotbugs | 3m 56s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 118m 0s | phoenix-core in the patch failed. |
| +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. |
| | | 182m 29s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | phoenix.end2end.PermissionNSDisabledWithCustomAccessControllerIT |
| | phoenix.end2end.UpsertSelectIT |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-349/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix/pull/349 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaserebuild hbaseanti checkstyle compile |
| uname | Linux 4c1eb8b054b3 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-personality.sh |
| git revision | master / 6ed3caf |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-349/1/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-349/1/testReport/ |
| Max. process+thread count | 9636 (vs. ulimit of 30000) |
| modules | C: phoenix-core U: phoenix-core |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-349/1/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> Snappy compression benefit is lost when generate hash cache RPC
> ---------------------------------------------------------------
>
> Key: PHOENIX-4902
> URL: https://issues.apache.org/jira/browse/PHOENIX-4902
> Project: Phoenix
> Issue Type: Bug
> Reporter: Marcell Ortutay
> Assignee: Marcell Ortutay
> Priority: Minor
>
> Phoenix uses snappy compression on hash caches before it sends them to region server:
> {code}
> int maxCompressedSize = Snappy.maxCompressedLength(baOut.size());
> byte[] compressed = new byte[maxCompressedSize]; // size for worst case
> int compressedSize = Snappy.compress(baOut.getBuffer(), 0, baOut.size(), compressed, 0);
> // Last realloc to size of compressed buffer.
> ptr.set(compressed,0,compressedSize);
> {code}
> However, looking at debug output, it seems like the serialized protobuf that it sends to region servers does not have the benefits of snappy compression. Below is an excerpt of some debug output I put in:
> {code}
> Building an RPC with a cache ptr of size: 39MB // The compressed size is 39MB
> Done serializing the AddServerCacheRequest RPC, size is 206MB // However the serialized RPC is 206MB
> And the cache ptr size is: 206MB // And specifically, the byte array that contains the serialized hash cache is 206MB
> {code}
> I've made a simple test codebase to attempt to reproduce this bug. It shows similar behavior:
> {code}
> bytes size: 10000 bytes
> compressed bytes size: 721 bytes
> message size: 10003 bytes
> compressed message size: 11701 bytes
> {code}
> The code for the simplified example is here: https://github.com/ortutay/snappy-bytes-buffer/blob/master/src/main/java/testprotobuf/Main.java
> I observed this behavior in Phoenix 4.14.1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)