You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Vijay (Created) (JIRA)" <ji...@apache.org> on 2011/12/11 18:02:39 UTC
[jira] [Created] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Checksum improvement for CompressedRandomAccessReader
-----------------------------------------------------
Key: CASSANDRA-3610
URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 1.1
Environment: JVM
Reporter: Vijay
Assignee: Vijay
Priority: Minor
Fix For: 1.1
When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
http://goo.gl/Qz2FW3
in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174968#comment-13174968 ]
Jonathan Ellis commented on CASSANDRA-3610:
-------------------------------------------
Hmm, I'm not quite sure what to make of that table. Is it rendering correctly?
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32.patch
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173434#comment-13173434 ]
Jonathan Ellis commented on CASSANDRA-3610:
-------------------------------------------
Didn't the Hadoop guys benchmark zip.CRC32 as being faster for > 32 bytes?
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32.patch
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174952#comment-13174952 ]
Vijay commented on CASSANDRA-3610:
----------------------------------
i modified the test class from HADOOP-6148 (bigger is better)
the result is:
||bytes||PureJava MB/sec||Native MB/sec||Random PureJava MB/sec||Native MB/sec||
| 1 |40.525 |12.118 |
| 2 |151.706 |23.046 |
| 4 |202.082 |46.778 |
| 8 |309.846 |88.030 |
| 16 |330.308 |152.373 |
| 32 |384.488 |247.181 |
| 64 |413.354 |350.407 || PureJava |0-64 |260.692 MB/sec|| Native |0-64 |198.142 MB/sec|
| 128 |429.670 |450.514 || PureJava |0-128 |323.232 MB/sec|| Native |0-128 |299.766 MB/sec|
| 256 |437.959 |521.985 || PureJava |0-256 |376.471 MB/sec|| Native |0-256 |405.063 MB/sec|
| 512 |439.036 |564.909 || PureJava |0-512 |410.257 MB/sec|| Native |0-512 |490.422 MB/sec|
| 1024 |441.101 |587.137 || PureJava |0-1024 |426.669 MB/sec|| Native |0-1024 |549.357 MB/sec|
| 2048 |443.893 |601.072 || PureJava |0-2048 |436.865 MB/sec|| Native |0-2048 |579.190 MB/sec|
| 4096 |444.964 |608.434 || PureJava |0-4096 |441.389 MB/sec|| Native |0-4096 |600.953 MB/sec|
| 8192 |445.737 |610.422 || PureJava |0-8192 |444.453 MB/sec|| Native |0-8192 |609.527 MB/sec|
| 16384 |447.157 |615.396 || PureJava |0-16384 |447.575 MB/sec|| Native |0-16384 |615.407 MB/sec|
| 32768 |446.494 |615.482 || PureJava |0-32768 |446.071 MB/sec|| Native |0-32768 |615.393 MB/sec|
| 65536 |446.550 |616.318 || PureJava |0-65536 |446.011 MB/sec|| Native |0-65536 |615.462 MB/sec|
| 131072 |446.603 |616.220 || PureJava |0-131072 |446.256 MB/sec|| Native |0-131072 |615.432 MB/sec|
| 262144 |445.461 |613.127 || PureJava |0-262144 |446.049 MB/sec|| Native |0-262144 |616.090 MB/sec|
| 524288 |445.208 |612.968 || PureJava |0-524288 |445.257 MB/sec|| Native |0-524288 |610.429 MB/sec|
| 1048576 |438.190 |607.029 || PureJava |0-1048576 |434.039 MB/sec|| Native |0-1048576 |603.446 MB/sec|
| 2097152 |428.816 |603.694 || PureJava |0-2097152 |423.804 MB/sec|| Native |0-2097152 |607.624 MB/sec|
| 4194304 |421.485 |606.262 || PureJava |0-4194304 |439.061 MB/sec|| Native |0-4194304 |609.371 MB/sec|
| 8388608 |424.566 |598.640 || PureJava |0-8388608 |426.544 MB/sec|| Native |0-8388608 |608.484 MB/sec|
| 16777216 |433.218 |602.733 || PureJava |0-16777216 |434.810 MB/sec|| Native |0-16777216 |601.179 MB/sec|
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32.patch
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204225#comment-13204225 ]
Vijay commented on CASSANDRA-3610:
----------------------------------
Hi Jonathan, Will do, There was also a discussion/issue noticed (not related to cassandra) @ Netflix where we saw a lock contention in JNI lib which was causing additional over head, hence I will run tests on Open JDK and also extend this test to be multi-threaded and see the difference.
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32-v3.patch, 0001-use-pure-java-CRC32.patch, TestCrc32Performance.java
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3610) Checksum improvement for
CommitLog
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3610:
--------------------------------------
Summary: Checksum improvement for CommitLog (was: Checksum improvement for CompressedRandomAccessReader)
> Checksum improvement for CommitLog
> ----------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1.0
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32-v3.patch, 0001-use-pure-java-CRC32.patch, TestCrc32Performance.java, TestCrc32Performance.java, crc32Test.xlsx
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-3610:
-----------------------------
Attachment: crc32Test.xlsx
TestCrc32Performance.java
Looks like the Open JDK 6 and Sun JDK 7 has the almost same performance on the JNI. Attached is the Multi-Threaded test (10 threads) for both the versions. hence closing this issue as not an issue as we expect users to upgrade to JDK 7
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1.0
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32-v3.patch, 0001-use-pure-java-CRC32.patch, TestCrc32Performance.java, TestCrc32Performance.java, crc32Test.xlsx
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178825#comment-13178825 ]
Vijay commented on CASSANDRA-3610:
----------------------------------
for this ticket, Should we just check if it is sunJDK and if true then use the pure java impl else regular CRC32?
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32-v3.patch, 0001-use-pure-java-CRC32.patch, TestCrc32Performance.java
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175035#comment-13175035 ]
Vijay commented on CASSANDRA-3610:
----------------------------------
Ooops pasted the wrong data.... the above data is without any Heap settings hence GC becomes a bottleneck... Plz see the below :)
/usr/java/latest/jre/bin/java -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8G -Xmx8G -Xmn2G -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -jar TestCRC32Performance.jar
||bytes||PureJava MB/sec||Native MB/sec||Random PureJava MB/sec||Native MB/sec||
| 1 |121.124 |11.866 |
| 2 |161.981 |23.851 |
| 4 |204.718 |45.486 |
| 8 |297.229 |76.296 |
| 16 |379.268 |117.326 |
| 32 |440.153 |157.711 |
| 64 |468.143 |193.304 || PureJava |0-64 |272.921 MB/sec|| Native |0-64 |145.289 MB/sec|
| 128 |500.006 |219.657 || PureJava |0-128 |367.816 MB/sec|| Native |0-128 |186.861 MB/sec|
| 256 |511.572 |234.052 || PureJava |0-256 |432.433 MB/sec|| Native |0-256 |214.047 MB/sec|
| 512 |517.550 |242.634 || PureJava |0-512 |474.074 MB/sec|| Native |0-512 |231.047 MB/sec|
| 1024 |516.994 |246.424 || PureJava |0-1024 |498.055 MB/sec|| Native |0-1024 |241.056 MB/sec|
| 2048 |518.095 |248.529 || PureJava |0-2048 |509.960 MB/sec|| Native |0-2048 |245.683 MB/sec|
| 4096 |522.002 |249.755 || PureJava |0-4096 |518.226 MB/sec|| Native |0-4096 |248.062 MB/sec|
| 8192 |522.795 |250.316 || PureJava |0-8192 |520.326 MB/sec|| Native |0-8192 |249.519 MB/sec|
| 16384 |522.521 |250.484 || PureJava |0-16384 |522.480 MB/sec|| Native |0-16384 |250.002 MB/sec|
| 32768 |521.098 |250.604 || PureJava |0-32768 |520.349 MB/sec|| Native |0-32768 |250.494 MB/sec|
| 65536 |520.973 |250.837 || PureJava |0-65536 |520.392 MB/sec|| Native |0-65536 |249.063 MB/sec|
| 131072 |510.129 |248.949 || PureJava |0-131072 |516.246 MB/sec|| Native |0-131072 |249.535 MB/sec|
| 262144 |513.534 |249.506 || PureJava |0-262144 |514.407 MB/sec|| Native |0-262144 |250.617 MB/sec|
| 524288 |519.554 |250.696 || PureJava |0-524288 |520.402 MB/sec|| Native |0-524288 |251.048 MB/sec|
| 1048576 |519.559 |250.557 || PureJava |0-1048576 |520.403 MB/sec|| Native |0-1048576 |250.734 MB/sec|
| 2097152 |519.259 |250.456 || PureJava |0-2097152 |519.337 MB/sec|| Native |0-2097152 |250.299 MB/sec|
| 4194304 |518.649 |250.470 || PureJava |0-4194304 |518.495 MB/sec|| Native |0-4194304 |250.523 MB/sec|
| 8388608 |501.986 |248.044 || PureJava |0-8388608 |509.521 MB/sec|| Native |0-8388608 |248.626 MB/sec|
| 16777216 |508.201 |247.587 || PureJava |0-16777216 |505.258 MB/sec|| Native |0-16777216 |249.558 MB/sec|
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32.patch
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Brandon Williams (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167153#comment-13167153 ]
Brandon Williams commented on CASSANDRA-3610:
---------------------------------------------
Above link is broken for me.
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://goo.gl/Qz2FW3
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172543#comment-13172543 ]
Sylvain Lebresne commented on CASSANDRA-3610:
---------------------------------------------
The patch lgtm, but pulling hadoop-commons for just that feels a bit too much imo. I'd prefer extracting the code from there (The initial patch were they added their optimized CRC32 was HADOOP-6148).
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32.patch
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-3610:
-----------------------------
Attachment: 0001-use-pure-java-CRC32-v3.patch
rebased
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32-v3.patch, 0001-use-pure-java-CRC32.patch, TestCrc32Performance.java
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172557#comment-13172557 ]
Vijay commented on CASSANDRA-3610:
----------------------------------
Sure will do, and i will also add the footnote about the same.
Just a note there is already a dependency in "org.apache.cassandra.hadoop" package.
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32.patch
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3610) Checksum improvement for
CommitLog
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3610:
--------------------------------------
Affects Version/s: (was: 1.1.0)
> Checksum improvement for CommitLog
> ----------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-CASSANDRA-3610-v4.patch, 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32-v3.patch, 0001-use-pure-java-CRC32.patch, TestCrc32Performance.java, TestCrc32Performance.java, crc32Test.xlsx
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-3610:
-----------------------------
Attachment: TestCrc32Performance.java
Plz try the attached test. Plz note it is a copied and modified to test only 2 types.
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32.patch, TestCrc32Performance.java
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175063#comment-13175063 ]
Vijay commented on CASSANDRA-3610:
----------------------------------
No, the take away is OpenJDK JNI is 2x faster :) The first result is from OpenJDK and the second is by SunJDK
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32.patch, TestCrc32Performance.java
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-3610:
-----------------------------
Description:
When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
was:
When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
http://goo.gl/Qz2FW3
in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175050#comment-13175050 ]
Vijay commented on CASSANDRA-3610:
----------------------------------
I might have miss commuicated the results... here is the results without any setting. SunJDK vs OpenJDK is the only difference.
[vijay_tcasstest@vijay_tcass--1a-i-aad629c8 ~]$ java -jar TestCRC32Performance.jar
||bytes||PureJava MB/sec||Native MB/sec||Random PureJava MB/sec||Native MB/sec||
| 1 |40.582 |11.678 |
| 2 |152.297 |23.060 |
| 4 |206.689 |46.501 |
| 8 |311.654 |87.895 |
| 16 |333.174 |151.716 |
| 32 |386.252 |246.780 |
| 64 |414.639 |353.623 || PureJava |0-64 |256.513 MB/sec|| Native |0-64 |196.319 MB/sec|
| 128 |429.995 |449.165 || PureJava |0-128 |324.051 MB/sec|| Native |0-128 |306.220 MB/sec|
| 256 |438.206 |519.658 || PureJava |0-256 |377.582 MB/sec|| Native |0-256 |410.256 MB/sec|
| 512 |442.210 |565.027 || PureJava |0-512 |408.946 MB/sec|| Native |0-512 |494.210 MB/sec|
| 1024 |453.627 |591.082 || PureJava |0-1024 |428.094 MB/sec|| Native |0-1024 |551.725 MB/sec|
| 2048 |455.738 |604.581 || PureJava |0-2048 |436.861 MB/sec|| Native |0-2048 |573.995 MB/sec|
| 4096 |443.916 |610.058 || PureJava |0-4096 |444.456 MB/sec|| Native |0-4096 |595.364 MB/sec|
| 8192 |449.964 |613.482 || PureJava |0-8192 |446.008 MB/sec|| Native |0-8192 |606.647 MB/sec|
| 16384 |447.484 |615.583 || PureJava |0-16384 |446.013 MB/sec|| Native |0-16384 |609.534 MB/sec|
| 32768 |446.452 |616.336 || PureJava |0-32768 |447.556 MB/sec|| Native |0-32768 |612.477 MB/sec|
| 65536 |446.351 |616.431 || PureJava |0-65536 |446.171 MB/sec|| Native |0-65536 |612.508 MB/sec|
| 131072 |446.457 |616.208 || PureJava |0-131072 |444.713 MB/sec|| Native |0-131072 |615.538 MB/sec|
| 262144 |445.521 |613.368 || PureJava |0-262144 |446.434 MB/sec|| Native |0-262144 |615.559 MB/sec|
| 524288 |445.233 |612.942 || PureJava |0-524288 |444.488 MB/sec|| Native |0-524288 |613.167 MB/sec|
| 1048576 |445.278 |613.044 || PureJava |0-1048576 |445.561 MB/sec|| Native |0-1048576 |614.500 MB/sec|
| 2097152 |443.896 |611.457 || PureJava |0-2097152 |440.685 MB/sec|| Native |0-2097152 |601.230 MB/sec|
| 4194304 |425.962 |595.604 || PureJava |0-4194304 |440.787 MB/sec|| Native |0-4194304 |610.587 MB/sec|
| 8388608 |430.688 |608.036 || PureJava |0-8388608 |439.621 MB/sec|| Native |0-8388608 |608.256 MB/sec|
| 16777216 |436.099 |599.706 || PureJava |0-16777216 |435.695 MB/sec|| Native |0-16777216 |603.244 MB/sec|
[vijay_tcasstest@vijay_tcass--1a-i-aad629c8 ~]$ /usr/java/latest/jre/bin/java -jar TestCRC32Performance.jar ||bytes||PureJava MB/sec||Native MB/sec||Random PureJava MB/sec||Native MB/sec||
| 1 |78.345 |10.897 |
| 2 |157.171 |20.341 |
| 4 |197.572 |40.119 |
| 8 |295.368 |67.051 |
| 16 |379.664 |104.782 |
| 32 |440.257 |147.490 |
| 64 |478.970 |187.356 || PureJava |0-64 |268.908 MB/sec|| Native |0-64 |141.436 MB/sec|
| 128 |495.235 |211.237 || PureJava |0-128 |363.636 MB/sec|| Native |0-128 |182.596 MB/sec|
| 256 |511.961 |231.697 || PureJava |0-256 |432.433 MB/sec|| Native |0-256 |211.921 MB/sec|
| 512 |517.852 |240.495 || PureJava |0-512 |475.837 MB/sec|| Native |0-512 |229.803 MB/sec|
| 1024 |516.815 |246.001 || PureJava |0-1024 |496.127 MB/sec|| Native |0-1024 |240.152 MB/sec|
| 2048 |520.283 |248.616 || PureJava |0-2048 |509.963 MB/sec|| Native |0-2048 |245.212 MB/sec|
| 4096 |521.950 |249.806 || PureJava |0-4096 |518.219 MB/sec|| Native |0-4096 |247.586 MB/sec|
| 8192 |516.635 |249.594 || PureJava |0-8192 |520.342 MB/sec|| Native |0-8192 |249.513 MB/sec|
| 16384 |522.743 |250.757 || PureJava |0-16384 |522.467 MB/sec|| Native |0-16384 |250.493 MB/sec|
| 32768 |521.264 |250.807 || PureJava |0-32768 |520.336 MB/sec|| Native |0-32768 |250.503 MB/sec|
| 65536 |521.192 |250.916 || PureJava |0-65536 |520.443 MB/sec|| Native |0-65536 |250.505 MB/sec|
| 131072 |520.264 |249.842 || PureJava |0-131072 |514.098 MB/sec|| Native |0-131072 |250.517 MB/sec|
| 262144 |519.953 |250.719 || PureJava |0-262144 |520.510 MB/sec|| Native |0-262144 |249.682 MB/sec|
| 524288 |512.457 |250.282 || PureJava |0-524288 |516.266 MB/sec|| Native |0-524288 |250.072 MB/sec|
| 1048576 |514.005 |249.813 || PureJava |0-1048576 |516.444 MB/sec|| Native |0-1048576 |250.548 MB/sec|
| 2097152 |519.207 |250.680 || PureJava |0-2097152 |519.260 MB/sec|| Native |0-2097152 |250.526 MB/sec|
| 4194304 |518.882 |250.608 || PureJava |0-4194304 |517.022 MB/sec|| Native |0-4194304 |250.538 MB/sec|
| 8388608 |517.444 |250.182 || PureJava |0-8388608 |518.502 MB/sec|| Native |0-8388608 |249.994 MB/sec|
| 16777216 |511.875 |249.008 || PureJava |0-16777216 |514.823 MB/sec|| Native |0-16777216 |249.477 MB/sec|
[vijay_tcasstest@vijay_tcass--1a-i-aad629c8 ~]$ java -version
java version "1.6.0_17"
OpenJDK Runtime Environment (IcedTea6 1.7.10) (rhel-1.21.b17.el5-x86_64)
OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode)
[vijay_tcasstest@vijay_tcass--1a-i-aad629c8 ~]$ /usr/java/latest/jre/bin/java -versionjava version "1.6.0_27"
Java(TM) SE Runtime Environment (build 1.6.0_27-b07)
Java HotSpot(TM) 64-Bit Server VM (build 20.2-b06, mixed mode)
[vijay_tcasstest@vijay_tcass--1a-i-aad629c8 ~]$
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32.patch, TestCrc32Performance.java
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175047#comment-13175047 ]
Jonathan Ellis commented on CASSANDRA-3610:
-------------------------------------------
Why would increasing heap size make the Native version perform *worse*? Something seems fishy here.
Also: GC overhead is part of "real world" workloads, so increasing heap size so that heap is large enough to complete small runs w/ no GC seems like stacking the deck. :)
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32.patch
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-3610:
-----------------------------
Attachment: 0001-use-pure-java-CRC32.patch
This patch needs hadoop-commons to be included in the lib path.
over head comes down from 51% cpu to 31%
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32.patch
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175059#comment-13175059 ]
Jonathan Ellis commented on CASSANDRA-3610:
-------------------------------------------
So... Sun JDK JNI is 2x as fast as OpenJDK, is that the takeaway?
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32.patch, TestCrc32Performance.java
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3610) Checksum improvement
for CompressedRandomAccessReader
Posted by "Vijay (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175035#comment-13175035 ]
Vijay edited comment on CASSANDRA-3610 at 12/22/11 8:30 PM:
------------------------------------------------------------
Ooops pasted the wrong data.... the above data is without any Heap settings and on Open JDK.. Plz see the below :)
/usr/java/latest/jre/bin/java -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8G -Xmx8G -Xmn2G -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -jar TestCRC32Performance.jar
||bytes||PureJava MB/sec||Native MB/sec||Random PureJava MB/sec||Native MB/sec||
| 1 |121.124 |11.866 |
| 2 |161.981 |23.851 |
| 4 |204.718 |45.486 |
| 8 |297.229 |76.296 |
| 16 |379.268 |117.326 |
| 32 |440.153 |157.711 |
| 64 |468.143 |193.304 || PureJava |0-64 |272.921 MB/sec|| Native |0-64 |145.289 MB/sec|
| 128 |500.006 |219.657 || PureJava |0-128 |367.816 MB/sec|| Native |0-128 |186.861 MB/sec|
| 256 |511.572 |234.052 || PureJava |0-256 |432.433 MB/sec|| Native |0-256 |214.047 MB/sec|
| 512 |517.550 |242.634 || PureJava |0-512 |474.074 MB/sec|| Native |0-512 |231.047 MB/sec|
| 1024 |516.994 |246.424 || PureJava |0-1024 |498.055 MB/sec|| Native |0-1024 |241.056 MB/sec|
| 2048 |518.095 |248.529 || PureJava |0-2048 |509.960 MB/sec|| Native |0-2048 |245.683 MB/sec|
| 4096 |522.002 |249.755 || PureJava |0-4096 |518.226 MB/sec|| Native |0-4096 |248.062 MB/sec|
| 8192 |522.795 |250.316 || PureJava |0-8192 |520.326 MB/sec|| Native |0-8192 |249.519 MB/sec|
| 16384 |522.521 |250.484 || PureJava |0-16384 |522.480 MB/sec|| Native |0-16384 |250.002 MB/sec|
| 32768 |521.098 |250.604 || PureJava |0-32768 |520.349 MB/sec|| Native |0-32768 |250.494 MB/sec|
| 65536 |520.973 |250.837 || PureJava |0-65536 |520.392 MB/sec|| Native |0-65536 |249.063 MB/sec|
| 131072 |510.129 |248.949 || PureJava |0-131072 |516.246 MB/sec|| Native |0-131072 |249.535 MB/sec|
| 262144 |513.534 |249.506 || PureJava |0-262144 |514.407 MB/sec|| Native |0-262144 |250.617 MB/sec|
| 524288 |519.554 |250.696 || PureJava |0-524288 |520.402 MB/sec|| Native |0-524288 |251.048 MB/sec|
| 1048576 |519.559 |250.557 || PureJava |0-1048576 |520.403 MB/sec|| Native |0-1048576 |250.734 MB/sec|
| 2097152 |519.259 |250.456 || PureJava |0-2097152 |519.337 MB/sec|| Native |0-2097152 |250.299 MB/sec|
| 4194304 |518.649 |250.470 || PureJava |0-4194304 |518.495 MB/sec|| Native |0-4194304 |250.523 MB/sec|
| 8388608 |501.986 |248.044 || PureJava |0-8388608 |509.521 MB/sec|| Native |0-8388608 |248.626 MB/sec|
| 16777216 |508.201 |247.587 || PureJava |0-16777216 |505.258 MB/sec|| Native |0-16777216 |249.558 MB/sec|
[vijay_tcasstest@vijay_tcass--1a-i-aad629c8 ~]$ /usr/java/latest/jre/bin/java -version
java version "1.6.0_27"
Java(TM) SE Runtime Environment (build 1.6.0_27-b07)
Java HotSpot(TM) 64-Bit Server VM (build 20.2-b06, mixed mode)
[vijay_tcasstest@vijay_tcass--1a-i-aad629c8 ~]$
was (Author: vijay2win@yahoo.com):
Ooops pasted the wrong data.... the above data is without any Heap settings hence GC becomes a bottleneck... Plz see the below :)
/usr/java/latest/jre/bin/java -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8G -Xmx8G -Xmn2G -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -jar TestCRC32Performance.jar
||bytes||PureJava MB/sec||Native MB/sec||Random PureJava MB/sec||Native MB/sec||
| 1 |121.124 |11.866 |
| 2 |161.981 |23.851 |
| 4 |204.718 |45.486 |
| 8 |297.229 |76.296 |
| 16 |379.268 |117.326 |
| 32 |440.153 |157.711 |
| 64 |468.143 |193.304 || PureJava |0-64 |272.921 MB/sec|| Native |0-64 |145.289 MB/sec|
| 128 |500.006 |219.657 || PureJava |0-128 |367.816 MB/sec|| Native |0-128 |186.861 MB/sec|
| 256 |511.572 |234.052 || PureJava |0-256 |432.433 MB/sec|| Native |0-256 |214.047 MB/sec|
| 512 |517.550 |242.634 || PureJava |0-512 |474.074 MB/sec|| Native |0-512 |231.047 MB/sec|
| 1024 |516.994 |246.424 || PureJava |0-1024 |498.055 MB/sec|| Native |0-1024 |241.056 MB/sec|
| 2048 |518.095 |248.529 || PureJava |0-2048 |509.960 MB/sec|| Native |0-2048 |245.683 MB/sec|
| 4096 |522.002 |249.755 || PureJava |0-4096 |518.226 MB/sec|| Native |0-4096 |248.062 MB/sec|
| 8192 |522.795 |250.316 || PureJava |0-8192 |520.326 MB/sec|| Native |0-8192 |249.519 MB/sec|
| 16384 |522.521 |250.484 || PureJava |0-16384 |522.480 MB/sec|| Native |0-16384 |250.002 MB/sec|
| 32768 |521.098 |250.604 || PureJava |0-32768 |520.349 MB/sec|| Native |0-32768 |250.494 MB/sec|
| 65536 |520.973 |250.837 || PureJava |0-65536 |520.392 MB/sec|| Native |0-65536 |249.063 MB/sec|
| 131072 |510.129 |248.949 || PureJava |0-131072 |516.246 MB/sec|| Native |0-131072 |249.535 MB/sec|
| 262144 |513.534 |249.506 || PureJava |0-262144 |514.407 MB/sec|| Native |0-262144 |250.617 MB/sec|
| 524288 |519.554 |250.696 || PureJava |0-524288 |520.402 MB/sec|| Native |0-524288 |251.048 MB/sec|
| 1048576 |519.559 |250.557 || PureJava |0-1048576 |520.403 MB/sec|| Native |0-1048576 |250.734 MB/sec|
| 2097152 |519.259 |250.456 || PureJava |0-2097152 |519.337 MB/sec|| Native |0-2097152 |250.299 MB/sec|
| 4194304 |518.649 |250.470 || PureJava |0-4194304 |518.495 MB/sec|| Native |0-4194304 |250.523 MB/sec|
| 8388608 |501.986 |248.044 || PureJava |0-8388608 |509.521 MB/sec|| Native |0-8388608 |248.626 MB/sec|
| 16777216 |508.201 |247.587 || PureJava |0-16777216 |505.258 MB/sec|| Native |0-16777216 |249.558 MB/sec|
[vijay_tcasstest@vijay_tcass--1a-i-aad629c8 ~]$ /usr/java/latest/jre/bin/java -version
java version "1.6.0_27"
Java(TM) SE Runtime Environment (build 1.6.0_27-b07)
Java HotSpot(TM) 64-Bit Server VM (build 20.2-b06, mixed mode)
[vijay_tcasstest@vijay_tcass--1a-i-aad629c8 ~]$
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32.patch
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Jonathan Ellis (Reopened) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reopened CASSANDRA-3610:
---------------------------------------
We still do a lot of int and long CRC updates in the commitlog though. Seems like that would be a better fit for the pure java implementation, which still does better for <= 64 bytes.
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1.0
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32-v3.patch, 0001-use-pure-java-CRC32.patch, TestCrc32Performance.java, TestCrc32Performance.java, crc32Test.xlsx
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-3610:
-----------------------------
Attachment: 0001-use-pure-java-CRC32-v2.patch
Attached is the patch for CRC32 without the dependency of Hadoop.
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32.patch
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3610) Checksum improvement for
CommitLog
Posted by "Vijay (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay updated CASSANDRA-3610:
-----------------------------
Attachment: 0001-CASSANDRA-3610-v4.patch
Commitlog Patch to use CRC32 for commitlog alone.... Thanks!
> Checksum improvement for CommitLog
> ----------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1.0
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-CASSANDRA-3610-v4.patch, 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32-v3.patch, 0001-use-pure-java-CRC32.patch, TestCrc32Performance.java, TestCrc32Performance.java, crc32Test.xlsx
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172572#comment-13172572 ]
Vijay commented on CASSANDRA-3610:
----------------------------------
Sure i can copy it but do we need CRC32C (HADOOP-7443) and make it configurable? i think we should be fine by just using CRC32.
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32.patch
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175017#comment-13175017 ]
Jonathan Ellis commented on CASSANDRA-3610:
-------------------------------------------
So, it looks like you're seeing similar results, that for larger values native is faster. Specifically, for arrays >= 128 bytes.
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32.patch
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167160#comment-13167160 ]
Vijay commented on CASSANDRA-3610:
----------------------------------
updated the slide share link. Thank!
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204221#comment-13204221 ]
Jonathan Ellis commented on CASSANDRA-3610:
-------------------------------------------
Dunno, this seems a little fragile to me. What does it look like with OpenJDK7 / Oracle JDK 7, which is what people are going to start transitioning to soon as JDK6 gets EOLed? (Oracle J7 is built from the OpenJDK tree so I would expect much more similarity there.)
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32-v3.patch, 0001-use-pure-java-CRC32.patch, TestCrc32Performance.java
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172567#comment-13172567 ]
Jonathan Ellis commented on CASSANDRA-3610:
-------------------------------------------
bq. The initial patch were they added their optimized CRC32 was HADOOP-6148
Related: HADOOP-7443
bq. there is already a dependency in "org.apache.cassandra.hadoop" package
But, that's only required if you're going to use the hadoop classes -- we don't ship any hadoop jars, and the rest of C* works fine w/o them.
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32.patch
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CommitLog
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401826#comment-13401826 ]
Jonathan Ellis commented on CASSANDRA-3610:
-------------------------------------------
Vijay, you might want to have a look at HDFS-2080. Looks like Todd's implemented a wrapper for a hardware crc implementation (as well as a pure java one) for even bigger improvements.
> Checksum improvement for CommitLog
> ----------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1.1
>
> Attachments: 0001-CASSANDRA-3610-v4.patch, 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32-v3.patch, 0001-use-pure-java-CRC32.patch, TestCrc32Performance.java, TestCrc32Performance.java, crc32Test.xlsx
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173448#comment-13173448 ]
Jonathan Ellis commented on CASSANDRA-3610:
-------------------------------------------
bq. The summary is that, on Sun's JDK (which most people use), the pure Java implementation is faster for all chunk sizes less than 32 bytes (by a high factor for the smaller end of the spectrum) and about 33% slower for chunk sizes larger than that.
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32.patch
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3610) Checksum improvement
for CompressedRandomAccessReader
Posted by "Vijay (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175035#comment-13175035 ]
Vijay edited comment on CASSANDRA-3610 at 12/22/11 8:29 PM:
------------------------------------------------------------
Ooops pasted the wrong data.... the above data is without any Heap settings hence GC becomes a bottleneck... Plz see the below :)
/usr/java/latest/jre/bin/java -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8G -Xmx8G -Xmn2G -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -jar TestCRC32Performance.jar
||bytes||PureJava MB/sec||Native MB/sec||Random PureJava MB/sec||Native MB/sec||
| 1 |121.124 |11.866 |
| 2 |161.981 |23.851 |
| 4 |204.718 |45.486 |
| 8 |297.229 |76.296 |
| 16 |379.268 |117.326 |
| 32 |440.153 |157.711 |
| 64 |468.143 |193.304 || PureJava |0-64 |272.921 MB/sec|| Native |0-64 |145.289 MB/sec|
| 128 |500.006 |219.657 || PureJava |0-128 |367.816 MB/sec|| Native |0-128 |186.861 MB/sec|
| 256 |511.572 |234.052 || PureJava |0-256 |432.433 MB/sec|| Native |0-256 |214.047 MB/sec|
| 512 |517.550 |242.634 || PureJava |0-512 |474.074 MB/sec|| Native |0-512 |231.047 MB/sec|
| 1024 |516.994 |246.424 || PureJava |0-1024 |498.055 MB/sec|| Native |0-1024 |241.056 MB/sec|
| 2048 |518.095 |248.529 || PureJava |0-2048 |509.960 MB/sec|| Native |0-2048 |245.683 MB/sec|
| 4096 |522.002 |249.755 || PureJava |0-4096 |518.226 MB/sec|| Native |0-4096 |248.062 MB/sec|
| 8192 |522.795 |250.316 || PureJava |0-8192 |520.326 MB/sec|| Native |0-8192 |249.519 MB/sec|
| 16384 |522.521 |250.484 || PureJava |0-16384 |522.480 MB/sec|| Native |0-16384 |250.002 MB/sec|
| 32768 |521.098 |250.604 || PureJava |0-32768 |520.349 MB/sec|| Native |0-32768 |250.494 MB/sec|
| 65536 |520.973 |250.837 || PureJava |0-65536 |520.392 MB/sec|| Native |0-65536 |249.063 MB/sec|
| 131072 |510.129 |248.949 || PureJava |0-131072 |516.246 MB/sec|| Native |0-131072 |249.535 MB/sec|
| 262144 |513.534 |249.506 || PureJava |0-262144 |514.407 MB/sec|| Native |0-262144 |250.617 MB/sec|
| 524288 |519.554 |250.696 || PureJava |0-524288 |520.402 MB/sec|| Native |0-524288 |251.048 MB/sec|
| 1048576 |519.559 |250.557 || PureJava |0-1048576 |520.403 MB/sec|| Native |0-1048576 |250.734 MB/sec|
| 2097152 |519.259 |250.456 || PureJava |0-2097152 |519.337 MB/sec|| Native |0-2097152 |250.299 MB/sec|
| 4194304 |518.649 |250.470 || PureJava |0-4194304 |518.495 MB/sec|| Native |0-4194304 |250.523 MB/sec|
| 8388608 |501.986 |248.044 || PureJava |0-8388608 |509.521 MB/sec|| Native |0-8388608 |248.626 MB/sec|
| 16777216 |508.201 |247.587 || PureJava |0-16777216 |505.258 MB/sec|| Native |0-16777216 |249.558 MB/sec|
[vijay_tcasstest@vijay_tcass--1a-i-aad629c8 ~]$ /usr/java/latest/jre/bin/java -version
java version "1.6.0_27"
Java(TM) SE Runtime Environment (build 1.6.0_27-b07)
Java HotSpot(TM) 64-Bit Server VM (build 20.2-b06, mixed mode)
[vijay_tcasstest@vijay_tcass--1a-i-aad629c8 ~]$
was (Author: vijay2win@yahoo.com):
Ooops pasted the wrong data.... the above data is without any Heap settings hence GC becomes a bottleneck... Plz see the below :)
/usr/java/latest/jre/bin/java -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8G -Xmx8G -Xmn2G -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -jar TestCRC32Performance.jar
||bytes||PureJava MB/sec||Native MB/sec||Random PureJava MB/sec||Native MB/sec||
| 1 |121.124 |11.866 |
| 2 |161.981 |23.851 |
| 4 |204.718 |45.486 |
| 8 |297.229 |76.296 |
| 16 |379.268 |117.326 |
| 32 |440.153 |157.711 |
| 64 |468.143 |193.304 || PureJava |0-64 |272.921 MB/sec|| Native |0-64 |145.289 MB/sec|
| 128 |500.006 |219.657 || PureJava |0-128 |367.816 MB/sec|| Native |0-128 |186.861 MB/sec|
| 256 |511.572 |234.052 || PureJava |0-256 |432.433 MB/sec|| Native |0-256 |214.047 MB/sec|
| 512 |517.550 |242.634 || PureJava |0-512 |474.074 MB/sec|| Native |0-512 |231.047 MB/sec|
| 1024 |516.994 |246.424 || PureJava |0-1024 |498.055 MB/sec|| Native |0-1024 |241.056 MB/sec|
| 2048 |518.095 |248.529 || PureJava |0-2048 |509.960 MB/sec|| Native |0-2048 |245.683 MB/sec|
| 4096 |522.002 |249.755 || PureJava |0-4096 |518.226 MB/sec|| Native |0-4096 |248.062 MB/sec|
| 8192 |522.795 |250.316 || PureJava |0-8192 |520.326 MB/sec|| Native |0-8192 |249.519 MB/sec|
| 16384 |522.521 |250.484 || PureJava |0-16384 |522.480 MB/sec|| Native |0-16384 |250.002 MB/sec|
| 32768 |521.098 |250.604 || PureJava |0-32768 |520.349 MB/sec|| Native |0-32768 |250.494 MB/sec|
| 65536 |520.973 |250.837 || PureJava |0-65536 |520.392 MB/sec|| Native |0-65536 |249.063 MB/sec|
| 131072 |510.129 |248.949 || PureJava |0-131072 |516.246 MB/sec|| Native |0-131072 |249.535 MB/sec|
| 262144 |513.534 |249.506 || PureJava |0-262144 |514.407 MB/sec|| Native |0-262144 |250.617 MB/sec|
| 524288 |519.554 |250.696 || PureJava |0-524288 |520.402 MB/sec|| Native |0-524288 |251.048 MB/sec|
| 1048576 |519.559 |250.557 || PureJava |0-1048576 |520.403 MB/sec|| Native |0-1048576 |250.734 MB/sec|
| 2097152 |519.259 |250.456 || PureJava |0-2097152 |519.337 MB/sec|| Native |0-2097152 |250.299 MB/sec|
| 4194304 |518.649 |250.470 || PureJava |0-4194304 |518.495 MB/sec|| Native |0-4194304 |250.523 MB/sec|
| 8388608 |501.986 |248.044 || PureJava |0-8388608 |509.521 MB/sec|| Native |0-8388608 |248.626 MB/sec|
| 16777216 |508.201 |247.587 || PureJava |0-16777216 |505.258 MB/sec|| Native |0-16777216 |249.558 MB/sec|
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32.patch
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173445#comment-13173445 ]
Vijay commented on CASSANDRA-3610:
----------------------------------
I think it is other way around they benchmarked zip.CRC32 to be slower than pure java implementation.
In my tests it reduces the CPU util by 20% without any other change.
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32.patch
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175014#comment-13175014 ]
Vijay commented on CASSANDRA-3610:
----------------------------------
Fields:
1 -> bytes sent to checksum
2 -> Mb/s (trials * size / 1024.0 / 1024.0) by pure java implementation
3 -> Mb/s by Native CRC
5 -> Pure Jave Random bytes from and till
6 -> Pure Java Mb/S
8 -> Native Random bytes from and till
9 -> Native Java Mb/S
Hope that makes sense... i should have separated the two tests random and constant byte test.
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32.patch
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for
CompressedRandomAccessReader
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173038#comment-13173038 ]
Sylvain Lebresne commented on CASSANDRA-3610:
---------------------------------------------
Let's not bother with CRC32C for now.
> Checksum improvement for CompressedRandomAccessReader
> -----------------------------------------------------
>
> Key: CASSANDRA-3610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3610
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.1
> Environment: JVM
> Reporter: Vijay
> Assignee: Vijay
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 0001-use-pure-java-CRC32.patch
>
>
> When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library.
> Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that.
> http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717
> in our test env it provided 50% improvement over native implementation which uses jni to call the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira