You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ethan Rose (Jira)" <ji...@apache.org> on 2024/08/13 16:38:00 UTC

[jira] [Assigned] (HDDS-11077) Optimize checksum calculations in container merkle tree

     [ https://issues.apache.org/jira/browse/HDDS-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ethan Rose reassigned HDDS-11077:
---------------------------------

    Assignee: Ethan Rose

> Optimize checksum calculations in container merkle tree
> -------------------------------------------------------
>
>                 Key: HDDS-11077
>                 URL: https://issues.apache.org/jira/browse/HDDS-11077
>             Project: Apache Ozone
>          Issue Type: Sub-task
>            Reporter: Ethan Rose
>            Assignee: Ethan Rose
>            Priority: Major
>
> *Choosing an Implementation*
> There are two main places we can get our checksum implementations from:
> * {{java.util.zip.CRC32[C\]}} which use native code.
> * {{PureJavaCrc32[C\]}} which has implementations in Ozone, Hadoop, and Apache Commons that are all more or less copied from each other.
> The considerations in choosing an implementation are:
> * CRC32C is a general improvement over CRC32.
> * {{java.util.zip.CRC32C}} does not exist until Java 9. Java 8 only has {{CRC32}}.
> * {{java.util.Checksum#update(ByteBuffer)}} does not exist until Java 9. This is why Ozone has the {{ChecksumByteBuffer}} wrapper class.
> Previous work to determine which checksum to use on data in Ozone was done [here|https://github.com/apache/ozone/pull/1910#issuecomment-775165462] and [here|https://github.com/apache/ozone/pull/1950]. These links explain the decision to default to {{java.util.zip.CRC32}} in Ozone. They also implement the ability to swap between {{PureJavaCrc32C}} and {{java.util.zip.CRC32C}} when CRC32C is specified based on the Java version.
> *Choosing an update method*
> It looks like {{java.util.Checksum#update(int)}} only reads the first byte out of the int. This is based on the [Java 9 javadoc for CRC32C|https://docs.oracle.com/javase%2F9%2Fdocs%2Fapi%2F%2F/java/util/zip/CRC32C.html#update-int-]. Other implementations do not specify whether the whole int is read or not. Since this is a single byte put, I'm not sure this is any better than using a byte buffer/array to either roll the longs into the checksum one by one, or batch the checksum computation on a buffer of all the longs under a tree node.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org