You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2009/07/01 02:30:47 UTC

[jira] Commented: (HDFS-297) Implement a pure Java CRC32 calculator

    [ https://issues.apache.org/jira/browse/HDFS-297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12725861#action_12725861 ] 

Todd Lipcon commented on HDFS-297:
----------------------------------

Hi Scott,

I'm looking over your patch. One question:
{code}
    int seed = 0xEDB88320;
    for (int i = 0; i < 256; i++) {
      // T1 is the ~crc for each single byte
      // create this with the crc seed 0xdebb20e3 for crc32
{code}

The seed indicated in the comments doesn't line up with the seed in the code. Can you please explain this?

Also, I'm glad to see the code that generates the tables T1 through T4, but I'm not certain it belongs in the actual patch, given it's essentially "dead code". Perhaps it's best to just put the code in this JIRA as a separate program with a main() function for future reference. What do others think?

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HDFS-297
>                 URL: https://issues.apache.org/jira/browse/HDFS-297
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.