You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "olivier gillet (JIRA)" <ji...@apache.org> on 2009/11/13 13:15:39 UTC
[jira] Created: (HADOOP-6372) MurmurHash does not yield the same
results as the reference C++ implementation when size % 4 >= 2
MurmurHash does not yield the same results as the reference C++ implementation when size % 4 >= 2
-------------------------------------------------------------------------------------------------
Key: HADOOP-6372
URL: https://issues.apache.org/jira/browse/HADOOP-6372
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: 0.20.1
Reporter: olivier gillet
Priority: Trivial
Last rounds of MurmurHash are done in reverse order. data[length - 3], data[length - 2] and data[length - 1] in the block processing the remaining bytes should be data[len_m +2], data[len_m + 1], data[len_m].
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6372) MurmurHash does not yield the same
results as the reference C++ implementation when size % 4 >= 2
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778222#action_12778222 ]
dhruba borthakur commented on HADOOP-6372:
------------------------------------------
I was referring to recent changes via HADOOP-6148 and HDFS-496
> MurmurHash does not yield the same results as the reference C++ implementation when size % 4 >= 2
> -------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6372
> URL: https://issues.apache.org/jira/browse/HADOOP-6372
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.20.1
> Reporter: olivier gillet
> Priority: Trivial
> Attachments: HADOOP-6372.patch, murmur.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Last rounds of MurmurHash are done in reverse order. data[length - 3], data[length - 2] and data[length - 1] in the block processing the remaining bytes should be data[len_m +2], data[len_m + 1], data[len_m].
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6372) MurmurHash does not yield the same
results as the reference C++ implementation when size % 4 >= 2
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrzej Bialecki updated HADOOP-6372:
--------------------------------------
Attachment: murmur.patch
This patch includes the fix from Olivier, and updates the hash to Murmur 2.0A that provides better handling of empty keys.
> MurmurHash does not yield the same results as the reference C++ implementation when size % 4 >= 2
> -------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6372
> URL: https://issues.apache.org/jira/browse/HADOOP-6372
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.20.1
> Reporter: olivier gillet
> Priority: Trivial
> Attachments: HADOOP-6372.patch, murmur.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Last rounds of MurmurHash are done in reverse order. data[length - 3], data[length - 2] and data[length - 1] in the block processing the remaining bytes should be data[len_m +2], data[len_m + 1], data[len_m].
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6372) MurmurHash does not yield the same
results as the reference C++ implementation when size % 4 >= 2
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777539#action_12777539 ]
Andrzej Bialecki commented on HADOOP-6372:
-------------------------------------------
I confirm the bug, and the bugfix works as expected.
> MurmurHash does not yield the same results as the reference C++ implementation when size % 4 >= 2
> -------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6372
> URL: https://issues.apache.org/jira/browse/HADOOP-6372
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.20.1
> Reporter: olivier gillet
> Priority: Trivial
> Attachments: HADOOP-6372.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Last rounds of MurmurHash are done in reverse order. data[length - 3], data[length - 2] and data[length - 1] in the block processing the remaining bytes should be data[len_m +2], data[len_m + 1], data[len_m].
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6372) MurmurHash does not yield the same
results as the reference C++ implementation when size % 4 >= 2
Posted by "David Rosenstrauch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857927#action_12857927 ]
David Rosenstrauch commented on HADOOP-6372:
--------------------------------------------
I just ran into this problem as well. Are there any plans to release a fix?
> MurmurHash does not yield the same results as the reference C++ implementation when size % 4 >= 2
> -------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6372
> URL: https://issues.apache.org/jira/browse/HADOOP-6372
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.20.1
> Reporter: olivier gillet
> Priority: Trivial
> Attachments: HADOOP-6372.patch, murmur.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Last rounds of MurmurHash are done in reverse order. data[length - 3], data[length - 2] and data[length - 1] in the block processing the remaining bytes should be data[len_m +2], data[len_m + 1], data[len_m].
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-6372) MurmurHash does not yield the same
results as the reference C++ implementation when size % 4 >= 2
Posted by "olivier gillet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777578#action_12777578 ]
olivier gillet commented on HADOOP-6372:
----------------------------------------
It would be nice if the update to 2.0A was also done on HBase's side. Is it common to run a version of HBase not in sync with the matching version of Hadoop-common? If not, could the few classes using org.apache.hadoop.hbase.util.Hash use org.apache.hadoop.util.hash.Hash instead?
> MurmurHash does not yield the same results as the reference C++ implementation when size % 4 >= 2
> -------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6372
> URL: https://issues.apache.org/jira/browse/HADOOP-6372
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.20.1
> Reporter: olivier gillet
> Priority: Trivial
> Attachments: HADOOP-6372.patch, murmur.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Last rounds of MurmurHash are done in reverse order. data[length - 3], data[length - 2] and data[length - 1] in the block processing the remaining bytes should be data[len_m +2], data[len_m + 1], data[len_m].
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6372) MurmurHash does not yield the same
results as the reference C++ implementation when size % 4 >= 2
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778223#action_12778223 ]
dhruba borthakur commented on HADOOP-6372:
------------------------------------------
But you are right, that was mostly a Crc32 computation change and has nothing to do with murmur hash.
> MurmurHash does not yield the same results as the reference C++ implementation when size % 4 >= 2
> -------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6372
> URL: https://issues.apache.org/jira/browse/HADOOP-6372
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.20.1
> Reporter: olivier gillet
> Priority: Trivial
> Attachments: HADOOP-6372.patch, murmur.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Last rounds of MurmurHash are done in reverse order. data[length - 3], data[length - 2] and data[length - 1] in the block processing the remaining bytes should be data[len_m +2], data[len_m + 1], data[len_m].
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6372) MurmurHash does not yield the same
results as the reference C++ implementation when size % 4 >= 2
Posted by "olivier gillet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778189#action_12778189 ]
olivier gillet commented on HADOOP-6372:
----------------------------------------
I was not aware hashes from util.hash were used for HDFS checksumming. Could you point me to the code using them?
> MurmurHash does not yield the same results as the reference C++ implementation when size % 4 >= 2
> -------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6372
> URL: https://issues.apache.org/jira/browse/HADOOP-6372
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.20.1
> Reporter: olivier gillet
> Priority: Trivial
> Attachments: HADOOP-6372.patch, murmur.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Last rounds of MurmurHash are done in reverse order. data[length - 3], data[length - 2] and data[length - 1] in the block processing the remaining bytes should be data[len_m +2], data[len_m + 1], data[len_m].
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6372) MurmurHash does not yield the same
results as the reference C++ implementation when size % 4 >= 2
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778185#action_12778185 ]
dhruba borthakur commented on HADOOP-6372:
------------------------------------------
Does this mean that HDFS data blocks that were checksummed using this algorithm is now unread-able with this bug fix? How likely is this possibility?
> MurmurHash does not yield the same results as the reference C++ implementation when size % 4 >= 2
> -------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6372
> URL: https://issues.apache.org/jira/browse/HADOOP-6372
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.20.1
> Reporter: olivier gillet
> Priority: Trivial
> Attachments: HADOOP-6372.patch, murmur.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Last rounds of MurmurHash are done in reverse order. data[length - 3], data[length - 2] and data[length - 1] in the block processing the remaining bytes should be data[len_m +2], data[len_m + 1], data[len_m].
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6372) MurmurHash does not yield the same
results as the reference C++ implementation when size % 4 >= 2
Posted by "olivier gillet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
olivier gillet updated HADOOP-6372:
-----------------------------------
Attachment: HADOOP-6372.patch
> MurmurHash does not yield the same results as the reference C++ implementation when size % 4 >= 2
> -------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6372
> URL: https://issues.apache.org/jira/browse/HADOOP-6372
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.20.1
> Reporter: olivier gillet
> Priority: Trivial
> Attachments: HADOOP-6372.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Last rounds of MurmurHash are done in reverse order. data[length - 3], data[length - 2] and data[length - 1] in the block processing the remaining bytes should be data[len_m +2], data[len_m + 1], data[len_m].
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.