You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Vaibhav Aggarwal (JIRA)" <ji...@apache.org> on 2010/09/10 22:17:34 UTC
[jira] Created: (HIVE-1629) Patch to fix hashCode method in
DoubleWritable class
Patch to fix hashCode method in DoubleWritable class
----------------------------------------------------
Key: HIVE-1629
URL: https://issues.apache.org/jira/browse/HIVE-1629
Project: Hadoop Hive
Issue Type: Bug
Reporter: Vaibhav Aggarwal
A patch to fix the hashCode() method of DoubleWritable class of Hive.
It prevents the HashMap (of type DoubleWritable) from behaving as LinkedList.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1629) Patch to fix hashCode method in
DoubleWritable class
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908622#action_12908622 ]
John Sichi commented on HIVE-1629:
----------------------------------
Ning, does this change introduce incompatibility with any persistent storage (e.g. bucketing)?
> Patch to fix hashCode method in DoubleWritable class
> ----------------------------------------------------
>
> Key: HIVE-1629
> URL: https://issues.apache.org/jira/browse/HIVE-1629
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-1629.patch
>
>
> A patch to fix the hashCode() method of DoubleWritable class of Hive.
> It prevents the HashMap (of type DoubleWritable) from behaving as LinkedList.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-1629) Patch to fix hashCode method in
DoubleWritable class
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang resolved HIVE-1629.
------------------------------
Fix Version/s: 0.7.0
Resolution: Fixed
Committed. Thanks Vaibhav!
> Patch to fix hashCode method in DoubleWritable class
> ----------------------------------------------------
>
> Key: HIVE-1629
> URL: https://issues.apache.org/jira/browse/HIVE-1629
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Fix For: 0.7.0
>
> Attachments: HIVE-1629.patch
>
>
> A patch to fix the hashCode() method of DoubleWritable class of Hive.
> It prevents the HashMap (of type DoubleWritable) from behaving as LinkedList.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1629) Patch to fix hashCode method in
DoubleWritable class
Posted by "Vaibhav Aggarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908210#action_12908210 ]
Vaibhav Aggarwal commented on HIVE-1629:
----------------------------------------
Hi
The doubleToLongBits converts the double value into IEEE 754 floating-point "double format" bit layout.
Furthermore the XOR operator prevents returning 0 for values less than 2^32.
This is the hashCode function used by standard java implementation.
I was noticing unexpected delay in one of the operations related to double data types.
After some debugging I realized that the HashMap puts and gets were extremely slow.
That pointed me to the hashCode implementatoin in DoubleWritable which turned out to be the cause of slow HashMap IO.
That is why I propose to use the standard java implmenetation of HashCode for double type.
Thanks
Vaibhav
> Patch to fix hashCode method in DoubleWritable class
> ----------------------------------------------------
>
> Key: HIVE-1629
> URL: https://issues.apache.org/jira/browse/HIVE-1629
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Attachments: HIVE-1629.patch
>
>
> A patch to fix the hashCode() method of DoubleWritable class of Hive.
> It prevents the HashMap (of type DoubleWritable) from behaving as LinkedList.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1629) Patch to fix hashCode method in
DoubleWritable class
Posted by "Vaibhav Aggarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vaibhav Aggarwal updated HIVE-1629:
-----------------------------------
Attachment: HIVE-1629.patch
> Patch to fix hashCode method in DoubleWritable class
> ----------------------------------------------------
>
> Key: HIVE-1629
> URL: https://issues.apache.org/jira/browse/HIVE-1629
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Attachments: HIVE-1629.patch
>
>
> A patch to fix the hashCode() method of DoubleWritable class of Hive.
> It prevents the HashMap (of type DoubleWritable) from behaving as LinkedList.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1629) Patch to fix hashCode method in
DoubleWritable class
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang reassigned HIVE-1629:
--------------------------------
Assignee: Vaibhav Aggarwal
> Patch to fix hashCode method in DoubleWritable class
> ----------------------------------------------------
>
> Key: HIVE-1629
> URL: https://issues.apache.org/jira/browse/HIVE-1629
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-1629.patch
>
>
> A patch to fix the hashCode() method of DoubleWritable class of Hive.
> It prevents the HashMap (of type DoubleWritable) from behaving as LinkedList.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1629) Patch to fix hashCode method in
DoubleWritable class
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908639#action_12908639 ]
Ning Zhang commented on HIVE-1629:
----------------------------------
Good question John. I think this patch doesn't affect bucketing, which is implemented using ObjectInspectorUtils.hashCode(). Actually the hash function used there for Double is the same as the one provided in this patch. But I'll double check with Zheng/Namit tomorrow.
> Patch to fix hashCode method in DoubleWritable class
> ----------------------------------------------------
>
> Key: HIVE-1629
> URL: https://issues.apache.org/jira/browse/HIVE-1629
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Fix For: 0.7.0
>
> Attachments: HIVE-1629.patch
>
>
> A patch to fix the hashCode() method of DoubleWritable class of Hive.
> It prevents the HashMap (of type DoubleWritable) from behaving as LinkedList.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1629) Patch to fix hashCode method in
DoubleWritable class
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908599#action_12908599 ]
Ning Zhang commented on HIVE-1629:
----------------------------------
+1 Will commit if tests pass.
> Patch to fix hashCode method in DoubleWritable class
> ----------------------------------------------------
>
> Key: HIVE-1629
> URL: https://issues.apache.org/jira/browse/HIVE-1629
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Attachments: HIVE-1629.patch
>
>
> A patch to fix the hashCode() method of DoubleWritable class of Hive.
> It prevents the HashMap (of type DoubleWritable) from behaving as LinkedList.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1629) Patch to fix hashCode method in
DoubleWritable class
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908194#action_12908194 ]
Ning Zhang commented on HIVE-1629:
----------------------------------
+ long v = Double.doubleToLongBits(value);
+ return (int) (v ^ (v >>> 32));
won't this return 0 for all long values less than 2^32?
Search on the web and it seems the following 64 bit to 32 bit hash is a good one
http://www.cris.com/~ttwang/tech/inthash.htm
> Patch to fix hashCode method in DoubleWritable class
> ----------------------------------------------------
>
> Key: HIVE-1629
> URL: https://issues.apache.org/jira/browse/HIVE-1629
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Attachments: HIVE-1629.patch
>
>
> A patch to fix the hashCode() method of DoubleWritable class of Hive.
> It prevents the HashMap (of type DoubleWritable) from behaving as LinkedList.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.