You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "James Aley (JIRA)" <ji...@apache.org> on 2015/07/29 10:30:04 UTC

[jira] [Commented] (SPARK-8786) Create a wrapper for BinaryType

    [ https://issues.apache.org/jira/browse/SPARK-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14645697#comment-14645697 ] 

James Aley commented on SPARK-8786:
-----------------------------------

Will this change mean that joins, distinct, equality etc on binary columns will work?

We serialise UUIDs as binary in our analytics system, as we have to attach them to every event, of which there are billions, therefore the space saving vs a string is fairly significant. Right now, our SQL queries and analysis jobs use a UDF we've written to read them back into UUID strings on the fly, so that we can compare them.

Would I be right in thinking that once this issue is solved, we can skip that transformation and should therefore see some speed-up?

> Create a wrapper for BinaryType
> -------------------------------
>
>                 Key: SPARK-8786
>                 URL: https://issues.apache.org/jira/browse/SPARK-8786
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>            Reporter: Davies Liu
>
> The hashCode and equals() of Array[Byte] does check the bytes, we should create a wrapper (internally) to do that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org