[jira] [Commented] (CRUNCH-368) TupleWritable.Comparator

["Chao Shi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CRUNCH-368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13956592#comment-13956592 ] 

Chao Shi commented on CRUNCH-368:
---------------------------------

bq. But in the snippet you provided, wouldn't comparing to a non-IntWritable type fail w/a ClassCastException?

Yes, it will fail.

Before this patch, I think the compareTo will fail if v1 and v2 are of different type. Is it intended?

{code}
        Writable v1 = ta.get(index);
        Writable v2 = tb.get(index);
        if (v1 != v2 && (v1 != null && !v1.equals(v2))) {
          if (v1 instanceof WritableComparable && v2 instanceof WritableComparable) {
            int cmp = ((WritableComparable) v1).compareTo((WritableComparable) v2);
            if (cmp != 0) {
              return order * cmp;
            }
          } else {
            int cmp = v1.hashCode() - v2.hashCode();
            if (cmp != 0) {
              return order * cmp;
            }
          }
        }
{code}

> TupleWritable.Comparator
> ------------------------
>
>                 Key: CRUNCH-368
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-368
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.10.0, 0.8.3
>            Reporter: Chao Shi
>            Assignee: Chao Shi
>         Attachments: crunch-368 benchmark.pdf, crunch-368.patch, gen_data.py
>
>
> This patch should improve comparison performance on TupleWritables. It saves the deserialization overhead. It is particularly useful when the input tuple are large, e.g. contains long strings.
> Please note that this changes the binary format of TupleWritable. It adds a var-int indicating size of field after each type code. This is a limitation of the writable system. We do not know the size of each field until fully desalinizing it. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)