[jira] [Created] (HIVE-20962) CommonMergeJoinOperator cannot join on complex keys

["Clemens Valiente (JIRA)" <ji...@apache.org>]

Clemens Valiente created HIVE-20962:
---------------------------------------

             Summary: CommonMergeJoinOperator cannot join on complex keys
                 Key: HIVE-20962
                 URL: https://issues.apache.org/jira/browse/HIVE-20962
             Project: Hive
          Issue Type: Bug
          Components: Query Processor
    Affects Versions: 2.3.4
            Reporter: Clemens Valiente


CommonMergeJoinOperator fails to perform joins on complex keys, e.g.

 
{code:sql}
CREATE TABLE complex_key (
`key` struct<id:bigint, country:string> 
value int)
PARTITIONED BY (date int);

SELECT t1.key, t1.value, t2.value FROM
complex_key t1
FULL OUTER JOIN
complex_key t2
ON (t1.date=20181121 and t2.date =20181122 AND t1.key=t2.key);
{code}
This causes a ClassCastException:
{code:java}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=1) {"key":{"reducesinkkey0":{"id":1,"country":"DK"}},"value":{"_col0":1489}}
	at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:357)
	at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:279)
	... 22 more
Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast to org.apache.hadoop.io.WritableComparable
	at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.compareKeys(CommonMergeJoinOperator.java:543)
	at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.processKey(CommonMergeJoinOperator.java:516)
	at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:212)
	at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:348)
{code}
 
Because the compareKeys() method tries to cast each key to a WritableComparable but e.g. the StandardStructObjectInspector would return our key field as an Arraylist.
https://github.com/apache/hive/blob/66f97da9de65b1c7151ec57bdf9ada937855bd75/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonMergeJoinOperator.java#L590

Proper way to do it would probably be to use the KeyWrapperFactory to convert the keys to something easily comparable?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)