You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Hao Ren <h....@claravista.fr> on 2013/08/07 11:50:11 UTC

Issue: join two tables containing Array.

Hi,

I encounter a problem when joining two tables which contain Array field.

Some test queries:

drop table t1;
create table t1 as
select 1 as id, Array('1','2') as list, 2 as nb from t_data limit 1;

drop table t2;
create table t2 as
select 1 as id, Array('3','4') as list, 2 as nb from t_data limit 1;

select * from t1 join t2 on(t1.id = t2.id)

The last join query ends with an exception:

java.lang.ClassCastException: 
org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryArray cannot be cast 
to [Ljava.lang.Object;
     at 
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector.getList(StandardListObjectInspector.java:98)
     at 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:434)
     at 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:381)
     at 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:365)
     at 
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:568)
     at 
shark.execution.FileSinkOperator$$anonfun$processPartition$1.apply(FileSinkOperator.scala:73)
     at 
shark.execution.FileSinkOperator$$anonfun$processPartition$1.apply(FileSinkOperator.scala:72)
     at scala.collection.Iterator$class.foreach(Iterator.scala:772)
     at scala.collection.Iterator$$anon$19.foreach(Iterator.scala:399)
     at 
shark.execution.FileSinkOperator.processPartition(FileSinkOperator.scala:72)
     at 
shark.execution.FileSinkOperator$.writeFiles$1(FileSinkOperator.scala:133)
     at 
shark.execution.FileSinkOperator$$anonfun$executeProcessFileSinkPartition$1.apply(FileSinkOperator.scala:138)
     at 
shark.execution.FileSinkOperator$$anonfun$executeProcessFileSinkPartition$1.apply(FileSinkOperator.scala:138)
     at spark.scheduler.ResultTask.run(ResultTask.scala:77)
     at spark.executor.Executor$TaskRunner.run(Executor.scala:98)
     at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
     at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
     at java.lang.Thread.run(Thread.java:724)

Any workaround there ?

Hao

-- 
Hao Ren
ClaraVista
www.claravista.fr