You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "zhoukang (Jira)" <ji...@apache.org> on 2020/06/10 09:34:00 UTC
[jira] [Updated] (HIVE-23670) hive on spark may encounter error
when use map join
[ https://issues.apache.org/jira/browse/HIVE-23670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhoukang updated HIVE-23670:
----------------------------
Description:
when using spark as hive engine, some task may failed with exception below.
The cause is that when we load table, we may issued with:
{code:java}
if (tableContainer == null) {
tableContainer = useOptimizedContainer ?
new MapJoinBytesTableContainer(hconf, valueContext, -1, 0) :
create(name, metaData);
}
{code}
then cast exception will issued with code below:
{code:java}
@Override
public JoinUtil.JoinResult setFromOther(ReusableGetAdaptor other) {
assert other instanceof GetAdaptor;
GetAdaptor other2 = (GetAdaptor)other;
this.key = other2.key;
this.isFirstKey = other2.isFirstKey;
this.currentValue = mHash.get(key);
if (this.currentValue == null) {
return JoinUtil.JoinResult.NOMATCH;
}
else {
return JoinUtil.JoinResult.MATCH;
}
}
{code}
and it is because spark HashTableLoader is based on the files on the given path.
Then it will generate HashMapWrapper:
{code:java}
// Get an empty container when the small table is empty.
private static MapJoinTableContainer getDefaultEmptyContainer(MapJoinObjectSerDeContext keyCtx,
MapJoinObjectSerDeContext valCtx) throws SerDeException {
MapJoinTableContainer container = new HashMapWrapper();
container.setSerde(keyCtx, valCtx);
container.seal();
return container;
}
{code}
{code:java}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception from MapJoinOperator : org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$GetAdaptor cannot be cast to org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper$GetAdaptor
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:465)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:148)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:547)
... 19 more
{code}
was:
when using spark as hive engine, some task may failed with exception below.
The cause is that when we load table, we may issued with:
{code:java}
if (tableContainer == null) {
tableContainer = useOptimizedContainer ?
new MapJoinBytesTableContainer(hconf, valueContext, -1, 0) :
create(name, metaData);
}
{code}
then cast exception will issued with code below:
{code:java}
@Override
public JoinUtil.JoinResult setFromOther(ReusableGetAdaptor other) {
assert other instanceof GetAdaptor;
GetAdaptor other2 = (GetAdaptor)other;
this.key = other2.key;
this.isFirstKey = other2.isFirstKey;
this.currentValue = mHash.get(key);
if (this.currentValue == null) {
return JoinUtil.JoinResult.NOMATCH;
}
else {
return JoinUtil.JoinResult.MATCH;
}
}
{code}
{code:java}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception from MapJoinOperator : org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$GetAdaptor cannot be cast to org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper$GetAdaptor
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:465)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:148)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:547)
... 19 more
{code}
> hive on spark may encounter error when use map join
> ---------------------------------------------------
>
> Key: HIVE-23670
> URL: https://issues.apache.org/jira/browse/HIVE-23670
> Project: Hive
> Issue Type: Bug
> Affects Versions: 2.4.0
> Reporter: zhoukang
> Priority: Major
>
> when using spark as hive engine, some task may failed with exception below.
> The cause is that when we load table, we may issued with:
> {code:java}
> if (tableContainer == null) {
> tableContainer = useOptimizedContainer ?
> new MapJoinBytesTableContainer(hconf, valueContext, -1, 0) :
> create(name, metaData);
> }
> {code}
> then cast exception will issued with code below:
> {code:java}
> @Override
> public JoinUtil.JoinResult setFromOther(ReusableGetAdaptor other) {
> assert other instanceof GetAdaptor;
> GetAdaptor other2 = (GetAdaptor)other;
> this.key = other2.key;
> this.isFirstKey = other2.isFirstKey;
> this.currentValue = mHash.get(key);
> if (this.currentValue == null) {
> return JoinUtil.JoinResult.NOMATCH;
> }
> else {
> return JoinUtil.JoinResult.MATCH;
> }
> }
> {code}
> and it is because spark HashTableLoader is based on the files on the given path.
> Then it will generate HashMapWrapper:
> {code:java}
> // Get an empty container when the small table is empty.
> private static MapJoinTableContainer getDefaultEmptyContainer(MapJoinObjectSerDeContext keyCtx,
> MapJoinObjectSerDeContext valCtx) throws SerDeException {
> MapJoinTableContainer container = new HashMapWrapper();
> container.setSerde(keyCtx, valCtx);
> container.seal();
> return container;
> }
> {code}
> {code:java}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception from MapJoinOperator : org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$GetAdaptor cannot be cast to org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper$GetAdaptor
> at org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:465)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:148)
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:547)
> ... 19 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)