You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Xuefu Zhang (JIRA)" <ji...@apache.org> on 2014/04/29 01:11:15 UTC
[jira] [Updated] (HIVE-6984) Analyzing partitioned table with NULL
values for the partition column failed with NPE
[ https://issues.apache.org/jira/browse/HIVE-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xuefu Zhang updated HIVE-6984:
------------------------------
Description:
The following describes how to produce the bug:
{code}
hive> desc test2;
name string
age int
hive> select * from test2;
6666666666666666666 NULL
5555555555555555555 NULL
tom 15
john NULL
mayr 40
30
NULL
hive> create table test3(name string) partitioned by (age int);
hive> from test2 insert overwrite table test3 partition(age) select test2.name, test2.age;
Loading data to table default.test3 partition (age=null)
Loading partition {age=40}
Loading partition {age=__HIVE_DEFAULT_PARTITION__}
Loading partition {age=30}
Loading partition {age=15}
Partition default.test3{age=15} stats: [numFiles=1, numRows=1, totalSize=4, rawDataSize=3]
Partition default.test3{age=30} stats: [numFiles=1, numRows=1, totalSize=1, rawDataSize=0]
Partition default.test3{age=40} stats: [numFiles=1, numRows=1, totalSize=5, rawDataSize=4]
Partition default.test3{age=__HIVE_DEFAULT_PARTITION__} stats: [numFiles=1, numRows=4, totalSize=46, rawDataSize=42]
hive> analyze table test3 partition(age) compute statistics;
...
Task with the most failures(4):
-----
Diagnostic Messages for this Task:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"name":"6666666666666666666","age":null,"raw__data__size":19}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"name":"6666666666666666666","age":null,"raw__data__size":19}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
at org.apache.hado
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
{code}
The following is the stack trace in mapper log:
{code}
2014-04-28 15:39:25,073 FATAL org.apache.hadoop.hive.ql.exec.mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"name":"6666666666666666666","age":null,"raw__data__size":19}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.TableScanOperator.gatherStats(TableScanOperator.java:149)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:90)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:539)
... 9 more
{code}
was:
The following describes how to produce the bug:
{code}
hive> desc test2;
name string
age int
hive> select * from test2;
Query ID = xzhang_20140428154343_bb921bce-7282-47b6-bfe8-669c7edaabe7
OK
6666666666666666666 NULL
5555555555555555555 NULL
tom 15
john NULL
mayr 40
30
NULL
hive> create table test3(name string) partitioned by (age int);
hive> from test2 insert overwrite table test3 partition(age) select test2.name, test2.age;
Loading data to table default.test3 partition (age=null)
Loading partition {age=40}
Loading partition {age=__HIVE_DEFAULT_PARTITION__}
Loading partition {age=30}
Loading partition {age=15}
Partition default.test3{age=15} stats: [numFiles=1, numRows=1, totalSize=4, rawDataSize=3]
Partition default.test3{age=30} stats: [numFiles=1, numRows=1, totalSize=1, rawDataSize=0]
Partition default.test3{age=40} stats: [numFiles=1, numRows=1, totalSize=5, rawDataSize=4]
Partition default.test3{age=__HIVE_DEFAULT_PARTITION__} stats: [numFiles=1, numRows=4, totalSize=46, rawDataSize=42]
hive> analyze table test3 partition(age) compute statistics;
Task with the most failures(4):
-----
Task ID:
task_201403041834_0063_m_000000
-----
Diagnostic Messages for this Task:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"name":"6666666666666666666","age":null,"raw__data__size":19}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"name":"6666666666666666666","age":null,"raw__data__size":19}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
at org.apache.hado
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
{code}
The following is the stack trace in mapper log:
{code}
2014-04-28 15:39:25,073 FATAL org.apache.hadoop.hive.ql.exec.mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"name":"6666666666666666666","age":null,"raw__data__size":19}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.TableScanOperator.gatherStats(TableScanOperator.java:149)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:90)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:539)
... 9 more
{code}
> Analyzing partitioned table with NULL values for the partition column failed with NPE
> -------------------------------------------------------------------------------------
>
> Key: HIVE-6984
> URL: https://issues.apache.org/jira/browse/HIVE-6984
> Project: Hive
> Issue Type: Bug
> Components: Statistics
> Affects Versions: 0.14.0
> Reporter: Xuefu Zhang
>
> The following describes how to produce the bug:
> {code}
> hive> desc test2;
> name string
> age int
> hive> select * from test2;
> 6666666666666666666 NULL
> 5555555555555555555 NULL
> tom 15
> john NULL
> mayr 40
> 30
> NULL
> hive> create table test3(name string) partitioned by (age int);
> hive> from test2 insert overwrite table test3 partition(age) select test2.name, test2.age;
> Loading data to table default.test3 partition (age=null)
> Loading partition {age=40}
> Loading partition {age=__HIVE_DEFAULT_PARTITION__}
> Loading partition {age=30}
> Loading partition {age=15}
> Partition default.test3{age=15} stats: [numFiles=1, numRows=1, totalSize=4, rawDataSize=3]
> Partition default.test3{age=30} stats: [numFiles=1, numRows=1, totalSize=1, rawDataSize=0]
> Partition default.test3{age=40} stats: [numFiles=1, numRows=1, totalSize=5, rawDataSize=4]
> Partition default.test3{age=__HIVE_DEFAULT_PARTITION__} stats: [numFiles=1, numRows=4, totalSize=46, rawDataSize=42]
> hive> analyze table test3 partition(age) compute statistics;
> ...
> Task with the most failures(4):
> -----
> Diagnostic Messages for this Task:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"name":"6666666666666666666","age":null,"raw__data__size":19}
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"name":"6666666666666666666","age":null,"raw__data__size":19}
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
> at org.apache.hado
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> {code}
> The following is the stack trace in mapper log:
> {code}
> 2014-04-28 15:39:25,073 FATAL org.apache.hadoop.hive.ql.exec.mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"name":"6666666666666666666","age":null,"raw__data__size":19}
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.TableScanOperator.gatherStats(TableScanOperator.java:149)
> at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:90)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:539)
> ... 9 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)