You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2013/10/17 23:10:43 UTC
[jira] [Commented] (HIVE-5580) push down predicates with an
and-operator between non-SARGable predicates will get NPE
[ https://issues.apache.org/jira/browse/HIVE-5580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13798424#comment-13798424 ]
Owen O'Malley commented on HIVE-5580:
-------------------------------------
The error looks like:
{code}
hive> select count(*) from web_sales where ws_wholesale_cost=66.29 and ws_list_price=174.34;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_1381995768224_0505, Tracking URL =
Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1381995768224_0505
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2013-10-17 06:28:27,545 Stage-1 map = 0%, reduce = 0%
2013-10-17 06:28:45,373 Stage-1 map = 100%, reduce = 100%
Ended Job = job_1381995768224_0505 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1381995768224_0505_m_000000 (and more) from job job_1381995768224_0505
Task with the most failures(4):
-----
Task ID:
task_1381995768224_0505_m_000000
URL:
-----
Diagnostic Messages for this Task:
Error: java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:343)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:290)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:404)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:556)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:167)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:408)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:329)
... 11 more
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.pickRowGroups(RecordReaderImpl.java:1723)
at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:1759)
at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:2148)
at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:2190)
at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.<init>(RecordReaderImpl.java:130)
at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rows(ReaderImpl.java:335)
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.<init>(OrcInputFormat.java:100)
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:205)
at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65)
... 16 more
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
{code}
> push down predicates with an and-operator between non-SARGable predicates will get NPE
> --------------------------------------------------------------------------------------
>
> Key: HIVE-5580
> URL: https://issues.apache.org/jira/browse/HIVE-5580
> Project: Hive
> Issue Type: Bug
> Components: File Formats
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
>
> When all of the predicates in an AND-operator in a SARG expression get removed by the SARG builder, evaluation can end up with a NPE. Sub-expressions are typically removed from AND-operators because they aren't SARGable.
--
This message was sent by Atlassian JIRA
(v6.1#6144)