You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Peter Rozsa (Jira)" <ji...@apache.org> on 2024/11/14 16:08:00 UTC
[jira] [Resolved] (IMPALA-13467) test_min_max_filters() failed due to NullPointerException
[ https://issues.apache.org/jira/browse/IMPALA-13467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Rozsa resolved IMPALA-13467.
----------------------------------
Fix Version/s: Impala 4.5.0
Resolution: Fixed
> test_min_max_filters() failed due to NullPointerException
> ---------------------------------------------------------
>
> Key: IMPALA-13467
> URL: https://issues.apache.org/jira/browse/IMPALA-13467
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 4.5.0
> Reporter: Fang-Yu Rao
> Assignee: Peter Rozsa
> Priority: Critical
> Labels: broken-build
> Fix For: Impala 4.5.0
>
>
> We found that the following query in [min_max_filters.test|https://github.com/apache/impala/blame/master/testdata/workloads/functional-query/queries/QueryTest/min_max_filters.test] could fail due to NullPointerException.
> {code:java}
> ---- QUERY
> SET RUNTIME_FILTER_WAIT_TIME_MS=$RUNTIME_FILTER_WAIT_TIME_MS;
> select * from functional_parquet.iceberg_partitioned i1,
> functional_parquet.iceberg_partitioned i2
> where i1.action = i2.action and
> i1.id = i2.id and
> i2.event_time = '2020-01-01 10:00:00';
> ---- RUNTIME_PROFILE
> row_regex:.* RF00.\[min_max\] -> i1\.action.*
> {code}
> The stack trace was below.
> {code:java}
> I1018 18:26:21.967474 15092 Frontend.java:2190] 2449ca58b6c7b2c3:20e13eca00000000] Analyzing query: select * from functional_parquet.iceberg_partitioned i1,
> functional_parquet.iceberg_partitioned i2
> where i1.action = i2.action and
> i1.id = i2.id and
> i2.event_time = '2020-01-01 10:00:00' db: functional_kudu
> I1018 18:26:21.967491 15092 Frontend.java:2202] 2449ca58b6c7b2c3:20e13eca00000000] The original executor group sets from executor membership snapshot: [TExecutorGroupSet(curr_num_ex
> I1018 18:26:21.967509 15092 RequestPoolService.java:200] 2449ca58b6c7b2c3:20e13eca00000000] Default pool only, scheduler allocation is not specified.
> I1018 18:26:21.967532 15092 Frontend.java:2222] 2449ca58b6c7b2c3:20e13eca00000000] A total of 2 executor group sets to be considered for auto-scaling: [TExecutorGroupSet(curr_num_ex
> I1018 18:26:21.967546 15092 Frontend.java:2263] 2449ca58b6c7b2c3:20e13eca00000000] Consider executor group set: TExecutorGroupSet(curr_num_executors:3, expected_num_executors:20, ex
> I1018 18:26:21.968324 15092 AnalysisContext.java:521] 2449ca58b6c7b2c3:20e13eca00000000] Analysis took 0 ms
> I1018 18:26:21.968353 15092 BaseAuthorizationChecker.java:114] 2449ca58b6c7b2c3:20e13eca00000000] Authorization check took 0 ms
> I1018 18:26:21.968367 15092 Frontend.java:2599] 2449ca58b6c7b2c3:20e13eca00000000] Analysis and authorization finished.
> I1018 18:26:21.968899 15092 IcebergScanPlanner.java:846] 2449ca58b6c7b2c3:20e13eca00000000] Push down the predicate: ref(name="event_time") == 1577901600000000 to iceberg
> I1018 18:26:21.969009 15092 SnapshotScan.java:124] 2449ca58b6c7b2c3:20e13eca00000000] Scanning table hdfs://localhost:20500/test-warehouse/iceberg_test/iceberg_partitioned snapshot
> I1018 18:26:21.969400 15092 LoggingMetricsReporter.java:38] 2449ca58b6c7b2c3:20e13eca00000000] Received metrics report: ScanReport{tableName=hdfs://localhost:20500/test-warehouse/ic
> I1018 18:26:21.969846 15092 jni-util.cc:321] 2449ca58b6c7b2c3:20e13eca00000000] java.lang.NullPointerException
> at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:903)
> at org.apache.impala.planner.HdfsScanNode.initOverlapPredicate(HdfsScanNode.java:845)
> at org.apache.impala.planner.RuntimeFilterGenerator.assignRuntimeFilters(RuntimeFilterGenerator.java:1257)
> at org.apache.impala.planner.RuntimeFilterGenerator.generateFiltersRecursive(RuntimeFilterGenerator.java:1159)
> at org.apache.impala.planner.RuntimeFilterGenerator.generateFiltersRecursive(RuntimeFilterGenerator.java:1162)
> at org.apache.impala.planner.RuntimeFilterGenerator.generateFiltersRecursive(RuntimeFilterGenerator.java:1157)
> at org.apache.impala.planner.RuntimeFilterGenerator.generateFiltersRecursive(RuntimeFilterGenerator.java:1162)
> at org.apache.impala.planner.RuntimeFilterGenerator.generateFilters(RuntimeFilterGenerator.java:1091)
> at org.apache.impala.planner.RuntimeFilterGenerator.generateRuntimeFilters(RuntimeFilterGenerator.java:918)
> at org.apache.impala.planner.Planner.createPlanFragments(Planner.java:160)
> at org.apache.impala.planner.Planner.createPlans(Planner.java:310)
> at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1969)
> at org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:2968)
> at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:2730)
> at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:2269)
> at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:2030)
> at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:175)
> {code}
>
> We recently had changes in [HdfsScanNode.java|https://github.com/apache/impala/blame/master/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java] in IMPALA-12861 so maybe it's related.
>
> The NullPointerException was thrown in initOverlapPredicate() of [HdfsScanNode.java|https://github.com/apache/impala/blame/master/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java] due to 'statsTuple_' being null.
> {code:java}
> public void initOverlapPredicate(Analyzer analyzer) {
> if (!allParquet_) return;
> Preconditions.checkNotNull(statsTuple_);
> ..
> }
> {code}
> 'stats' is written in computeStatsTupleAndConjuncts() of [HdfsScanNode.java|https://github.com/apache/impala/blame/master/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java], which in turn is called in init() of [HdfsScanNode.java|https://github.com/apache/impala/blame/master/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java] and will be called only if hasParquet(fileFormats_) or hasOrc(fileFormats_) evaluate to true. I am wondering if it's possible that for some reason 'stats' is not populated.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org