You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Stamatis Zampetakis (Jira)" <ji...@apache.org> on 2026/04/13 13:06:00 UTC
[jira] [Resolved] (HIVE-29516) NPE in StatsUtils.updateStats when removing semijoin by benefit and column statistics are missing
[ https://issues.apache.org/jira/browse/HIVE-29516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stamatis Zampetakis resolved HIVE-29516.
----------------------------------------
Assignee: Shubham Sharma
Resolution: Fixed
Fixed in https://github.com/apache/hive/commit/876021b99fb3debee5f82a5d73c362b486d29abc
Many thanks for the PR [~shubh_init]!
> NPE in StatsUtils.updateStats when removing semijoin by benefit and column statistics are missing
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-29516
> URL: https://issues.apache.org/jira/browse/HIVE-29516
> Project: Hive
> Issue Type: Bug
> Components: Query Processor, Statistics
> Affects Versions: 4.2.0
> Reporter: Shubham Sharma
> Assignee: Shubham Sharma
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> h3. Problem
> Query compilation fails with {{NullPointerException}} in {{StatsUtils.updateStats()}} when column statistics are not available for certain operators. This occurs during the semijoin optimization phase in {{{}TezCompiler.removeSemijoinOptimizationByBenefit(){}}}.
> The issue is reproducible with TPC-DS queries at scale factors of 100GB or higher, where column-level statistics may be incomplete or unavailable for some tables.
>
>
> {code:java}
> java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.stats.StatsUtils.updateStats(StatsUtils.java:2067)
> at org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1982)
> at org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:539)
> at org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:238)
> at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:174)
> at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.compilePlan(SemanticAnalyzer.java:12521)
> at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12739)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:460)
> ... {code}
> h2. How to Reproduce
> # Generate TPC-DS dataset at 100GB or larger scale
> # Run TPC-DS queries that involve semijoin optimizations (queries with subqueries or complex joins, eg: 10 17 19 23 24 25 29 32)
> # Ensure column statistics are not fully computed for all tables
> # Observe NPE during query compilation
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)