[jira] [Created] (HIVE-20317) Spark Dynamic Partition Pruning - Use Stats to Determine Partition Count

["BELUGA BEHR (JIRA)" <ji...@apache.org>]

BELUGA BEHR created HIVE-20317:
----------------------------------

             Summary: Spark Dynamic Partition Pruning - Use Stats to Determine Partition Count
                 Key: HIVE-20317
                 URL: https://issues.apache.org/jira/browse/HIVE-20317
             Project: Hive
          Issue Type: Improvement
          Components: Spark
    Affects Versions: 3.1.0, 4.0.0
            Reporter: BELUGA BEHR


{code:xml|hive-site.xml}
<!-- HMS -->
<property>
<name>hive.metastore.limit.partition.request</name>
<value>2</value>
</property>
{code}

{code:sql}
CREATE TABLE partitioned_user(
	firstname VARCHAR(64),
	lastname  VARCHAR(64)
) PARTITIONED BY (country VARCHAR(64))
STORED AS PARQUET;

CREATE TABLE country(
    name VARCHAR(64)
) STORED AS PARQUET;

insert into partitioned_user partition (country='USA') values ("John", "Doe");
insert into partitioned_user partition (country='UK') values ("Sir", "Arthur");
insert into partitioned_user partition (country='FR') values ("Jacque", "Martin");

insert into country values ('USA');

set hive.execution.engine=spark;
set hive.spark.dynamic.partition.pruning=true;
explain select * from partitioned_user u where u.country in (select c.name from country c);
-- Error while compiling statement: FAILED: SemanticException MetaException(message:Number of partitions scanned (=3) on table 'partitioned_user' exceeds limit (=2). This is controlled on the metastore server by hive.metastore.limit.partition.request.)
{code}

The EXPLAIN plan generation fails because there are three partitions involved in this query.  However, since Spark DPP is enabled, Hive should be able to use table stats to know that the {{country}} table only has one record and therefore there will only need to be one partitioned scanned and allow this query to execute.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)