[jira] [Created] (SPARK-17179) Consider improving partition pruning in HiveMetastoreCatalog

["Rajesh Balamohan (JIRA)" <ji...@apache.org>]

Rajesh Balamohan created SPARK-17179:
----------------------------------------

             Summary: Consider improving partition pruning in HiveMetastoreCatalog
                 Key: SPARK-17179
                 URL: https://issues.apache.org/jira/browse/SPARK-17179
             Project: Spark
          Issue Type: Improvement
          Components: SQL
            Reporter: Rajesh Balamohan
            Priority: Critical



Issue:
- Create an external table with 1000s of partition
- Running simple query with partition details ends up listing all files for caching in ListingFileCatalog.  This would turn out to be very slow in cloud based FS access (e.g S3). Even though, ListingFileCatalog supports
multi-threading, it would end up unncessarily listing 1000+ files when user is just interested in 1 partition.
- This adds up additional overhead in HiveMetastoreCatalog as it queries all partitions in convertToLogicalRelation (metastoreRelation.getHiveQlPartitions()).  Partition related details
are not passed in here, so ends up overloading hive metastore.
- Also even if any partition changes, cache would be dirtied and have to be re-populated.  It would be nice to prune the partitions in metastore layer itself, so that few partitions are looked up via FileSystem and only few items are cached.

{noformat}
"CREATE EXTERNAL TABLE `ca_par_ext`(
  `customer_id` bigint,
  `account_id` bigint)
PARTITIONED BY (
  `effective_date` date)
ROW FORMAT SERDE
  'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
  's3a://bucket_details/ca_par'"

explain select count(*) from ca_par_ext where effective_date between '2015-12-17' and '2015-12-18';

{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org