[jira] [Updated] (HIVE-24263) Create an HMS endpoint to list partition locations

["Szehon Ho (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-24263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Szehon Ho updated HIVE-24263:
-----------------------------
    Description: 
In our company, we have a use-case to get quickly a list of partition locations.  Currently it is done via listPartitions, which is a very heavy operation in terms of memory and performance.

This JIRA proposes an API: Map<String, String> listPartitionLocations(String db, String table, short max) that returns a map of partition names to locations.

For example, we have an integration from output of a Hive pipeline to Spark jobs that consume directly from HDFS.  The Spark job scheduler needs to know the partition paths that are available for consumption (the partition name is not sufficient as it's input is HDFS path), and so we have to do heavy listPartitions() for this.

Another use-case is for a HDFS data removal tool that does a nightly crawl to see if there are associated hive partitions mapped to a given partition path.  The nightly crawling job could be much less resource-intensive if we had a listPartitionLocations().

As there is already an internal method in the ObjectStore for this done for dropPartitions, it is only a matter of exposing this API to HiveMetaStoreClient.



  was:
In our company, we have a use-case to get quickly a list of partition locations.  Currently it is done via listPartitions, which is a very heavy operation in terms of memory and performance.

For example, we have an integration from output of a Hive pipeline to Spark jobs that consume directly from HDFS.  It needs to know the partition paths that are available for consumation, and does repeated listPartitions() for this.

As there is already an internal method in the ObjectStore for this done for dropPartitions, it is only a matter of exposing this API to HiveMetaStoreClient.




> Create an HMS endpoint to list partition locations
> --------------------------------------------------
>
>                 Key: HIVE-24263
>                 URL: https://issues.apache.org/jira/browse/HIVE-24263
>             Project: Hive
>          Issue Type: Improvement
>          Components: Standalone Metastore
>            Reporter: Szehon Ho
>            Priority: Major
>
> In our company, we have a use-case to get quickly a list of partition locations.  Currently it is done via listPartitions, which is a very heavy operation in terms of memory and performance.
> This JIRA proposes an API: Map<String, String> listPartitionLocations(String db, String table, short max) that returns a map of partition names to locations.
> For example, we have an integration from output of a Hive pipeline to Spark jobs that consume directly from HDFS.  The Spark job scheduler needs to know the partition paths that are available for consumption (the partition name is not sufficient as it's input is HDFS path), and so we have to do heavy listPartitions() for this.
> Another use-case is for a HDFS data removal tool that does a nightly crawl to see if there are associated hive partitions mapped to a given partition path.  The nightly crawling job could be much less resource-intensive if we had a listPartitionLocations().
> As there is already an internal method in the ObjectStore for this done for dropPartitions, it is only a matter of exposing this API to HiveMetaStoreClient.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)