[jira] [Updated] (HIVE-27224) Enhance drop table/partition command

["Taraka Rama Rao Lethavadla (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-27224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Taraka Rama Rao Lethavadla updated HIVE-27224:
----------------------------------------------
    Description: 
{*}Problem Statement{*}:

If the table has a large number of partitions, then drop table command will take a lot of time to finish. To improve the command we have the following proposals 
 * Perform all the queries(HMS->DB) in drop table in batches(not just partitions table) so that query will not fail throwing exceptions like transaction id not found or any other timeout issues as this is directly proportional to backend database performance
 * Display what action is happening as part of drop table, so that user will know what step is taking more time or how many steps completed so far. we should have loggers(DEBUG's at least) in clients to know how many partitions/batches being processed & current iterations to estimate approx. timeout for such large HMS operation.
 ** It would be great to add time taken for each HMS API call which implies response time from backend database
 * support retry option, if for some reason drop table command fails performing some of the operations, the next time it is run, it should proceed with next operations instead of failing due to missing/stale entries

  was:
{*}Problem Statement{*}:

If the table has a large number of partitions, then drop table command will take a lot of time to finish. To improve the command we have the following proposals 
 * Perform all the queries(HMS->DB) in drop table in batches(not just partitions table) so that query will not fail throwing exceptions like transaction id not found or any other timeout issues as this is directly proportional to backend database performance
 * Display what action is happening as part of drop table, so that user will know what step is taking more time or how many steps completed so far. we should have loggers(DEBUG's at least) in clients to know how many partitions/batches being processed & current iterations to estimate approx. timeout for such large HMS operation.
 * support retry option, if for some reason drop table command fails performing some of the operations, the next time it is run, it should proceed with next operations instead of failing due to missing/stale entries


> Enhance drop table/partition command
> ------------------------------------
>
>                 Key: HIVE-27224
>                 URL: https://issues.apache.org/jira/browse/HIVE-27224
>             Project: Hive
>          Issue Type: Improvement
>          Components: Hive, Standalone Metastore
>            Reporter: Taraka Rama Rao Lethavadla
>            Priority: Major
>
> {*}Problem Statement{*}:
> If the table has a large number of partitions, then drop table command will take a lot of time to finish. To improve the command we have the following proposals 
>  * Perform all the queries(HMS->DB) in drop table in batches(not just partitions table) so that query will not fail throwing exceptions like transaction id not found or any other timeout issues as this is directly proportional to backend database performance
>  * Display what action is happening as part of drop table, so that user will know what step is taking more time or how many steps completed so far. we should have loggers(DEBUG's at least) in clients to know how many partitions/batches being processed & current iterations to estimate approx. timeout for such large HMS operation.
>  ** It would be great to add time taken for each HMS API call which implies response time from backend database
>  * support retry option, if for some reason drop table command fails performing some of the operations, the next time it is run, it should proceed with next operations instead of failing due to missing/stale entries



--
This message was sent by Atlassian Jira
(v8.20.10#820010)