You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Taraka Rama Rao Lethavadla (Jira)" <ji...@apache.org> on 2023/02/13 06:26:00 UTC
[jira] [Updated] (HIVE-27072) create an sql query to validate a given table for partitions and list out any discrepancies in files/folders, list out empty files etc
[ https://issues.apache.org/jira/browse/HIVE-27072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Taraka Rama Rao Lethavadla updated HIVE-27072:
----------------------------------------------
Description:
There are couple of issues when partitions were corrupted or have additional unwanted files that will intervene query execution and fail.
If we run query like "validate table table_name [partition(partition=a,..)]", the output should list
* any unwanted files like empty/metadata files(like _SUCCESS etc)
* any unwanted folders not confirming to the partition naming convention like test_folder where actual partition name looks like test=23
* Too many staging directories, if we find many then cleanup is not happening properly after query execution
* any file permission related issues like table has one owner, partition has another owner etc(Optional)
We have something like this in Impala [Invalidate metadata and Refresh commands|https://impala.apache.org/docs/build/html/topics/impala_invalidate_metadata.html]
So we can have something similar to that functionality in Hive
was:
There are couple of issues when partitions were corrupted or have additional unwanted files that will intervene query execution and fail.
If we run query like "validate table table_name [partition(partition=a,..)]", the output should list
* any unwanted files like empty/metadata files(like _SUCCESS etc)
* any unwanted folders not confirming to the partition naming convention like test_folder where actual partition name looks like test=23
* Too many staging directories, if we find many then cleanup is not happening properly after query execution
* any file permission related issues like table has one owner, partition has another owner etc(Optional)
> create an sql query to validate a given table for partitions and list out any discrepancies in files/folders, list out empty files etc
> --------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-27072
> URL: https://issues.apache.org/jira/browse/HIVE-27072
> Project: Hive
> Issue Type: New Feature
> Components: HiveServer2
> Reporter: Taraka Rama Rao Lethavadla
> Priority: Major
>
> There are couple of issues when partitions were corrupted or have additional unwanted files that will intervene query execution and fail.
> If we run query like "validate table table_name [partition(partition=a,..)]", the output should list
> * any unwanted files like empty/metadata files(like _SUCCESS etc)
> * any unwanted folders not confirming to the partition naming convention like test_folder where actual partition name looks like test=23
> * Too many staging directories, if we find many then cleanup is not happening properly after query execution
> * any file permission related issues like table has one owner, partition has another owner etc(Optional)
> We have something like this in Impala [Invalidate metadata and Refresh commands|https://impala.apache.org/docs/build/html/topics/impala_invalidate_metadata.html]
> So we can have something similar to that functionality in Hive
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)