You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Srinivasu alla (Jira)" <ji...@apache.org> on 2023/02/20 03:07:00 UTC
[jira] [Commented] (HIVE-27072) create an sql query to validate a given table for partitions and list out any discrepancies in files/folders, list out empty files etc
[ https://issues.apache.org/jira/browse/HIVE-27072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17690994#comment-17690994 ]
Srinivasu alla commented on HIVE-27072:
---------------------------------------
[~tarak271] , , can't most of these be achieved with MSCK REPAIR and ANALYZE TABLE COMPUTE STATS ..
How is _SUCCESS an unwanted file ?
Though your usecase may seem valid for managed tables, we shouldn't be controlling the External tables data/info from hive .
Happy to get the inputs and work on it.
> create an sql query to validate a given table for partitions and list out any discrepancies in files/folders, list out empty files etc
> --------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-27072
> URL: https://issues.apache.org/jira/browse/HIVE-27072
> Project: Hive
> Issue Type: New Feature
> Components: HiveServer2
> Reporter: Taraka Rama Rao Lethavadla
> Priority: Major
>
> There are couple of issues when partitions were corrupted or have additional unwanted files that will intervene query execution and fail.
> If we run query like "validate table table_name [partition(partition=a,..)]", the output should list
> * any unwanted files like empty/metadata files(like _SUCCESS etc)
> * any unwanted folders not confirming to the partition naming convention like test_folder where actual partition name looks like test=23
> * Too many staging directories, if we find many then cleanup is not happening properly after query execution
> * any file permission related issues like table has one owner, partition has another owner etc(Optional)
> We have something like this in Impala [Invalidate metadata and Refresh commands|https://impala.apache.org/docs/build/html/topics/impala_invalidate_metadata.html]
> So we can have something similar to that functionality in Hive
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)