[jira] [Commented] (HIVE-12052) automatically populate file metadata to HBase metastore based on config or table properties

["Sergey Shelukhin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-12052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359659#comment-15359659 ] 

Sergey Shelukhin commented on HIVE-12052:
-----------------------------------------

working on a patch for this... it's going to be a large patch :)

> automatically populate file metadata to HBase metastore based on config or table properties
> -------------------------------------------------------------------------------------------
>
>                 Key: HIVE-12052
>                 URL: https://issues.apache.org/jira/browse/HIVE-12052
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>
> As discussed in HIVE-11500
> Should use a table property similar to auto.purge.
> Then, when this setting is set, partitions are added (convertToMPart is a good source to find all the paths for that), after compactions, after load/non-ACID insert, and periodically (configurable), the storage locations should be scanned for new files and cache updated accordingly. All the updates should probably be in the background thread and taken from queue (high pri from most ops, low pri from enabling the property and from periodic updates) to avoid high load on HDFS from metastore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)