[jira] [Issue Comment Deleted] (IMPALA-6994) Avoid reloading a table's HMS data for file-only operations

["Pranay Singh (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pranay Singh updated IMPALA-6994:
---------------------------------
    Comment: was deleted

(was: [~jeszyb] I have made the below change, that is currently in review, here is a brief description of the change made.

      The problem is that while inserting a  new row to an unpartitioned HDFS table,
      or to an existing partition, the catalogd makes unecessary call to Hive Meta
      Store to do a getTable() and list the partitions of the table, when in fact
      only the file metadata for HDFS tables needs to be updated.

      This extra call to HMS introduces a point of failure, we need to handle
      for error scenario when Hive MetaStore crashes. This change removes the
      extra call to Hive Meta Store for the case when a row is inserted to an
      existing partition in HDFS table or when a row is added to unpartitioned
      table. Thus, an optimization as it reduces the call to Hive Meta Store
      during Update of Catalog)

> Avoid reloading a table's HMS data for file-only operations
> -----------------------------------------------------------
>
>                 Key: IMPALA-6994
>                 URL: https://issues.apache.org/jira/browse/IMPALA-6994
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Catalog
>    Affects Versions: Impala 2.12.0
>            Reporter: Balazs Jeszenszky
>            Assignee: Pranay Singh
>            Priority: Major
>
> Reloading file metadata for HDFS tables (e.g. as a final step in an 'insert') is done via
> https://github.com/apache/impala/blob/branch-2.12.0/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L628
> , which calls
> https://github.com/apache/impala/blob/branch-2.12.0/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java#L1243
> HdfsTable.load has no option to only load file metadata. HMS metadata will also be reloaded every time, which is an unnecessary overhead (and potential point of failure) when adding files to existing locations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org