[jira] [Resolved] (IMPALA-5325) Update HdfsTable#totalHdfsBytes_ and HdfsTable#numHdfsFiles_ atomically.

["bharath v (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

bharath v resolved IMPALA-5325.
-------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 2.8.0

 Author: Bharath Vissapragada <bh...@cloudera.com>
  Date:   2017-05-26 (Fri, 26 May 2017)

  Changed paths:
    M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
    M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java

  Log Message:
  -----------
  IMPALA-5325: Do not update totalHdfsBytes_/numHdfsFiles_ on Catalogd

We need not account these hdfs table metrics on the Catalog
server as they are eventually calculated again on the Impalads
while unpacking the corresponding thrift table.

This fix can potentially affect the frontend tests that directly load
the Catalog's version of HdfsTable without the loadFromThrift() call
paths that do the accounting. That is fixed by adding a separate call
that computes these values and is called from
ImpaladTestCatalog.getTable().

Testing: Enough tests already cover these code paths like show stats/
table or partition refresh tests etc. No new tests are added.

Change-Id: I03cc6f9e9b2c03cafb87029ea0802dfdb2745be1
Reviewed-on: http://gerrit.cloudera.org:8080/6970
Reviewed-by: Bharath Vissapragada <bh...@cloudera.com>
Tested-by: Impala Public Jenkins

> Update HdfsTable#totalHdfsBytes_ and HdfsTable#numHdfsFiles_ atomically.
> ------------------------------------------------------------------------
>
>                 Key: IMPALA-5325
>                 URL: https://issues.apache.org/jira/browse/IMPALA-5325
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Catalog
>    Affects Versions: Impala 2.8.0, Impala 2.9.0
>            Reporter: bharath v
>            Assignee: bharath v
>             Fix For: Impala 2.8.0
>
>
> In the following snippet from HdfsTable.java
> {noformat}
>  private void loadPartitionFileMetadata(StorageDescriptor storageDescriptor,
>       HdfsPartition partition) throws Exception {
>     Preconditions.checkNotNull(storageDescriptor);
>     Preconditions.checkNotNull(partition);
>     Path partDirPath = new Path(storageDescriptor.getLocation());
>     FileSystem fs = partDirPath.getFileSystem(CONF);
>     if (!fs.exists(partDirPath)) return;
>     numHdfsFiles_ -= partition.getNumFileDescriptors();
>     totalHdfsBytes_ -= partition.getSize();
>     Preconditions.checkState(numHdfsFiles_ >= 0 && totalHdfsBytes_ >= 0);
>     refreshFileMetadata(partition);
>   }
> {noformat}
> If {{refreshFileMetadata}} throws an exception before updating {{numHdfsFiles_}} and {{totalHdfsBytes_}}, the table could get into an inconsistent state and could possibly hit a preconditions check in the next block metadata update.
> Fix: Update {{totalHdfsBytes_}} and {{numHdfsFiles_}} in an atomic fashion.
> Workaround: Run "invalidate table" to clear the full table state.
> This bug also affects older versions around method {{updatePartitionFds}}
> This issue can manifest itself as eventually hitting the Preconditions check, leading to this error message:
> {code}
> TableLoadingException: Failed to load metadata for table: t CAUSED BY: IllegalStateException: null 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)