[jira] [Created] (IMPALA-9214) REFRESH with sync_ddl may fail with concurrent INVALIDATE METADATA

["Quanlong Huang (Jira)" <ji...@apache.org>]

Quanlong Huang created IMPALA-9214:
--------------------------------------

             Summary: REFRESH with sync_ddl may fail with concurrent INVALIDATE METADATA
                 Key: IMPALA-9214
                 URL: https://issues.apache.org/jira/browse/IMPALA-9214
             Project: IMPALA
          Issue Type: Bug
          Components: Catalog
            Reporter: Quanlong Huang


The call trace for executing a REFRESH statement in Catalogd is
{code:java}
JniCatalog#resetMetadata
  CatalogOpExecutor#execResetMetadata
    CatalogServiceCatalog#reloadTable
    CatalogServiceCatalog#waitForSyncDdlVersion
{code}
In CatalogServiceCatalog#reloadTable(), the {{Tbl}} object may be stale if there's a concurrent reset, i.e. INVALIDATE METADATA, running. Then {{CatalogServiceCatalog#reloadTable}} will return the thrift object of a stale Table. It can't be found in the catalog cache and the {{topicUpdateLog_}}, so {{waitForSyncDdlVersion}} will finally hang or run out of attempts.

Here is an example. Let's say table1 is an unpartitioned table and is loaded. Two queries, "Refresh table1" and "Invalidate metadata" are running concurrently.

Thread-1 (Refresh):
 * Gets the {{Table}} object in CatalogServiceCatalog#execResetMetadata and goes into {{reloadTable}}. The catalog version of table1 is 50.
 * Waiting for both version lock and table lock here: [https://github.com/apache/impala/blob/a1588e44980c648cb7f9263cbd0409abfbaeacf7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2023]

Thread-2 (Invalidate Metadata):
 * Holds the version lock and replace the whole catalog cache with a new one. Makes all existing catalog objects stale. Now the catalog version of table1 is 90.
 * Release the version lock.

Thread-1 (Refresh):
 * Gets the version lock and table lock
 * Get a new catalog version, let's say 100. Then release version lock.
 * Load the metadata into the stale Table object. Bump its catalog version from 50 to 100.
 * Return the thrift object of the updated stale object from {{reloadTable}}
 * Goes into {{waitForSyncDdlVersion}}. Wait for an update of table1 is sent and the sent version >= 100.

However, table1 in the catalog cache is with version 90. Unless there's another update on this table, Thread-1 will hang or run out of attempts for waiting the expected update.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org