[jira] [Updated] (IMPALA-9211) CreateTable with sync_ddl may fail with concurrent INVALIDATE METADATA

["Quanlong Huang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Quanlong Huang updated IMPALA-9211:
-----------------------------------
    Labels: concurrency  (was: )

> CreateTable with sync_ddl may fail with concurrent INVALIDATE METADATA
> ----------------------------------------------------------------------
>
>                 Key: IMPALA-9211
>                 URL: https://issues.apache.org/jira/browse/IMPALA-9211
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Catalog
>    Affects Versions: Impala 3.0, Impala 3.1.0, Impala 3.2.0, Impala 3.3.0
>            Reporter: Quanlong Huang
>            Assignee: Quanlong Huang
>            Priority: Major
>              Labels: concurrency
>             Fix For: Impala 3.4.0
>
>
> The execution flow for CreateTable is
> {code:java}
> hold metastoreDdlLock_
> |  create table in HMS
> |  if (success) add Table to catalog cache by calling CatalogServiceCatalog.addTable()
> |--free metastoreDdlLock_{code}
> The execution flow of CatalogServiceCatalog.addTable() is
> {code:java}
> get db from dbCache
> init an IncompleteTable for the table
> hold writeLock of versionLock_
> |   incr catalog version and assign it to the table
> |   add table to the db
> |-- free writeLock of versionLock_
> {code}
> The execution flow for INVALIDATE METADATA is (major part in reset()):
> {code:java}
> hold writeLock of versionLock_
> |   create a new dbCache
> |   for each db init a new item in new dbCache
> |     get table names in db from HMS
> |     for each table in the db
> |       init a new table object and put it into the dbCache
> |   replace original dbCache with the new one
> |-- free writeLock of versionLock_{code}
> *Buggy scenario:*
> {code:java}
> Thread-1(CreateTable):
>  - create table in HMS
>  - get db from dbCache
>  - waiting for writeLock of versionLock_
> Thread-2(Invalidate Metadata):
>  - holds the writeLock of versionLock_
>  - replacing dbCache with a new one.
>  - find the new table in HMS, create an IncompleteTable for it using catalogVersion=900
>  - add it into the new db object
> Thread-1(CreateTable):
>  - get the writeLock of versionLock_
>  - Incr catalog version to 1000 and assign it to the table.
>  - Add the table to the stale db object.
>  - Then wait in waitForSyncDdlVersion() for the update to be sent.
> Thread-3(catalog-update-gathering):
>  - get current catalog verion (1000) as toVersion.
>  - Collecting updates in version range (800, 1000].
>  - Sent the created table with catalogVersion=900.
> Thread-1(CreateTable):
>  - Find an older version (900) is sent.
>  - Keep waiting a verion > 1000 for this table to be sent. Let's say no other modifications on the table happen. It remains catalogVersion=900.
> Thread-3(catalog-update-gathering):
>  - Collecting updates in version range (1000, 1234].
>  - Do nothing for the table since its version not in range.
> {code}
> The created Table is added into a stale Db object. Reset() op already creates a new Db object. Thread-1 will finally run out of waiting attemps and throw an exception as described in IMPALA-9135.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org