[jira] [Commented] (IMPALA-9211) CreateTable with sync_ddl may fail with concurrent INVALIDATE METADATA

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989228#comment-16989228 ] 

ASF subversion and git services commented on IMPALA-9211:
---------------------------------------------------------

Commit f9a52ca0e0cdfd7c23a36273b557c6385827b71b in impala's branch refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=f9a52ca ]

IMPALA-9211: Fix adding new table to stale db due to concurrent reset

When adding a new table to the catalog, we first get the db object from
dbCache and then add the table into it. When doing reset(), i.e. global
INVALIDATE METADATA, we replace the whole dbCache with a new one and
load db and table names from HMS. These two operations have race
conflicts so should both be protected by a exclusive lock, i.e. write
lock of version lock.

Currently, CatalogServiceCatalog.addTable() does not get the db object
within the write lock, which may get a stale db object and add new table
into it. This patch moves the operations into the protection of write
lock.

Tests:
 - Ran test_concurrent_ddls.py without errors for CreateTable like
   IMPALA-9135 found.
 - Ran Core tests

Change-Id: I83d2e5f00eabe61a42c948ec1685ce29cdea1592
Reviewed-on: http://gerrit.cloudera.org:8080/14820
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> CreateTable with sync_ddl may fail with concurrent INVALIDATE METADATA
> ----------------------------------------------------------------------
>
>                 Key: IMPALA-9211
>                 URL: https://issues.apache.org/jira/browse/IMPALA-9211
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Catalog
>            Reporter: Quanlong Huang
>            Assignee: Quanlong Huang
>            Priority: Major
>
> The execution flow for CreateTable is
> {code:java}
> hold metastoreDdlLock_
> |  create table in HMS
> |  if (success) add Table to catalog cache by calling CatalogServiceCatalog.addTable()
> |--free metastoreDdlLock_{code}
> The execution flow of CatalogServiceCatalog.addTable() is
> {code:java}
> get db from dbCache
> init an IncompleteTable for the table
> hold writeLock of versionLock_
> |   incr catalog version and assign it to the table
> |   add table to the db
> |-- free writeLock of versionLock_
> {code}
> The execution flow for INVALIDATE METADATA is (major part in reset()):
> {code:java}
> hold writeLock of versionLock_
> |   create a new dbCache
> |   for each db init a new item in new dbCache
> |     get table names in db from HMS
> |     for each table in the db
> |       init a new table object and put it into the dbCache
> |   replace original dbCache with the new one
> |-- free writeLock of versionLock_{code}
> *Buggy scenario:*
> {code:java}
> Thread-1(CreateTable):
>  - create table in HMS
>  - get db from dbCache
>  - waiting for writeLock of versionLock_
> Thread-2(Invalidate Metadata):
>  - holds the writeLock of versionLock_
>  - replacing dbCache with a new one.
>  - find the new table in HMS, create an IncompleteTable for it using catalogVersion=900
>  - add it into the new db object
> Thread-1(CreateTable):
>  - get the writeLock of versionLock_
>  - Incr catalog version to 1000 and assign it to the table.
>  - Add the table to the stale db object.
>  - Then wait in waitForSyncDdlVersion() for the update to be sent.
> Thread-3(catalog-update-gathering):
>  - get current catalog verion (1000) as toVersion.
>  - Collecting updates in version range (800, 1000].
>  - Sent the created table with catalogVersion=900.
> Thread-1(CreateTable):
>  - Find an older version (900) is sent.
>  - Keep waiting a verion > 1000 for this table to be sent. Let's say no other modifications on the table happen. It remains catalogVersion=900.
> Thread-3(catalog-update-gathering):
>  - Collecting updates in version range (1000, 1234].
>  - Do nothing for the table since its version not in range.
> {code}
> The created Table is added into a stale Db object. Reset() op already creates a new Db object. Thread-1 will finally run out of waiting attemps and throw an exception as described in IMPALA-9135.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org