[jira] [Commented] (IMPALA-7530) Re-plan queries on InconsistentMetadataException

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-7530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16618328#comment-16618328 ] 

ASF subversion and git services commented on IMPALA-7530:
---------------------------------------------------------

Commit 94cfdac664cbc9644c4873407783727ff0b87710 in impala's branch refs/heads/master from [~tlipcon]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=94cfdac ]

IMPALA-7530, IMPALA-7509. Re-plan queries if they fetch inconsistent metadata

Given the granular caching in CatalogdMetaProvider, it's possible to
have a situation like the following:

- impalad has cached the table list including some table "t"
- some other daemon has issued deletion of "t"
- we try to access "t" before the invalidation has reached us via the
  statestore

In this case, we'll get an error back when we try to fetch the table
"t". This can confuse the planning process since it previously
determined that the table exists. Note that this may occur either when
"t" is first referenced or later during planning (eg when fetching a
specific partition post-pruning), so it wouldn't be possible to simply
convert it into a 'table does not exist' result.

The solution here is to throw InconsistentMetadataFetchException after
invalidating the table list and associated object. This then triggers a
re-plan of the query.

This patch implements such re-planning up to 10 times before eventually
giving up. Given that each attempt invalidates the inconsistent cache we
would normally expect it to succeed on the first such retry. The limit
of 10 retries is just to avoid infinite loops in the case of a bug.

Change-Id: I17389954780fa22d7866224c17e128458fffa545
Reviewed-on: http://gerrit.cloudera.org:8080/11403
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Re-plan queries on InconsistentMetadataException
> ------------------------------------------------
>
>                 Key: IMPALA-7530
>                 URL: https://issues.apache.org/jira/browse/IMPALA-7530
>             Project: IMPALA
>          Issue Type: Sub-task
>            Reporter: Todd Lipcon
>            Assignee: Vuk Ercegovac
>            Priority: Critical
>
> In the case that the impalad has cached some information about a table, but the table has since been modified or deleted on the catalogd, the metaprovider can throw an InconsistentMetadataException after invalidating the cached information. On a re-plan, we'll see the new version and avoid the issue. The front-end needs to catch the exception and trigger the replan.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org