[Impala-ASF-CR] IMPALA-5476: Fix catalogd restart brings stale metadata

["liuyao (Code Review)" <ge...@cloudera.org>]

Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#25).

Change subject: IMPALA-5476: Fix catalogd restart brings stale metadata
......................................................................

IMPALA-5476: Fix catalogd restart brings stale metadata

ImpaladCatalog#updateCatalog() doesn't trigger a full topic update
request when detecting catalogServiceId changes. It just updates the
local catalogServiceId and throws an exception to abort applying the
DDL/DML results. This causes a problem when catalogd is restarted and
the DDL/DML is executed on the restarted instance. In this case, only
the local catalogServiceId is updated to the latest. The local catalog
remains stale. Then when dealing with the following updates from
statestore, the catalogServiceId always matches, so updates will be
applied without exceptions. However, the catalog objects usually won't
be updated since they have higher versions (from the old catalogd
instance) than those in the update. This brings the local catalog out
of sync until the catalog version of the new catalogd grows larger
enough.

Note that in dealing with the catalog updates from statestore, if the
catalogServiceId unmatches, impalad will request a full topic update.
See more in ImpalaServer::CatalogUpdateCallback().

This patch fixes this issue by checking the catalogServiceId before
invoking UpdateCatalogCache() of FE. If catalogServiceId doesn't match
the one in the DDL/DML result, wait until it changes. The following
update from statestore will change it and unblocks the DDL/DML thread.

Testing

add several tests in
tests/custom_cluster/test_restart_services.py

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 150 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/25
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 25
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>