You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org> on 2018/09/28 07:05:27 UTC
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Vuk Ercegovac has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11539
Change subject: IMPALA-7599: make the number of local cache retries configurable
......................................................................
IMPALA-7599: make the number of local cache retries configurable
Under heavy read/write load, the number of retries needed for queries
in order to skip over inconsistent metadata exceptions needs to be set
higher. This change makes the number of retries configurable. It can be
set with the newly added flag --local_catalog_max_retries.
In addition, this change increases the default from 10 to 40, which was
sufficient when handling several workloads with high read/write load.
Made several fixes to exception messages.
Testing:
- manual tests
- added an e2e test that sets the flag and checks for inconsistent metadata
Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
---
M be/src/runtime/exec-env.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/common/impala_test_suite.py
M tests/custom_cluster/test_local_catalog.py
7 files changed, 66 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/11539/1
--
To view, visit http://gerrit.cloudera.org:8080/11539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Gerrit-Change-Number: 11539
Gerrit-PatchSet: 1
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11539 )
Change subject: IMPALA-7599: make the number of local cache retries configurable
......................................................................
Patch Set 6: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/11539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Gerrit-Change-Number: 11539
Gerrit-PatchSet: 6
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Sat, 29 Sep 2018 04:41:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11539 )
Change subject: IMPALA-7599: make the number of local cache retries configurable
......................................................................
Patch Set 1:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/11539/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11539/1//COMMIT_MSG@15
PS1, Line 15: Made several fixes to exception messages.
whats this?
http://gerrit.cloudera.org:8080/#/c/11539/1/be/src/runtime/exec-env.cc
File be/src/runtime/exec-env.cc:
http://gerrit.cloudera.org:8080/#/c/11539/1/be/src/runtime/exec-env.cc@97
PS1, Line 97: local_catalog_max_retries
Make it more clear, --local_catalog_max_metadata_fetch_retries / --local_catalog_max_plan_retries or something like that? Doesn't tell what is being retried.
http://gerrit.cloudera.org:8080/#/c/11539/1/be/src/runtime/exec-env.cc@99
PS1, Line 99: frontend
Just to be sure, do we plan to skip these retries for non-fe paths? Like directly accessing the Catalog
http://gerrit.cloudera.org:8080/#/c/11539/1/tests/custom_cluster/test_local_catalog.py
File tests/custom_cluster/test_local_catalog.py:
http://gerrit.cloudera.org:8080/#/c/11539/1/tests/custom_cluster/test_local_catalog.py@220
PS1, Line 220: Tests that the flag to limit the number of retries works
Maybe I'm missing something but we don't seem to be doing this? We set max_retries=0 and asserting total_ex_throws > 0.
http://gerrit.cloudera.org:8080/#/c/11539/1/tests/custom_cluster/test_local_catalog.py@251
PS1, Line 251:
else, do we need to re-throw and fail?
--
To view, visit http://gerrit.cloudera.org:8080/11539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Gerrit-Change-Number: 11539
Gerrit-PatchSet: 1
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Sep 2018 15:28:05 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Posted by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11539
to look at the new patch set (#2).
Change subject: IMPALA-7599: make the number of local cache retries configurable
......................................................................
IMPALA-7599: make the number of local cache retries configurable
Under heavy read/write load, the number of retries needed for queries
in order to skip over inconsistent metadata exceptions needs to be set
higher. This change makes the number of retries configurable. It can be
set with the newly added flag --local_catalog_max_retries.
In addition, this change increases the default from 10 to 40, which was
sufficient when handling several workloads with high read/write load.
Made several fixes to exception messages.
Testing:
- manual tests
- added an e2e test that sets the flag and checks for inconsistent metadata
Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
---
M be/src/runtime/exec-env.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/common/impala_test_suite.py
M tests/custom_cluster/test_local_catalog.py
8 files changed, 71 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/11539/2
--
To view, visit http://gerrit.cloudera.org:8080/11539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Gerrit-Change-Number: 11539
Gerrit-PatchSet: 2
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Posted by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org>.
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11539 )
Change subject: IMPALA-7599: make the number of local cache retries configurable
......................................................................
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/11539/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11539/3//COMMIT_MSG@16
PS3, Line 16:
> May be mention that all other retries (outside of Frontend) will come in a
Done. Note that most of these are from Frontend as well, but there are several cases outside as well.
http://gerrit.cloudera.org:8080/#/c/11539/3/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/11539/3/fe/src/main/java/org/apache/impala/service/Frontend.java@1090
PS3, Line 1090: timeline.markEvent(
: String.format("Retrying query planning due to inconsistent metadata "
: + "fetch, attempt %s of %s: ",
: attempt, INCONSISTENT_METADATA_NUM_RETRIES)
> While we are here, can we move this out of the loop? messes up run-time pro
avoided the repetition by printing a summary to the profile. note that it'll only be the last exception that's printed... conceivably, we may fail for a variety of reasons but I think its ok to get this level of detail from the logs.
--
To view, visit http://gerrit.cloudera.org:8080/11539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Gerrit-Change-Number: 11539
Gerrit-PatchSet: 3
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Sep 2018 23:42:53 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11539 )
Change subject: IMPALA-7599: make the number of local cache retries configurable
......................................................................
Patch Set 6:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3248/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/11539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Gerrit-Change-Number: 11539
Gerrit-PatchSet: 6
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Sat, 29 Sep 2018 00:42:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Posted by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org>.
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11539 )
Change subject: IMPALA-7599: make the number of local cache retries configurable
......................................................................
Patch Set 2:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/11539/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11539/1//COMMIT_MSG@15
PS1, Line 15: Made several fixes to exception messages.
> whats this?
whoops, added to a wrong branch... thanks for catching it.
when invalidate exception is thrown, the message was messed up due to {} instead of %s. the "name" of the object did not make sense either and was inconsistent with the other case where there this exception is thrown.
the change is in CatalogMetaProvider.
http://gerrit.cloudera.org:8080/#/c/11539/1/be/src/runtime/exec-env.cc
File be/src/runtime/exec-env.cc:
http://gerrit.cloudera.org:8080/#/c/11539/1/be/src/runtime/exec-env.cc@97
PS1, Line 97: local_catalog_max_fetch_r
> Make it more clear, --local_catalog_max_metadata_fetch_retries / --local_ca
added 'fetch'. metadata and catalog in the same name seems redundant.
http://gerrit.cloudera.org:8080/#/c/11539/1/be/src/runtime/exec-env.cc@99
PS1, Line 99: frontend
> Just to be sure, do we plan to skip these retries for non-fe paths? Like di
clarified that its use is for accessing the local catalog's objects.
http://gerrit.cloudera.org:8080/#/c/11539/1/tests/custom_cluster/test_local_catalog.py
File tests/custom_cluster/test_local_catalog.py:
http://gerrit.cloudera.org:8080/#/c/11539/1/tests/custom_cluster/test_local_catalog.py@220
PS1, Line 220: for c in [client1, client2]]
> Maybe I'm missing something but we don't seem to be doing this? We set max_
tightened this to check that we never retry (since the retries are set to 0).
http://gerrit.cloudera.org:8080/#/c/11539/1/tests/custom_cluster/test_local_catalog.py@251
PS1, Line 251:
> else, do we need to re-throw and fail?
the counter will be incorrect, so it'll fail. I'm not excluding other exceptions here.
--
To view, visit http://gerrit.cloudera.org:8080/11539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Gerrit-Change-Number: 11539
Gerrit-PatchSet: 2
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Sep 2018 16:49:26 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Posted by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11539
to look at the new patch set (#5).
Change subject: IMPALA-7599: make the number of local cache retries configurable
......................................................................
IMPALA-7599: make the number of local cache retries configurable
Under heavy read/write load, the number of retries needed for queries
in order to skip over inconsistent metadata exceptions needs to be set
higher. This change makes the number of retries configurable. It can be
set with the newly added flag --local_catalog_max_fetch_retries.
In addition, this change increases the default from 10 to 40, which was
sufficient when handling several workloads with high read/write load.
Follow-up change for IMPALA-7597 will make use of this configuration
when retrying for cases other than analyzing queries.
Made several fixes to exception messages.
Testing:
- manual tests
- added an e2e test that sets the flag and checks for inconsistent metadata
Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
---
M be/src/runtime/exec-env.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/common/impala_test_suite.py
M tests/custom_cluster/test_local_catalog.py
8 files changed, 93 insertions(+), 12 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/11539/5
--
To view, visit http://gerrit.cloudera.org:8080/11539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Gerrit-Change-Number: 11539
Gerrit-PatchSet: 5
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11539 )
Change subject: IMPALA-7599: make the number of local cache retries configurable
......................................................................
Patch Set 6: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/11539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Gerrit-Change-Number: 11539
Gerrit-PatchSet: 6
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Sat, 29 Sep 2018 00:42:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11539 )
Change subject: IMPALA-7599: make the number of local cache retries configurable
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/872/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Gerrit-Change-Number: 11539
Gerrit-PatchSet: 4
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Sat, 29 Sep 2018 00:00:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11539 )
Change subject: IMPALA-7599: make the number of local cache retries configurable
......................................................................
IMPALA-7599: make the number of local cache retries configurable
Under heavy read/write load, the number of retries needed for queries
in order to skip over inconsistent metadata exceptions needs to be set
higher. This change makes the number of retries configurable. It can be
set with the newly added flag --local_catalog_max_fetch_retries.
In addition, this change increases the default from 10 to 40, which was
sufficient when handling several workloads with high read/write load.
Follow-up change for IMPALA-7597 will make use of this configuration
when retrying for cases other than analyzing queries.
Made several fixes to exception messages.
Testing:
- manual tests
- added an e2e test that sets the flag and checks for inconsistent metadata
Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Reviewed-on: http://gerrit.cloudera.org:8080/11539
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/runtime/exec-env.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/common/impala_test_suite.py
M tests/custom_cluster/test_local_catalog.py
8 files changed, 93 insertions(+), 12 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/11539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Gerrit-Change-Number: 11539
Gerrit-PatchSet: 7
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11539 )
Change subject: IMPALA-7599: make the number of local cache retries configurable
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/861/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Gerrit-Change-Number: 11539
Gerrit-PatchSet: 1
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Sep 2018 07:55:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Posted by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11539
to look at the new patch set (#3).
Change subject: IMPALA-7599: make the number of local cache retries configurable
......................................................................
IMPALA-7599: make the number of local cache retries configurable
Under heavy read/write load, the number of retries needed for queries
in order to skip over inconsistent metadata exceptions needs to be set
higher. This change makes the number of retries configurable. It can be
set with the newly added flag --local_catalog_max_fetch_retries.
In addition, this change increases the default from 10 to 40, which was
sufficient when handling several workloads with high read/write load.
Made several fixes to exception messages.
Testing:
- manual tests
- added an e2e test that sets the flag and checks for inconsistent metadata
Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
---
M be/src/runtime/exec-env.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/common/impala_test_suite.py
M tests/custom_cluster/test_local_catalog.py
8 files changed, 71 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/11539/3
--
To view, visit http://gerrit.cloudera.org:8080/11539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Gerrit-Change-Number: 11539
Gerrit-PatchSet: 3
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Posted by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11539
to look at the new patch set (#4).
Change subject: IMPALA-7599: make the number of local cache retries configurable
......................................................................
IMPALA-7599: make the number of local cache retries configurable
Under heavy read/write load, the number of retries needed for queries
in order to skip over inconsistent metadata exceptions needs to be set
higher. This change makes the number of retries configurable. It can be
set with the newly added flag --local_catalog_max_fetch_retries.
In addition, this change increases the default from 10 to 40, which was
sufficient when handling several workloads with high read/write load.
Follow-up change for IMPALA-7597 will make use of this configuration
when retrying for cases other than analyzing queries.
Made several fixes to exception messages.
Testing:
- manual tests
- added an e2e test that sets the flag and checks for inconsistent metadata
Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
---
M be/src/runtime/exec-env.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/common/impala_test_suite.py
M tests/custom_cluster/test_local_catalog.py
8 files changed, 87 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/11539/4
--
To view, visit http://gerrit.cloudera.org:8080/11539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Gerrit-Change-Number: 11539
Gerrit-PatchSet: 4
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11539 )
Change subject: IMPALA-7599: make the number of local cache retries configurable
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/873/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Gerrit-Change-Number: 11539
Gerrit-PatchSet: 5
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Sat, 29 Sep 2018 01:27:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Posted by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org>.
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11539 )
Change subject: IMPALA-7599: make the number of local cache retries configurable
......................................................................
Patch Set 5: Code-Review+2
carry +2
--
To view, visit http://gerrit.cloudera.org:8080/11539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Gerrit-Change-Number: 11539
Gerrit-PatchSet: 5
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Sat, 29 Sep 2018 00:40:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11539 )
Change subject: IMPALA-7599: make the number of local cache retries configurable
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/865/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Gerrit-Change-Number: 11539
Gerrit-PatchSet: 2
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Sep 2018 17:22:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7599: make the number of local cache retries configurable
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11539 )
Change subject: IMPALA-7599: make the number of local cache retries configurable
......................................................................
Patch Set 3: Code-Review+2
(2 comments)
http://gerrit.cloudera.org:8080/#/c/11539/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11539/3//COMMIT_MSG@16
PS3, Line 16:
May be mention that all other retries (outside of Frontend) will come in a separate patch?
http://gerrit.cloudera.org:8080/#/c/11539/3/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/11539/3/fe/src/main/java/org/apache/impala/service/Frontend.java@1090
PS3, Line 1090: timeline.markEvent(
: String.format("Retrying query planning due to inconsistent metadata "
: + "fetch, attempt %s of %s: ",
: attempt, INCONSISTENT_METADATA_NUM_RETRIES)
While we are here, can we move this out of the loop? messes up run-time profile if logged for 40 times. May be we include the number of retries it took for it to be successful?
--
To view, visit http://gerrit.cloudera.org:8080/11539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f14d5a8728f3cb07c7710589c44c2cd52478ba8
Gerrit-Change-Number: 11539
Gerrit-PatchSet: 3
Gerrit-Owner: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Sep 2018 23:17:12 +0000
Gerrit-HasComments: Yes