You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org> on 2022/03/04 10:31:52 UTC
[Impala-ASF-CR] IMPALA-11153: Expose LOCK RETRIES/LOCK RETRY WAIT SECONDS settings for user
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18289
Change subject: IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user
......................................................................
IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user
Currently Impala is using private static final values for lock
retry/wait:
// Number of retries to acquire an HMS ACID lock.
private static final int LOCK_RETRIES = 10;
// Time interval between retries of acquiring an HMS ACID lock
private static final int LOCK_RETRY_WAIT_SECONDS = 3;
This patch makes the above values tunable via query options.
Testing:
* added e2e tests
Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/query_test/test_acid.py
10 files changed, 143 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/18289/1
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gergely Fürnstáhl, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18289
to look at the new patch set (#6).
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
IMPALA-11153: Make lock wait time configurable for the users
Currently Impala is using private static final values for lock
retry/wait:
// Number of retries to acquire an HMS ACID lock.
private static final int LOCK_RETRIES = 10;
// Time interval between retries of acquiring an HMS ACID lock
private static final int LOCK_RETRY_WAIT_SECONDS = 3;
This patch changes the logic for waiting on locks. It does an
exponential backoff starting from 50 ms up to 30 seconds, and the
users can configure a maximum total wait time for the locks.
Testing:
* added e2e tests
Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
---
M be/src/exec/catalog-op-executor.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/query_test/test_acid.py
11 files changed, 153 insertions(+), 58 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/18289/6
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 11: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 11
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Apr 2022 19:53:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10355/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Mar 2022 16:50:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 8: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7996/
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 8
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 31 Mar 2022 17:20:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 10: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8010/
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 02 Apr 2022 17:32:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 5: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18289/5/common/thrift/CatalogService.thrift
File common/thrift/CatalogService.thrift:
http://gerrit.cloudera.org:8080/#/c/18289/5/common/thrift/CatalogService.thrift@161
PS5, Line 161: // True if SYNC_DDL is set in query options
: 22: required bool sync_ddl
Sync_ddl also comes from a query option
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Mar 2022 15:41:27 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 9:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8007/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 9
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Apr 2022 20:10:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gergely Fürnstáhl, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18289
to look at the new patch set (#12).
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
IMPALA-11153: Make lock wait time configurable for the users
Currently Impala is using private static final values for lock
retry/wait:
// Number of retries to acquire an HMS ACID lock.
private static final int LOCK_RETRIES = 10;
// Time interval between retries of acquiring an HMS ACID lock
private static final int LOCK_RETRY_WAIT_SECONDS = 3;
This patch changes the logic for waiting on locks. It does an
exponential backoff starting from 50 ms up to 30 seconds, and the
users can configure a maximum total wait time for the locks. This
total maximum wait time is 5 minutes (the earlier 30 seconds
was too short in real use cases).
Testing:
* added e2e tests
Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
---
M be/src/exec/catalog-op-executor.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/catalog/AlterDatabaseTest.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/query_test/test_acid.py
13 files changed, 179 insertions(+), 58 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/18289/12
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 12
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 8: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 8
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 31 Mar 2022 12:56:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 4:
(4 comments)
Thanks for the comments!
http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
File fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java:
http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java@620
PS2, Line 620: * lock doesn't belong to a transaction.
> nit: alignment is off here, but as I see we don't have a consistent style f
Done
http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@483
PS2, Line 483: ddlRequest.getLock_max_wait_time_s());
> Just an idea to make passing these parameters simpler: the params in ddlReq
I haven't done it in the current PS as I decreased the number of new query options.
But I can do it in a new PS if you still think it'd make sense.
http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/Frontend.java@2195
PS2, Line 2195:
> missing comment of queryOptions param
Done
http://gerrit.cloudera.org:8080/#/c/18289/2/tests/query_test/test_acid.py
File tests/query_test/test_acid.py:
http://gerrit.cloudera.org:8080/#/c/18289/2/tests/query_test/test_acid.py@334
PS2, Line 334: assert elapsed < 5
> this should be < 3 I think, as >3 && < 5 would mean we had 1 retry
I changed the code since then. Now there's only one query option.
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Mar 2022 17:58:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 12:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10392/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 12
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Apr 2022 16:14:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Expose LOCK RETRIES/LOCK RETRY WAIT SECONDS settings for user
Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user
......................................................................
Patch Set 2:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
File fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java:
http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java@620
PS2, Line 620: * @param lockWaitSeconds Time interval between retries of acquiring an HMS ACID lock.
nit: alignment is off here, but as I see we don't have a consistent style for it
http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/Frontend.java@2195
PS2, Line 2195:
missing comment of queryOptions param
http://gerrit.cloudera.org:8080/#/c/18289/2/tests/query_test/test_acid.py
File tests/query_test/test_acid.py:
http://gerrit.cloudera.org:8080/#/c/18289/2/tests/query_test/test_acid.py@334
PS2, Line 334: assert elapsed < 5
this should be < 3 I think, as >3 && < 5 would mean we had 1 retry
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Mar 2022 14:38:31 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gergely Fürnstáhl, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18289
to look at the new patch set (#4).
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
IMPALA-11153: Make lock wait time configurable for the users
Currently Impala is using private static final values for lock
retry/wait:
// Number of retries to acquire an HMS ACID lock.
private static final int LOCK_RETRIES = 10;
// Time interval between retries of acquiring an HMS ACID lock
private static final int LOCK_RETRY_WAIT_SECONDS = 3;
This patch changes the logic for waiting on locks. It does an
exponential backoff starting from 100 ms up to 30 seconds, and the
users can configure a maximum total wait time for the locks.
Testing:
* added e2e tests
Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/query_test/test_acid.py
10 files changed, 131 insertions(+), 40 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/18289/4
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 14:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8028/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 14
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Apr 2022 12:20:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 8:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7996/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 8
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 31 Mar 2022 12:56:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Expose LOCK RETRIES/LOCK RETRY WAIT SECONDS settings for user
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10347/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Mar 2022 15:00:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18289/5/common/thrift/CatalogService.thrift
File common/thrift/CatalogService.thrift:
http://gerrit.cloudera.org:8080/#/c/18289/5/common/thrift/CatalogService.thrift@161
PS5, Line 161: // Parameters for GRANT/REVOKE privilege
: 21: optional JniCatalog.TG
> Sync_ddl also comes from a query option
Done
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Mar 2022 16:30:59 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 7: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Mar 2022 11:29:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gergely Fürnstáhl, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18289
to look at the new patch set (#9).
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
IMPALA-11153: Make lock wait time configurable for the users
Currently Impala is using private static final values for lock
retry/wait:
// Number of retries to acquire an HMS ACID lock.
private static final int LOCK_RETRIES = 10;
// Time interval between retries of acquiring an HMS ACID lock
private static final int LOCK_RETRY_WAIT_SECONDS = 3;
This patch changes the logic for waiting on locks. It does an
exponential backoff starting from 50 ms up to 30 seconds, and the
users can configure a maximum total wait time for the locks.
Testing:
* added e2e tests
Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
---
M be/src/exec/catalog-op-executor.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/catalog/AlterDatabaseTest.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/query_test/test_acid.py
13 files changed, 178 insertions(+), 58 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/18289/9
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 9
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11153: Expose LOCK RETRIES/LOCK RETRY WAIT SECONDS settings for user
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user
......................................................................
Patch Set 3:
(1 comment)
PS3 is only a rebase.
http://gerrit.cloudera.org:8080/#/c/18289/2/common/thrift/CatalogService.thrift
File common/thrift/CatalogService.thrift:
http://gerrit.cloudera.org:8080/#/c/18289/2/common/thrift/CatalogService.thrift@171
PS2, Line 171: lock_retry_wait_seconds
> Aren't we pessimistic about "seconds"? I don't know how much overhead do we
From the name of the config at first I thought Hive uses 1 minute sleep interval...
Then I realized it is only a maximum value, and Hive does exponential backoff: https://github.com/apache/hive/blob/7b3ecf617a6d46f48a3b6f77e0339fd4ad95a420/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbLockManager.java#L95-L97
I think it would be reasonable for us as well, so I switched to a backoff logic.
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Mar 2022 14:39:43 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gergely Fürnstáhl, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18289
to look at the new patch set (#5).
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
IMPALA-11153: Make lock wait time configurable for the users
Currently Impala is using private static final values for lock
retry/wait:
// Number of retries to acquire an HMS ACID lock.
private static final int LOCK_RETRIES = 10;
// Time interval between retries of acquiring an HMS ACID lock
private static final int LOCK_RETRY_WAIT_SECONDS = 3;
This patch changes the logic for waiting on locks. It does an
exponential backoff starting from 50 ms up to 30 seconds, and the
users can configure a maximum total wait time for the locks.
Testing:
* added e2e tests
Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
---
M be/src/exec/catalog-op-executor.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/query_test/test_acid.py
11 files changed, 144 insertions(+), 49 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/18289/5
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10385/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 02 Apr 2022 13:28:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 10:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/18289/10/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/18289/10/common/thrift/Query.thrift@593
PS10, Line 593: 300
I think this should be 30
http://gerrit.cloudera.org:8080/#/c/18289/10/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
File fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java:
http://gerrit.cloudera.org:8080/#/c/18289/10/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java@107
PS10, Line 107: private static final long MAX_SLEEP_INTERVAL_MS = 30000;
Do we need this? It is provided by the caller.
http://gerrit.cloudera.org:8080/#/c/18289/10/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java@655
PS10, Line 655: sleepIntervalMs = Math.min(MAX_SLEEP_INTERVAL_MS,
: sleepIntervalMs * 2);
We don't really need Math.min here, overshooting will be catched on line 649 anyways.
http://gerrit.cloudera.org:8080/#/c/18289/10/tests/query_test/test_acid.py
File tests/query_test/test_acid.py:
http://gerrit.cloudera.org:8080/#/c/18289/10/tests/query_test/test_acid.py@327
PS10, Line 327: try:
could add a test case before the first set, to check 30
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Apr 2022 15:34:21 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gergely Fürnstáhl, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18289
to look at the new patch set (#13).
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
IMPALA-11153: Make lock wait time configurable for the users
Currently Impala is using private static final values for lock
retry/wait:
// Number of retries to acquire an HMS ACID lock.
private static final int LOCK_RETRIES = 10;
// Time interval between retries of acquiring an HMS ACID lock
private static final int LOCK_RETRY_WAIT_SECONDS = 3;
This patch changes the logic for waiting on locks. It does an
exponential backoff starting from 50 ms up to 30 seconds, and the
users can configure a maximum total wait time for the locks. This
total maximum wait time is 5 minutes (the earlier 30 seconds
was too short in real use cases).
Testing:
* added e2e tests
Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
---
M be/src/exec/catalog-op-executor.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/catalog/AlterDatabaseTest.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/query_test/test_acid.py
13 files changed, 182 insertions(+), 58 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/18289/13
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 13
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 13: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 13
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Apr 2022 12:11:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 14: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 14
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Apr 2022 16:46:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Expose LOCK RETRIES/LOCK RETRY WAIT SECONDS settings for user
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18289/2/common/thrift/CatalogService.thrift
File common/thrift/CatalogService.thrift:
http://gerrit.cloudera.org:8080/#/c/18289/2/common/thrift/CatalogService.thrift@171
PS2, Line 171: lock_retry_wait_seconds
Aren't we pessimistic about "seconds"? I don't know how much overhead do we add with the RPCs to Hive, but I think that smaller retry times like 200 ms could be reasonable.
We could use lock_retry_wait_ms instead.
http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@483
PS2, Line 483: ddlRequest.getLock_max_retries(), ddlRequest.getLock_retry_wait_seconds()
Just an idea to make passing these parameters simpler: the params in ddlRequest that come from query options could be collected to a struct and that could be passed here.
e.g. it could contain debug_action too, which may become useful in the future in RPCs
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Mar 2022 17:42:03 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11153: Expose LOCK RETRIES/LOCK RETRY WAIT SECONDS settings for user
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10252/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Mar 2022 11:04:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10348/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Mar 2022 18:17:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 5:
(2 comments)
Thanks for the comments!
http://gerrit.cloudera.org:8080/#/c/18289/4/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
File fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java:
http://gerrit.cloudera.org:8080/#/c/18289/4/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java@641
PS4, Line 641: long sleepIntervalMs = 100;
> noticed that Hive starts with 50 ms - maybe we should use the same?
Done
http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@483
PS2, Line 483: ddlRequest.getQuery_options().getLock_max_wait_time_s());
> I still think that collecting query options that are passed in TDdlExecRequ
Done
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Mar 2022 15:39:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11153: Expose LOCK RETRIES/LOCK RETRY WAIT SECONDS settings for user
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10250/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Mar 2022 10:51:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 13: Code-Review+1
Looks good to me
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 13
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Apr 2022 09:06:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Expose LOCK RETRIES/LOCK RETRY WAIT SECONDS settings for user
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gergely Fürnstáhl, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18289
to look at the new patch set (#3).
Change subject: IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user
......................................................................
IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user
Currently Impala is using private static final values for lock
retry/wait:
// Number of retries to acquire an HMS ACID lock.
private static final int LOCK_RETRIES = 10;
// Time interval between retries of acquiring an HMS ACID lock
private static final int LOCK_RETRY_WAIT_SECONDS = 3;
This patch makes the above values tunable via query options.
Testing:
* added e2e tests
Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/query_test/test_acid.py
10 files changed, 142 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/18289/3
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 10:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8010/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 02 Apr 2022 13:09:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 10:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/18289/10/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/18289/10/common/thrift/Query.thrift@593
PS10, Line 593: 300
> I think this should be 30
Yeah, sorry, I didn't mention it in the comment message (until now), but I decided to increase the total lock wait time, as 30 seconds was too short in practice.
http://gerrit.cloudera.org:8080/#/c/18289/10/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
File fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java:
http://gerrit.cloudera.org:8080/#/c/18289/10/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java@107
PS10, Line 107: private static final long MAX_SLEEP_INTERVAL_MS = 30000;
> Do we need this? It is provided by the caller.
The caller provides the max total wait time. This is the sleep interval between 'checkLock()' calls.
http://gerrit.cloudera.org:8080/#/c/18289/10/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java@655
PS10, Line 655: sleepIntervalMs = Math.min(MAX_SLEEP_INTERVAL_MS,
: sleepIntervalMs * 2);
> We don't really need Math.min here, overshooting will be catched on line 64
This limits the sleep interval to 30 seconds (MAX_SLEEP_INTERVAL_MS).
At L649 we limit the sleep interval to the time left until 'timeoutTime'.
So if the user sets a very long timeout time (1 hour), then we'd sleep more than MAX_SLEEP_INTERVAL_MS without this Math.min().
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Apr 2022 15:53:37 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10383/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 9
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Apr 2022 20:29:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10354/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Mar 2022 15:58:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 7:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7981/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Mar 2022 11:29:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 14: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 14
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Apr 2022 12:20:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Expose LOCK RETRIES/LOCK RETRY WAIT SECONDS settings for user
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18289
to look at the new patch set (#2).
Change subject: IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user
......................................................................
IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user
Currently Impala is using private static final values for lock
retry/wait:
// Number of retries to acquire an HMS ACID lock.
private static final int LOCK_RETRIES = 10;
// Time interval between retries of acquiring an HMS ACID lock
private static final int LOCK_RETRY_WAIT_SECONDS = 3;
This patch makes the above values tunable via query options.
Testing:
* added e2e tests
Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/query_test/test_acid.py
10 files changed, 142 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/18289/2
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 11:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8018/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 11
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Apr 2022 15:23:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gergely Fürnstáhl, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18289
to look at the new patch set (#10).
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
IMPALA-11153: Make lock wait time configurable for the users
Currently Impala is using private static final values for lock
retry/wait:
// Number of retries to acquire an HMS ACID lock.
private static final int LOCK_RETRIES = 10;
// Time interval between retries of acquiring an HMS ACID lock
private static final int LOCK_RETRY_WAIT_SECONDS = 3;
This patch changes the logic for waiting on locks. It does an
exponential backoff starting from 50 ms up to 30 seconds, and the
users can configure a maximum total wait time for the locks.
Testing:
* added e2e tests
Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
---
M be/src/exec/catalog-op-executor.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/catalog/AlterDatabaseTest.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/query_test/test_acid.py
13 files changed, 179 insertions(+), 58 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/18289/10
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 10:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18289/10/tests/query_test/test_acid.py
File tests/query_test/test_acid.py:
http://gerrit.cloudera.org:8080/#/c/18289/10/tests/query_test/test_acid.py@327
PS10, Line 327: try:
> could add a test case before the first set, to check 30
So the default is now 5 min and I didn't want the test to take too long. But now I've added a check in exhaustive test runs.
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Apr 2022 15:56:02 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 13:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10393/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 13
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Apr 2022 16:15:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 9: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8007/
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 9
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 02 Apr 2022 00:40:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
IMPALA-11153: Make lock wait time configurable for the users
Currently Impala is using private static final values for lock
retry/wait:
// Number of retries to acquire an HMS ACID lock.
private static final int LOCK_RETRIES = 10;
// Time interval between retries of acquiring an HMS ACID lock
private static final int LOCK_RETRY_WAIT_SECONDS = 3;
This patch changes the logic for waiting on locks. It does an
exponential backoff starting from 50 ms up to 30 seconds, and the
users can configure a maximum total wait time for the locks. This
total maximum wait time is 5 minutes (the earlier 30 seconds
was too short in real use cases).
Testing:
* added e2e tests
Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Reviewed-on: http://gerrit.cloudera.org:8080/18289
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/catalog-op-executor.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/catalog/AlterDatabaseTest.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/query_test/test_acid.py
13 files changed, 182 insertions(+), 58 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 15
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 4: Code-Review+2
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18289/4/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
File fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java:
http://gerrit.cloudera.org:8080/#/c/18289/4/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java@641
PS4, Line 641: long sleepIntervalMs = 100;
noticed that Hive starts with 50 ms - maybe we should use the same?
https://github.com/apache/hive/blob/7b3ecf617a6d46f48a3b6f77e0339fd4ad95a420/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbLockManager.java#L93
http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@483
PS2, Line 483: ddlRequest.getLock_max_wait_time_s());
> I haven't done it in the current PS as I decreased the number of new query
I still think that collecting query options that are passed in TDdlExecRequest in a struct would be nice in the long term.
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Mar 2022 10:53:09 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 6: Code-Review+2
Carry +2
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Mar 2022 11:29:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )
Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................
Patch Set 7: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7981/
--
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Mar 2022 15:53:47 +0000
Gerrit-HasComments: No