You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org> on 2022/03/04 10:31:52 UTC

[Impala-ASF-CR] IMPALA-11153: Expose LOCK RETRIES/LOCK RETRY WAIT SECONDS settings for user

Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18289


Change subject: IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user
......................................................................

IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user

Currently Impala is using private static final values for lock
retry/wait:

 // Number of retries to acquire an HMS ACID lock.
 private static final int LOCK_RETRIES = 10;

 // Time interval between retries of acquiring an HMS ACID lock
 private static final int LOCK_RETRY_WAIT_SECONDS = 3;

This patch makes the above values tunable via query options.

Testing:
 * added e2e tests

Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/query_test/test_acid.py
10 files changed, 143 insertions(+), 31 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/18289/1
-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gergely Fürnstáhl, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18289

to look at the new patch set (#6).

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................

IMPALA-11153: Make lock wait time configurable for the users

Currently Impala is using private static final values for lock
retry/wait:

 // Number of retries to acquire an HMS ACID lock.
 private static final int LOCK_RETRIES = 10;

 // Time interval between retries of acquiring an HMS ACID lock
 private static final int LOCK_RETRY_WAIT_SECONDS = 3;

This patch changes the logic for waiting on locks. It does an
exponential backoff starting from 50 ms up to 30 seconds, and the
users can configure a maximum total wait time for the locks.

Testing:
 * added e2e tests

Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
---
M be/src/exec/catalog-op-executor.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/query_test/test_acid.py
11 files changed, 153 insertions(+), 58 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/18289/6
-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 11: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 11
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Apr 2022 19:53:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10355/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Mar 2022 16:50:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 8: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7996/


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 8
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 31 Mar 2022 17:20:22 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 10: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8010/


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 02 Apr 2022 17:32:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 5: Code-Review+2

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18289/5/common/thrift/CatalogService.thrift
File common/thrift/CatalogService.thrift:

http://gerrit.cloudera.org:8080/#/c/18289/5/common/thrift/CatalogService.thrift@161
PS5, Line 161:   // True if SYNC_DDL is set in query options
             :   22: required bool sync_ddl
Sync_ddl also comes from a query option



-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Mar 2022 15:41:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 9:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8007/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 9
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Apr 2022 20:10:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gergely Fürnstáhl, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18289

to look at the new patch set (#12).

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................

IMPALA-11153: Make lock wait time configurable for the users

Currently Impala is using private static final values for lock
retry/wait:

 // Number of retries to acquire an HMS ACID lock.
 private static final int LOCK_RETRIES = 10;

 // Time interval between retries of acquiring an HMS ACID lock
 private static final int LOCK_RETRY_WAIT_SECONDS = 3;

This patch changes the logic for waiting on locks. It does an
exponential backoff starting from 50 ms up to 30 seconds, and the
users can configure a maximum total wait time for the locks. This
total maximum wait time is 5 minutes (the earlier 30 seconds
was too short in real use cases).

Testing:
 * added e2e tests

Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
---
M be/src/exec/catalog-op-executor.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/catalog/AlterDatabaseTest.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/query_test/test_acid.py
13 files changed, 179 insertions(+), 58 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/18289/12
-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 12
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 8: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 8
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 31 Mar 2022 12:56:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 4:

(4 comments)

Thanks for the comments!

http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
File fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java:

http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java@620
PS2, Line 620:    *                             lock doesn't belong to a transaction.
> nit: alignment is off here, but as I see we don't have a consistent style f
Done


http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@483
PS2, Line 483: ddlRequest.getLock_max_wait_time_s());
> Just an idea to make passing these parameters simpler: the params in ddlReq
I haven't done it in the current PS as I decreased the number of new query options.

But I can do it in a new PS if you still think it'd make sense.


http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:

http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/Frontend.java@2195
PS2, Line 2195:  
> missing comment of queryOptions param
Done


http://gerrit.cloudera.org:8080/#/c/18289/2/tests/query_test/test_acid.py
File tests/query_test/test_acid.py:

http://gerrit.cloudera.org:8080/#/c/18289/2/tests/query_test/test_acid.py@334
PS2, Line 334:       assert elapsed < 5
> this should be < 3 I think, as >3 && < 5 would mean we had 1 retry
I changed the code since then. Now there's only one query option.



-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Mar 2022 17:58:34 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 12:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10392/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 12
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Apr 2022 16:14:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Expose LOCK RETRIES/LOCK RETRY WAIT SECONDS settings for user

Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user
......................................................................


Patch Set 2:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
File fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java:

http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java@620
PS2, Line 620:    * @param lockWaitSeconds Time interval between retries of acquiring an HMS ACID lock.
nit: alignment is off here, but as I see we don't have a consistent style for it


http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:

http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/Frontend.java@2195
PS2, Line 2195:  
missing comment of queryOptions param


http://gerrit.cloudera.org:8080/#/c/18289/2/tests/query_test/test_acid.py
File tests/query_test/test_acid.py:

http://gerrit.cloudera.org:8080/#/c/18289/2/tests/query_test/test_acid.py@334
PS2, Line 334:       assert elapsed < 5
this should be < 3 I think, as >3 && < 5 would mean we had 1 retry



-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Mar 2022 14:38:31 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gergely Fürnstáhl, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18289

to look at the new patch set (#4).

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................

IMPALA-11153: Make lock wait time configurable for the users

Currently Impala is using private static final values for lock
retry/wait:

 // Number of retries to acquire an HMS ACID lock.
 private static final int LOCK_RETRIES = 10;

 // Time interval between retries of acquiring an HMS ACID lock
 private static final int LOCK_RETRY_WAIT_SECONDS = 3;

This patch changes the logic for waiting on locks. It does an
exponential backoff starting from 100 ms up to 30 seconds, and the
users can configure a maximum total wait time for the locks.

Testing:
 * added e2e tests

Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/query_test/test_acid.py
10 files changed, 131 insertions(+), 40 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/18289/4
-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 14:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8028/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 14
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Apr 2022 12:20:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 8:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7996/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 8
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 31 Mar 2022 12:56:45 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Expose LOCK RETRIES/LOCK RETRY WAIT SECONDS settings for user

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10347/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Mar 2022 15:00:00 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18289/5/common/thrift/CatalogService.thrift
File common/thrift/CatalogService.thrift:

http://gerrit.cloudera.org:8080/#/c/18289/5/common/thrift/CatalogService.thrift@161
PS5, Line 161:   // Parameters for GRANT/REVOKE privilege
             :   21: optional JniCatalog.TG
> Sync_ddl also comes from a query option
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Mar 2022 16:30:59 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 7: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Mar 2022 11:29:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gergely Fürnstáhl, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18289

to look at the new patch set (#9).

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................

IMPALA-11153: Make lock wait time configurable for the users

Currently Impala is using private static final values for lock
retry/wait:

 // Number of retries to acquire an HMS ACID lock.
 private static final int LOCK_RETRIES = 10;

 // Time interval between retries of acquiring an HMS ACID lock
 private static final int LOCK_RETRY_WAIT_SECONDS = 3;

This patch changes the logic for waiting on locks. It does an
exponential backoff starting from 50 ms up to 30 seconds, and the
users can configure a maximum total wait time for the locks.

Testing:
 * added e2e tests

Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
---
M be/src/exec/catalog-op-executor.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/catalog/AlterDatabaseTest.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/query_test/test_acid.py
13 files changed, 178 insertions(+), 58 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/18289/9
-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 9
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-11153: Expose LOCK RETRIES/LOCK RETRY WAIT SECONDS settings for user

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user
......................................................................


Patch Set 3:

(1 comment)

PS3 is only a rebase.

http://gerrit.cloudera.org:8080/#/c/18289/2/common/thrift/CatalogService.thrift
File common/thrift/CatalogService.thrift:

http://gerrit.cloudera.org:8080/#/c/18289/2/common/thrift/CatalogService.thrift@171
PS2, Line 171: lock_retry_wait_seconds
> Aren't we pessimistic about "seconds"? I don't know how much overhead do we
From the name of the config at first I thought Hive uses 1 minute sleep interval...

Then I realized it is only a maximum value, and Hive does exponential backoff: https://github.com/apache/hive/blob/7b3ecf617a6d46f48a3b6f77e0339fd4ad95a420/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbLockManager.java#L95-L97

I think it would be reasonable for us as well, so I switched to a backoff logic.



-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Mar 2022 14:39:43 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gergely Fürnstáhl, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18289

to look at the new patch set (#5).

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................

IMPALA-11153: Make lock wait time configurable for the users

Currently Impala is using private static final values for lock
retry/wait:

 // Number of retries to acquire an HMS ACID lock.
 private static final int LOCK_RETRIES = 10;

 // Time interval between retries of acquiring an HMS ACID lock
 private static final int LOCK_RETRY_WAIT_SECONDS = 3;

This patch changes the logic for waiting on locks. It does an
exponential backoff starting from 50 ms up to 30 seconds, and the
users can configure a maximum total wait time for the locks.

Testing:
 * added e2e tests

Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
---
M be/src/exec/catalog-op-executor.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/query_test/test_acid.py
11 files changed, 144 insertions(+), 49 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/18289/5
-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 10:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10385/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 02 Apr 2022 13:28:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 10:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/18289/10/common/thrift/Query.thrift
File common/thrift/Query.thrift:

http://gerrit.cloudera.org:8080/#/c/18289/10/common/thrift/Query.thrift@593
PS10, Line 593: 300
I think this should be 30


http://gerrit.cloudera.org:8080/#/c/18289/10/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
File fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java:

http://gerrit.cloudera.org:8080/#/c/18289/10/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java@107
PS10, Line 107:   private static final long MAX_SLEEP_INTERVAL_MS = 30000;
Do we need this? It is provided by the caller.


http://gerrit.cloudera.org:8080/#/c/18289/10/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java@655
PS10, Line 655:           sleepIntervalMs = Math.min(MAX_SLEEP_INTERVAL_MS,
              :                                      sleepIntervalMs * 2);
We don't really need Math.min here, overshooting will be catched on line 649 anyways.


http://gerrit.cloudera.org:8080/#/c/18289/10/tests/query_test/test_acid.py
File tests/query_test/test_acid.py:

http://gerrit.cloudera.org:8080/#/c/18289/10/tests/query_test/test_acid.py@327
PS10, Line 327:     try:
could add a test case before the first set, to check 30



-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Apr 2022 15:34:21 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gergely Fürnstáhl, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18289

to look at the new patch set (#13).

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................

IMPALA-11153: Make lock wait time configurable for the users

Currently Impala is using private static final values for lock
retry/wait:

 // Number of retries to acquire an HMS ACID lock.
 private static final int LOCK_RETRIES = 10;

 // Time interval between retries of acquiring an HMS ACID lock
 private static final int LOCK_RETRY_WAIT_SECONDS = 3;

This patch changes the logic for waiting on locks. It does an
exponential backoff starting from 50 ms up to 30 seconds, and the
users can configure a maximum total wait time for the locks. This
total maximum wait time is 5 minutes (the earlier 30 seconds
was too short in real use cases).

Testing:
 * added e2e tests

Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
---
M be/src/exec/catalog-op-executor.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/catalog/AlterDatabaseTest.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/query_test/test_acid.py
13 files changed, 182 insertions(+), 58 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/18289/13
-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 13
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 13: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 13
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Apr 2022 12:11:43 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 14: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 14
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Apr 2022 16:46:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Expose LOCK RETRIES/LOCK RETRY WAIT SECONDS settings for user

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user
......................................................................


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/18289/2/common/thrift/CatalogService.thrift
File common/thrift/CatalogService.thrift:

http://gerrit.cloudera.org:8080/#/c/18289/2/common/thrift/CatalogService.thrift@171
PS2, Line 171: lock_retry_wait_seconds
Aren't we pessimistic about "seconds"? I don't know how much overhead do we add with the RPCs to Hive, but I think that smaller retry times like 200 ms could be reasonable.

We could use lock_retry_wait_ms instead.


http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@483
PS2, Line 483: ddlRequest.getLock_max_retries(), ddlRequest.getLock_retry_wait_seconds()
Just an idea to make passing these parameters simpler: the params in ddlRequest that come from query options could be collected to a struct and that could be passed here.

e.g. it could contain debug_action too, which may become useful in the future in RPCs



-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Mar 2022 17:42:03 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11153: Expose LOCK RETRIES/LOCK RETRY WAIT SECONDS settings for user

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10252/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Mar 2022 11:04:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10348/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Mar 2022 18:17:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 5:

(2 comments)

Thanks for the comments!

http://gerrit.cloudera.org:8080/#/c/18289/4/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
File fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java:

http://gerrit.cloudera.org:8080/#/c/18289/4/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java@641
PS4, Line 641: long sleepIntervalMs = 100;
> noticed that Hive starts with 50 ms - maybe we should use the same?
Done


http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@483
PS2, Line 483: ddlRequest.getQuery_options().getLock_max_wait_time_s());
> I still think that collecting query options that are passed in TDdlExecRequ
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Mar 2022 15:39:34 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11153: Expose LOCK RETRIES/LOCK RETRY WAIT SECONDS settings for user

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10250/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Mar 2022 10:51:21 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 13: Code-Review+1

Looks good to me


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 13
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Apr 2022 09:06:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Expose LOCK RETRIES/LOCK RETRY WAIT SECONDS settings for user

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gergely Fürnstáhl, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18289

to look at the new patch set (#3).

Change subject: IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user
......................................................................

IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user

Currently Impala is using private static final values for lock
retry/wait:

 // Number of retries to acquire an HMS ACID lock.
 private static final int LOCK_RETRIES = 10;

 // Time interval between retries of acquiring an HMS ACID lock
 private static final int LOCK_RETRY_WAIT_SECONDS = 3;

This patch makes the above values tunable via query options.

Testing:
 * added e2e tests

Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/query_test/test_acid.py
10 files changed, 142 insertions(+), 31 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/18289/3
-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 10:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8010/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 02 Apr 2022 13:09:32 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 10:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/18289/10/common/thrift/Query.thrift
File common/thrift/Query.thrift:

http://gerrit.cloudera.org:8080/#/c/18289/10/common/thrift/Query.thrift@593
PS10, Line 593: 300
> I think this should be 30
Yeah, sorry, I didn't mention it in the comment message (until now), but I decided to increase the total lock wait time, as 30 seconds was too short in practice.


http://gerrit.cloudera.org:8080/#/c/18289/10/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
File fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java:

http://gerrit.cloudera.org:8080/#/c/18289/10/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java@107
PS10, Line 107:   private static final long MAX_SLEEP_INTERVAL_MS = 30000;
> Do we need this? It is provided by the caller.
The caller provides the max total wait time. This is the sleep interval between 'checkLock()' calls.


http://gerrit.cloudera.org:8080/#/c/18289/10/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java@655
PS10, Line 655:           sleepIntervalMs = Math.min(MAX_SLEEP_INTERVAL_MS,
              :                                      sleepIntervalMs * 2);
> We don't really need Math.min here, overshooting will be catched on line 64
This limits the sleep interval to 30 seconds (MAX_SLEEP_INTERVAL_MS).

At L649 we limit the sleep interval to the time left until 'timeoutTime'.

So if the user sets a very long timeout time (1 hour), then we'd sleep more than MAX_SLEEP_INTERVAL_MS without this Math.min().



-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Apr 2022 15:53:37 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 9:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10383/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 9
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Apr 2022 20:29:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10354/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Mar 2022 15:58:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 7:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7981/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Mar 2022 11:29:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 14: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 14
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Apr 2022 12:20:39 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Expose LOCK RETRIES/LOCK RETRY WAIT SECONDS settings for user

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18289

to look at the new patch set (#2).

Change subject: IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user
......................................................................

IMPALA-11153: Expose LOCK_RETRIES/LOCK_RETRY_WAIT_SECONDS settings for user

Currently Impala is using private static final values for lock
retry/wait:

 // Number of retries to acquire an HMS ACID lock.
 private static final int LOCK_RETRIES = 10;

 // Time interval between retries of acquiring an HMS ACID lock
 private static final int LOCK_RETRY_WAIT_SECONDS = 3;

This patch makes the above values tunable via query options.

Testing:
 * added e2e tests

Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/query_test/test_acid.py
10 files changed, 142 insertions(+), 31 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/18289/2
-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 11:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8018/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 11
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Apr 2022 15:23:24 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Gergely Fürnstáhl, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18289

to look at the new patch set (#10).

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................

IMPALA-11153: Make lock wait time configurable for the users

Currently Impala is using private static final values for lock
retry/wait:

 // Number of retries to acquire an HMS ACID lock.
 private static final int LOCK_RETRIES = 10;

 // Time interval between retries of acquiring an HMS ACID lock
 private static final int LOCK_RETRY_WAIT_SECONDS = 3;

This patch changes the logic for waiting on locks. It does an
exponential backoff starting from 50 ms up to 30 seconds, and the
users can configure a maximum total wait time for the locks.

Testing:
 * added e2e tests

Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
---
M be/src/exec/catalog-op-executor.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/catalog/AlterDatabaseTest.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/query_test/test_acid.py
13 files changed, 179 insertions(+), 58 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/18289/10
-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 10:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18289/10/tests/query_test/test_acid.py
File tests/query_test/test_acid.py:

http://gerrit.cloudera.org:8080/#/c/18289/10/tests/query_test/test_acid.py@327
PS10, Line 327:     try:
> could add a test case before the first set, to check 30
So the default is now 5 min and I didn't want the test to take too long. But now I've added a check in exhaustive test runs.



-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Apr 2022 15:56:02 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 13:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10393/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 13
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Apr 2022 16:15:14 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 9: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8007/


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 9
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 02 Apr 2022 00:40:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................

IMPALA-11153: Make lock wait time configurable for the users

Currently Impala is using private static final values for lock
retry/wait:

 // Number of retries to acquire an HMS ACID lock.
 private static final int LOCK_RETRIES = 10;

 // Time interval between retries of acquiring an HMS ACID lock
 private static final int LOCK_RETRY_WAIT_SECONDS = 3;

This patch changes the logic for waiting on locks. It does an
exponential backoff starting from 50 ms up to 30 seconds, and the
users can configure a maximum total wait time for the locks. This
total maximum wait time is 5 minutes (the earlier 30 seconds
was too short in real use cases).

Testing:
 * added e2e tests

Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Reviewed-on: http://gerrit.cloudera.org:8080/18289
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/catalog-op-executor.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/catalog/AlterDatabaseTest.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/query_test/test_acid.py
13 files changed, 182 insertions(+), 58 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 15
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 4: Code-Review+2

(2 comments)

http://gerrit.cloudera.org:8080/#/c/18289/4/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
File fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java:

http://gerrit.cloudera.org:8080/#/c/18289/4/fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java@641
PS4, Line 641: long sleepIntervalMs = 100;
noticed that Hive starts with 50 ms - maybe we should use the same?
https://github.com/apache/hive/blob/7b3ecf617a6d46f48a3b6f77e0339fd4ad95a420/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbLockManager.java#L93


http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/18289/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@483
PS2, Line 483: ddlRequest.getLock_max_wait_time_s());
> I haven't done it in the current PS as I decreased the number of new query 
I still think that collecting query options that are passed in TDdlExecRequest in a struct would be nice in the long term.



-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Mar 2022 10:53:09 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 6: Code-Review+2

Carry +2


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Mar 2022 11:29:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 )

Change subject: IMPALA-11153: Make lock wait time configurable for the users
......................................................................


Patch Set 7: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7981/


-- 
To view, visit http://gerrit.cloudera.org:8080/18289
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc
Gerrit-Change-Number: 18289
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Mar 2022 15:53:47 +0000
Gerrit-HasComments: No