You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Denys Kuzmenko via Review Board <no...@reviews.apache.org> on 2020/04/20 07:23:26 UTC
Review Request 72387: Locks: Add new lock implementations for always
zero-wait readers
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/
-----------------------------------------------------------
Review request for hive, Marton Bod and Peter Vary.
Bugs: HIVE-19369
https://issues.apache.org/jira/browse/HIVE-19369
Repository: hive-git
Description
-------
Hive Locking with Micro-managed and full-ACID tables needs a better locking implementation which allows for no-wait readers always.
EXCL_DROP
EXCL_WRITE
SHARED_WRITE
SHARED_READ
Short write-up
EXCL_DROP is a "drop partition" or "drop table" and waits for all others to exit
EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to exit.
SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different threads).
SHARED_READ does not wait for any lock - it fails fast for a pending EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no semantic reason to wait for them to succeed before going ahead with a SHARED_WRITE.
a select * => SHARED_READ
an insert into => SHARED_WRITE
an insert overwrite or MERGE => EXCL_WRITE
a drop table => EXCL_DROP
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7b3acad511
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b
ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 1d211857bf
ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 73d3b91585
ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f
ql/src/test/results/clientpositive/explain_locks.q.out 3183533478
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java 22902a9c20
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java PRE-CREATION
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java 331fd4cc8d
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java f928bf781b
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java 3d69a6e7dc
Diff: https://reviews.apache.org/r/72387/diff/1/
Testing
-------
Added number of tests under TestDbTxnManager2, TestTxnHandler & extended explain_locks.q test.
Thanks,
Denys Kuzmenko
Re: Review Request 72387: Locks: Add new lock implementations for
always zero-wait readers
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/
-----------------------------------------------------------
(Updated April 22, 2020, 3:27 p.m.)
Review request for hive, Marton Bod and Peter Vary.
Bugs: HIVE-19369
https://issues.apache.org/jira/browse/HIVE-19369
Repository: hive-git
Description
-------
Hive Locking with Micro-managed and full-ACID tables needs a better locking implementation which allows for no-wait readers always.
EXCL_DROP
EXCL_WRITE
SHARED_WRITE
SHARED_READ
Short write-up
EXCL_DROP is a "drop partition" or "drop table" and waits for all others to exit
EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to exit.
SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different threads).
SHARED_READ does not wait for any lock - it fails fast for a pending EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no semantic reason to wait for them to succeed before going ahead with a SHARED_WRITE.
a select * => SHARED_READ
an insert into => SHARED_WRITE
an insert overwrite or MERGE => EXCL_WRITE
a drop table => EXCL_DROP
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df
ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java 4885e437aa
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b
ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 1d211857bf
ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 73d3b91585
ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f
ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java 22902a9c20
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java PRE-CREATION
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java d080df417b
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java f928bf781b
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java 3d69a6e7dc
Diff: https://reviews.apache.org/r/72387/diff/6/
Changes: https://reviews.apache.org/r/72387/diff/5-6/
Testing
-------
Added number of tests under TestDbTxnManager2, TestTxnHandler & extended explain_locks.q test.
Thanks,
Denys Kuzmenko
Re: Review Request 72387: Locks: Add new lock implementations for
always zero-wait readers
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
> On April 22, 2020, 3:22 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java
> > Line 113 (original), 112 (patched)
> > <https://reviews.apache.org/r/72387/diff/5/?file=2224739#file2224739line117>
> >
> > Isn't reference level check is risky here?
good catch, need to reorder :)
- Denys
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/#review220426
-----------------------------------------------------------
On April 22, 2020, 2:29 p.m., Denys Kuzmenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72387/
> -----------------------------------------------------------
>
> (Updated April 22, 2020, 2:29 p.m.)
>
>
> Review request for hive, Marton Bod and Peter Vary.
>
>
> Bugs: HIVE-19369
> https://issues.apache.org/jira/browse/HIVE-19369
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Hive Locking with Micro-managed and full-ACID tables needs a better locking implementation which allows for no-wait readers always.
>
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
>
> Short write-up
>
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different threads).
>
> SHARED_READ does not wait for any lock - it fails fast for a pending EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no semantic reason to wait for them to succeed before going ahead with a SHARED_WRITE.
>
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df
> ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java 4885e437aa
> ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 1d211857bf
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 73d3b91585
> ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f
> ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java 22902a9c20
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java PRE-CREATION
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java d080df417b
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java f928bf781b
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java 3d69a6e7dc
>
>
> Diff: https://reviews.apache.org/r/72387/diff/5/
>
>
> Testing
> -------
>
> Added number of tests under TestDbTxnManager2, TestTxnHandler & extended explain_locks.q test.
>
>
> Thanks,
>
> Denys Kuzmenko
>
>
Re: Review Request 72387: Locks: Add new lock implementations for
always zero-wait readers
Posted by Peter Vary via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/#review220426
-----------------------------------------------------------
Ship it!
ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java
Lines 96 (patched)
<https://reviews.apache.org/r/72387/#comment308814>
nit: new line?
ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java
Line 113 (original), 112 (patched)
<https://reviews.apache.org/r/72387/#comment308813>
Isn't reference level check is risky here?
- Peter Vary
On ápr. 22, 2020, 2:29 du, Denys Kuzmenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72387/
> -----------------------------------------------------------
>
> (Updated ápr. 22, 2020, 2:29 du)
>
>
> Review request for hive, Marton Bod and Peter Vary.
>
>
> Bugs: HIVE-19369
> https://issues.apache.org/jira/browse/HIVE-19369
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Hive Locking with Micro-managed and full-ACID tables needs a better locking implementation which allows for no-wait readers always.
>
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
>
> Short write-up
>
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different threads).
>
> SHARED_READ does not wait for any lock - it fails fast for a pending EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no semantic reason to wait for them to succeed before going ahead with a SHARED_WRITE.
>
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df
> ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java 4885e437aa
> ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 1d211857bf
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 73d3b91585
> ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f
> ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java 22902a9c20
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java PRE-CREATION
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java d080df417b
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java f928bf781b
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java 3d69a6e7dc
>
>
> Diff: https://reviews.apache.org/r/72387/diff/5/
>
>
> Testing
> -------
>
> Added number of tests under TestDbTxnManager2, TestTxnHandler & extended explain_locks.q test.
>
>
> Thanks,
>
> Denys Kuzmenko
>
>
Re: Review Request 72387: Locks: Add new lock implementations for
always zero-wait readers
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/
-----------------------------------------------------------
(Updated April 22, 2020, 2:29 p.m.)
Review request for hive, Marton Bod and Peter Vary.
Changes
-------
reorder checks & fixed isValidTxnListState
Bugs: HIVE-19369
https://issues.apache.org/jira/browse/HIVE-19369
Repository: hive-git
Description
-------
Hive Locking with Micro-managed and full-ACID tables needs a better locking implementation which allows for no-wait readers always.
EXCL_DROP
EXCL_WRITE
SHARED_WRITE
SHARED_READ
Short write-up
EXCL_DROP is a "drop partition" or "drop table" and waits for all others to exit
EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to exit.
SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different threads).
SHARED_READ does not wait for any lock - it fails fast for a pending EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no semantic reason to wait for them to succeed before going ahead with a SHARED_WRITE.
a select * => SHARED_READ
an insert into => SHARED_WRITE
an insert overwrite or MERGE => EXCL_WRITE
a drop table => EXCL_DROP
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df
ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java 4885e437aa
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b
ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 1d211857bf
ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 73d3b91585
ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f
ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java 22902a9c20
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java PRE-CREATION
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java d080df417b
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java f928bf781b
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java 3d69a6e7dc
Diff: https://reviews.apache.org/r/72387/diff/5/
Changes: https://reviews.apache.org/r/72387/diff/4-5/
Testing
-------
Added number of tests under TestDbTxnManager2, TestTxnHandler & extended explain_locks.q test.
Thanks,
Denys Kuzmenko
Re: Review Request 72387: Locks: Add new lock implementations for
always zero-wait readers
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/
-----------------------------------------------------------
(Updated April 21, 2020, 12:14 p.m.)
Review request for hive, Marton Bod and Peter Vary.
Changes
-------
added quotes
Bugs: HIVE-19369
https://issues.apache.org/jira/browse/HIVE-19369
Repository: hive-git
Description
-------
Hive Locking with Micro-managed and full-ACID tables needs a better locking implementation which allows for no-wait readers always.
EXCL_DROP
EXCL_WRITE
SHARED_WRITE
SHARED_READ
Short write-up
EXCL_DROP is a "drop partition" or "drop table" and waits for all others to exit
EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to exit.
SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different threads).
SHARED_READ does not wait for any lock - it fails fast for a pending EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no semantic reason to wait for them to succeed before going ahead with a SHARED_WRITE.
a select * => SHARED_READ
an insert into => SHARED_WRITE
an insert overwrite or MERGE => EXCL_WRITE
a drop table => EXCL_DROP
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b
ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 1d211857bf
ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 73d3b91585
ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f
ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java 22902a9c20
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java PRE-CREATION
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java d080df417b
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java f928bf781b
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java 3d69a6e7dc
Diff: https://reviews.apache.org/r/72387/diff/4/
Changes: https://reviews.apache.org/r/72387/diff/3-4/
Testing
-------
Added number of tests under TestDbTxnManager2, TestTxnHandler & extended explain_locks.q test.
Thanks,
Denys Kuzmenko
Re: Review Request 72387: Locks: Add new lock implementations for
always zero-wait readers
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/
-----------------------------------------------------------
(Updated April 20, 2020, 8:50 p.m.)
Review request for hive, Marton Bod and Peter Vary.
Bugs: HIVE-19369
https://issues.apache.org/jira/browse/HIVE-19369
Repository: hive-git
Description
-------
Hive Locking with Micro-managed and full-ACID tables needs a better locking implementation which allows for no-wait readers always.
EXCL_DROP
EXCL_WRITE
SHARED_WRITE
SHARED_READ
Short write-up
EXCL_DROP is a "drop partition" or "drop table" and waits for all others to exit
EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to exit.
SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different threads).
SHARED_READ does not wait for any lock - it fails fast for a pending EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no semantic reason to wait for them to succeed before going ahead with a SHARED_WRITE.
a select * => SHARED_READ
an insert into => SHARED_WRITE
an insert overwrite or MERGE => EXCL_WRITE
a drop table => EXCL_DROP
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b
ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 1d211857bf
ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 73d3b91585
ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f
ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java 22902a9c20
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java PRE-CREATION
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java d080df417b
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java f928bf781b
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java 3d69a6e7dc
Diff: https://reviews.apache.org/r/72387/diff/2/
Changes: https://reviews.apache.org/r/72387/diff/1-2/
Testing
-------
Added number of tests under TestDbTxnManager2, TestTxnHandler & extended explain_locks.q test.
Thanks,
Denys Kuzmenko
Re: Review Request 72387: Locks: Add new lock implementations for
always zero-wait readers
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
> On April 20, 2020, 4:55 p.m., Peter Vary wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 2727-2730 (original), 2727-2730 (patched)
> > <https://reviews.apache.org/r/72387/diff/1/?file=2223800#file2223800line2727>
> >
> > Is this config still needed in this case?
yes, in order to be backward compatible, when user goes with an exclusive writes and desides to lower the lock level for the insert overwrite from exclusive to exclusive write (insert overwrite can execute in parallel with insert - current functionality)
> On April 20, 2020, 4:55 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
> > Line 3027 (original), 3027-3028 (patched)
> > <https://reviews.apache.org/r/72387/diff/1/?file=2223801#file2223801line3027>
> >
> > Maybe get once, and store?
fixed
- Denys
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/#review220374
-----------------------------------------------------------
On April 20, 2020, 8:50 p.m., Denys Kuzmenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72387/
> -----------------------------------------------------------
>
> (Updated April 20, 2020, 8:50 p.m.)
>
>
> Review request for hive, Marton Bod and Peter Vary.
>
>
> Bugs: HIVE-19369
> https://issues.apache.org/jira/browse/HIVE-19369
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Hive Locking with Micro-managed and full-ACID tables needs a better locking implementation which allows for no-wait readers always.
>
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
>
> Short write-up
>
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different threads).
>
> SHARED_READ does not wait for any lock - it fails fast for a pending EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no semantic reason to wait for them to succeed before going ahead with a SHARED_WRITE.
>
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df
> ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 1d211857bf
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 73d3b91585
> ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f
> ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java 22902a9c20
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java PRE-CREATION
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java d080df417b
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java f928bf781b
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java 3d69a6e7dc
>
>
> Diff: https://reviews.apache.org/r/72387/diff/2/
>
>
> Testing
> -------
>
> Added number of tests under TestDbTxnManager2, TestTxnHandler & extended explain_locks.q test.
>
>
> Thanks,
>
> Denys Kuzmenko
>
>
Re: Review Request 72387: Locks: Add new lock implementations for
always zero-wait readers
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
> On April 20, 2020, 4:55 p.m., Peter Vary wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 2732 (patched)
> > <https://reviews.apache.org/r/72387/diff/1/?file=2223800#file2223800line2732>
> >
> > Maybe rephrase?
> > Manages concurrency levels for ACID resoruces. Enables users enable parallel queries by enabling write-write conflict resolution happen only at commit phase
> > - If true - no commit phase conflict resolution:
> > - INSERT OVERWRITE requests EXCLUSIVE locks
> > - UPDATE/DELETE requests EXCL_WRITE lock
> > - INSERT requests SHARED_READ lock
> > - If false - write might fail when committed on conflict check:
> > - INSERT OVERWRITE requests EXCL_WRITE locks
> > - UPDATE/DELETE/INSERT requests SHARED_READ lock
fixed
- Denys
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/#review220374
-----------------------------------------------------------
On April 20, 2020, 8:50 p.m., Denys Kuzmenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72387/
> -----------------------------------------------------------
>
> (Updated April 20, 2020, 8:50 p.m.)
>
>
> Review request for hive, Marton Bod and Peter Vary.
>
>
> Bugs: HIVE-19369
> https://issues.apache.org/jira/browse/HIVE-19369
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Hive Locking with Micro-managed and full-ACID tables needs a better locking implementation which allows for no-wait readers always.
>
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
>
> Short write-up
>
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different threads).
>
> SHARED_READ does not wait for any lock - it fails fast for a pending EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no semantic reason to wait for them to succeed before going ahead with a SHARED_WRITE.
>
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16bae920df
> ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 1d211857bf
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 73d3b91585
> ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f
> ql/src/test/results/clientpositive/llap/explain_locks.q.out 3183533478
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java 22902a9c20
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java PRE-CREATION
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java d080df417b
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java f928bf781b
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java 3d69a6e7dc
>
>
> Diff: https://reviews.apache.org/r/72387/diff/2/
>
>
> Testing
> -------
>
> Added number of tests under TestDbTxnManager2, TestTxnHandler & extended explain_locks.q test.
>
>
> Thanks,
>
> Denys Kuzmenko
>
>
Re: Review Request 72387: Locks: Add new lock implementations for
always zero-wait readers
Posted by Peter Vary via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72387/#review220374
-----------------------------------------------------------
Some nits, did not checked all the unit tests changes. Anything I have to check particularly?
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 2727-2730 (original), 2727-2730 (patched)
<https://reviews.apache.org/r/72387/#comment308732>
Is this config still needed in this case?
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 2732 (patched)
<https://reviews.apache.org/r/72387/#comment308730>
Maybe rephrase?
Manages concurrency levels for ACID resoruces. Enables users enable parallel queries by enabling write-write conflict resolution happen only at commit phase
- If true - no commit phase conflict resolution:
- INSERT OVERWRITE requests EXCLUSIVE locks
- UPDATE/DELETE requests EXCL_WRITE lock
- INSERT requests SHARED_READ lock
- If false - write might fail when committed on conflict check:
- INSERT OVERWRITE requests EXCL_WRITE locks
- UPDATE/DELETE/INSERT requests SHARED_READ lock
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
Line 3027 (original), 3027-3028 (patched)
<https://reviews.apache.org/r/72387/#comment308731>
Maybe get once, and store?
ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java
Lines 758 (patched)
<https://reviews.apache.org/r/72387/#comment308733>
nit: remove spaces
ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java
Line 1142 (original), 1265 (patched)
<https://reviews.apache.org/r/72387/#comment308734>
nit: space too
- Peter Vary
On ápr. 20, 2020, 7:23 de, Denys Kuzmenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72387/
> -----------------------------------------------------------
>
> (Updated ápr. 20, 2020, 7:23 de)
>
>
> Review request for hive, Marton Bod and Peter Vary.
>
>
> Bugs: HIVE-19369
> https://issues.apache.org/jira/browse/HIVE-19369
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Hive Locking with Micro-managed and full-ACID tables needs a better locking implementation which allows for no-wait readers always.
>
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
>
> Short write-up
>
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different threads).
>
> SHARED_READ does not wait for any lock - it fails fast for a pending EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no semantic reason to wait for them to succeed before going ahead with a SHARED_WRITE.
>
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7b3acad511
> ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 77878ca40b
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 1d211857bf
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 73d3b91585
> ql/src/test/queries/clientpositive/explain_locks.q 3c11560c5f
> ql/src/test/results/clientpositive/explain_locks.q.out 3183533478
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockRequestBuilder.java 22902a9c20
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/LockTypeComparator.java PRE-CREATION
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java 331fd4cc8d
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtil.java f928bf781b
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/LockTypeUtilTest.java 3d69a6e7dc
>
>
> Diff: https://reviews.apache.org/r/72387/diff/1/
>
>
> Testing
> -------
>
> Added number of tests under TestDbTxnManager2, TestTxnHandler & extended explain_locks.q test.
>
>
> Thanks,
>
> Denys Kuzmenko
>
>