You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Denys Kuzmenko via Review Board <no...@reviews.apache.org> on 2019/11/20 12:20:26 UTC
Review Request 71792: COMPLETED_TXN_COMPONENTS table is never cleaned
up unless Compactor runs
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71792/
-----------------------------------------------------------
Review request for hive, Laszlo Pinter and Peter Vary.
Bugs: HIVE-21917
https://issues.apache.org/jira/browse/HIVE-21917
Repository: hive-git
Description
-------
The Initiator thread in the metastore repeatedly loops over entries in the COMPLETED_TXN_COMPONENTS table to determine which partitions / tables might need to be compacted. However, entries are never removed from this table except by a completed Compactor run.
In a cluster where most tables / partitions are write-once read-many, this results in stale entries in this table never being cleaned up. In a small test cluster, we have observed approximately 45k entries in this table (virtually equal to the number of partitions in the cluster) while < 100 of these tables have delta files at all. Since most of the tables will never get enough writes to trigger a compaction (and in fact have only ever been written to once), the initiator thread keeps trying to evaluate them on every loop.
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 610cf05204
ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java b28b57779b
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java 8253ccb9c9
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java 6281208247
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java e840758c9d
Diff: https://reviews.apache.org/r/71792/diff/1/
Testing
-------
Unit tests
Thanks,
Denys Kuzmenko
Re: Review Request 71792: COMPLETED_TXN_COMPONENTS table is never
cleaned up unless Compactor runs
Posted by Peter Vary via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71792/#review218762
-----------------------------------------------------------
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
Lines 83 (patched)
<https://reviews.apache.org/r/71792/#comment306628>
Please use the constants here.
- Peter Vary
On nov. 21, 2019, 5:35 du, Denys Kuzmenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71792/
> -----------------------------------------------------------
>
> (Updated nov. 21, 2019, 5:35 du)
>
>
> Review request for hive, Laszlo Pinter and Peter Vary.
>
>
> Bugs: HIVE-21917
> https://issues.apache.org/jira/browse/HIVE-21917
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> The Initiator thread in the metastore repeatedly loops over entries in the COMPLETED_TXN_COMPONENTS table to determine which partitions / tables might need to be compacted. However, entries are never removed from this table except by a completed Compactor run.
>
> In a cluster where most tables / partitions are write-once read-many, this results in stale entries in this table never being cleaned up. In a small test cluster, we have observed approximately 45k entries in this table (virtually equal to the number of partitions in the cluster) while < 100 of these tables have delta files at all. Since most of the tables will never get enough writes to trigger a compaction (and in fact have only ever been written to once), the initiator thread keeps trying to evaluate them on every loop.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 610cf05204
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java b28b57779b
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java 8253ccb9c9
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java 268038795b
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java e840758c9d
>
>
> Diff: https://reviews.apache.org/r/71792/diff/2/
>
>
> Testing
> -------
>
> Unit tests
>
>
> Thanks,
>
> Denys Kuzmenko
>
>
Re: Review Request 71792: COMPLETED_TXN_COMPONENTS table is never
cleaned up unless Compactor runs
Posted by Laszlo Pinter via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71792/#review218754
-----------------------------------------------------------
Ship it!
Lgtm +1
- Laszlo Pinter
On Nov. 21, 2019, 5:35 p.m., Denys Kuzmenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71792/
> -----------------------------------------------------------
>
> (Updated Nov. 21, 2019, 5:35 p.m.)
>
>
> Review request for hive, Laszlo Pinter and Peter Vary.
>
>
> Bugs: HIVE-21917
> https://issues.apache.org/jira/browse/HIVE-21917
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> The Initiator thread in the metastore repeatedly loops over entries in the COMPLETED_TXN_COMPONENTS table to determine which partitions / tables might need to be compacted. However, entries are never removed from this table except by a completed Compactor run.
>
> In a cluster where most tables / partitions are write-once read-many, this results in stale entries in this table never being cleaned up. In a small test cluster, we have observed approximately 45k entries in this table (virtually equal to the number of partitions in the cluster) while < 100 of these tables have delta files at all. Since most of the tables will never get enough writes to trigger a compaction (and in fact have only ever been written to once), the initiator thread keeps trying to evaluate them on every loop.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 610cf05204
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java b28b57779b
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java 8253ccb9c9
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java 268038795b
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java e840758c9d
>
>
> Diff: https://reviews.apache.org/r/71792/diff/2/
>
>
> Testing
> -------
>
> Unit tests
>
>
> Thanks,
>
> Denys Kuzmenko
>
>
Re: Review Request 71792: COMPLETED_TXN_COMPONENTS table is never
cleaned up unless Compactor runs
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71792/
-----------------------------------------------------------
(Updated Nov. 21, 2019, 5:35 p.m.)
Review request for hive, Laszlo Pinter and Peter Vary.
Bugs: HIVE-21917
https://issues.apache.org/jira/browse/HIVE-21917
Repository: hive-git
Description
-------
The Initiator thread in the metastore repeatedly loops over entries in the COMPLETED_TXN_COMPONENTS table to determine which partitions / tables might need to be compacted. However, entries are never removed from this table except by a completed Compactor run.
In a cluster where most tables / partitions are write-once read-many, this results in stale entries in this table never being cleaned up. In a small test cluster, we have observed approximately 45k entries in this table (virtually equal to the number of partitions in the cluster) while < 100 of these tables have delta files at all. Since most of the tables will never get enough writes to trigger a compaction (and in fact have only ever been written to once), the initiator thread keeps trying to evaluate them on every loop.
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 610cf05204
ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java b28b57779b
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java 8253ccb9c9
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java 268038795b
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java e840758c9d
Diff: https://reviews.apache.org/r/71792/diff/2/
Changes: https://reviews.apache.org/r/71792/diff/1-2/
Testing
-------
Unit tests
Thanks,
Denys Kuzmenko
Re: Review Request 71792: COMPLETED_TXN_COMPONENTS table is never
cleaned up unless Compactor runs
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
> On Nov. 20, 2019, 3:19 p.m., Denys Kuzmenko wrote:
> > Not ready. Need to handle aborted and currently active compactions.
Handling above cases would complicate the Initiator logic and make preliminare check longer. Not sure how critial it is that in case of unsuccessful compaction attempt, on next run we won't retry unless there is some change to the selected table/partiotion. Any thoughts on this?
- Denys
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71792/#review218723
-----------------------------------------------------------
On Nov. 20, 2019, 12:20 p.m., Denys Kuzmenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71792/
> -----------------------------------------------------------
>
> (Updated Nov. 20, 2019, 12:20 p.m.)
>
>
> Review request for hive, Laszlo Pinter and Peter Vary.
>
>
> Bugs: HIVE-21917
> https://issues.apache.org/jira/browse/HIVE-21917
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> The Initiator thread in the metastore repeatedly loops over entries in the COMPLETED_TXN_COMPONENTS table to determine which partitions / tables might need to be compacted. However, entries are never removed from this table except by a completed Compactor run.
>
> In a cluster where most tables / partitions are write-once read-many, this results in stale entries in this table never being cleaned up. In a small test cluster, we have observed approximately 45k entries in this table (virtually equal to the number of partitions in the cluster) while < 100 of these tables have delta files at all. Since most of the tables will never get enough writes to trigger a compaction (and in fact have only ever been written to once), the initiator thread keeps trying to evaluate them on every loop.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 610cf05204
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java b28b57779b
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java 8253ccb9c9
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java 6281208247
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java e840758c9d
>
>
> Diff: https://reviews.apache.org/r/71792/diff/1/
>
>
> Testing
> -------
>
> Unit tests
>
>
> Thanks,
>
> Denys Kuzmenko
>
>
Re: Review Request 71792: COMPLETED_TXN_COMPONENTS table is never
cleaned up unless Compactor runs
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
> On Nov. 20, 2019, 3:19 p.m., Denys Kuzmenko wrote:
> > Not ready. Need to handle aborted and currently active compactions.
>
> Denys Kuzmenko wrote:
> Handling above cases would complicate the Initiator logic and make preliminare check longer. Not sure how critial it is that in case of unsuccessful compaction attempt, on next run we won't retry unless there is some change to the selected table/partiotion. Any thoughts on this?
Changed findPotentialCompactions query to:
select distinct ctc_database, ctc_table, ctc_partition from COMPLETED_TXN_COMPONENTS where
(select CC_STATE from COMPLETED_COMPACTIONS where ctc_database = CC_DATABASE and ctc_table = CC_TABLE and (ctc_partition is null or ctc_partition = cc_partition)
order by cc_id desc limit 1) IN ('a', 'f') || ctc_timestamp < current_timestamp
however this still won't cover skipped compactions due to already running one
- Denys
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71792/#review218723
-----------------------------------------------------------
On Nov. 20, 2019, 12:20 p.m., Denys Kuzmenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71792/
> -----------------------------------------------------------
>
> (Updated Nov. 20, 2019, 12:20 p.m.)
>
>
> Review request for hive, Laszlo Pinter and Peter Vary.
>
>
> Bugs: HIVE-21917
> https://issues.apache.org/jira/browse/HIVE-21917
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> The Initiator thread in the metastore repeatedly loops over entries in the COMPLETED_TXN_COMPONENTS table to determine which partitions / tables might need to be compacted. However, entries are never removed from this table except by a completed Compactor run.
>
> In a cluster where most tables / partitions are write-once read-many, this results in stale entries in this table never being cleaned up. In a small test cluster, we have observed approximately 45k entries in this table (virtually equal to the number of partitions in the cluster) while < 100 of these tables have delta files at all. Since most of the tables will never get enough writes to trigger a compaction (and in fact have only ever been written to once), the initiator thread keeps trying to evaluate them on every loop.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 610cf05204
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java b28b57779b
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java 8253ccb9c9
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java 6281208247
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java e840758c9d
>
>
> Diff: https://reviews.apache.org/r/71792/diff/1/
>
>
> Testing
> -------
>
> Unit tests
>
>
> Thanks,
>
> Denys Kuzmenko
>
>
Re: Review Request 71792: COMPLETED_TXN_COMPONENTS table is never
cleaned up unless Compactor runs
Posted by Denys Kuzmenko via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71792/#review218723
-----------------------------------------------------------
Not ready. Need to handle aborted and currently active compactions.
- Denys Kuzmenko
On Nov. 20, 2019, 12:20 p.m., Denys Kuzmenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71792/
> -----------------------------------------------------------
>
> (Updated Nov. 20, 2019, 12:20 p.m.)
>
>
> Review request for hive, Laszlo Pinter and Peter Vary.
>
>
> Bugs: HIVE-21917
> https://issues.apache.org/jira/browse/HIVE-21917
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> The Initiator thread in the metastore repeatedly loops over entries in the COMPLETED_TXN_COMPONENTS table to determine which partitions / tables might need to be compacted. However, entries are never removed from this table except by a completed Compactor run.
>
> In a cluster where most tables / partitions are write-once read-many, this results in stale entries in this table never being cleaned up. In a small test cluster, we have observed approximately 45k entries in this table (virtually equal to the number of partitions in the cluster) while < 100 of these tables have delta files at all. Since most of the tables will never get enough writes to trigger a compaction (and in fact have only ever been written to once), the initiator thread keeps trying to evaluate them on every loop.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 610cf05204
> ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java b28b57779b
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java 8253ccb9c9
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java 6281208247
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java e840758c9d
>
>
> Diff: https://reviews.apache.org/r/71792/diff/1/
>
>
> Testing
> -------
>
> Unit tests
>
>
> Thanks,
>
> Denys Kuzmenko
>
>