You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2021/09/01 09:38:15 UTC
[GitHub] [hive] kasakrisz commented on a change in pull request #2612: added dedup for the COMPLETED_TXN_COMPONENTS
kasakrisz commented on a change in pull request #2612:
URL: https://github.com/apache/hive/pull/2612#discussion_r699351152
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##########
@@ -93,6 +93,11 @@ public CompactionTxnHandler() {
try {
dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
stmt = dbConn.createStatement();
+ // De-duplicate
+ stmt.execute("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE CTC_WRITEID\" NOT IN ("+
Review comment:
As far as I know compaction is also deletes records from the `COMPLETED_TXN_COMPONENTS` table. So after a source table of a materialized view is compacted incremental rebuild of the MV is no longer possible.
Currently the only way to exploit the benefits fo incremental MV rebuild is synchronising compaction and MV rebuild schedule.
If `AcidHouseKeeperService` runs more frequent than compaction it can lower the chances of an incremental MV rebuild.
If I understand correctly this extra delete statement will be run only when a compaction is in progress. Can we keep it here? @pvary
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##########
@@ -93,6 +93,11 @@ public CompactionTxnHandler() {
try {
dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
stmt = dbConn.createStatement();
+ // De-duplicate
+ stmt.execute("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE CTC_WRITEID\" NOT IN ("+
Review comment:
A default config value for `AcidHouseKeeperService` which equals the default compaction schedule is also good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org