You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Peter Varga (Jira)" <ji...@apache.org> on 2020/11/18 19:44:00 UTC
[jira] [Updated] (HIVE-24403) change min_history_level schema
change to be compatible with previous version
[ https://issues.apache.org/jira/browse/HIVE-24403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Varga updated HIVE-24403:
-------------------------------
Description:
In some configurations the HMS backend DB is used by HMS services with different versions.
HIVE-23107 dropped the min_history_level table from the backend DB making the new schema version incompatible with the older HMS services.
It is possible to modify that change to keep the compatibility
* Keep the min_history_level table
* Add the new fields for the compaction_queue the same way
* Create a feature flag for min_history_level and if it is on
* Keep the logic inserting to the table during openTxn
* Keep the logic removing the records at commitTxn and abortTxn
* Change the logic in the cleaner, to get the highwatermark the old way
* But still change it to not start the cleaning before that
* Change the logic in AcidHouseKeeper to clean the txn_to_write_id the old way
* This feature flag can be automatically setup based on the existence of the min_history level table, this way if the table will be dropped all HMS-s can switch to the new functionality without restart
was:
In some configurations the HMS backend DB is used by HMS services with different versions.
HIVE-23107 dropped the min_history_level table from the backend DB making the new schema version incompatible with the older HMS services.
It is possible to modify that change to keep the compatibility
* Keep the min_history_level table
* Add the new fields for the compaction_queue the same way
* Create a feature flag for min_history_level and if it is on
* Keep the logic inserting to the table during openTxn
* Change the logic in the cleaner, to get the highwatermark the old way
* But still change it to not start the cleaning before that
* Keep the min_history level delete after cleaner
* Change the logic in AcidHouseKeeper to clean the txn_to_write_id the old way
* This feature flag can be automatically setup based on the existence of the min_history level table, this way if the table will be dropped all HMS-s can switch to the new functionality without restart
> change min_history_level schema change to be compatible with previous version
> -----------------------------------------------------------------------------
>
> Key: HIVE-24403
> URL: https://issues.apache.org/jira/browse/HIVE-24403
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Peter Varga
> Assignee: Peter Varga
> Priority: Major
>
> In some configurations the HMS backend DB is used by HMS services with different versions.
> HIVE-23107 dropped the min_history_level table from the backend DB making the new schema version incompatible with the older HMS services.
> It is possible to modify that change to keep the compatibility
> * Keep the min_history_level table
> * Add the new fields for the compaction_queue the same way
> * Create a feature flag for min_history_level and if it is on
> * Keep the logic inserting to the table during openTxn
> * Keep the logic removing the records at commitTxn and abortTxn
> * Change the logic in the cleaner, to get the highwatermark the old way
> * But still change it to not start the cleaning before that
> * Change the logic in AcidHouseKeeper to clean the txn_to_write_id the old way
> * This feature flag can be automatically setup based on the existence of the min_history level table, this way if the table will be dropped all HMS-s can switch to the new functionality without restart
--
This message was sent by Atlassian Jira
(v8.3.4#803005)