You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "VenuReddy2103 (via GitHub)" <gi...@apache.org> on 2026/04/08 11:27:52 UTC
[PR] [WIP]Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
VenuReddy2103 opened a new pull request, #6416:
URL: https://github.com/apache/hive/pull/6416
### What changes were proposed in this pull request?
Introduce a configuration to enable/disable ACID functionality in Hive Metastore
### Why are the changes needed?
With the increasing adoption of the Apache Iceberg table format, many deployments no longer rely on ACID (transactional) tables provided by Apache Hive. In such environments, ACID-related functionality in the Hive Metastore becomes unnecessary.
Currently, Hive Metastore initializes and runs several background threads and services that are specifically designed to support ACID operations (e.g., compaction handling, transaction management, cleanup tasks). These components continue to run regardless of whether ACID tables are actually used in the deployment.
Running ACID-related services in environments that do not use ACID tables leads to:
1. Unnecessary resource consumption (CPU, memory, threads)
2. Increased operational overhead
3. Additional complexity in debugging and monitoring
4. Longer startup times and maintenance burden
Providing a mechanism to disable ACID functionality would streamline such deployments and improve efficiency.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added unittests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
Posted by "VenuReddy2103 (via GitHub)" <gi...@apache.org>.
VenuReddy2103 commented on code in PR #6416:
URL: https://github.com/apache/hive/pull/6416#discussion_r3159181072
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java:
##########
@@ -614,6 +614,26 @@ private static final Path getDefaultPath(IHMSHandler hmsHandler, Database db, St
}
+ private void transformToExternalIfAcidNotSupported(Table table) throws MetaException {
+ Map<String, String> params = table.getParameters();
+ boolean isSupportAcid = MetastoreConf.getBoolVar(hmsHandler.getConf(),
+ ConfVars.METASTORE_SUPPORT_ACID);
+ if (!isSupportAcid) {
+ if (Boolean.parseBoolean(params.get(TABLE_IS_TRANSACTIONAL))) {
+ throw new MetaException("ACID tables are not permitted when the "
Review Comment:
We do not allow transactional tables when acid is disabled on HMS. Throwing exception to engine as an indication of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6416:
URL: https://github.com/apache/hive/pull/6416#discussion_r3157914861
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java:
##########
@@ -614,6 +614,26 @@ private static final Path getDefaultPath(IHMSHandler hmsHandler, Database db, St
}
+ private void transformToExternalIfAcidNotSupported(Table table) throws MetaException {
+ Map<String, String> params = table.getParameters();
+ boolean isSupportAcid = MetastoreConf.getBoolVar(hmsHandler.getConf(),
+ ConfVars.METASTORE_SUPPORT_ACID);
+ if (!isSupportAcid) {
+ if (Boolean.parseBoolean(params.get(TABLE_IS_TRANSACTIONAL))) {
+ throw new MetaException("ACID tables are not permitted when the "
Review Comment:
why do we throw the exception since we are able to transform a managed transactional table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP]HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6416:
URL: https://github.com/apache/hive/pull/6416#issuecomment-4212624857
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6416) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6416&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6416&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6416&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6416&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6416&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6416)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP]HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6416:
URL: https://github.com/apache/hive/pull/6416#discussion_r3083684655
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/HouseKeepingTasks.java:
##########
@@ -69,6 +69,12 @@ public List<MetastoreTaskThread> getRemoteOnlyTasks() throws Exception {
Collection<String> taskNames =
MetastoreConf.getStringCollection(configuration, MetastoreConf.ConfVars.TASK_THREADS_REMOTE_ONLY);
+ if (!MetastoreConf.getBoolVar(configuration, MetastoreConf.ConfVars.METASTORE_SUPPORT_ACID)) {
+ taskNames.removeAll(ImmutableSet.of(MetastoreConf.ACID_OPEN_TXNS_COUNTER_SERVICE_CLASS,
+ MetastoreConf.ACID_HOUSEKEEPER_SERVICE_CLASS,
Review Comment:
can we honor this `METASTORE_SUPPORT_ACID` per task, such as `CompactionHouseKeeperService.java`. If the task is disabled completely, then make `runFrequency` return 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
Posted by "VenuReddy2103 (via GitHub)" <gi...@apache.org>.
VenuReddy2103 commented on code in PR #6416:
URL: https://github.com/apache/hive/pull/6416#discussion_r3123091899
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/HouseKeepingTasks.java:
##########
@@ -69,6 +69,12 @@ public List<MetastoreTaskThread> getRemoteOnlyTasks() throws Exception {
Collection<String> taskNames =
MetastoreConf.getStringCollection(configuration, MetastoreConf.ConfVars.TASK_THREADS_REMOTE_ONLY);
+ if (!MetastoreConf.getBoolVar(configuration, MetastoreConf.ConfVars.METASTORE_SUPPORT_ACID)) {
+ taskNames.removeAll(ImmutableSet.of(MetastoreConf.ACID_OPEN_TXNS_COUNTER_SERVICE_CLASS,
+ MetastoreConf.ACID_HOUSEKEEPER_SERVICE_CLASS,
+ MetastoreConf.ACID_TXN_CLEANER_SERVICE_CLASS,
+ MetastoreConf.MATERIALZIATIONS_REBUILD_LOCK_CLEANER_TASK_CLASS));
Review Comment:
When acid function is disabled, we support materialized views only in iceberg format(i.e, cannot have mv in HIVE ACID format). MV in iceberg format do not use `MATERIALIZATION_REBUILD_LOCKS`. Have refactored this code for another comment to disable the task when `runFrequency()` returns 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6416:
URL: https://github.com/apache/hive/pull/6416#issuecomment-4297852868
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6416) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6416&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6416&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6416&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6416&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6416&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6416)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6416:
URL: https://github.com/apache/hive/pull/6416#discussion_r3192729518
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java:
##########
@@ -614,6 +614,26 @@ private static final Path getDefaultPath(IHMSHandler hmsHandler, Database db, St
}
+ private void transformToExternalIfAcidNotSupported(Table table) throws MetaException {
+ Map<String, String> params = table.getParameters();
+ boolean isSupportAcid = MetastoreConf.getBoolVar(hmsHandler.getConf(),
+ ConfVars.METASTORE_SUPPORT_ACID);
+ if (!isSupportAcid) {
+ if (Boolean.parseBoolean(params.get(TABLE_IS_TRANSACTIONAL))) {
+ throw new MetaException("ACID tables are not permitted when the "
+ + ConfVars.METASTORE_SUPPORT_ACID.getHiveName() + " property is set to false");
+ }
+ if (TableType.MANAGED_TABLE.name().equals(table.getTableType())) {
+ table.setTableType(TableType.EXTERNAL_TABLE.toString());
Review Comment:
this means all tables' data cannot be removed upon dropping the table, until we drop these table files directly, is it supposed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
Posted by "VenuReddy2103 (via GitHub)" <gi...@apache.org>.
VenuReddy2103 commented on code in PR #6416:
URL: https://github.com/apache/hive/pull/6416#discussion_r3159149464
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java:
##########
@@ -614,6 +614,26 @@ private static final Path getDefaultPath(IHMSHandler hmsHandler, Database db, St
}
+ private void transformToExternalIfAcidNotSupported(Table table) throws MetaException {
+ Map<String, String> params = table.getParameters();
+ boolean isSupportAcid = MetastoreConf.getBoolVar(hmsHandler.getConf(),
+ ConfVars.METASTORE_SUPPORT_ACID);
+ if (!isSupportAcid) {
+ if (Boolean.parseBoolean(params.get(TABLE_IS_TRANSACTIONAL))) {
+ throw new MetaException("ACID tables are not permitted when the "
+ + ConfVars.METASTORE_SUPPORT_ACID.getHiveName() + " property is set to false");
+ }
+ if (TableType.MANAGED_TABLE.name().equals(table.getTableType())) {
+ table.setTableType(TableType.EXTERNAL_TABLE.toString());
Review Comment:
It is intentionally done. When the acid functionality is disabled, they are purely external tables. They are not manged tables transalated to external ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
Posted by "VenuReddy2103 (via GitHub)" <gi...@apache.org>.
VenuReddy2103 commented on code in PR #6416:
URL: https://github.com/apache/hive/pull/6416#discussion_r3159149464
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java:
##########
@@ -614,6 +614,26 @@ private static final Path getDefaultPath(IHMSHandler hmsHandler, Database db, St
}
+ private void transformToExternalIfAcidNotSupported(Table table) throws MetaException {
+ Map<String, String> params = table.getParameters();
+ boolean isSupportAcid = MetastoreConf.getBoolVar(hmsHandler.getConf(),
+ ConfVars.METASTORE_SUPPORT_ACID);
+ if (!isSupportAcid) {
+ if (Boolean.parseBoolean(params.get(TABLE_IS_TRANSACTIONAL))) {
+ throw new MetaException("ACID tables are not permitted when the "
+ + ConfVars.METASTORE_SUPPORT_ACID.getHiveName() + " property is set to false");
+ }
+ if (TableType.MANAGED_TABLE.name().equals(table.getTableType())) {
+ table.setTableType(TableType.EXTERNAL_TABLE.toString());
Review Comment:
It is intentionally done. When the acid functionality is disabled, they are purely external tables. They are not transalated to external ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP]HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
Posted by "VenuReddy2103 (via GitHub)" <gi...@apache.org>.
VenuReddy2103 commented on PR #6416:
URL: https://github.com/apache/hive/pull/6416#issuecomment-4206030156
> why do we need new toggle for that?
`metastore.support.acid` is a startup configuration. If the configuration is set to `false`, We can avoid starting acid related threads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP]HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6416:
URL: https://github.com/apache/hive/pull/6416#issuecomment-4206283715
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6416) **Quality Gate passed**
Issues
 [12 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6416&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6416&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6416&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6416&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6416&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6416)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6416:
URL: https://github.com/apache/hive/pull/6416#issuecomment-4345155354
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6416) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6416&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6416&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6416&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6416&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6416&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6416)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP]HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6416:
URL: https://github.com/apache/hive/pull/6416#discussion_r3051121614
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/HouseKeepingTasks.java:
##########
@@ -69,6 +69,12 @@ public List<MetastoreTaskThread> getRemoteOnlyTasks() throws Exception {
Collection<String> taskNames =
MetastoreConf.getStringCollection(configuration, MetastoreConf.ConfVars.TASK_THREADS_REMOTE_ONLY);
+ if (!MetastoreConf.getBoolVar(configuration, MetastoreConf.ConfVars.METASTORE_SUPPORT_ACID)) {
+ taskNames.removeAll(ImmutableSet.of(MetastoreConf.ACID_OPEN_TXNS_COUNTER_SERVICE_CLASS,
+ MetastoreConf.ACID_HOUSEKEEPER_SERVICE_CLASS,
+ MetastoreConf.ACID_TXN_CLEANER_SERVICE_CLASS,
+ MetastoreConf.MATERIALZIATIONS_REBUILD_LOCK_CLEANER_TASK_CLASS));
Review Comment:
Hive provides support for iceberg tables materialized views
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6416:
URL: https://github.com/apache/hive/pull/6416#discussion_r3157904817
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java:
##########
@@ -614,6 +614,26 @@ private static final Path getDefaultPath(IHMSHandler hmsHandler, Database db, St
}
+ private void transformToExternalIfAcidNotSupported(Table table) throws MetaException {
+ Map<String, String> params = table.getParameters();
+ boolean isSupportAcid = MetastoreConf.getBoolVar(hmsHandler.getConf(),
+ ConfVars.METASTORE_SUPPORT_ACID);
+ if (!isSupportAcid) {
+ if (Boolean.parseBoolean(params.get(TABLE_IS_TRANSACTIONAL))) {
+ throw new MetaException("ACID tables are not permitted when the "
+ + ConfVars.METASTORE_SUPPORT_ACID.getHiveName() + " property is set to false");
+ }
+ if (TableType.MANAGED_TABLE.name().equals(table.getTableType())) {
+ table.setTableType(TableType.EXTERNAL_TABLE.toString());
Review Comment:
if we translate the manage table to external just by changing the table type, the table data wouldn't be deleted when the table dropped, as the following params are missing:
```
params.put(HiveMetaHook.EXTERNAL, "TRUE");
params.put(EXTERNAL_TABLE_PURGE, "TRUE");
params.put(HiveMetaHook.TRANSLATED_TO_EXTERNAL, "TRUE");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP]HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
Posted by "VenuReddy2103 (via GitHub)" <gi...@apache.org>.
VenuReddy2103 commented on code in PR #6416:
URL: https://github.com/apache/hive/pull/6416#discussion_r3086565588
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/HouseKeepingTasks.java:
##########
@@ -69,6 +69,12 @@ public List<MetastoreTaskThread> getRemoteOnlyTasks() throws Exception {
Collection<String> taskNames =
MetastoreConf.getStringCollection(configuration, MetastoreConf.ConfVars.TASK_THREADS_REMOTE_ONLY);
+ if (!MetastoreConf.getBoolVar(configuration, MetastoreConf.ConfVars.METASTORE_SUPPORT_ACID)) {
+ taskNames.removeAll(ImmutableSet.of(MetastoreConf.ACID_OPEN_TXNS_COUNTER_SERVICE_CLASS,
+ MetastoreConf.ACID_HOUSEKEEPER_SERVICE_CLASS,
Review Comment:
Have tried to handle at `LeaderElection.LeadershipStateListener` level most of the cases like `CompactorTasks`, `HouseKeepingTasks` instead of leaf level (`MetastoreTaskThread`). Thought we can avoid all those instance creations, `setConf()`/`init() `which internally gets `TxnUtils.getTxnStore(conf)` and internally `setConf()` of txnHandler etc if we do at `LeadershipStateListener` level.
But in case of `CompactionHouseKeeperService`, a part of its functionality((i.e., `txnHandler::purgeCompactionHistory` alone) is required when `MetastoreConf.ConfVars.METASTORE_SUPPORT_ACID` is disabled. So have handled at leaf level.
What is your opinion ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
Posted by "VenuReddy2103 (via GitHub)" <gi...@apache.org>.
VenuReddy2103 commented on PR #6416:
URL: https://github.com/apache/hive/pull/6416#issuecomment-4614086078
Have created https://github.com/apache/hive/pull/6524
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
Posted by "VenuReddy2103 (via GitHub)" <gi...@apache.org>.
VenuReddy2103 commented on code in PR #6416:
URL: https://github.com/apache/hive/pull/6416#discussion_r3349842392
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java:
##########
@@ -614,6 +614,26 @@ private static final Path getDefaultPath(IHMSHandler hmsHandler, Database db, St
}
+ private void transformToExternalIfAcidNotSupported(Table table) throws MetaException {
+ Map<String, String> params = table.getParameters();
+ boolean isSupportAcid = MetastoreConf.getBoolVar(hmsHandler.getConf(),
+ ConfVars.METASTORE_SUPPORT_ACID);
+ if (!isSupportAcid) {
+ if (Boolean.parseBoolean(params.get(TABLE_IS_TRANSACTIONAL))) {
+ throw new MetaException("ACID tables are not permitted when the "
+ + ConfVars.METASTORE_SUPPORT_ACID.getHiveName() + " property is set to false");
+ }
+ if (TableType.MANAGED_TABLE.name().equals(table.getTableType())) {
+ table.setTableType(TableType.EXTERNAL_TABLE.toString());
Review Comment:
That would be an issue. Have fixed it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
Posted by "VenuReddy2103 (via GitHub)" <gi...@apache.org>.
VenuReddy2103 closed pull request #6416: HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore
URL: https://github.com/apache/hive/pull/6416
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
Posted by "VenuReddy2103 (via GitHub)" <gi...@apache.org>.
VenuReddy2103 commented on PR #6416:
URL: https://github.com/apache/hive/pull/6416#issuecomment-4613302220
Will create a new PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP]HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6416:
URL: https://github.com/apache/hive/pull/6416#issuecomment-4206002174
why do we need toggle for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore [hive]
Posted by "VenuReddy2103 (via GitHub)" <gi...@apache.org>.
VenuReddy2103 commented on code in PR #6416:
URL: https://github.com/apache/hive/pull/6416#discussion_r3123057462
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/HouseKeepingTasks.java:
##########
@@ -69,6 +69,12 @@ public List<MetastoreTaskThread> getRemoteOnlyTasks() throws Exception {
Collection<String> taskNames =
MetastoreConf.getStringCollection(configuration, MetastoreConf.ConfVars.TASK_THREADS_REMOTE_ONLY);
+ if (!MetastoreConf.getBoolVar(configuration, MetastoreConf.ConfVars.METASTORE_SUPPORT_ACID)) {
+ taskNames.removeAll(ImmutableSet.of(MetastoreConf.ACID_OPEN_TXNS_COUNTER_SERVICE_CLASS,
+ MetastoreConf.ACID_HOUSEKEEPER_SERVICE_CLASS,
Review Comment:
> can we honor this `METASTORE_SUPPORT_ACID` per task, such as `CompactionHouseKeeperService.java`. If the task is disabled completely, then make `runFrequency` return 0.
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org