You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "wecharyu (via GitHub)" <gi...@apache.org> on 2025/04/21 17:01:51 UTC

[PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

wecharyu opened a new pull request, #5780:
URL: https://github.com/apache/hive/pull/5780

   ### What changes were proposed in this pull request?
   Fix the bug in `ObjectStore#openTransaction` and `ObjectStore#rollbackTransaction`.
   
   
   ### Why are the changes needed?
   Currently `rollbackTransaction()` always reset `openTransactionCalls` to 0, but it would cause unbalanced issue in nested transaction calls, for example:
   ```bash
   openTransaction()
     // new function1
     {
       openTransaction()
       commitTransaction() # place1
     }
   
     // new function2
     {
       openTransaction()
       commitTransaction()
     }
   commitTransaction() # place2
   ```
   If the new function1 rollbackTransaction in place1, then it would get unbalanced exception in place2.
   
   
   ### Does this PR introduce _any_ user-facing change?
   No.
   
   ### How was this patch tested?
   Add a unit test:
   ```bash
   mvn test -Dtest.groups= -Dtest=org.apache.hadoop.hive.metastore.TestObjectStore#testNestedTransaction -pl :hive-standalone-metastore-server
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5780:
URL: https://github.com/apache/hive/pull/5780#discussion_r2079670620


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -561,17 +561,10 @@ public void shutdown() {
   @Override
   public boolean openTransaction() {
     openTrasactionCalls++;
-    if (openTrasactionCalls == 1) {
+    if (currentTransaction == null || !currentTransaction.isActive()){

Review Comment:
   Why do we check for `!currentTransaction.isActive()`? What if we called rollback on the interior txn? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on PR #5780:
URL: https://github.com/apache/hive/pull/5780#issuecomment-2826235004

   Some thoughts:
   -  at `place1`, the `openTrasactionCalls` should be 2, so `commitTransaction` is just changing the `openTrasactionCalls` to 1, we don't reach the end of the transaction here.
   -  if we rollback at the `place1`, doesn't it mean we get an unexpected exception here, why do we continue to execute to `place2`? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #5780:
URL: https://github.com/apache/hive/pull/5780#issuecomment-3462589108

   @wecharyu, sorry for abandoning the review! @dengzhhu653 we should probably revisit this PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5780:
URL: https://github.com/apache/hive/pull/5780#discussion_r2135650183


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -662,7 +655,7 @@ public void rollbackTransaction() {
         currentTransaction.rollback();
       }
     } finally {
-      openTrasactionCalls = 0;
+      openTrasactionCalls--;

Review Comment:
   that is what we support now, isn't ? the inner transactions are somewhat similar to @Transactional(REQUIRED) annotation



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5780:
URL: https://github.com/apache/hive/pull/5780#discussion_r2079665820


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java:
##########
@@ -191,6 +184,10 @@ public void alterTable(RawStore msdb, Warehouse wh, String catName, String dbnam
             TableName.getQualified(catName, dbname, name) + " doesn't exist");
       }
 
+      String expectedKey = environmentContext != null && environmentContext.getProperties() != null ?

Review Comment:
   why move non-transactional logic under the txn boundary?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5780:
URL: https://github.com/apache/hive/pull/5780#discussion_r2135653957


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -3254,31 +3247,38 @@ protected List<Partition> getJdoResult(GetHelper<List<Partition>> ctx) throws Me
   public Partition getPartitionWithAuth(String catName, String dbName, String tblName,
       List<String> partVals, String user_name, List<String> group_names)
       throws NoSuchObjectException, MetaException, InvalidObjectException {
-    boolean success = false;
-    try {
-      openTransaction();
-      MPartition mpart = getMPartition(catName, dbName, tblName, partVals, null);
-      if (mpart == null) {
-        commitTransaction();
-        throw new NoSuchObjectException("partition values="
-            + partVals.toString());
+    return new GetHelper<Partition>(catName, dbName, tblName, false, true) {

Review Comment:
   does it even run in txn? where do we open it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5780:
URL: https://github.com/apache/hive/pull/5780#discussion_r2079660465


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java:
##########
@@ -983,12 +983,11 @@ public void alter_catalog(AlterCatalogRequest rqst) throws TException {
     GetCatalogResponse oldCat = null;
 
     try {
+      ms.openTransaction();

Review Comment:
   why do we need to open txn before the `PreAlterCatalogEvent`? are those transactional as well?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5780:
URL: https://github.com/apache/hive/pull/5780#discussion_r2079689026


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -3254,31 +3247,38 @@ protected List<Partition> getJdoResult(GetHelper<List<Partition>> ctx) throws Me
   public Partition getPartitionWithAuth(String catName, String dbName, String tblName,
       List<String> partVals, String user_name, List<String> group_names)
       throws NoSuchObjectException, MetaException, InvalidObjectException {
-    boolean success = false;
-    try {
-      openTransaction();
-      MPartition mpart = getMPartition(catName, dbName, tblName, partVals, null);
-      if (mpart == null) {
-        commitTransaction();
-        throw new NoSuchObjectException("partition values="
-            + partVals.toString());
+    return new GetHelper<Partition>(catName, dbName, tblName, false, true) {

Review Comment:
   why refactor?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #5780:
URL: https://github.com/apache/hive/pull/5780#discussion_r2085001834


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java:
##########
@@ -1928,9 +1928,8 @@ private void create_dataconnector_core(RawStore ms, final DataConnector connecto
     boolean success = false;
     Map<String, String> transactionalListenersResponses = Collections.emptyMap();
     try {
-      firePreEvent(new PreCreateDataConnectorEvent(connector, this));
-
       ms.openTransaction();
+      firePreEvent(new PreCreateDataConnectorEvent(connector, this));

Review Comment:
   ditto, `firePreEvent` may throw exception and not call `openTransaction()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5780:
URL: https://github.com/apache/hive/pull/5780#issuecomment-3169572462

   This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
   Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5780:
URL: https://github.com/apache/hive/pull/5780#discussion_r2079658212


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java:
##########
@@ -1928,9 +1928,8 @@ private void create_dataconnector_core(RawStore ms, final DataConnector connecto
     boolean success = false;
     Map<String, String> transactionalListenersResponses = Collections.emptyMap();
     try {
-      firePreEvent(new PreCreateDataConnectorEvent(connector, this));
-
       ms.openTransaction();
+      firePreEvent(new PreCreateDataConnectorEvent(connector, this));

Review Comment:
   why the change?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5780:
URL: https://github.com/apache/hive/pull/5780#discussion_r2079670620


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -561,17 +561,10 @@ public void shutdown() {
   @Override
   public boolean openTransaction() {
     openTrasactionCalls++;
-    if (openTrasactionCalls == 1) {
+    if (currentTransaction == null || !currentTransaction.isActive()){

Review Comment:
   Why do we check for `!currentTransaction.isActive()`? Can we get `openTrasactionCalls > 1` for a newly opened txn?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #5780:
URL: https://github.com/apache/hive/pull/5780#discussion_r2085022464


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -561,17 +561,10 @@ public void shutdown() {
   @Override
   public boolean openTransaction() {
     openTrasactionCalls++;
-    if (openTrasactionCalls == 1) {
+    if (currentTransaction == null || !currentTransaction.isActive()){

Review Comment:
   To open transaction if it's not opened. It would reopen a new transaction if has already called rollback.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on PR #5780:
URL: https://github.com/apache/hive/pull/5780#issuecomment-2839402119

   > if we rollback at the place1, doesn't it mean we get an unexpected exception here, why do we continue to execute to place2?
   
   @dengzhhu653  There are some cases where rollback transaction does not need fail the api calls, for example `create_xxx` will always check if it already exists by `get_xxx` that throw expected `NoSuchObjectException`, in this case we should continue other queries.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5780:
URL: https://github.com/apache/hive/pull/5780#issuecomment-2819115039

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5780) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [5 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5780&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5780&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5780&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5780&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5780&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5780)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5780:
URL: https://github.com/apache/hive/pull/5780#issuecomment-2839557318

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5780) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [6 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5780&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5780&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5780&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5780&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5780&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5780)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on PR #5780:
URL: https://github.com/apache/hive/pull/5780#issuecomment-2840815511

   > Per my understanding, this shouldn't happen, imaging we rollback the transaction at place1, then the ObjectStore 
    openTrasactionCalls is reset to 0, if we continue to execute, then means we have one openTransaction and two commitTransaction, the commitTransaction at place2 should throw the exception:
   > ```java
   > if (openTrasactionCalls <= 0) {
   >      RuntimeException e = new RuntimeException("commitTransaction was called but openTransactionCalls = "
   >          + openTrasactionCalls + ". This probably indicates that there are unbalanced " +
   >          "calls to openTransaction/commitTransaction");
   >      LOG.error("Unbalanced calls to open/commit Transaction", e);
   >      throw e;
   > }
   > ```
   >the Metastore client call should get this exception, but our tests work fine.
   
   Yes it works now because current code does not put `get_xxx` and `create_xxx` into the same transaction, it would fail if we    openTransaction at the top of a create_xxx api.
   
   Additionally, the failed tests in this PR reveal following unbalanced calls that are hidden by current implementation:
   - `commitTransaction()` twice for a `openTransaction()`
   - `rollbackTransaction()` even it does not call `openTransaction()`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on PR #5780:
URL: https://github.com/apache/hive/pull/5780#issuecomment-2840565842

   > > if we rollback at the place1, doesn't it mean we get an unexpected exception here, why do we continue to execute to place2?
   > 
   > @dengzhhu653 There are some cases where rollback transaction does not need fail the api calls, for example `create_xxx` will always check if it already exists by `get_xxx` that throw expected `NoSuchObjectException`, in this case we should continue other queries.
   
   Per my understanding, this shouldn't happen, imaging we rollback the transaction at place1, then the ObjectStore `openTrasactionCalls` is reset to 0, if we continue to execute, then means we have one `openTransaction` and two `commitTransaction`, the `commitTransaction` at place2 should throw the exception:
   ```
   if (openTrasactionCalls <= 0) {
         RuntimeException e = new RuntimeException("commitTransaction was called but openTransactionCalls = "
             + openTrasactionCalls + ". This probably indicates that there are unbalanced " +
             "calls to openTransaction/commitTransaction");
         LOG.error("Unbalanced calls to open/commit Transaction", e);
         throw e;
    }
   ```
   the Metastore client call should get this exception, but our tests work fine.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5780:
URL: https://github.com/apache/hive/pull/5780#issuecomment-2821829542

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5780) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [5 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5780&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5780&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5780&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5780&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5780&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5780)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5780:
URL: https://github.com/apache/hive/pull/5780#issuecomment-2825136112

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5780) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [7 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5780&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5780&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5780&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5780&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5780&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5780)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5780:
URL: https://github.com/apache/hive/pull/5780#discussion_r2079656878


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java:
##########
@@ -1299,7 +1297,9 @@ private void create_database_core(RawStore ms, final Database db)
       success = ms.commitTransaction();
     } finally {
       if (!success) {
-        ms.rollbackTransaction();
+        if (openTransaction) {

Review Comment:
   why are you checking for an `openTransaction` flag here? I don't see the same check in other places. 
   ````
   if (isActiveTransaction() && transactionStatus != TXN_STATUS.ROLLBACK) {
       currentTransaction.rollback();
   }
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5780:
URL: https://github.com/apache/hive/pull/5780#discussion_r2079684168


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -629,13 +622,13 @@ public boolean commitTransaction() {
       LOG.error("Unbalanced calls to open/commit Transaction", e);
       throw e;
     }
-    openTrasactionCalls--;
-    debugLog("Commit transaction: count = " + openTrasactionCalls + ", isactive "+ currentTransaction.isActive());
 
-    if ((openTrasactionCalls == 0) && currentTransaction.isActive()) {
+    debugLog("Commit transaction: count = " + openTrasactionCalls + ", isactive "+ currentTransaction.isActive());
+    if ((openTrasactionCalls == 1) && currentTransaction.isActive()) {
       transactionStatus = TXN_STATUS.COMMITED;
       currentTransaction.commit();
     }
+    openTrasactionCalls--;

Review Comment:
   what if commit throws RuntimeException, db txn would be closed, but openTrasactionCalls won't be 0



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5780:
URL: https://github.com/apache/hive/pull/5780#discussion_r2135633264


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java:
##########
@@ -1299,7 +1297,9 @@ private void create_database_core(RawStore ms, final Database db)
       success = ms.commitTransaction();
     } finally {
       if (!success) {
-        ms.rollbackTransaction();
+        if (openTransaction) {

Review Comment:
   won't `isActiveTransaction()` return false?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5780:
URL: https://github.com/apache/hive/pull/5780#discussion_r2135638784


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java:
##########
@@ -1928,9 +1928,8 @@ private void create_dataconnector_core(RawStore ms, final DataConnector connecto
     boolean success = false;
     Map<String, String> transactionalListenersResponses = Collections.emptyMap();
     try {
-      firePreEvent(new PreCreateDataConnectorEvent(connector, this));
-
       ms.openTransaction();
+      firePreEvent(new PreCreateDataConnectorEvent(connector, this));

Review Comment:
   What is the point? We open a txn, throw an exception in `firePreEvent`, and then abort txn?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5780:
URL: https://github.com/apache/hive/pull/5780#discussion_r2054381088


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -3252,7 +3252,7 @@ public Partition getPartitionWithAuth(String catName, String dbName, String tblN
       openTransaction();
       MPartition mpart = getMPartition(catName, dbName, tblName, partVals, null);
       if (mpart == null) {
-        commitTransaction();
+        success = commitTransaction();

Review Comment:
   why not just move `commitTransaction()` to finally and remove the flag completely? this is a simple select, there is nothing to rollback



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #5780:
URL: https://github.com/apache/hive/pull/5780#discussion_r2056541693


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -3252,7 +3252,7 @@ public Partition getPartitionWithAuth(String catName, String dbName, String tblN
       openTransaction();
       MPartition mpart = getMPartition(catName, dbName, tblName, partVals, null);
       if (mpart == null) {
-        commitTransaction();
+        success = commitTransaction();

Review Comment:
   Addressed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #5780:
URL: https://github.com/apache/hive/pull/5780#discussion_r2084999446


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java:
##########
@@ -1299,7 +1297,9 @@ private void create_database_core(RawStore ms, final Database db)
       success = ms.commitTransaction();
     } finally {
       if (!success) {
-        ms.rollbackTransaction();
+        if (openTransaction) {

Review Comment:
   In `create_database_core` it may throw exception before calling `openTransaction()`, so it may not need call `rollbackTransaction()` in finally block.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #5780:
URL: https://github.com/apache/hive/pull/5780#discussion_r2085049414


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -662,7 +655,7 @@ public void rollbackTransaction() {
         currentTransaction.rollback();
       }
     } finally {
-      openTrasactionCalls = 0;
+      openTrasactionCalls--;

Review Comment:
   Our unit test could not pass if is 0 here, the key point seems that if we should support such usage:
   ```bash
   openTransaction()
     // new function1
     {
       openTransaction()
       commitTransaction() # place1
     }
   
     // new function2
     {
       openTransaction()
       commitTransaction()
     }
   commitTransaction() # place2
   ```



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -629,13 +622,13 @@ public boolean commitTransaction() {
       LOG.error("Unbalanced calls to open/commit Transaction", e);
       throw e;
     }
-    openTrasactionCalls--;
-    debugLog("Commit transaction: count = " + openTrasactionCalls + ", isactive "+ currentTransaction.isActive());
 
-    if ((openTrasactionCalls == 0) && currentTransaction.isActive()) {
+    debugLog("Commit transaction: count = " + openTrasactionCalls + ", isactive "+ currentTransaction.isActive());
+    if ((openTrasactionCalls == 1) && currentTransaction.isActive()) {
       transactionStatus = TXN_STATUS.COMMITED;
       currentTransaction.commit();
     }
+    openTrasactionCalls--;

Review Comment:
   If commit throw RuntimeException, it should call rollback in the finally block of previous layer where the openTransactionCalls would be decreased.



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -3254,31 +3247,38 @@ protected List<Partition> getJdoResult(GetHelper<List<Partition>> ctx) throws Me
   public Partition getPartitionWithAuth(String catName, String dbName, String tblName,
       List<String> partVals, String user_name, List<String> group_names)
       throws NoSuchObjectException, MetaException, InvalidObjectException {
-    boolean success = false;
-    try {
-      openTransaction();
-      MPartition mpart = getMPartition(catName, dbName, tblName, partVals, null);
-      if (mpart == null) {
-        commitTransaction();
-        throw new NoSuchObjectException("partition values="
-            + partVals.toString());
+    return new GetHelper<Partition>(catName, dbName, tblName, false, true) {

Review Comment:
   It may throw exception, so we need such logic:
   ```java
   try {
     openTransaction();
   
     success = commitTransaction();
   } finally {
     if (!success) rollbackTransaction();
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5780:
URL: https://github.com/apache/hive/pull/5780#discussion_r2079685769


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -662,7 +655,7 @@ public void rollbackTransaction() {
         currentTransaction.rollback();
       }
     } finally {
-      openTrasactionCalls = 0;
+      openTrasactionCalls--;

Review Comment:
   afyter rollback openTrasactionCalls should be 0



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -662,7 +655,7 @@ public void rollbackTransaction() {
         currentTransaction.rollback();
       }
     } finally {
-      openTrasactionCalls = 0;
+      openTrasactionCalls--;

Review Comment:
   after rollback openTrasactionCalls should be 0



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5780:
URL: https://github.com/apache/hive/pull/5780#discussion_r2135641056


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -561,17 +561,10 @@ public void shutdown() {
   @Override
   public boolean openTransaction() {
     openTrasactionCalls++;
-    if (openTrasactionCalls == 1) {
+    if (currentTransaction == null || !currentTransaction.isActive()){

Review Comment:
   if the interior txn was rolled back, should we rollback everything?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction [hive]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #5780: HIVE-28916: Fix unbalannced calls in ObjectStore rollback transaction
URL: https://github.com/apache/hive/pull/5780


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org