You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "abstractdog (via GitHub)" <gi...@apache.org> on 2026/08/04 13:12:36 UTC

[PR] HIVE-29798: NumberFormatException while reading a table having UNION subdirs after ACID conversion [hive]

abstractdog opened a new pull request, #6675:
URL: https://github.com/apache/hive/pull/6675

   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29798: NumberFormatException while reading a table having UNION subdirs after ACID conversion [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6675:
URL: https://github.com/apache/hive/pull/6675#discussion_r3925593291


##########
ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java:
##########
@@ -146,12 +146,18 @@ public void flattenUnionSubdirectories(Path sourcePath) throws HiveException {
           Path path = i.next().getPath();
           Path parent = path.getParent();
           if (parent.getName().startsWith(prefix)) {
-            // We do rename by including the name of parent directory into the filename so that there are no clashes
-            // when we move the files to the parent directory. Ex. HIVE_UNION_SUBDIR_1/000000_0 -> 1_000000_0
+            // Fold the subdir index into the filename so there are no clashes when the files
+            // are moved to the parent directory, and so that the resulting name still matches
+            // the "original data file" convention that both the ACID reader (AcidUtils.
+            // ORIGINAL_PATTERN_COPY) and the metastore's non-ACID→ACID validator (Transactional
+            // ValidationListener.ORIGINAL_PATTERN_COPY) recognize:
+            //   HIVE_UNION_SUBDIR_1/000000_0 -> 000000_0_copy_1
+            // See Utilities.COPY_KEYWORD.
             String parentOfParent = parent.getParent().toString();
             String parentNameSuffix = parent.getName().substring(prefix.length());
 
-            fs.rename(path, new Path(parentOfParent + "/" + parentNameSuffix + "_" + path.getName()));
+            fs.rename(path,
+                new Path(parentOfParent + "/" + path.getName() + Utilities.COPY_KEYWORD + parentNameSuffix));

Review Comment:
   very good catch, I had to redesign the logic used in movetask, addressed in: https://github.com/apache/hive/pull/6675/changes/2057a99182e129f5d1023976395e54375321b1f2



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29798: NumberFormatException while reading a table having UNION subdirs after ACID conversion [hive]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #6675:
URL: https://github.com/apache/hive/pull/6675#discussion_r3854385383


##########
ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java:
##########
@@ -146,12 +146,18 @@ public void flattenUnionSubdirectories(Path sourcePath) throws HiveException {
           Path path = i.next().getPath();
           Path parent = path.getParent();
           if (parent.getName().startsWith(prefix)) {
-            // We do rename by including the name of parent directory into the filename so that there are no clashes
-            // when we move the files to the parent directory. Ex. HIVE_UNION_SUBDIR_1/000000_0 -> 1_000000_0
+            // Fold the subdir index into the filename so there are no clashes when the files
+            // are moved to the parent directory, and so that the resulting name still matches
+            // the "original data file" convention that both the ACID reader (AcidUtils.
+            // ORIGINAL_PATTERN_COPY) and the metastore's non-ACID→ACID validator (Transactional
+            // ValidationListener.ORIGINAL_PATTERN_COPY) recognize:
+            //   HIVE_UNION_SUBDIR_1/000000_0 -> 000000_0_copy_1
+            // See Utilities.COPY_KEYWORD.
             String parentOfParent = parent.getParent().toString();
             String parentNameSuffix = parent.getName().substring(prefix.length());
 
-            fs.rename(path, new Path(parentOfParent + "/" + parentNameSuffix + "_" + path.getName()));
+            fs.rename(path,
+                new Path(parentOfParent + "/" + path.getName() + Utilities.COPY_KEYWORD + parentNameSuffix));

Review Comment:
   This new name already contains a `_copy_<N>` suffix, but the subsequent `Hive.loadTable` move still treats it as an ordinary basename. On non-atomic filesystems, `Hive.pickDestFilePath` unconditionally appends its per-query suffix, producing names such as `000000_0_copy_1_copy_<tag>`; on atomic filesystems the same happens whenever a prior flattened insert has already created `000000_0_copy_1`. Those names do not match `ORIGINAL_PATTERN_COPY`, so flattening can still make later inserts unreadable or cause ACID conversion validation to fail. Please coordinate the destination-name handling to preserve a single valid copy suffix (and add a flatten-on repeated-insert/non-atomic regression test).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29798: NumberFormatException while reading a table having UNION subdirs after ACID conversion [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6675:
URL: https://github.com/apache/hive/pull/6675#issuecomment-5531848520

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6675) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [6 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6675&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6675&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6675&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6675&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6675&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6675)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29798: NumberFormatException while reading a table having UNION subdirs after ACID conversion [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6675:
URL: https://github.com/apache/hive/pull/6675#discussion_r3925593291


##########
ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java:
##########
@@ -146,12 +146,18 @@ public void flattenUnionSubdirectories(Path sourcePath) throws HiveException {
           Path path = i.next().getPath();
           Path parent = path.getParent();
           if (parent.getName().startsWith(prefix)) {
-            // We do rename by including the name of parent directory into the filename so that there are no clashes
-            // when we move the files to the parent directory. Ex. HIVE_UNION_SUBDIR_1/000000_0 -> 1_000000_0
+            // Fold the subdir index into the filename so there are no clashes when the files
+            // are moved to the parent directory, and so that the resulting name still matches
+            // the "original data file" convention that both the ACID reader (AcidUtils.
+            // ORIGINAL_PATTERN_COPY) and the metastore's non-ACID→ACID validator (Transactional
+            // ValidationListener.ORIGINAL_PATTERN_COPY) recognize:
+            //   HIVE_UNION_SUBDIR_1/000000_0 -> 000000_0_copy_1
+            // See Utilities.COPY_KEYWORD.
             String parentOfParent = parent.getParent().toString();
             String parentNameSuffix = parent.getName().substring(prefix.length());
 
-            fs.rename(path, new Path(parentOfParent + "/" + parentNameSuffix + "_" + path.getName()));
+            fs.rename(path,
+                new Path(parentOfParent + "/" + path.getName() + Utilities.COPY_KEYWORD + parentNameSuffix));

Review Comment:
   very good catch, I had to redesign the logic used in movetask, addressed in: https://github.com/apache/hive/pull/6675/changes/100134a1517d5a5750f60f528d28b490d41b52c7



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29798: NumberFormatException while reading a table having UNION subdirs after ACID conversion [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6675:
URL: https://github.com/apache/hive/pull/6675#issuecomment-5341236862

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6675) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [4 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6675&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6675&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6675&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6675&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6675&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6675)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29798: NumberFormatException while reading a table having UNION subdirs after ACID conversion [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6675:
URL: https://github.com/apache/hive/pull/6675#issuecomment-5180083670

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6675) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6675&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6675&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6675&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6675&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6675&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6675)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29798: NumberFormatException while reading a table having UNION subdirs after ACID conversion [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6675:
URL: https://github.com/apache/hive/pull/6675#issuecomment-5543232348

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6675) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6675&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6675&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6675&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6675&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6675&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6675)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29798: NumberFormatException while reading a table having UNION subdirs after ACID conversion [hive]

Posted by "github-actions[bot] (via GitHub) via gitbox" <gi...@hive.apache.org>.
github-actions[bot] closed pull request #6675: HIVE-29798: NumberFormatException while reading a table having UNION subdirs after ACID conversion
URL: https://github.com/apache/hive/pull/6675


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29798: NumberFormatException while reading a table having UNION subdirs after ACID conversion [hive]

Posted by "github-actions[bot] (via GitHub) via gitbox" <gi...@hive.apache.org>.
github-actions[bot] commented on PR #6675:
URL: https://github.com/apache/hive/pull/6675#issuecomment-6619074313

   This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
   Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org