You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org> on 2025/05/26 18:17:55 UTC
[PR] HIVE-28971: DirectSql for msck repair is failing because of wrong casting [hive]
Aggarwal-Raghav opened a new pull request, #5827:
URL: https://github.com/apache/hive/pull/5827
### What changes were proposed in this pull request?
Please check [HIVE-28971](https://issues.apache.org/jira/browse/HIVE-28971), for more details.
ERROR Message:
```
2025-05-26 23:07:36,111 WARN metastore.ObjectStore: Falling back to ORM path due to direct SQL failure (this is not an error): class javax.jdo.identity.LongIdentity cannot be cast to class org.datanucleus.identity.DatastoreId (javax.jdo.identity.LongIdentity and org.datanucleus.identity.DatastoreId are in unnamed module of loader 'app') at org.apache.hadoop.hive.metastore.DirectSqlInsertPart.addPartitions(DirectSqlInsertPart.java:758) at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.addPartitions(MetaStoreDirectSql.java:516) at org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:2747)
```
### Why are the changes needed?
To leverage directSql for _msck repair_ command
### Does this PR introduce _any_ user-facing change?
NO
### How was this patch tested?
On local cluster (single node)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28971: DirectSql for msck repair is failing because of wrong casting [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #5827:
URL: https://github.com/apache/hive/pull/5827#discussion_r2109448198
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlInsertPart.java:
##########
@@ -754,7 +754,7 @@ public void addPartitions(List<MPartition> parts, List<List<MPartitionPrivilege>
serdeIdToSerDeInfo.put(serDeId, sd.getSerDeInfo());
Long cdId;
- DatastoreId storeId = (DatastoreId) pm.getObjectId(sd.getCD());
+ LongIdentity storeId = (LongIdentity) pm.getObjectId(sd.getCD());
Review Comment:
Yes, this is a regression. I checked with and without [HIVE-28909](https://issues.apache.org/jira/browse/HIVE-28909) in master branch with JDK8, datanucleus 5.x as well. Attaching screenshots for reference:
**Without HIVE-28909:**
<img width="703" alt="DataNucleus_5 x_without_28909" src="https://github.com/user-attachments/assets/ff742f7f-cd9d-4a5b-94da-0f4b73a47a75" />
**With HIVE-28909**
<img width="817" alt="DataNucleus_5 x_with_28909" src="https://github.com/user-attachments/assets/ad50679f-87b6-4a9c-be2e-5040201d7f3b" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28971: DirectSql for msck repair is failing because of wrong casting [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5827:
URL: https://github.com/apache/hive/pull/5827#issuecomment-2910555897
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5827) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5827&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5827&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5827&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5827&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5827&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5827)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28971: DirectSql for msck repair is failing because of wrong casting [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #5827:
URL: https://github.com/apache/hive/pull/5827#issuecomment-2910392832
CC @deniskuzZ @saihemanth-cloudera
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28971: DirectSql for msck repair is failing because of wrong casting [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5827:
URL: https://github.com/apache/hive/pull/5827#issuecomment-2913251867
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5827) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5827&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5827&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5827&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5827&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5827&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5827)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28971: DirectSql for msck repair is failing because of wrong casting [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #5827:
URL: https://github.com/apache/hive/pull/5827#issuecomment-2912667395
> @Aggarwal-Raghav, is it possible to add a simple unit test to cover this scenario?
Sure. Let me get back to you on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28971: DirectSql for msck repair is failing because of wrong casting [hive]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #5827:
URL: https://github.com/apache/hive/pull/5827#discussion_r2109732526
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlInsertPart.java:
##########
@@ -754,7 +754,7 @@ public void addPartitions(List<MPartition> parts, List<List<MPartitionPrivilege>
serdeIdToSerDeInfo.put(serDeId, sd.getSerDeInfo());
Long cdId;
- DatastoreId storeId = (DatastoreId) pm.getObjectId(sd.getCD());
+ LongIdentity storeId = (LongIdentity) pm.getObjectId(sd.getCD());
Review Comment:
Most of the hive tests just check the query output, not like what route it took to complete.
the case with DirectSql is like if it fails, Hive automatically fallbacks to the JDO layer, without failing the query, so unless the response via JDO is different as compared to DirectSql, which it shouldn’t be unless there is bug. I belive the tests won’t fail, just a warning log in the test log, which we don’t check :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28971: DirectSql for msck repair is failing because of wrong casting [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on PR #5827:
URL: https://github.com/apache/hive/pull/5827#issuecomment-2928367516
I created HIVE-28985 to improve the test coverage for direct sql
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28971: DirectSql for msck repair is failing because of wrong casting [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #5827:
URL: https://github.com/apache/hive/pull/5827#issuecomment-2912634611
@Aggarwal-Raghav, is it possible to add a simple unit test to cover this scenario?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28971: DirectSql for msck repair is failing because of wrong casting [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5827:
URL: https://github.com/apache/hive/pull/5827#discussion_r2109301490
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlInsertPart.java:
##########
@@ -754,7 +754,7 @@ public void addPartitions(List<MPartition> parts, List<List<MPartitionPrivilege>
serdeIdToSerDeInfo.put(serDeId, sd.getSerDeInfo());
Long cdId;
- DatastoreId storeId = (DatastoreId) pm.getObjectId(sd.getCD());
+ LongIdentity storeId = (LongIdentity) pm.getObjectId(sd.getCD());
Review Comment:
Did https://github.com/apache/hive/commit/fbe090f5edf098e573749e40fdb6519cdc615682 cause that?
Looping in @ayushtkn, who's coordinating the Java 17 upgrade effort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28971: DirectSql for msck repair is failing because of wrong casting [hive]
Posted by "saihemanth-cloudera (via GitHub)" <gi...@apache.org>.
saihemanth-cloudera commented on code in PR #5827:
URL: https://github.com/apache/hive/pull/5827#discussion_r2109705039
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlInsertPart.java:
##########
@@ -754,7 +754,7 @@ public void addPartitions(List<MPartition> parts, List<List<MPartitionPrivilege>
serdeIdToSerDeInfo.put(serDeId, sd.getSerDeInfo());
Long cdId;
- DatastoreId storeId = (DatastoreId) pm.getObjectId(sd.getCD());
+ LongIdentity storeId = (LongIdentity) pm.getObjectId(sd.getCD());
Review Comment:
DirectSQL is enabled by default and we have test coverage for msck repair queries, so not sure why didn't catch this earlier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28971: DirectSql for msck repair is failing because of wrong casting [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 merged PR #5827:
URL: https://github.com/apache/hive/pull/5827
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28971: DirectSql for msck repair is failing because of wrong casting [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #5827:
URL: https://github.com/apache/hive/pull/5827#issuecomment-2913485771
@deniskuzZ @ayushtkn @saihemanth-cloudera , I have filed [HIVE-28972](https://issues.apache.org/jira/browse/HIVE-28972), I am observing HMS performace degradation for alter table add column cascade query. Please check the JIRA for details. I have the details and provided steps to repro as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28971: DirectSql for msck repair is failing because of wrong casting [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5827:
URL: https://github.com/apache/hive/pull/5827#discussion_r2109785804
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlInsertPart.java:
##########
@@ -754,7 +754,7 @@ public void addPartitions(List<MPartition> parts, List<List<MPartitionPrivilege>
serdeIdToSerDeInfo.put(serDeId, sd.getSerDeInfo());
Long cdId;
- DatastoreId storeId = (DatastoreId) pm.getObjectId(sd.getCD());
+ LongIdentity storeId = (LongIdentity) pm.getObjectId(sd.getCD());
Review Comment:
we should probably disable fallback in DirectSql tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28971: DirectSql for msck repair is failing because of wrong casting [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5827:
URL: https://github.com/apache/hive/pull/5827#discussion_r2109301490
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlInsertPart.java:
##########
@@ -754,7 +754,7 @@ public void addPartitions(List<MPartition> parts, List<List<MPartitionPrivilege>
serdeIdToSerDeInfo.put(serDeId, sd.getSerDeInfo());
Long cdId;
- DatastoreId storeId = (DatastoreId) pm.getObjectId(sd.getCD());
+ LongIdentity storeId = (LongIdentity) pm.getObjectId(sd.getCD());
Review Comment:
Did https://github.com/apache/hive/commit/fbe090f5edf098e573749e40fdb6519cdc615682 cause that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org