You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "tanishq-chugh (via GitHub)" <gi...@apache.org> on 2024/12/29 12:47:57 UTC
[PR] HIVE-28682: Multiple Avro Versions on Classpath Can Cause Potential C… [hive]
tanishq-chugh opened a new pull request, #5593:
URL: https://github.com/apache/hive/pull/5593
…onflicts
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
Align avro to a single version - 1.11.4
The same can be achieved by adding avro in dependencyManagement section in standalone-metastore pom as this will ensure the transitive version of avro coming in from hadoop-common is 1.11.4
### Why are the changes needed?
Multiple avro versions on classpath can cause potential conflicts
### Does this PR introduce _any_ user-facing change?
No
### Is the change a dependency upgrade?
Yes
Old Dependency Tree:
[dpn_old.txt](https://github.com/user-attachments/files/18269623/dpn_old.txt)
New Dependency Tree:
[dpn_new.txt](https://github.com/user-attachments/files/18269624/dpn_new.txt)
### How was this patch tested?
Manual Testing by running few queries after local compilation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28682: Multiple Avro Versions on Classpath Can Cause Potential C… [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #5593:
URL: https://github.com/apache/hive/pull/5593#issuecomment-2564765420
@tanishq-chugh, i don't think in hive avro-1.7.7.jar is shipped anywhere, i checked both:
1. `standalone-metastore/metastore-server/target/apache-hive-metastore-4.1.0-SNAPSHOT-bin/lib`
2. `packaging/target/apache-hive-4.1.0-SNAPSHOT-bin/apache-hive-4.1.0-SNAPSHOT-bin/lib`
and in _standalone-metastore_ module/submodules, we are not using any avro api's which means that metastore don't have to depend on avro.
Also, in terms of CVE's mentioned in JIRA, SonaType scan only picks the jars that are getting shipped (based on my understanding) so we are good from that front as well.
Thoughts on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28682: Multiple Avro Versions on Classpath Can Cause Potential C… [hive]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5593:
URL: https://github.com/apache/hive/pull/5593#issuecomment-2708582728
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28682: Multiple Avro Versions on Classpath Can Cause Potential C… [hive]
Posted by "tanishq-chugh (via GitHub)" <gi...@apache.org>.
tanishq-chugh commented on PR #5593:
URL: https://github.com/apache/hive/pull/5593#issuecomment-2566193369
@Aggarwal-Raghav Agreed.
I think since avro 1.7.7 jars are present only for compile-time classpath, and since there are no direct API calls from standalone metastore, successful compilation with exclusion should verify the same. Also, we are not shipping 1.7.7 jars that means at runtime classes are being loaded by 1.11.4 jars itself currently. So, that classpath and runtime compatibility remains unaffected.
Rest Let's see how the precommit goes and committers opinion as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28682: Multiple Avro Versions on Classpath Can Cause Potential C… [hive]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #5593: HIVE-28682: Multiple Avro Versions on Classpath Can Cause Potential C…
URL: https://github.com/apache/hive/pull/5593
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28682: Multiple Avro Versions on Classpath Can Cause Potential C… [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5593:
URL: https://github.com/apache/hive/pull/5593#issuecomment-2564728863
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5593) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5593&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5593&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5593&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5593&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5593&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5593)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28682: Multiple Avro Versions on Classpath Can Cause Potential C… [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #5593:
URL: https://github.com/apache/hive/pull/5593#issuecomment-2574686340
`This can cause potential classpath conflicts in dynamic class-loading with classes coming in from Avro v1.11.4 jars.`
That statement is not legit since `hadoop-common` avro dependency has compile scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28682: Multiple Avro Versions on Classpath Can Cause Potential C… [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5593:
URL: https://github.com/apache/hive/pull/5593#issuecomment-2567088312
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5593) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5593&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5593&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5593&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5593&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5593&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5593)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28682: Multiple Avro Versions on Classpath Can Cause Potential C… [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5593:
URL: https://github.com/apache/hive/pull/5593#issuecomment-2566227475
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5593) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5593&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5593&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5593&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5593&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5593&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5593)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28682: Multiple Avro Versions on Classpath Can Cause Potential C… [hive]
Posted by "tanishq-chugh (via GitHub)" <gi...@apache.org>.
tanishq-chugh commented on PR #5593:
URL: https://github.com/apache/hive/pull/5593#issuecomment-2575112967
@deniskuzZ yes, my bad.
In confusion of considering that v1.7.7 jars are being packaged, had created the JIRA. As @Aggarwal-Raghav as well pointed out that we are not packaging v1.7.7 jars so they won't be present at runtime at all.
At this point, we are not using these 1.7.7 avro jars not even at compile time. So, this change is a good-to-have not critical, as it would clean up compile-time class path, no more affect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28682: Multiple Avro Versions on Classpath Can Cause Potential C… [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5593:
URL: https://github.com/apache/hive/pull/5593#issuecomment-2567327726
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5593) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5593&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5593&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5593&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5593&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5593&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5593)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28682: Multiple Avro Versions on Classpath Can Cause Potential C… [hive]
Posted by "tanishq-chugh (via GitHub)" <gi...@apache.org>.
tanishq-chugh commented on PR #5593:
URL: https://github.com/apache/hive/pull/5593#issuecomment-2565181347
@Aggarwal-Raghav Thanks for pointing this out! Avro v1.7.7 is present only in compile-time classpath and it isn't getting packaged as per maven dependency resolution rules as v1.11.4 occurs at a lower level as a direct dependency. The main concern here was its compile time presence for standalone-metastore sub-modules as the META-INF of sub-modules showed a dependency on v1.7.7 . But, as you correctly pointed out, we are not using any API's from avro in standalone-metastore, so there is no usage of this avro jar.
IMO, we should rather exclude this avro dependency from hadoop-common in standalone-metastore. Just to make the compile-time classpath cleaner. What are your thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28682: Multiple Avro Versions on Classpath Can Cause Potential C… [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #5593:
URL: https://github.com/apache/hive/pull/5593#issuecomment-2565253097
@tanishq-chugh, **IMO, excluding avro from hadoop dependencies can help as we are not shipping the avro-1.7.7 jar.**
Example Scenario:
`User avro related Query -> standalone-metastore code -> hadoop-commons jar -> avro api's`
Either of 2 things are happening which can be confirmed with `-verbose:class` JVM flag:
1. The avro-1.7.7 jar is loaded first and should be provided by hadoop project.
2. If hive avro version 1.11.4 is picked first then avro api's used in hadoop should be compatible.
I would also suggest to take someone else opinion on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org