You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/09/13 13:56:00 UTC
[GitHub] [iceberg] ajantha-bhat opened a new pull request, #5754: Spark: Fix runtime jars packaging scala library files
ajantha-bhat opened a new pull request, #5754:
URL: https://github.com/apache/iceberg/pull/5754
https://github.com/apache/iceberg/pull/4009 adds a dependency on `scala.collection.compat` which is bringing the `scala-library` dependencies and causing the runtime jars to be packaged with scala-library files.
So, far two issues reported that scala files packaged with run time jar is conflicting with their environment scala files.
Fixes #5732
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on pull request #5754: Spark: Fix runtime jars packaging scala library files
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on PR #5754:
URL: https://github.com/apache/iceberg/pull/5754#issuecomment-1246928750
@KarlManong also verified these changes as mentioned in #5732
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko merged pull request #5754: Spark: Fix runtime jars packaging scala library files
Posted by GitBox <gi...@apache.org>.
Fokko merged PR #5754:
URL: https://github.com/apache/iceberg/pull/5754
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on pull request #5754: Spark: Fix runtime jars packaging scala library files
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on PR #5754:
URL: https://github.com/apache/iceberg/pull/5754#issuecomment-1245662374
cc: @fqaiser94, @rdblue, @Fokko, @RussellSpitzer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pan3793 commented on a diff in pull request #5754: Spark: Fix runtime jars packaging scala library files
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #5754:
URL: https://github.com/apache/iceberg/pull/5754#discussion_r970225950
##########
spark/v3.2/build.gradle:
##########
@@ -58,7 +58,9 @@ project(":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}") {
implementation project(':iceberg-orc')
implementation project(':iceberg-parquet')
implementation project(':iceberg-arrow')
- implementation("org.scala-lang.modules:scala-collection-compat_${scalaVersion}")
+ implementation("org.scala-lang.modules:scala-collection-compat_${scalaVersion}") {
Review Comment:
`scala-collection-compat` is included into spark binary release tarball, so `compileOnly` should be used here IMO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on pull request #5754: Spark: Fix runtime jars packaging scala library files
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on PR #5754:
URL: https://github.com/apache/iceberg/pull/5754#issuecomment-1246234605
> Have a quick check of spark runtime jar, `httpclient5` is included but w/o proper relocation, could you please fix it as well?
Will handle it in the follow-up PR as it affects all the spark versions and usually I get comments to separate the independent issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on pull request #5754: Spark: Fix runtime jars packaging scala library files
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on PR #5754:
URL: https://github.com/apache/iceberg/pull/5754#issuecomment-1246240026
> Will handle it in the follow-up PR as it affects all the spark versions and usually I get comments to separate the independent issues.
@pan3793: Handled in the below PR, please review.
https://github.com/apache/iceberg/pull/5761
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pan3793 commented on pull request #5754: Spark: Fix runtime jars packaging scala library files
Posted by GitBox <gi...@apache.org>.
pan3793 commented on PR #5754:
URL: https://github.com/apache/iceberg/pull/5754#issuecomment-1246132624
Have a quick check of spark runtime jar, `httpclient5` is included by w/o proper relocation, could you please fix it as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pan3793 commented on a diff in pull request #5754: Spark: Fix runtime jars packaging scala library files
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #5754:
URL: https://github.com/apache/iceberg/pull/5754#discussion_r970313020
##########
spark/v3.2/build.gradle:
##########
@@ -58,7 +58,9 @@ project(":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}") {
implementation project(':iceberg-orc')
implementation project(':iceberg-parquet')
implementation project(':iceberg-arrow')
- implementation("org.scala-lang.modules:scala-collection-compat_${scalaVersion}")
+ implementation("org.scala-lang.modules:scala-collection-compat_${scalaVersion}") {
Review Comment:
Then I think we need not touch this part, just exclude it from the runtime jar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pan3793 commented on a diff in pull request #5754: Spark: Fix runtime jars packaging scala library files
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #5754:
URL: https://github.com/apache/iceberg/pull/5754#discussion_r970381019
##########
spark/v3.3/build.gradle:
##########
@@ -193,8 +193,12 @@ project(":iceberg-spark:iceberg-spark-runtime-${sparkMajorVersion}_${scalaVersio
dependencies {
api project(':iceberg-api')
- implementation project(":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}")
- implementation project(":iceberg-spark:iceberg-spark-extensions-${sparkMajorVersion}_${scalaVersion}")
+ implementation(project(":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}")) {
+ exclude group: 'org.scala-lang', module: 'scala-library'
Review Comment:
My previous comment may be unclear, my suggestion is to add the following part after line 190
```
exclude group: 'org.scala-lang'
exclude group: 'org.scala-lang.modules'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on a diff in pull request #5754: Spark: Fix runtime jars packaging scala library files
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on code in PR #5754:
URL: https://github.com/apache/iceberg/pull/5754#discussion_r970466593
##########
spark/v3.3/build.gradle:
##########
@@ -193,8 +193,12 @@ project(":iceberg-spark:iceberg-spark-runtime-${sparkMajorVersion}_${scalaVersio
dependencies {
api project(':iceberg-api')
- implementation project(":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}")
- implementation project(":iceberg-spark:iceberg-spark-extensions-${sparkMajorVersion}_${scalaVersion}")
+ implementation(project(":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}")) {
+ exclude group: 'org.scala-lang', module: 'scala-library'
Review Comment:
sounds good to me as spark is providing the `scala.collection.compat` also.
https://files.slack.com/files-pri/T025T8UC953-F042D9QPEMA/image.png
Updated the PR. Manually verified that there is no scala folder with the runtime jars. let's wait for the build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on a diff in pull request #5754: Spark: Fix runtime jars packaging scala library files
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on code in PR #5754:
URL: https://github.com/apache/iceberg/pull/5754#discussion_r970467042
##########
spark/v3.2/build.gradle:
##########
@@ -58,7 +58,9 @@ project(":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}") {
implementation project(':iceberg-orc')
implementation project(':iceberg-parquet')
implementation project(':iceberg-arrow')
- implementation("org.scala-lang.modules:scala-collection-compat_${scalaVersion}")
+ implementation("org.scala-lang.modules:scala-collection-compat_${scalaVersion}") {
Review Comment:
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on a diff in pull request #5754: Spark: Fix runtime jars packaging scala library files
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on code in PR #5754:
URL: https://github.com/apache/iceberg/pull/5754#discussion_r970299451
##########
spark/v3.2/build.gradle:
##########
@@ -58,7 +58,9 @@ project(":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}") {
implementation project(':iceberg-orc')
implementation project(':iceberg-parquet')
implementation project(':iceberg-arrow')
- implementation("org.scala-lang.modules:scala-collection-compat_${scalaVersion}")
+ implementation("org.scala-lang.modules:scala-collection-compat_${scalaVersion}") {
Review Comment:
I too thought of it. But spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ExtendedDistributionAndOrderingUtils.scala needs it at runtime to support scala 2.13 (from PR#4009). So, I have just excluded scala-library files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on a diff in pull request #5754: Spark: Fix runtime jars packaging scala library files
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on code in PR #5754:
URL: https://github.com/apache/iceberg/pull/5754#discussion_r970299451
##########
spark/v3.2/build.gradle:
##########
@@ -58,7 +58,9 @@ project(":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}") {
implementation project(':iceberg-orc')
implementation project(':iceberg-parquet')
implementation project(':iceberg-arrow')
- implementation("org.scala-lang.modules:scala-collection-compat_${scalaVersion}")
+ implementation("org.scala-lang.modules:scala-collection-compat_${scalaVersion}") {
Review Comment:
I too thought of it. But `ExtendedDistributionAndOrderingUtils.scala` needs it at runtime to support scala 2.13 (from PR#4009). So, I have just excluded scala-library files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on a diff in pull request #5754: Spark: Fix runtime jars packaging scala library files
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on code in PR #5754:
URL: https://github.com/apache/iceberg/pull/5754#discussion_r970466593
##########
spark/v3.3/build.gradle:
##########
@@ -193,8 +193,12 @@ project(":iceberg-spark:iceberg-spark-runtime-${sparkMajorVersion}_${scalaVersio
dependencies {
api project(':iceberg-api')
- implementation project(":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}")
- implementation project(":iceberg-spark:iceberg-spark-extensions-${sparkMajorVersion}_${scalaVersion}")
+ implementation(project(":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}")) {
+ exclude group: 'org.scala-lang', module: 'scala-library'
Review Comment:
sounds good to me as spark is providing the `scala.collection.compat` also.
<img width="992" alt="image" src="https://user-images.githubusercontent.com/5889404/190136254-6b5aa236-09a8-481d-a00f-4c290a4baf08.png">
Updated the PR. Manually verified that there is no scala folder with the runtime jars. let's wait for the build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org