You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "cfmcgrady (via GitHub)" <gi...@apache.org> on 2023/08/24 02:56:49 UTC
[GitHub] [incubator-celeborn] cfmcgrady opened a new pull request, #1831: [CELEBORN-906] Aligning dependencies between SBT and Maven
cfmcgrady opened a new pull request, #1831:
URL: https://github.com/apache/incubator-celeborn/pull/1831
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
As title
### Why are the changes needed?
This PR ensures dependency alignment between SBT and Maven, based on the audit results implemented in https://github.com/apache/incubator-celeborn/pull/1797
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass GA and Review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1831: [CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1831:
URL: https://github.com/apache/incubator-celeborn/pull/1831#discussion_r1305504678
##########
project/CelebornBuild.scala:
##########
@@ -62,34 +62,46 @@ object Dependencies {
val protocVersion = "3.19.2"
val protoVersion = "3.19.2"
- val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion
+ val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion excludeAll(
+ ExclusionRule("net.java.dev.jna", "jna"))
val commonsIo = "commons-io" % "commons-io" % commonsIoVersion
val commonsLang3 = "org.apache.commons" % "commons-lang3" % commonsLang3Version
val findbugsJsr305 = "com.google.code.findbugs" % "jsr305" % findbugsVersion
val guava = "com.google.guava" % "guava" % guavaVersion
- val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion
- val hadoopClientRuntime = "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion
- val ioDropwizardMetricsCore = "io.dropwizard.metrics" % "metrics-core" % metricsVersion
- val ioDropwizardMetricsGraphite = "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion
- val ioDropwizardMetricsJvm = "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion
- val ioNetty = "io.netty" % "netty-all" % nettyVersion
+ val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion excludeAll(
+ ExclusionRule("org.slf4j", "slf4j-api"))
Review Comment:
addressed in https://github.com/apache/incubator-celeborn/pull/1831/commits/269b9cedb28be15a0d60843fe3b1c293af6d699a, it looks more clearly now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1831: [CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1831:
URL: https://github.com/apache/incubator-celeborn/pull/1831#discussion_r1303787787
##########
project/CelebornBuild.scala:
##########
@@ -62,34 +62,46 @@ object Dependencies {
val protocVersion = "3.19.2"
val protoVersion = "3.19.2"
- val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion
+ val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion excludeAll(
+ ExclusionRule("net.java.dev.jna", "jna"))
val commonsIo = "commons-io" % "commons-io" % commonsIoVersion
val commonsLang3 = "org.apache.commons" % "commons-lang3" % commonsLang3Version
val findbugsJsr305 = "com.google.code.findbugs" % "jsr305" % findbugsVersion
val guava = "com.google.guava" % "guava" % guavaVersion
- val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion
- val hadoopClientRuntime = "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion
- val ioDropwizardMetricsCore = "io.dropwizard.metrics" % "metrics-core" % metricsVersion
- val ioDropwizardMetricsGraphite = "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion
- val ioDropwizardMetricsJvm = "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion
- val ioNetty = "io.netty" % "netty-all" % nettyVersion
+ val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion excludeAll(
+ ExclusionRule("org.slf4j", "slf4j-api"))
Review Comment:
Perform local testing. the `intransitive()` cannot achieve this functionality.
```
[warn] Found intransitive dependency (org.apache.hadoop:hadoop-client-api:3.2.4 intransitive) while publishMavenStyle is true, but Maven repositories
[warn] do not support intransitive dependencies. Use exclusions instead so transitive dependencies
[warn] will be correctly excluded in dependent projects.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1831: [CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1831:
URL: https://github.com/apache/incubator-celeborn/pull/1831#discussion_r1305439182
##########
project/CelebornBuild.scala:
##########
@@ -62,34 +62,46 @@ object Dependencies {
val protocVersion = "3.19.2"
val protoVersion = "3.19.2"
- val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion
+ val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion excludeAll(
+ ExclusionRule("net.java.dev.jna", "jna"))
val commonsIo = "commons-io" % "commons-io" % commonsIoVersion
val commonsLang3 = "org.apache.commons" % "commons-lang3" % commonsLang3Version
val findbugsJsr305 = "com.google.code.findbugs" % "jsr305" % findbugsVersion
val guava = "com.google.guava" % "guava" % guavaVersion
- val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion
- val hadoopClientRuntime = "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion
- val ioDropwizardMetricsCore = "io.dropwizard.metrics" % "metrics-core" % metricsVersion
- val ioDropwizardMetricsGraphite = "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion
- val ioDropwizardMetricsJvm = "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion
- val ioNetty = "io.netty" % "netty-all" % nettyVersion
+ val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion excludeAll(
+ ExclusionRule("org.slf4j", "slf4j-api"))
Review Comment:
Explicitly Exclude from all deps looks ugly. `dependencyOverrides` seems as clean as `force()`? if yes, let's try this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on pull request #1831: [CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on PR #1831:
URL: https://github.com/apache/incubator-celeborn/pull/1831#issuecomment-1690997056
FYI:
SBT's approach to handling dependency conflicts differs from Maven's.
Learn more: [SBT Library Management - Eviction Warning](https://www.scala-sbt.org/1.x/docs/Library-Management.html#Eviction+warning).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1831: [CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1831:
URL: https://github.com/apache/incubator-celeborn/pull/1831#discussion_r1303771880
##########
project/CelebornBuild.scala:
##########
@@ -62,34 +62,46 @@ object Dependencies {
val protocVersion = "3.19.2"
val protoVersion = "3.19.2"
- val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion
+ val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion excludeAll(
+ ExclusionRule("net.java.dev.jna", "jna"))
val commonsIo = "commons-io" % "commons-io" % commonsIoVersion
val commonsLang3 = "org.apache.commons" % "commons-lang3" % commonsLang3Version
val findbugsJsr305 = "com.google.code.findbugs" % "jsr305" % findbugsVersion
val guava = "com.google.guava" % "guava" % guavaVersion
- val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion
- val hadoopClientRuntime = "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion
- val ioDropwizardMetricsCore = "io.dropwizard.metrics" % "metrics-core" % metricsVersion
- val ioDropwizardMetricsGraphite = "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion
- val ioDropwizardMetricsJvm = "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion
- val ioNetty = "io.netty" % "netty-all" % nettyVersion
+ val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion excludeAll(
+ ExclusionRule("org.slf4j", "slf4j-api"))
Review Comment:
an other approach, [Disable Transitivity](https://www.scala-sbt.org/1.x/docs/Library-Management.html#Disable+Transitivity)
> By default, these declarations fetch all project dependencies, transitively. In some instances, you may find that the dependencies listed for a project aren’t necessary for it to build. Projects using the Felix OSGI framework, for instance, only explicitly require its main jar to compile and run. Avoid fetching artifact dependencies with either intransitive() or notTransitive(), as in this example:
>
> ```
> libraryDependencies += "org.apache.felix" % "org.apache.felix.framework" % "1.8.0" intransitive()
> ```
##########
project/CelebornBuild.scala:
##########
@@ -62,34 +62,46 @@ object Dependencies {
val protocVersion = "3.19.2"
val protoVersion = "3.19.2"
- val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion
+ val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion excludeAll(
+ ExclusionRule("net.java.dev.jna", "jna"))
val commonsIo = "commons-io" % "commons-io" % commonsIoVersion
val commonsLang3 = "org.apache.commons" % "commons-lang3" % commonsLang3Version
val findbugsJsr305 = "com.google.code.findbugs" % "jsr305" % findbugsVersion
val guava = "com.google.guava" % "guava" % guavaVersion
- val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion
- val hadoopClientRuntime = "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion
- val ioDropwizardMetricsCore = "io.dropwizard.metrics" % "metrics-core" % metricsVersion
- val ioDropwizardMetricsGraphite = "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion
- val ioDropwizardMetricsJvm = "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion
- val ioNetty = "io.netty" % "netty-all" % nettyVersion
+ val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion excludeAll(
+ ExclusionRule("org.slf4j", "slf4j-api"))
Review Comment:
an other approach, [Disable Transitivity](https://www.scala-sbt.org/1.x/docs/Library-Management.html#Disable+Transitivity)
> By default, these declarations fetch all project dependencies, transitively. In some instances, you may find that the dependencies listed for a project aren’t necessary for it to build. Projects using the Felix OSGI framework, for instance, only explicitly require its main jar to compile and run. Avoid fetching artifact dependencies with either intransitive() or notTransitive(), as in this example:
>
> ```
> libraryDependencies += "org.apache.felix" % "org.apache.felix.framework" % "1.8.0" intransitive()
> ```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1831: [CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1831:
URL: https://github.com/apache/incubator-celeborn/pull/1831#discussion_r1303760048
##########
project/CelebornBuild.scala:
##########
@@ -62,34 +62,46 @@ object Dependencies {
val protocVersion = "3.19.2"
val protoVersion = "3.19.2"
- val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion
+ val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion excludeAll(
+ ExclusionRule("net.java.dev.jna", "jna"))
val commonsIo = "commons-io" % "commons-io" % commonsIoVersion
val commonsLang3 = "org.apache.commons" % "commons-lang3" % commonsLang3Version
val findbugsJsr305 = "com.google.code.findbugs" % "jsr305" % findbugsVersion
val guava = "com.google.guava" % "guava" % guavaVersion
- val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion
- val hadoopClientRuntime = "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion
- val ioDropwizardMetricsCore = "io.dropwizard.metrics" % "metrics-core" % metricsVersion
- val ioDropwizardMetricsGraphite = "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion
- val ioDropwizardMetricsJvm = "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion
- val ioNetty = "io.netty" % "netty-all" % nettyVersion
+ val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion excludeAll(
+ ExclusionRule("org.slf4j", "slf4j-api"))
Review Comment:
does sbt support globally pin dep's version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1831: [CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1831:
URL: https://github.com/apache/incubator-celeborn/pull/1831#discussion_r1303857650
##########
project/CelebornBuild.scala:
##########
@@ -62,34 +62,46 @@ object Dependencies {
val protocVersion = "3.19.2"
val protoVersion = "3.19.2"
- val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion
+ val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion excludeAll(
+ ExclusionRule("net.java.dev.jna", "jna"))
val commonsIo = "commons-io" % "commons-io" % commonsIoVersion
val commonsLang3 = "org.apache.commons" % "commons-lang3" % commonsLang3Version
val findbugsJsr305 = "com.google.code.findbugs" % "jsr305" % findbugsVersion
val guava = "com.google.guava" % "guava" % guavaVersion
- val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion
- val hadoopClientRuntime = "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion
- val ioDropwizardMetricsCore = "io.dropwizard.metrics" % "metrics-core" % metricsVersion
- val ioDropwizardMetricsGraphite = "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion
- val ioDropwizardMetricsJvm = "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion
- val ioNetty = "io.netty" % "netty-all" % nettyVersion
+ val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion excludeAll(
+ ExclusionRule("org.slf4j", "slf4j-api"))
Review Comment:
I've also come across an alternative method for setting up `dependencyOverrides` described here: [Overriding a Version](https://www.scala-sbt.org/1.x/docs/Library-Management.html#Overriding+a+version).
Moreover, during my search, I observed that prominent projects employing SBT as their build tool – like [akka](https://github.com/akka/akka/blob/d7b17e9dec52cd27cd17192dda58598d229e3985/project/Dependencies.scala#L96), [play](https://github.com/playframework/playframework/blob/d07e378e88471a4ec3613ebc842d4581143b047b/project/Dependencies.scala#L285), and [delta](https://github.com/delta-io/delta/blob/23826a3b25f39cc1097d3b6174f009da496028b7/build.sbt#L434) – commonly adopt the `excludeAll` approach to exclude transitive dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] waitinfuture closed pull request #1831: [CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture closed pull request #1831: [CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven
URL: https://github.com/apache/incubator-celeborn/pull/1831
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1831: [CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1831:
URL: https://github.com/apache/incubator-celeborn/pull/1831#discussion_r1303808097
##########
project/CelebornBuild.scala:
##########
@@ -62,34 +62,46 @@ object Dependencies {
val protocVersion = "3.19.2"
val protoVersion = "3.19.2"
- val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion
+ val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion excludeAll(
+ ExclusionRule("net.java.dev.jna", "jna"))
val commonsIo = "commons-io" % "commons-io" % commonsIoVersion
val commonsLang3 = "org.apache.commons" % "commons-lang3" % commonsLang3Version
val findbugsJsr305 = "com.google.code.findbugs" % "jsr305" % findbugsVersion
val guava = "com.google.guava" % "guava" % guavaVersion
- val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion
- val hadoopClientRuntime = "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion
- val ioDropwizardMetricsCore = "io.dropwizard.metrics" % "metrics-core" % metricsVersion
- val ioDropwizardMetricsGraphite = "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion
- val ioDropwizardMetricsJvm = "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion
- val ioNetty = "io.netty" % "netty-all" % nettyVersion
+ val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion excludeAll(
+ ExclusionRule("org.slf4j", "slf4j-api"))
Review Comment:
Let's use `force()` way.
> 1. Forcing can create logical inconsistencies so it’s no longer recommended.
We know what we are doing, IMO, rules-based dependency derivation is more unpredictable.
> 2. this is an Ivy-only feature and cannot be included in a published pom.xml.
It does not affect our use case, since we only publish client-shaded modules, without transitive deps.
##########
project/CelebornBuild.scala:
##########
@@ -62,34 +62,46 @@ object Dependencies {
val protocVersion = "3.19.2"
val protoVersion = "3.19.2"
- val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion
+ val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion excludeAll(
+ ExclusionRule("net.java.dev.jna", "jna"))
val commonsIo = "commons-io" % "commons-io" % commonsIoVersion
val commonsLang3 = "org.apache.commons" % "commons-lang3" % commonsLang3Version
val findbugsJsr305 = "com.google.code.findbugs" % "jsr305" % findbugsVersion
val guava = "com.google.guava" % "guava" % guavaVersion
- val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion
- val hadoopClientRuntime = "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion
- val ioDropwizardMetricsCore = "io.dropwizard.metrics" % "metrics-core" % metricsVersion
- val ioDropwizardMetricsGraphite = "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion
- val ioDropwizardMetricsJvm = "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion
- val ioNetty = "io.netty" % "netty-all" % nettyVersion
+ val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion excludeAll(
+ ExclusionRule("org.slf4j", "slf4j-api"))
Review Comment:
Let's use `force()` way.
> 1. Forcing can create logical inconsistencies so it’s no longer recommended.
We know what we are doing, IMO, rules-based dependency derivation is more unpredictable.
> 2. this is an Ivy-only feature and cannot be included in a published pom.xml.
It does not affect our use case, since we only publish client-shaded modules, without transitive deps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1831: [CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1831:
URL: https://github.com/apache/incubator-celeborn/pull/1831#discussion_r1303767859
##########
project/CelebornBuild.scala:
##########
@@ -62,34 +62,46 @@ object Dependencies {
val protocVersion = "3.19.2"
val protoVersion = "3.19.2"
- val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion
+ val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion excludeAll(
+ ExclusionRule("net.java.dev.jna", "jna"))
val commonsIo = "commons-io" % "commons-io" % commonsIoVersion
val commonsLang3 = "org.apache.commons" % "commons-lang3" % commonsLang3Version
val findbugsJsr305 = "com.google.code.findbugs" % "jsr305" % findbugsVersion
val guava = "com.google.guava" % "guava" % guavaVersion
- val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion
- val hadoopClientRuntime = "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion
- val ioDropwizardMetricsCore = "io.dropwizard.metrics" % "metrics-core" % metricsVersion
- val ioDropwizardMetricsGraphite = "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion
- val ioDropwizardMetricsJvm = "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion
- val ioNetty = "io.netty" % "netty-all" % nettyVersion
+ val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion excludeAll(
+ ExclusionRule("org.slf4j", "slf4j-api"))
Review Comment:
Certainly, the [SBT](https://www.scala-sbt.org/1.x/docs/Library-Management.html#Forcing+a+revision+%28Not+recommended%29) does allow for the specification of a forced dependency revision, e.g.
```
libraryDependencies ++= Seq(
"log4j" % "log4j" % "1.2.14" force()
)
```
However, the docs also point out it is advised against, as indicated by the following considerations:
> 1. Forcing can create logical inconsistencies so it’s no longer recommended.
> 2. this is an Ivy-only feature and cannot be included in a published pom.xml.
##########
project/CelebornBuild.scala:
##########
@@ -62,34 +62,46 @@ object Dependencies {
val protocVersion = "3.19.2"
val protoVersion = "3.19.2"
- val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion
+ val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion excludeAll(
+ ExclusionRule("net.java.dev.jna", "jna"))
val commonsIo = "commons-io" % "commons-io" % commonsIoVersion
val commonsLang3 = "org.apache.commons" % "commons-lang3" % commonsLang3Version
val findbugsJsr305 = "com.google.code.findbugs" % "jsr305" % findbugsVersion
val guava = "com.google.guava" % "guava" % guavaVersion
- val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion
- val hadoopClientRuntime = "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion
- val ioDropwizardMetricsCore = "io.dropwizard.metrics" % "metrics-core" % metricsVersion
- val ioDropwizardMetricsGraphite = "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion
- val ioDropwizardMetricsJvm = "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion
- val ioNetty = "io.netty" % "netty-all" % nettyVersion
+ val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion excludeAll(
+ ExclusionRule("org.slf4j", "slf4j-api"))
Review Comment:
Certainly, the [SBT](https://www.scala-sbt.org/1.x/docs/Library-Management.html#Forcing+a+revision+%28Not+recommended%29) does allow for the specification of a forced dependency revision, e.g.
```
libraryDependencies ++= Seq(
"log4j" % "log4j" % "1.2.14" force()
)
```
However, the docs also point out it is advised against, as indicated by the following considerations:
> 1. Forcing can create logical inconsistencies so it’s no longer recommended.
> 2. this is an Ivy-only feature and cannot be included in a published pom.xml.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1831: [CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1831:
URL: https://github.com/apache/incubator-celeborn/pull/1831#discussion_r1305505743
##########
project/CelebornBuild.scala:
##########
@@ -62,34 +62,46 @@ object Dependencies {
val protocVersion = "3.19.2"
val protoVersion = "3.19.2"
- val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion
+ val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion excludeAll(
+ ExclusionRule("net.java.dev.jna", "jna"))
val commonsIo = "commons-io" % "commons-io" % commonsIoVersion
val commonsLang3 = "org.apache.commons" % "commons-lang3" % commonsLang3Version
val findbugsJsr305 = "com.google.code.findbugs" % "jsr305" % findbugsVersion
val guava = "com.google.guava" % "guava" % guavaVersion
- val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion
- val hadoopClientRuntime = "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion
- val ioDropwizardMetricsCore = "io.dropwizard.metrics" % "metrics-core" % metricsVersion
- val ioDropwizardMetricsGraphite = "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion
- val ioDropwizardMetricsJvm = "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion
- val ioNetty = "io.netty" % "netty-all" % nettyVersion
+ val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion excludeAll(
+ ExclusionRule("org.slf4j", "slf4j-api"))
Review Comment:
```
$ ./build/sbt -Pspark-3.3 celeborn-client-spark-3/dependencyTree | grep slf4j-api
[info] | | | +-org.slf4j:slf4j-api:1.7.36
[info] | | | | +-org.slf4j:slf4j-api:1.7.36
[info] | | | +-org.slf4j:slf4j-api:1.7.36
[info] | | | | +-org.slf4j:slf4j-api:1.7.36
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1831: [CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1831:
URL: https://github.com/apache/incubator-celeborn/pull/1831#discussion_r1305506786
##########
project/CelebornBuild.scala:
##########
@@ -62,34 +62,46 @@ object Dependencies {
val protocVersion = "3.19.2"
val protoVersion = "3.19.2"
- val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion
+ val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion excludeAll(
+ ExclusionRule("net.java.dev.jna", "jna"))
val commonsIo = "commons-io" % "commons-io" % commonsIoVersion
val commonsLang3 = "org.apache.commons" % "commons-lang3" % commonsLang3Version
val findbugsJsr305 = "com.google.code.findbugs" % "jsr305" % findbugsVersion
val guava = "com.google.guava" % "guava" % guavaVersion
- val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion
- val hadoopClientRuntime = "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion
- val ioDropwizardMetricsCore = "io.dropwizard.metrics" % "metrics-core" % metricsVersion
- val ioDropwizardMetricsGraphite = "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion
- val ioDropwizardMetricsJvm = "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion
- val ioNetty = "io.netty" % "netty-all" % nettyVersion
+ val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion excludeAll(
+ ExclusionRule("org.slf4j", "slf4j-api"))
Review Comment:
nice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] codecov[bot] commented on pull request #1831: [CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #1831:
URL: https://github.com/apache/incubator-celeborn/pull/1831#issuecomment-1690922953
## [Codecov](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1831?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#1831](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1831?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (9e3467d) into [main](https://app.codecov.io/gh/apache/incubator-celeborn/commit/aa35c1cafc4b2933531db10ea8c6d31c950c59a2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (aa35c1c) will **not change** coverage.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## main #1831 +/- ##
=======================================
Coverage 46.43% 46.43%
=======================================
Files 163 163
Lines 10135 10135
Branches 934 934
=======================================
Hits 4705 4705
Misses 5118 5118
Partials 312 312
```
[see 2 files with indirect coverage changes](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1831/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1831: [CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1831:
URL: https://github.com/apache/incubator-celeborn/pull/1831#discussion_r1303857650
##########
project/CelebornBuild.scala:
##########
@@ -62,34 +62,46 @@ object Dependencies {
val protocVersion = "3.19.2"
val protoVersion = "3.19.2"
- val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion
+ val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion excludeAll(
+ ExclusionRule("net.java.dev.jna", "jna"))
val commonsIo = "commons-io" % "commons-io" % commonsIoVersion
val commonsLang3 = "org.apache.commons" % "commons-lang3" % commonsLang3Version
val findbugsJsr305 = "com.google.code.findbugs" % "jsr305" % findbugsVersion
val guava = "com.google.guava" % "guava" % guavaVersion
- val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion
- val hadoopClientRuntime = "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion
- val ioDropwizardMetricsCore = "io.dropwizard.metrics" % "metrics-core" % metricsVersion
- val ioDropwizardMetricsGraphite = "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion
- val ioDropwizardMetricsJvm = "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion
- val ioNetty = "io.netty" % "netty-all" % nettyVersion
+ val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion excludeAll(
+ ExclusionRule("org.slf4j", "slf4j-api"))
Review Comment:
I've also come across an alternative method for setting up dependencyOverrides described here: [Overriding a Version](https://www.scala-sbt.org/1.x/docs/Library-Management.html#Overriding+a+version).
Moreover, during my search, I observed that prominent projects employing SBT as their build tool – like [akka](https://github.com/akka/akka/blob/d7b17e9dec52cd27cd17192dda58598d229e3985/project/Dependencies.scala#L96), [play](https://github.com/playframework/playframework/blob/d07e378e88471a4ec3613ebc842d4581143b047b/project/Dependencies.scala#L285), and [delta](https://github.com/delta-io/delta/blob/23826a3b25f39cc1097d3b6174f009da496028b7/build.sbt#L434) – commonly adopt the excludeAll approach to exclude transitive dependencies.
##########
project/CelebornBuild.scala:
##########
@@ -62,34 +62,46 @@ object Dependencies {
val protocVersion = "3.19.2"
val protoVersion = "3.19.2"
- val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion
+ val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion excludeAll(
+ ExclusionRule("net.java.dev.jna", "jna"))
val commonsIo = "commons-io" % "commons-io" % commonsIoVersion
val commonsLang3 = "org.apache.commons" % "commons-lang3" % commonsLang3Version
val findbugsJsr305 = "com.google.code.findbugs" % "jsr305" % findbugsVersion
val guava = "com.google.guava" % "guava" % guavaVersion
- val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion
- val hadoopClientRuntime = "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion
- val ioDropwizardMetricsCore = "io.dropwizard.metrics" % "metrics-core" % metricsVersion
- val ioDropwizardMetricsGraphite = "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion
- val ioDropwizardMetricsJvm = "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion
- val ioNetty = "io.netty" % "netty-all" % nettyVersion
+ val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion excludeAll(
+ ExclusionRule("org.slf4j", "slf4j-api"))
Review Comment:
I've also come across an alternative method for setting up dependencyOverrides described here: [Overriding a Version](https://www.scala-sbt.org/1.x/docs/Library-Management.html#Overriding+a+version).
Moreover, during my search, I observed that prominent projects employing SBT as their build tool – like [akka](https://github.com/akka/akka/blob/d7b17e9dec52cd27cd17192dda58598d229e3985/project/Dependencies.scala#L96), [play](https://github.com/playframework/playframework/blob/d07e378e88471a4ec3613ebc842d4581143b047b/project/Dependencies.scala#L285), and [delta](https://github.com/delta-io/delta/blob/23826a3b25f39cc1097d3b6174f009da496028b7/build.sbt#L434) – commonly adopt the excludeAll approach to exclude transitive dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1831: [CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1831:
URL: https://github.com/apache/incubator-celeborn/pull/1831#discussion_r1303771880
##########
project/CelebornBuild.scala:
##########
@@ -62,34 +62,46 @@ object Dependencies {
val protocVersion = "3.19.2"
val protoVersion = "3.19.2"
- val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion
+ val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion excludeAll(
+ ExclusionRule("net.java.dev.jna", "jna"))
val commonsIo = "commons-io" % "commons-io" % commonsIoVersion
val commonsLang3 = "org.apache.commons" % "commons-lang3" % commonsLang3Version
val findbugsJsr305 = "com.google.code.findbugs" % "jsr305" % findbugsVersion
val guava = "com.google.guava" % "guava" % guavaVersion
- val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion
- val hadoopClientRuntime = "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion
- val ioDropwizardMetricsCore = "io.dropwizard.metrics" % "metrics-core" % metricsVersion
- val ioDropwizardMetricsGraphite = "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion
- val ioDropwizardMetricsJvm = "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion
- val ioNetty = "io.netty" % "netty-all" % nettyVersion
+ val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion excludeAll(
+ ExclusionRule("org.slf4j", "slf4j-api"))
Review Comment:
~~another approach, [Disable Transitivity](https://www.scala-sbt.org/1.x/docs/Library-Management.html#Disable+Transitivity)~~
> By default, these declarations fetch all project dependencies, transitively. In some instances, you may find that the dependencies listed for a project aren’t necessary for it to build. Projects using the Felix OSGI framework, for instance, only explicitly require its main jar to compile and run. Avoid fetching artifact dependencies with either intransitive() or notTransitive(), as in this example:
>
> ```
> libraryDependencies += "org.apache.felix" % "org.apache.felix.framework" % "1.8.0" intransitive()
> ```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1831: [CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1831:
URL: https://github.com/apache/incubator-celeborn/pull/1831#discussion_r1303849418
##########
project/CelebornBuild.scala:
##########
@@ -62,34 +62,46 @@ object Dependencies {
val protocVersion = "3.19.2"
val protoVersion = "3.19.2"
- val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion
+ val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion excludeAll(
+ ExclusionRule("net.java.dev.jna", "jna"))
val commonsIo = "commons-io" % "commons-io" % commonsIoVersion
val commonsLang3 = "org.apache.commons" % "commons-lang3" % commonsLang3Version
val findbugsJsr305 = "com.google.code.findbugs" % "jsr305" % findbugsVersion
val guava = "com.google.guava" % "guava" % guavaVersion
- val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion
- val hadoopClientRuntime = "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion
- val ioDropwizardMetricsCore = "io.dropwizard.metrics" % "metrics-core" % metricsVersion
- val ioDropwizardMetricsGraphite = "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion
- val ioDropwizardMetricsJvm = "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion
- val ioNetty = "io.netty" % "netty-all" % nettyVersion
+ val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion excludeAll(
+ ExclusionRule("org.slf4j", "slf4j-api"))
Review Comment:
Test locally with commit https://github.com/apache/incubator-celeborn/pull/1831/commits/32f7ce052763a12c8f9f33515e26143f498ea297, The use of `force()` ineffective.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1831: [CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1831:
URL: https://github.com/apache/incubator-celeborn/pull/1831#discussion_r1303809425
##########
pom.xml:
##########
@@ -268,6 +268,16 @@
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-transport-native-epoll</artifactId>
Review Comment:
seems we can simply exclude `io.netty:*` as we manage netty deps by ourselves
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1831: [CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1831:
URL: https://github.com/apache/incubator-celeborn/pull/1831#discussion_r1303771880
##########
project/CelebornBuild.scala:
##########
@@ -62,34 +62,46 @@ object Dependencies {
val protocVersion = "3.19.2"
val protoVersion = "3.19.2"
- val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion
+ val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion excludeAll(
+ ExclusionRule("net.java.dev.jna", "jna"))
val commonsIo = "commons-io" % "commons-io" % commonsIoVersion
val commonsLang3 = "org.apache.commons" % "commons-lang3" % commonsLang3Version
val findbugsJsr305 = "com.google.code.findbugs" % "jsr305" % findbugsVersion
val guava = "com.google.guava" % "guava" % guavaVersion
- val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion
- val hadoopClientRuntime = "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion
- val ioDropwizardMetricsCore = "io.dropwizard.metrics" % "metrics-core" % metricsVersion
- val ioDropwizardMetricsGraphite = "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion
- val ioDropwizardMetricsJvm = "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion
- val ioNetty = "io.netty" % "netty-all" % nettyVersion
+ val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion excludeAll(
+ ExclusionRule("org.slf4j", "slf4j-api"))
Review Comment:
~~an other approach, [Disable Transitivity](https://www.scala-sbt.org/1.x/docs/Library-Management.html#Disable+Transitivity)~~
> By default, these declarations fetch all project dependencies, transitively. In some instances, you may find that the dependencies listed for a project aren’t necessary for it to build. Projects using the Felix OSGI framework, for instance, only explicitly require its main jar to compile and run. Avoid fetching artifact dependencies with either intransitive() or notTransitive(), as in this example:
>
> ```
> libraryDependencies += "org.apache.felix" % "org.apache.felix.framework" % "1.8.0" intransitive()
> ```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1831: [CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1831:
URL: https://github.com/apache/incubator-celeborn/pull/1831#discussion_r1303849418
##########
project/CelebornBuild.scala:
##########
@@ -62,34 +62,46 @@ object Dependencies {
val protocVersion = "3.19.2"
val protoVersion = "3.19.2"
- val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion
+ val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion excludeAll(
+ ExclusionRule("net.java.dev.jna", "jna"))
val commonsIo = "commons-io" % "commons-io" % commonsIoVersion
val commonsLang3 = "org.apache.commons" % "commons-lang3" % commonsLang3Version
val findbugsJsr305 = "com.google.code.findbugs" % "jsr305" % findbugsVersion
val guava = "com.google.guava" % "guava" % guavaVersion
- val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion
- val hadoopClientRuntime = "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion
- val ioDropwizardMetricsCore = "io.dropwizard.metrics" % "metrics-core" % metricsVersion
- val ioDropwizardMetricsGraphite = "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion
- val ioDropwizardMetricsJvm = "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion
- val ioNetty = "io.netty" % "netty-all" % nettyVersion
+ val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion excludeAll(
+ ExclusionRule("org.slf4j", "slf4j-api"))
Review Comment:
Test locally with commit https://github.com/apache/incubator-celeborn/pull/1831/commits/32f7ce052763a12c8f9f33515e26143f498ea297, The use of `force()` ineffective.
```
$ ./build/sbt -Pspark-3.3 celeborn-client-spark-3/dependencyTree | grep slf4j-api
[info] | | | +-org.slf4j:slf4j-api:1.7.22 (evicted by: 2.0.7)
[info] | | | +-org.slf4j:slf4j-api:2.0.7
[info] | | | | +-org.slf4j:slf4j-api:1.7.22 (evicted by: 2.0.7)
[info] | | | | +-org.slf4j:slf4j-api:2.0.7
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1831: [CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1831:
URL: https://github.com/apache/incubator-celeborn/pull/1831#discussion_r1305508384
##########
project/CelebornBuild.scala:
##########
@@ -62,34 +62,46 @@ object Dependencies {
val protocVersion = "3.19.2"
val protoVersion = "3.19.2"
- val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion
+ val commonsCrypto = "org.apache.commons" % "commons-crypto" % commonsCryptoVersion excludeAll(
+ ExclusionRule("net.java.dev.jna", "jna"))
val commonsIo = "commons-io" % "commons-io" % commonsIoVersion
val commonsLang3 = "org.apache.commons" % "commons-lang3" % commonsLang3Version
val findbugsJsr305 = "com.google.code.findbugs" % "jsr305" % findbugsVersion
val guava = "com.google.guava" % "guava" % guavaVersion
- val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion
- val hadoopClientRuntime = "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion
- val ioDropwizardMetricsCore = "io.dropwizard.metrics" % "metrics-core" % metricsVersion
- val ioDropwizardMetricsGraphite = "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion
- val ioDropwizardMetricsJvm = "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion
- val ioNetty = "io.netty" % "netty-all" % nettyVersion
+ val hadoopClientApi = "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion excludeAll(
+ ExclusionRule("org.slf4j", "slf4j-api"))
Review Comment:
BTW, `dependencyOverrides` seems does not support pattern, e.g. `io.netty:*`, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org