You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "cxzl25 (via GitHub)" <gi...@apache.org> on 2023/09/20 07:26:55 UTC
[GitHub] [incubator-celeborn] cxzl25 opened a new pull request, #1930: [CELEBORN-1002] add SBT MRClientProject
cxzl25 opened a new pull request, #1930:
URL: https://github.com/apache/incubator-celeborn/pull/1930
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1002] Add SBT MRClientProject [incubator-celeborn]
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on PR #1930:
URL: https://github.com/apache/incubator-celeborn/pull/1930#issuecomment-1751892295
thanks, merging to main(v0.4.0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cxzl25 commented on a diff in pull request #1930: [CELEBORN-1002] add SBT MRClientProject
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #1930:
URL: https://github.com/apache/incubator-celeborn/pull/1930#discussion_r1340410526
##########
project/CelebornBuild.scala:
##########
@@ -872,3 +882,89 @@ trait FlinkClientProjects {
)
}
}
+
+////////////////////////////////////////////////////////
+// MR Client //
+////////////////////////////////////////////////////////
+object MRClientProjects {
+
+ def mrClient: Project = {
+ Project("celeborn-client-mr", file("client-mr/mr"))
+ .dependsOn(CelebornCommon.common, CelebornClient.client)
+ .settings(
+ commonSettings,
+ libraryDependencies ++= Seq(
+ Dependencies.hadoopClientApi,
+ Dependencies.hadoopClientRuntime,
+ Dependencies.hadoopMapreduceClientApp
+ ) ++ commonUnitTestDependencies
+ )
+ }
+
+ def mrClientShade: Project = {
+ Project("celeborn-client-mr-shaded", file("client-mr/mr-shaded"))
+ .dependsOn(mrClient)
+ .settings(
+ commonSettings,
+
+ // align final shaded jar name with maven.
+ (assembly / assemblyJarName) := {
+ val extension = artifact.value.extension
+ s"${moduleName.value}_${scalaBinaryVersion.value}-${version.value}.$extension"
+ },
+
+ (assembly / test) := {},
+
+ (assembly / logLevel) := Level.Info,
+
+ // include `scala-library` from assembly.
+ (assembly / assemblyPackageScala / assembleArtifact) := true,
+
+ (assembly / assemblyExcludedJars) := {
+ val cp = (assembly / fullClasspath).value
+ cp filter { v =>
+ val name = v.data.getName
+ !(name.startsWith("celeborn-") ||
+ name.startsWith("protobuf-java-") ||
+ name.startsWith("guava-") ||
+ name.startsWith("netty-") ||
+ name.startsWith("commons-lang3-") ||
+ name.startsWith("RoaringBitmap-") ||
+ name.startsWith("lz4-java-") ||
+ name.startsWith("zstd-jni-") ||
+ name.startsWith("scala-library-"))
+ }
+ },
+
+ (assembly / assemblyShadeRules) := Seq(
+ ShadeRule.rename("com.google.protobuf.**" -> "org.apache.celeborn.shaded.com.google.protobuf.@1").inAll,
+ ShadeRule.rename("com.google.common.**" -> "org.apache.celeborn.shaded.com.google.common.@1").inAll,
+ ShadeRule.rename("io.netty.**" -> "org.apache.celeborn.shaded.io.netty.@1").inAll,
+ ShadeRule.rename("org.apache.commons.**" -> "org.apache.celeborn.shaded.org.apache.commons.@1").inAll,
+ ShadeRule.rename("org.roaringbitmap.**" -> "org.apache.celeborn.shaded.org.roaringbitmap.@1").inAll
+ ),
+
+ (assembly / assemblyMergeStrategy) := {
+ case m if m.toLowerCase(Locale.ROOT).endsWith("manifest.mf") => MergeStrategy.discard
+ case m if m.startsWith("META-INF/license/") => MergeStrategy.discard
+ case m if m == "META-INF/LICENSE.txt" => MergeStrategy.discard
+ case m if m == "META-INF/NOTICE.txt" => MergeStrategy.discard
+ case m if m == "LICENSE.txt" => MergeStrategy.discard
+ case m if m == "NOTICE.txt" => MergeStrategy.discard
Review Comment:
SBT Packaging Flink/Spark Excluding license related files will be implemented in the next PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1930: [CELEBORN-1002] add SBT MRClientProject
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1930:
URL: https://github.com/apache/incubator-celeborn/pull/1930#discussion_r1331375447
##########
project/CelebornBuild.scala:
##########
@@ -872,3 +882,82 @@ trait FlinkClientProjects {
)
}
}
+
+////////////////////////////////////////////////////////
+// MR Client //
+////////////////////////////////////////////////////////
+object MRClientProjects {
+
+ def mrClient: Project = {
+ Project("celeborn-client-mr", file("client-mr/mr"))
+ .dependsOn(CelebornCommon.common % "test->test;compile->compile")
+ .dependsOn(CelebornClient.client % "test->test;compile->compile")
+ .settings(
+ commonSettings,
+ libraryDependencies ++= Seq(
+ Dependencies.hadoopClientApi,
+ Dependencies.hadoopClientRuntime,
+ Dependencies.hadoopMapreduceClientApp
+ ) ++ commonUnitTestDependencies
+ )
+ }
+
+ def mrClientShade: Project = {
+ Project("celeborn-client-mr-shaded", file("client-mr/mr-shaded"))
+ .dependsOn(mrClient)
+ .settings(
+ commonSettings,
+
+ // align final shaded jar name with maven.
+ (assembly / assemblyJarName) := {
+ val extension = artifact.value.extension
+ s"${moduleName.value}_${scalaBinaryVersion.value}-${version.value}.$extension"
+ },
+
+ (assembly / test) := {},
+
+ (assembly / logLevel) := Level.Info,
+
+ // Exclude `scala-library` from assembly.
Review Comment:
I think we need to include the `scala-library`.
https://github.com/apache/incubator-celeborn/blob/8eba1b470e9cd54cf0cdcefc381433ce94d881a0/client-mr/mr-shaded/pom.xml#L62-L65
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1930: [CELEBORN-1002] add SBT MRClientProject
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1930:
URL: https://github.com/apache/incubator-celeborn/pull/1930#issuecomment-1738980909
https://github.com/apache/incubator-celeborn/pull/1933 updates the shading configuration, the SBT part should be updated accordingly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1002] Add SBT MRClientProject [incubator-celeborn]
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on PR #1930:
URL: https://github.com/apache/incubator-celeborn/pull/1930#issuecomment-1749935283
SBT and Maven have different `guava` versions.
```shell
./build/mvn dependency:tree -Pmr -am -pl client-mr/mr
[INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ celeborn-client-mr_2.12 ---
[INFO] org.apache.celeborn:celeborn-client-mr_2.12:jar:0.4.0-SNAPSHOT
[INFO] +- org.apache.celeborn:celeborn-common_2.12:jar:0.4.0-SNAPSHOT:compile
[INFO] | +- com.google.guava:guava:jar:14.0.1:compile
```
```shell
./build/sbt -Pmr celeborn-client-mr/dependencyTree | grep guava
[info] | +-com.google.guava:guava:14.0.1 (evicted by: 27.0-jre)
[info] | +-com.google.guava:guava:27.0-jre
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] FMX commented on a diff in pull request #1930: [CELEBORN-1002] add SBT MRClientProject
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #1930:
URL: https://github.com/apache/incubator-celeborn/pull/1930#discussion_r1335697482
##########
project/CelebornBuild.scala:
##########
@@ -872,3 +882,82 @@ trait FlinkClientProjects {
)
}
}
+
+////////////////////////////////////////////////////////
+// MR Client //
+////////////////////////////////////////////////////////
+object MRClientProjects {
+
+ def mrClient: Project = {
+ Project("celeborn-client-mr", file("client-mr/mr"))
+ .dependsOn(CelebornCommon.common % "test->test;compile->compile")
+ .dependsOn(CelebornClient.client % "test->test;compile->compile")
+ .settings(
+ commonSettings,
+ libraryDependencies ++= Seq(
+ Dependencies.hadoopClientApi,
+ Dependencies.hadoopClientRuntime,
+ Dependencies.hadoopMapreduceClientApp
+ ) ++ commonUnitTestDependencies
+ )
+ }
+
+ def mrClientShade: Project = {
+ Project("celeborn-client-mr-shaded", file("client-mr/mr-shaded"))
+ .dependsOn(mrClient)
+ .settings(
+ commonSettings,
+
+ // align final shaded jar name with maven.
+ (assembly / assemblyJarName) := {
+ val extension = artifact.value.extension
+ s"${moduleName.value}_${scalaBinaryVersion.value}-${version.value}.$extension"
+ },
+
+ (assembly / test) := {},
+
+ (assembly / logLevel) := Level.Info,
+
+ // Exclude `scala-library` from assembly.
+ (assembly / assemblyPackageScala / assembleArtifact) := false,
+
+ (assembly / assemblyExcludedJars) := {
+ val cp = (assembly / fullClasspath).value
+ cp filter { v =>
+ val name = v.data.getName
+ !(name.startsWith("celeborn-") ||
+ name.startsWith("protobuf-java-") ||
+ name.startsWith("guava-") ||
+ name.startsWith("netty-") ||
+ name.startsWith("commons-lang3-") ||
+ name.startsWith("RoaringBitmap-"))
Review Comment:
Looks like you have already added `zstd-jni`, I think it's OK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1930: [CELEBORN-1002] add SBT MRClientProject
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1930:
URL: https://github.com/apache/incubator-celeborn/pull/1930#discussion_r1331371742
##########
project/CelebornBuild.scala:
##########
@@ -872,3 +882,82 @@ trait FlinkClientProjects {
)
}
}
+
+////////////////////////////////////////////////////////
+// MR Client //
+////////////////////////////////////////////////////////
+object MRClientProjects {
+
+ def mrClient: Project = {
+ Project("celeborn-client-mr", file("client-mr/mr"))
+ .dependsOn(CelebornCommon.common % "test->test;compile->compile")
+ .dependsOn(CelebornClient.client % "test->test;compile->compile")
Review Comment:
```suggestion
.dependsOn(CelebornCommon.common, CelebornClient.client)
```
according to `client-mr/mr/pom.xml` we don't need the `test` dependency of project `common` and `client`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cxzl25 commented on a diff in pull request #1930: [CELEBORN-1002] add SBT MRClientProject
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #1930:
URL: https://github.com/apache/incubator-celeborn/pull/1930#discussion_r1331716983
##########
project/CelebornBuild.scala:
##########
@@ -872,3 +882,82 @@ trait FlinkClientProjects {
)
}
}
+
+////////////////////////////////////////////////////////
+// MR Client //
+////////////////////////////////////////////////////////
+object MRClientProjects {
+
+ def mrClient: Project = {
+ Project("celeborn-client-mr", file("client-mr/mr"))
+ .dependsOn(CelebornCommon.common % "test->test;compile->compile")
+ .dependsOn(CelebornClient.client % "test->test;compile->compile")
+ .settings(
+ commonSettings,
+ libraryDependencies ++= Seq(
+ Dependencies.hadoopClientApi,
+ Dependencies.hadoopClientRuntime,
+ Dependencies.hadoopMapreduceClientApp
+ ) ++ commonUnitTestDependencies
+ )
+ }
+
+ def mrClientShade: Project = {
+ Project("celeborn-client-mr-shaded", file("client-mr/mr-shaded"))
+ .dependsOn(mrClient)
+ .settings(
+ commonSettings,
+
+ // align final shaded jar name with maven.
+ (assembly / assemblyJarName) := {
+ val extension = artifact.value.extension
+ s"${moduleName.value}_${scalaBinaryVersion.value}-${version.value}.$extension"
+ },
+
+ (assembly / test) := {},
+
+ (assembly / logLevel) := Level.Info,
+
+ // Exclude `scala-library` from assembly.
+ (assembly / assemblyPackageScala / assembleArtifact) := false,
+
+ (assembly / assemblyExcludedJars) := {
+ val cp = (assembly / fullClasspath).value
+ cp filter { v =>
+ val name = v.data.getName
+ !(name.startsWith("celeborn-") ||
+ name.startsWith("protobuf-java-") ||
+ name.startsWith("guava-") ||
+ name.startsWith("netty-") ||
+ name.startsWith("commons-lang3-") ||
+ name.startsWith("RoaringBitmap-"))
Review Comment:
It may be because in the hadoop MR environment, there is no lz4-java.
https://issues.apache.org/jira/browse/HADOOP-17292
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1930: [CELEBORN-1002] add SBT MRClientProject
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1930:
URL: https://github.com/apache/incubator-celeborn/pull/1930#discussion_r1332388047
##########
project/CelebornBuild.scala:
##########
@@ -872,3 +882,82 @@ trait FlinkClientProjects {
)
}
}
+
+////////////////////////////////////////////////////////
+// MR Client //
+////////////////////////////////////////////////////////
+object MRClientProjects {
+
+ def mrClient: Project = {
+ Project("celeborn-client-mr", file("client-mr/mr"))
+ .dependsOn(CelebornCommon.common % "test->test;compile->compile")
+ .dependsOn(CelebornClient.client % "test->test;compile->compile")
+ .settings(
+ commonSettings,
+ libraryDependencies ++= Seq(
+ Dependencies.hadoopClientApi,
+ Dependencies.hadoopClientRuntime,
+ Dependencies.hadoopMapreduceClientApp
+ ) ++ commonUnitTestDependencies
+ )
+ }
+
+ def mrClientShade: Project = {
+ Project("celeborn-client-mr-shaded", file("client-mr/mr-shaded"))
+ .dependsOn(mrClient)
+ .settings(
+ commonSettings,
+
+ // align final shaded jar name with maven.
+ (assembly / assemblyJarName) := {
+ val extension = artifact.value.extension
+ s"${moduleName.value}_${scalaBinaryVersion.value}-${version.value}.$extension"
+ },
+
+ (assembly / test) := {},
+
+ (assembly / logLevel) := Level.Info,
+
+ // Exclude `scala-library` from assembly.
Review Comment:
then shall we remove this rule?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1002] Add SBT MRClientProject [incubator-celeborn]
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady closed pull request #1930: [CELEBORN-1002] Add SBT MRClientProject
URL: https://github.com/apache/incubator-celeborn/pull/1930
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] FMX commented on a diff in pull request #1930: [CELEBORN-1002] add SBT MRClientProject
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #1930:
URL: https://github.com/apache/incubator-celeborn/pull/1930#discussion_r1335696045
##########
project/CelebornBuild.scala:
##########
@@ -872,3 +882,82 @@ trait FlinkClientProjects {
)
}
}
+
+////////////////////////////////////////////////////////
+// MR Client //
+////////////////////////////////////////////////////////
+object MRClientProjects {
+
+ def mrClient: Project = {
+ Project("celeborn-client-mr", file("client-mr/mr"))
+ .dependsOn(CelebornCommon.common % "test->test;compile->compile")
+ .dependsOn(CelebornClient.client % "test->test;compile->compile")
+ .settings(
+ commonSettings,
+ libraryDependencies ++= Seq(
+ Dependencies.hadoopClientApi,
+ Dependencies.hadoopClientRuntime,
+ Dependencies.hadoopMapreduceClientApp
+ ) ++ commonUnitTestDependencies
+ )
+ }
+
+ def mrClientShade: Project = {
+ Project("celeborn-client-mr-shaded", file("client-mr/mr-shaded"))
+ .dependsOn(mrClient)
+ .settings(
+ commonSettings,
+
+ // align final shaded jar name with maven.
+ (assembly / assemblyJarName) := {
+ val extension = artifact.value.extension
+ s"${moduleName.value}_${scalaBinaryVersion.value}-${version.value}.$extension"
+ },
+
+ (assembly / test) := {},
+
+ (assembly / logLevel) := Level.Info,
+
+ // Exclude `scala-library` from assembly.
+ (assembly / assemblyPackageScala / assembleArtifact) := false,
+
+ (assembly / assemblyExcludedJars) := {
+ val cp = (assembly / fullClasspath).value
+ cp filter { v =>
+ val name = v.data.getName
+ !(name.startsWith("celeborn-") ||
+ name.startsWith("protobuf-java-") ||
+ name.startsWith("guava-") ||
+ name.startsWith("netty-") ||
+ name.startsWith("commons-lang3-") ||
+ name.startsWith("RoaringBitmap-"))
Review Comment:
I think Hadoop has `zstd-jni` already. I'll make a test to verify this later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cxzl25 commented on a diff in pull request #1930: [CELEBORN-1002] add SBT MRClientProject
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #1930:
URL: https://github.com/apache/incubator-celeborn/pull/1930#discussion_r1331709563
##########
project/CelebornBuild.scala:
##########
@@ -872,3 +882,82 @@ trait FlinkClientProjects {
)
}
}
+
+////////////////////////////////////////////////////////
+// MR Client //
+////////////////////////////////////////////////////////
+object MRClientProjects {
+
+ def mrClient: Project = {
+ Project("celeborn-client-mr", file("client-mr/mr"))
+ .dependsOn(CelebornCommon.common % "test->test;compile->compile")
+ .dependsOn(CelebornClient.client % "test->test;compile->compile")
+ .settings(
+ commonSettings,
+ libraryDependencies ++= Seq(
+ Dependencies.hadoopClientApi,
+ Dependencies.hadoopClientRuntime,
+ Dependencies.hadoopMapreduceClientApp
+ ) ++ commonUnitTestDependencies
+ )
+ }
+
+ def mrClientShade: Project = {
+ Project("celeborn-client-mr-shaded", file("client-mr/mr-shaded"))
+ .dependsOn(mrClient)
+ .settings(
+ commonSettings,
+
+ // align final shaded jar name with maven.
+ (assembly / assemblyJarName) := {
+ val extension = artifact.value.extension
+ s"${moduleName.value}_${scalaBinaryVersion.value}-${version.value}.$extension"
+ },
+
+ (assembly / test) := {},
+
+ (assembly / logLevel) := Level.Info,
+
+ // Exclude `scala-library` from assembly.
Review Comment:
It seems that this `relocation` does not take effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1930: [CELEBORN-1002] add SBT MRClientProject
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1930:
URL: https://github.com/apache/incubator-celeborn/pull/1930#discussion_r1331377495
##########
project/CelebornBuild.scala:
##########
@@ -872,3 +882,82 @@ trait FlinkClientProjects {
)
}
}
+
+////////////////////////////////////////////////////////
+// MR Client //
+////////////////////////////////////////////////////////
+object MRClientProjects {
+
+ def mrClient: Project = {
+ Project("celeborn-client-mr", file("client-mr/mr"))
+ .dependsOn(CelebornCommon.common % "test->test;compile->compile")
+ .dependsOn(CelebornClient.client % "test->test;compile->compile")
+ .settings(
+ commonSettings,
+ libraryDependencies ++= Seq(
+ Dependencies.hadoopClientApi,
+ Dependencies.hadoopClientRuntime,
+ Dependencies.hadoopMapreduceClientApp
+ ) ++ commonUnitTestDependencies
+ )
+ }
+
+ def mrClientShade: Project = {
+ Project("celeborn-client-mr-shaded", file("client-mr/mr-shaded"))
+ .dependsOn(mrClient)
+ .settings(
+ commonSettings,
+
+ // align final shaded jar name with maven.
+ (assembly / assemblyJarName) := {
+ val extension = artifact.value.extension
+ s"${moduleName.value}_${scalaBinaryVersion.value}-${version.value}.$extension"
+ },
+
+ (assembly / test) := {},
+
+ (assembly / logLevel) := Level.Info,
+
+ // Exclude `scala-library` from assembly.
+ (assembly / assemblyPackageScala / assembleArtifact) := false,
+
+ (assembly / assemblyExcludedJars) := {
+ val cp = (assembly / fullClasspath).value
+ cp filter { v =>
+ val name = v.data.getName
+ !(name.startsWith("celeborn-") ||
+ name.startsWith("protobuf-java-") ||
+ name.startsWith("guava-") ||
+ name.startsWith("netty-") ||
+ name.startsWith("commons-lang3-") ||
+ name.startsWith("RoaringBitmap-"))
Review Comment:
lz4?
https://github.com/apache/incubator-celeborn/blob/8eba1b470e9cd54cf0cdcefc381433ce94d881a0/client-mr/mr-shaded/pom.xml#L76-L85
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cxzl25 commented on a diff in pull request #1930: [CELEBORN-1002] add SBT MRClientProject
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #1930:
URL: https://github.com/apache/incubator-celeborn/pull/1930#discussion_r1336071605
##########
project/CelebornBuild.scala:
##########
@@ -872,3 +882,82 @@ trait FlinkClientProjects {
)
}
}
+
+////////////////////////////////////////////////////////
+// MR Client //
+////////////////////////////////////////////////////////
+object MRClientProjects {
+
+ def mrClient: Project = {
+ Project("celeborn-client-mr", file("client-mr/mr"))
+ .dependsOn(CelebornCommon.common % "test->test;compile->compile")
+ .dependsOn(CelebornClient.client % "test->test;compile->compile")
+ .settings(
+ commonSettings,
+ libraryDependencies ++= Seq(
+ Dependencies.hadoopClientApi,
+ Dependencies.hadoopClientRuntime,
+ Dependencies.hadoopMapreduceClientApp
+ ) ++ commonUnitTestDependencies
+ )
+ }
+
+ def mrClientShade: Project = {
+ Project("celeborn-client-mr-shaded", file("client-mr/mr-shaded"))
+ .dependsOn(mrClient)
+ .settings(
+ commonSettings,
+
+ // align final shaded jar name with maven.
+ (assembly / assemblyJarName) := {
+ val extension = artifact.value.extension
+ s"${moduleName.value}_${scalaBinaryVersion.value}-${version.value}.$extension"
+ },
+
+ (assembly / test) := {},
+
+ (assembly / logLevel) := Level.Info,
+
+ // Exclude `scala-library` from assembly.
+ (assembly / assemblyPackageScala / assembleArtifact) := false,
+
+ (assembly / assemblyExcludedJars) := {
+ val cp = (assembly / fullClasspath).value
+ cp filter { v =>
+ val name = v.data.getName
+ !(name.startsWith("celeborn-") ||
+ name.startsWith("protobuf-java-") ||
+ name.startsWith("guava-") ||
+ name.startsWith("netty-") ||
+ name.startsWith("commons-lang3-") ||
+ name.startsWith("RoaringBitmap-"))
Review Comment:
Hadoop uses the native zstd and does not use `zstd-ini`.
See https://issues.apache.org/jira/browse/HADOOP-13578
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] codecov[bot] commented on pull request #1930: [CELEBORN-1002] add SBT MRClientProject
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #1930:
URL: https://github.com/apache/incubator-celeborn/pull/1930#issuecomment-1727144929
## [Codecov](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1930?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#1930](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1930?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (cea7412) into [main](https://app.codecov.io/gh/apache/incubator-celeborn/commit/6b0addb93484b5c78b47d194251838748e4dd67f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (6b0addb) will **increase** coverage by `0.01%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## main #1930 +/- ##
==========================================
+ Coverage 46.68% 46.69% +0.01%
==========================================
Files 164 164
Lines 10280 10280
Branches 938 938
==========================================
+ Hits 4798 4799 +1
Misses 5168 5168
+ Partials 314 313 -1
```
[see 1 file with indirect coverage changes](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1930/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1930: [CELEBORN-1002] add SBT MRClientProject
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1930:
URL: https://github.com/apache/incubator-celeborn/pull/1930#discussion_r1331387509
##########
project/CelebornBuild.scala:
##########
@@ -872,3 +882,82 @@ trait FlinkClientProjects {
)
}
}
+
+////////////////////////////////////////////////////////
+// MR Client //
+////////////////////////////////////////////////////////
+object MRClientProjects {
+
+ def mrClient: Project = {
+ Project("celeborn-client-mr", file("client-mr/mr"))
+ .dependsOn(CelebornCommon.common % "test->test;compile->compile")
+ .dependsOn(CelebornClient.client % "test->test;compile->compile")
+ .settings(
+ commonSettings,
+ libraryDependencies ++= Seq(
+ Dependencies.hadoopClientApi,
+ Dependencies.hadoopClientRuntime,
+ Dependencies.hadoopMapreduceClientApp
+ ) ++ commonUnitTestDependencies
+ )
+ }
+
+ def mrClientShade: Project = {
+ Project("celeborn-client-mr-shaded", file("client-mr/mr-shaded"))
+ .dependsOn(mrClient)
+ .settings(
+ commonSettings,
+
+ // align final shaded jar name with maven.
+ (assembly / assemblyJarName) := {
+ val extension = artifact.value.extension
+ s"${moduleName.value}_${scalaBinaryVersion.value}-${version.value}.$extension"
+ },
+
+ (assembly / test) := {},
+
+ (assembly / logLevel) := Level.Info,
+
+ // Exclude `scala-library` from assembly.
+ (assembly / assemblyPackageScala / assembleArtifact) := false,
+
+ (assembly / assemblyExcludedJars) := {
+ val cp = (assembly / fullClasspath).value
+ cp filter { v =>
+ val name = v.data.getName
+ !(name.startsWith("celeborn-") ||
+ name.startsWith("protobuf-java-") ||
+ name.startsWith("guava-") ||
+ name.startsWith("netty-") ||
+ name.startsWith("commons-lang3-") ||
+ name.startsWith("RoaringBitmap-"))
Review Comment:
hmm... it seems we don't need the `lz4-java` dependency for MR shaded client? cc @FMX
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1002] Add SBT MRClientProject [incubator-celeborn]
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on PR #1930:
URL: https://github.com/apache/incubator-celeborn/pull/1930#issuecomment-1751892228
> we can align the `guava` versions in #1928.
addressed in https://github.com/apache/incubator-celeborn/pull/1930/commits/cbb8a4290af3af3f0d153b0d31181f80ed423a37
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1002] Add SBT MRClientProject [incubator-celeborn]
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on PR #1930:
URL: https://github.com/apache/incubator-celeborn/pull/1930#issuecomment-1749936451
we can align the `guava` versions in https://github.com/apache/incubator-celeborn/pull/1928.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1930: [CELEBORN-1002] add SBT MRClientProject
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1930:
URL: https://github.com/apache/incubator-celeborn/pull/1930#discussion_r1331375447
##########
project/CelebornBuild.scala:
##########
@@ -872,3 +882,82 @@ trait FlinkClientProjects {
)
}
}
+
+////////////////////////////////////////////////////////
+// MR Client //
+////////////////////////////////////////////////////////
+object MRClientProjects {
+
+ def mrClient: Project = {
+ Project("celeborn-client-mr", file("client-mr/mr"))
+ .dependsOn(CelebornCommon.common % "test->test;compile->compile")
+ .dependsOn(CelebornClient.client % "test->test;compile->compile")
+ .settings(
+ commonSettings,
+ libraryDependencies ++= Seq(
+ Dependencies.hadoopClientApi,
+ Dependencies.hadoopClientRuntime,
+ Dependencies.hadoopMapreduceClientApp
+ ) ++ commonUnitTestDependencies
+ )
+ }
+
+ def mrClientShade: Project = {
+ Project("celeborn-client-mr-shaded", file("client-mr/mr-shaded"))
+ .dependsOn(mrClient)
+ .settings(
+ commonSettings,
+
+ // align final shaded jar name with maven.
+ (assembly / assemblyJarName) := {
+ val extension = artifact.value.extension
+ s"${moduleName.value}_${scalaBinaryVersion.value}-${version.value}.$extension"
+ },
+
+ (assembly / test) := {},
+
+ (assembly / logLevel) := Level.Info,
+
+ // Exclude `scala-library` from assembly.
Review Comment:
I think we need to include the `scala-library` and relocate it.
https://github.com/apache/incubator-celeborn/blob/8eba1b470e9cd54cf0cdcefc381433ce94d881a0/client-mr/mr-shaded/pom.xml#L62-L65
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1930: [CELEBORN-1002] add SBT MRClientProject
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1930:
URL: https://github.com/apache/incubator-celeborn/pull/1930#discussion_r1332339537
##########
project/CelebornBuild.scala:
##########
@@ -872,3 +882,82 @@ trait FlinkClientProjects {
)
}
}
+
+////////////////////////////////////////////////////////
+// MR Client //
+////////////////////////////////////////////////////////
+object MRClientProjects {
+
+ def mrClient: Project = {
+ Project("celeborn-client-mr", file("client-mr/mr"))
+ .dependsOn(CelebornCommon.common % "test->test;compile->compile")
+ .dependsOn(CelebornClient.client % "test->test;compile->compile")
+ .settings(
+ commonSettings,
+ libraryDependencies ++= Seq(
+ Dependencies.hadoopClientApi,
+ Dependencies.hadoopClientRuntime,
+ Dependencies.hadoopMapreduceClientApp
+ ) ++ commonUnitTestDependencies
+ )
+ }
+
+ def mrClientShade: Project = {
+ Project("celeborn-client-mr-shaded", file("client-mr/mr-shaded"))
+ .dependsOn(mrClient)
+ .settings(
+ commonSettings,
+
+ // align final shaded jar name with maven.
+ (assembly / assemblyJarName) := {
+ val extension = artifact.value.extension
+ s"${moduleName.value}_${scalaBinaryVersion.value}-${version.value}.$extension"
+ },
+
+ (assembly / test) := {},
+
+ (assembly / logLevel) := Level.Info,
+
+ // Exclude `scala-library` from assembly.
+ (assembly / assemblyPackageScala / assembleArtifact) := false,
+
+ (assembly / assemblyExcludedJars) := {
+ val cp = (assembly / fullClasspath).value
+ cp filter { v =>
+ val name = v.data.getName
+ !(name.startsWith("celeborn-") ||
+ name.startsWith("protobuf-java-") ||
+ name.startsWith("guava-") ||
+ name.startsWith("netty-") ||
+ name.startsWith("commons-lang3-") ||
+ name.startsWith("RoaringBitmap-"))
Review Comment:
shall we also include `zstd-jni`? cc @FMX
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org