You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "cfmcgrady (via GitHub)" <gi...@apache.org> on 2023/07/24 16:30:10 UTC
[GitHub] [incubator-celeborn] cfmcgrady opened a new pull request, #1757: [WIP][CELEBORN-XXX][BUILD] Initial support sbt
cfmcgrady opened a new pull request, #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
TODO
pure sbt project
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#issuecomment-1652971175
@FMX
the `assembly` command only works on shaded-related modules now, let's switch the project first.
```shell
$ ./build/sbt -Pspark-3.4
sbt:celeborn> projects
[info] In file:/home/chenfu/workspace/celeborn/
[info] * celeborn
[info] celeborn-client-spark-3
[info] celeborn-client-spark-3-shaded
[info] client
[info] common
[info] master
[info] service
[info] spark-common
[info] spark-it
[info] worker
sbt:celeborn> project celeborn-client-spark-3-shaded
sbt:celeborn-client-spark-3-shaded> assembly
```
or
```shell
$ ./build/sbt -Pspark-3.4 celeborn-client-spark-3-shaded/assembly
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1274501474
##########
build/sbt-config/repositories-cn.template:
##########
@@ -1,7 +1,11 @@
+# As a Chinese developer facing network issues, you can accelerate the download speed of bootstrap/plugin/dependencies jar packages by executing the following command to configure the mirror:
Review Comment:
break the long line by 100 chars
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1757: [WIP][CELEBORN-XXX][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1272938558
##########
project/build.properties:
##########
@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Please update the version in appveyor-install-dependencies.ps1 together.
Review Comment:
Remove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#issuecomment-1651061536
@waitinfuture @RexXiong @FMX @AngersZhuuuu would you please take a look?
note: if you encounter network issue, run the following command to apply mirror
```
cp build/sbt-config/repositories-cn.template build/sbt-config/repositories-local
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#issuecomment-1657398665
I plan to add the doc, once the following task is finished. cc @waitinfuture
- [ ] https://github.com/apache/incubator-celeborn/pull/1764
- [ ] https://github.com/apache/incubator-celeborn/pull/1771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1275952899
##########
project/CelebornBuild.scala:
##########
@@ -0,0 +1,603 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.nio.file.Files
+import java.nio.charset.StandardCharsets.UTF_8
+import java.util.Locale
+
+import scala.util.Properties
+
+import sbtassembly.AssemblyPlugin.autoImport._
+import sbtprotoc.ProtocPlugin.autoImport._
+
+import sbt._
+import sbt.Keys._
+import Utils._
+import CelebornCommonSettings._
+// import sbt.Keys.streams
+
+object CelebornCommonSettings {
+
+ // Scala versions
+ val SCALA_2_11_12 = "2.11.12"
+ val SCALA_2_12_10 = "2.12.10"
+ val SCALA_2_12_15 = "2.12.15"
+ val SCALA_2_12_17 = "2.12.17"
+ val scala213 = "2.13.5"
+ val ALL_SCALA_VERSIONS = Seq(SCALA_2_11_12, SCALA_2_12_10, SCALA_2_12_15, SCALA_2_12_17, scala213)
+
+ val DEFAULT_SCALA_VERSION = SCALA_2_12_15
+
+ val projectScalaVersion = defaultScalaVersion()
+
+ val zstdJniVersion = sparkClientProjects.map(_.zstdJniVersion).getOrElse("1.5.2-1")
+ val lz4JavaVersion = sparkClientProjects.map(_.lz4JavaVersion).getOrElse("1.8.0")
+
+ // Dependent library versions
+ val commonsCryptoVersion = "1.0.0"
+ val commonsIoVersion = "2.13.0"
+ val commonsLang3Version = "3.12.0"
+ val findbugsVersion = "1.3.9"
+ val guavaVersion = "14.0.1"
+ val hadoopVersion = "3.2.4"
+ val javaxServletVersion = "3.1.0"
+ val leveldbJniVersion = "1.8"
+ val log4j2Version = "2.17.2"
+ val metricsVersion = "3.2.6"
+ val scalatestMockitoVersion = "1.17.14"
+ val nettyVersion = "4.1.93.Final"
+ val ratisVersion = "2.5.1"
+ val roaringBitmapVersion = "0.9.32"
+ val slf4jVersion = "1.7.36"
+ val snakeyamlVersion = "1.33"
+
+ // Versions for proto
+ val protocVersion = "3.19.2"
+ val protoVersion = "3.19.2"
+
+ scalaVersion := projectScalaVersion
+
+ autoScalaLibrary := false
+
+ // crossScalaVersions must be set to Nil on the root project
+ crossScalaVersions := Nil
+
+ lazy val commonSettings = Seq(
+ organization := "org.apache.celeborn",
+ scalaVersion := projectScalaVersion,
+ crossScalaVersions := ALL_SCALA_VERSIONS,
+ fork := true,
+ scalacOptions ++= Seq("-target:jvm-1.8"),
+ javacOptions ++= Seq("-encoding", UTF_8.name(), "-source", "1.8"),
+
+ // -target cannot be passed as a parameter to javadoc. See https://github.com/sbt/sbt/issues/355
+ Compile / compile / javacOptions ++= Seq("-target", "1.8"),
+
+ // Make sure any tests in any project that uses Spark is configured for running well locally
+ Test / javaOptions ++= Seq(
+ "-Xmx4g"
+ ),
+
+ testOptions += Tests.Argument("-oF"),
+
+ Test / testOptions += Tests.Argument("-oDF"),
+ Test / testOptions += Tests.Argument(TestFrameworks.JUnit, "-v", "-a"),
+
+ // Don't execute in parallel since we can't have multiple Sparks in the same JVM
+ Test / parallelExecution := false,
+
+ scalacOptions ++= Seq(
+ "-P:genjavadoc:strictVisibility=true" // hide package private types and methods in javadoc
+ ),
+
+ javaOptions += "-Xmx4g",
+
+ // Configurations to speed up tests and reduce memory footprint
+ Test / javaOptions ++= Seq(
+ "-Xmx4g"
+ ),
+
+ Test / envVars += ("IS_TESTING", "1")
+ )
+
+ lazy val protoSettings = Seq(
+ // Setting version for the protobuf compiler
+ PB.protocVersion := protocVersion,
+ // set proto sources path
+ Compile / PB.protoSources := Seq(sourceDirectory.value / "main" / "proto"),
+ Compile / PB.targets := Seq(PB.gens.java -> (Compile / sourceManaged).value)
+ )
+
+ lazy val commonUnitTestDependencies = Seq(
+ "org.mockito" % "mockito-core" % "4.11.0" % "test",
+ "org.scalatest" %% "scalatest" % "3.2.16" % "test",
+ "junit" % "junit" % "4.12" % "test",
+ // https://www.scala-sbt.org/1.x/docs/Testing.html
+ "com.github.sbt" % "junit-interface" % "0.13.3" % "test")
+}
+
+object CelebornBuild extends sbt.internal.BuildDef {
+ override def projectDefinitions(baseDirectory: File): Seq[Project] = {
+ Seq(
+ CelebornCommon.common,
+ CelebornClient.client,
+ CelebornService.service,
+ CelebornWorker.worker,
+ CelebornMaster.master) ++ maybeSparkClientModules
+ }
+
+ // ThisBuild / parallelExecution := false
+
+ // scalaVersion := "2.11.12"
+
+ // autoScalaLibrary := false
+
+ crossScalaVersions := Nil
+
+ // load user-defined Profiles
+ // loadProfiles()
+}
+
+object Utils {
+ val profiles = {
+ val profiles = Properties.envOrNone("SBT_MAVEN_PROFILES")
+ .orElse(Properties.propOrNone("sbt.maven.profiles")) match {
+ case None => Seq("sbt")
+ case Some(v) =>
+ v.split("(\\s+|,)").filterNot(_.isEmpty).map(_.trim.replaceAll("-P", "")).toSeq
+ }
+ if (profiles.contains("jdwp-test-debug")) {
+ sys.props.put("test.jdwp.enabled", "true")
+ }
+ profiles
+ }
+
+ val SPARK_VERSION = profiles.filter(_.startsWith("spark")).headOption
+
+ lazy val sparkClientProjects = SPARK_VERSION match {
+ case Some("spark-2.4") => Some(Spark24)
+ case Some("spark-3.0") => Some(Spark30)
+ case Some("spark-3.1") => Some(Spark31)
+ case Some("spark-3.2") => Some(Spark32)
+ case Some("spark-3.3") => Some(Spark33)
+ case Some("spark-3.4") => Some(Spark34)
+ case _ => None
+ }
+
+ lazy val maybeSparkClientModules: Seq[Project] = sparkClientProjects.map(_.modules).getOrElse(Seq.empty)
+
+ def defaultScalaVersion(): String = {
+ // 1. Inherit the scala version of the spark project
+ // 2. if the spark profile not specified, using the DEFAULT_SCALA_VERSION
+ val v = sparkClientProjects.map(_.sparkProjectScalaVersion).getOrElse(DEFAULT_SCALA_VERSION)
+ require(ALL_SCALA_VERSIONS.contains(v), s"found not allow scala version: $v")
+ v
+ }
+}
+
+object CelebornCommon {
+ lazy val common = Project("celeborn-common", file("common"))
+ .settings (
+ commonSettings,
+ protoSettings,
+ libraryDependencies ++= Seq(
+ "com.google.protobuf" % "protobuf-java" % protoVersion % "protobuf",
+ "com.google.code.findbugs" % "jsr305" % findbugsVersion,
+ "com.google.guava" % "guava" % guavaVersion,
+ "commons-io" % "commons-io" % commonsIoVersion,
+ "io.dropwizard.metrics" % "metrics-core" % metricsVersion,
+ "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion,
+ "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion,
+ "io.netty" % "netty-all" % nettyVersion,
+ "org.apache.commons" % "commons-crypto" % commonsCryptoVersion,
+ "org.apache.commons" % "commons-lang3" % commonsLang3Version,
+ "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion,
+ "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion,
+ "org.apache.ratis" % "ratis-client" % ratisVersion,
+ "org.apache.ratis" % "ratis-common" % ratisVersion,
+ "org.fusesource.leveldbjni" % "leveldbjni-all" % leveldbJniVersion,
+ "org.roaringbitmap" % "RoaringBitmap" % roaringBitmapVersion,
+ "org.scala-lang" % "scala-reflect" % scalaVersion.value,
+ "org.slf4j" % "jcl-over-slf4j" % slf4jVersion,
+ "org.slf4j" % "jul-to-slf4j" % slf4jVersion,
+ "org.slf4j" % "slf4j-api" % slf4jVersion,
+ "org.yaml" % "snakeyaml" % snakeyamlVersion,
+ "org.apache.logging.log4j" % "log4j-slf4j-impl" % log4j2Version % "test",
+ "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version % "test",
+
+ // Compiler plugins
+ // -- Bump up the genjavadoc version explicitly to 0.18 to work with Scala 2.12
+ compilerPlugin(
+ "com.typesafe.genjavadoc" %% "genjavadoc-plugin" % "0.18" cross CrossVersion.full)
+ ) ++ commonUnitTestDependencies,
+
+ Compile / sourceGenerators += Def.task {
+ val file = (Compile / sourceManaged).value / "org" / "apache" / "celeborn" / "package.scala"
+ streams.value.log.info("geneate version information file %s".format(file.toPath))
+ IO.write(file,
+ s"""package org.apache
+ |
+ |package object celeborn {
+ | val VERSION = "${version.value}"
+ |}
+ |""".stripMargin)
+ Seq(file)
+ // generate version task depends on PB generate to avoid concurrency generate source files
+ // otherwise we may encounter the error:
+ // ```
+ // [error] IO error while decoding ./celeborn/common/target/scala-2.12/src_managed/main/org/apache/celeborn/package.scala with UTF-8: ./celeborn/common/target/scala-2.12/src_managed/main/org/apache/celeborn/package.scala (No such file or directory)
+ // ```
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1274434766
##########
project/CelebornBuild.scala:
##########
@@ -0,0 +1,604 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.nio.file.Files
+import java.nio.charset.StandardCharsets.UTF_8
+import java.util.Locale
+
+import scala.util.Properties
+
+import sbtassembly.AssemblyPlugin.autoImport._
+import sbtprotoc.ProtocPlugin.autoImport._
+
+import sbt._
+import sbt.Keys._
+import Utils._
+import CelebornCommonSettings._
+// import sbt.Keys.streams
+
+object CelebornCommonSettings {
+
+ // Scala versions
+ val SCALA_2_11_12 = "2.11.12"
+ val SCALA_2_12_10 = "2.12.10"
+ val SCALA_2_12_15 = "2.12.15"
+ val SCALA_2_12_17 = "2.12.17"
+ val scala213 = "2.13.5"
+ val ALL_SCALA_VERSIONS = Seq(SCALA_2_11_12, SCALA_2_12_10, SCALA_2_12_15, SCALA_2_12_17, scala213)
+
+ val DEFAULT_SCALA_VERSION = SCALA_2_12_15
+
+ val projectScalaVersion = defaultScalaVersion()
+
+ val zstdJniVersion = sparkClientProjects.map(_.zstdJniVersion).getOrElse("1.5.2-1")
+ val lz4JavaVersion = sparkClientProjects.map(_.lz4JavaVersion).getOrElse("1.8.0")
+
+ // Dependent library versions
+ val commonsCryptoVersion = "1.0.0"
+ val commonsIoVersion = "2.13.0"
+ val commonsLang3Version = "3.12.0"
+ val findbugsVersion = "1.3.9"
+ val guavaVersion = "14.0.1"
+ val hadoopVersion = "3.2.4"
+ val javaxServletVersion = "3.1.0"
+ val leveldbJniVersion = "1.8"
+ val log4j2Version = "2.17.2"
+ val metricsVersion = "3.2.6"
+ val scalatestMockitoVersion = "1.17.14"
+ val nettyVersion = "4.1.93.Final"
+ val ratisVersion = "2.5.1"
+ val roaringBitmapVersion = "0.9.32"
+ val slf4jVersion = "1.7.36"
+ val snakeyamlVersion = "1.33"
+
+ // Versions for proto
+ val protocVersion = "3.19.2"
+ val protoVersion = "3.19.2"
+
+ scalaVersion := projectScalaVersion
+
+ autoScalaLibrary := false
+
+ // crossScalaVersions must be set to Nil on the root project
+ crossScalaVersions := Nil
+
+ lazy val commonSettings = Seq(
+ organization := "org.apache.celeborn",
+ scalaVersion := projectScalaVersion,
+ crossScalaVersions := ALL_SCALA_VERSIONS,
+ fork := true,
+ scalacOptions ++= Seq("-target:jvm-1.8"),
+ javacOptions ++= Seq("-encoding", UTF_8.name(), "-source", "1.8"),
+
+ // -target cannot be passed as a parameter to javadoc. See https://github.com/sbt/sbt/issues/355
+ Compile / compile / javacOptions ++= Seq("-target", "1.8"),
+
+ // Make sure any tests in any project that uses Spark is configured for running well locally
+ Test / javaOptions ++= Seq(
+ "-Xmx4g"
+ ),
+
+ testOptions += Tests.Argument("-oF"),
+
+ Test / testOptions += Tests.Argument("-oDF"),
+ Test / testOptions += Tests.Argument(TestFrameworks.JUnit, "-v", "-a"),
+
+ // Don't execute in parallel since we can't have multiple Sparks in the same JVM
+ Test / parallelExecution := false,
+
+ scalacOptions ++= Seq(
+ "-P:genjavadoc:strictVisibility=true" // hide package private types and methods in javadoc
+ ),
+
+ javaOptions += "-Xmx4g",
+
+ // Configurations to speed up tests and reduce memory footprint
+ Test / javaOptions ++= Seq(
+ "-Xmx4g"
+ ),
+
+ Test / envVars += ("IS_TESTING", "1")
+ )
+
+ lazy val protoSettings = Seq(
+ // Setting version for the protobuf compiler
+ PB.protocVersion := protocVersion,
+ // set proto sources path
+ Compile / PB.protoSources := Seq(sourceDirectory.value / "main" / "proto"),
+ Compile / PB.targets := Seq(PB.gens.java -> (Compile / sourceManaged).value)
+ )
+
+ lazy val commonUnitTestDependencies = Seq(
+ "org.mockito" % "mockito-core" % "4.11.0" % "test",
+ "org.scalatest" %% "scalatest" % "3.2.16" % "test",
+ "junit" % "junit" % "4.12" % "test",
+ // https://www.scala-sbt.org/1.x/docs/Testing.html
+ "com.github.sbt" % "junit-interface" % "0.13.3" % "test")
+}
+
+object CelebornBuild extends sbt.internal.BuildDef {
+ override def projectDefinitions(baseDirectory: File): Seq[Project] = {
+ Seq(
+ CelebornCommon.common,
+ CelebornClient.client,
+ CelebornService.service,
+ CelebornWorker.worker,
+ CelebornMaster.master) ++ maybeSparkClientModules
+ }
+
+ // ThisBuild / parallelExecution := false
+
+ // scalaVersion := "2.11.12"
+
+ // autoScalaLibrary := false
+
+ crossScalaVersions := Nil
+
+ // load user-defined Profiles
+ // loadProfiles()
+}
+
+object Utils {
+ val profiles = {
+ val profiles = Properties.envOrNone("SBT_MAVEN_PROFILES")
+ .orElse(Properties.propOrNone("sbt.maven.profiles")) match {
+ case None => Seq("sbt")
+ case Some(v) =>
+ v.split("(\\s+|,)").filterNot(_.isEmpty).map(_.trim.replaceAll("-P", "")).toSeq
+ }
+ if (profiles.contains("jdwp-test-debug")) {
+ sys.props.put("test.jdwp.enabled", "true")
+ }
+ profiles
+ }
+
+ val SPARK_VERSION = profiles.filter(_.startsWith("spark")).headOption
+
+ lazy val sparkClientProjects = SPARK_VERSION match {
+ case Some("spark-2.4") => Some(Spark24)
+ case Some("spark-3.0") => Some(Spark30)
+ case Some("spark-3.1") => Some(Spark31)
+ case Some("spark-3.2") => Some(Spark32)
+ case Some("spark-3.3") => Some(Spark33)
+ case Some("spark-3.4") => Some(Spark34)
+ case _ => None
+ }
+
+ lazy val maybeSparkClientModules: Seq[Project] = sparkClientProjects.map(_.modules).getOrElse(Seq.empty)
+
+ def defaultScalaVersion(): String = {
+ // 1. Inherit the scala version of the spark project
+ // 2. if the spark profile not specified, using the DEFAULT_SCALA_VERSION
+ val v = sparkClientProjects.map(_.sparkProjectScalaVersion).getOrElse(DEFAULT_SCALA_VERSION)
+ require(ALL_SCALA_VERSIONS.contains(v), s"found not allow scala version: $v")
+ v
+ }
+}
+
+object CelebornCommon {
+ lazy val common = (project in file("common"))
+ .settings (
+ name := "celeborn-common",
+ commonSettings,
+ protoSettings,
+ libraryDependencies ++= Seq(
+ "com.google.protobuf" % "protobuf-java" % protoVersion % "protobuf",
+ "com.google.code.findbugs" % "jsr305" % findbugsVersion,
+ "com.google.guava" % "guava" % guavaVersion,
+ "commons-io" % "commons-io" % commonsIoVersion,
+ "io.dropwizard.metrics" % "metrics-core" % metricsVersion,
+ "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion,
+ "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion,
+ "io.netty" % "netty-all" % nettyVersion,
+ "org.apache.commons" % "commons-crypto" % commonsCryptoVersion,
+ "org.apache.commons" % "commons-lang3" % commonsLang3Version,
+ "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion,
+ "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion,
+ "org.apache.ratis" % "ratis-client" % ratisVersion,
+ "org.apache.ratis" % "ratis-common" % ratisVersion,
+ "org.fusesource.leveldbjni" % "leveldbjni-all" % leveldbJniVersion,
+ "org.roaringbitmap" % "RoaringBitmap" % roaringBitmapVersion,
+ "org.scala-lang" % "scala-reflect" % scalaVersion.value,
+ "org.slf4j" % "jcl-over-slf4j" % slf4jVersion,
+ "org.slf4j" % "jul-to-slf4j" % slf4jVersion,
+ "org.slf4j" % "slf4j-api" % slf4jVersion,
+ "org.yaml" % "snakeyaml" % snakeyamlVersion,
+ "org.apache.logging.log4j" % "log4j-slf4j-impl" % log4j2Version % "test",
+ "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version % "test",
+
+ // Compiler plugins
+ // -- Bump up the genjavadoc version explicitly to 0.18 to work with Scala 2.12
+ compilerPlugin(
+ "com.typesafe.genjavadoc" %% "genjavadoc-plugin" % "0.18" cross CrossVersion.full)
+ ) ++ commonUnitTestDependencies,
+
+ Compile / sourceGenerators += Def.task {
+ val file = (Compile / sourceManaged).value / "org" / "apache" / "celeborn" / "package.scala"
+ streams.value.log.info("geneate version information file %s".format(file.toPath))
+ IO.write(file,
+ s"""package org.apache
+ |
+ |package object celeborn {
+ | val VERSION = "${version.value}"
+ |}
+ |""".stripMargin)
+ Seq(file)
+ // generate version task depends on PB generate to avoid concurrency generate source files
+ // otherwise we may encounter the error:
+ // ```
+ // [error] IO error while decoding ./celeborn/common/target/scala-2.12/src_managed/main/org/apache/celeborn/package.scala with UTF-8: ./celeborn/common/target/scala-2.12/src_managed/main/org/apache/celeborn/package.scala (No such file or directory)
+ // ```
+ }.dependsOn(Compile / PB.generate),
+
+ // a task to show current profiles
+ printProfiles := {
+ val message = profiles.mkString("", " ", "")
+ println("compile with profiles: %s".format(message))
+ }
+ )
+
+ lazy val printProfiles = taskKey[Unit]("Prints Profiles")
+}
+
+object CelebornClient {
+ lazy val client = (project in file("client"))
+ // ref: https://www.scala-sbt.org/1.x/docs/Multi-Project.html#Classpath+dependencies
+ .dependsOn(CelebornCommon.common % "test->test;compile->compile")
+ .settings (
+ name := "client",
+ commonSettings,
+ libraryDependencies ++= Seq(
+ "io.netty" % "netty-all" % nettyVersion,
+ "com.google.guava" % "guava" % guavaVersion,
+ "org.lz4" % "lz4-java" % lz4JavaVersion,
+ "com.github.luben" % "zstd-jni" % zstdJniVersion,
+ "org.apache.commons" % "commons-lang3" % commonsLang3Version,
+ "org.apache.logging.log4j" % "log4j-slf4j-impl" % log4j2Version % "test",
+ "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version % "test",
+
+ // Compiler plugins
+ // -- Bump up the genjavadoc version explicitly to 0.18 to work with Scala 2.12
+ compilerPlugin(
+ "com.typesafe.genjavadoc" %% "genjavadoc-plugin" % "0.18" cross CrossVersion.full)
+ ) ++ commonUnitTestDependencies
+ )
+}
+
+object CelebornService {
+ lazy val service = (project in file("service"))
+ .dependsOn(CelebornCommon.common)
+ .settings (
+ name := "service",
+ commonSettings,
+ libraryDependencies ++= Seq(
+ "com.google.code.findbugs" % "jsr305" % findbugsVersion,
+ "commons-io" % "commons-io" % commonsIoVersion,
+ "io.netty" % "netty-all" % nettyVersion,
+ "javax.servlet" % "javax.servlet-api" % javaxServletVersion,
+ "org.apache.commons" % "commons-crypto" % commonsCryptoVersion,
+ "org.slf4j" % "slf4j-api" % slf4jVersion,
+ "org.apache.logging.log4j" % "log4j-slf4j-impl" % log4j2Version % "test",
+ "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version % "test",
+
+ // Compiler plugins
+ // -- Bump up the genjavadoc version explicitly to 0.18 to work with Scala 2.12
+ compilerPlugin(
+ "com.typesafe.genjavadoc" %% "genjavadoc-plugin" % "0.18" cross CrossVersion.full)
+ ) ++ commonUnitTestDependencies
+ )
+}
+
+object CelebornMaster {
+ lazy val master = (project in file("master"))
+ .dependsOn(CelebornCommon.common, CelebornService.service)
+ .settings (
+ name := "master",
+ commonSettings,
+ protoSettings,
+ libraryDependencies ++= Seq(
+ "com.google.guava" % "guava" % guavaVersion,
+ "com.google.protobuf" % "protobuf-java" % protoVersion,
+ "io.netty" % "netty-all" % nettyVersion,
+ "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion,
+ "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version,
+ "org.apache.logging.log4j" % "log4j-slf4j-impl" % log4j2Version,
+ "org.apache.ratis" % "ratis-client" % ratisVersion,
+ "org.apache.ratis" % "ratis-common" % ratisVersion,
+ "org.apache.ratis" % "ratis-grpc" % ratisVersion,
+ "org.apache.ratis" % "ratis-netty" % ratisVersion,
+ "org.apache.ratis" % "ratis-server" % ratisVersion,
+ "org.apache.ratis" % "ratis-shell" % ratisVersion,
+
+ // Compiler plugins
+ // -- Bump up the genjavadoc version explicitly to 0.18 to work with Scala 2.12
+ compilerPlugin(
+ "com.typesafe.genjavadoc" %% "genjavadoc-plugin" % "0.18" cross CrossVersion.full)
+ ) ++ commonUnitTestDependencies
+ )
+}
+
+object CelebornWorker {
+ lazy val worker = (project in file("worker"))
+ .dependsOn(CelebornCommon.common, CelebornService.service)
+ .dependsOn(CelebornClient.client % "test->test;compile->compile")
+ .dependsOn(CelebornMaster.master % "test->test;compile->compile")
+ .settings (
+ name := "worker",
+ commonSettings,
+ libraryDependencies ++= Seq(
+ "com.google.guava" % "guava" % guavaVersion,
+ "commons-io" % "commons-io" % commonsIoVersion,
+ "io.netty" % "netty-all" % nettyVersion,
+ "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version,
+ "org.apache.logging.log4j" % "log4j-slf4j-impl" % log4j2Version,
+ "org.fusesource.leveldbjni" % "leveldbjni-all" % leveldbJniVersion,
+ "org.roaringbitmap" % "RoaringBitmap" % roaringBitmapVersion,
+ "org.mockito" %% "mockito-scala-scalatest" % scalatestMockitoVersion % "test",
+
+ // Compiler plugins
+ // -- Bump up the genjavadoc version explicitly to 0.18 to work with Scala 2.12
+ compilerPlugin(
+ "com.typesafe.genjavadoc" %% "genjavadoc-plugin" % "0.18" cross CrossVersion.full)
+ ) ++ commonUnitTestDependencies
+ )
+}
+
+////////////////////////////////////////////////////////
+// Spark Client //
+////////////////////////////////////////////////////////
+
+object Spark24 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-2"
+ val sparkClientProjectName = "celeborn-client-spark-2"
+ val sparkClientShadeProjectPath = "client-spark/spark-2-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-2-shaded"
+
+ // val jacksonVersion = "2.5.7"
+ // val jacksonDatabindVersion = "2.6.7.3"
+ val lz4JavaVersion = "1.4.0"
+ val sparkProjectScalaVersion = "2.11.12"
+ // scalaBinaryVersion
+ // val scalaBinaryVersion = "2.11"
+ val sparkVersion = "2.4.8"
+ val zstdJniVersion = "1.4.4-3"
+}
+
+object Spark30 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-3"
+ val sparkClientProjectName = "celeborn-client-spark-3"
+ val sparkClientShadeProjectPath = "client-spark/spark-3-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-3-shaded"
+
+ val lz4JavaVersion = "1.7.1"
+ val sparkProjectScalaVersion = "2.12.10"
+
+ val sparkVersion = "3.0.3"
+ val zstdJniVersion = "1.4.4-3"
+}
+
+object Spark31 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-3"
+ val sparkClientProjectName = "celeborn-client-spark-3"
+ val sparkClientShadeProjectPath = "client-spark/spark-3-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-3-shaded"
+
+ val lz4JavaVersion = "1.7.1"
+ val sparkProjectScalaVersion = "2.12.10"
+
+ val sparkVersion = "3.1.3"
+ val zstdJniVersion = "1.4.8-1"
+}
+
+object Spark32 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-3"
+ val sparkClientProjectName = "celeborn-client-spark-3"
+ val sparkClientShadeProjectPath = "client-spark/spark-3-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-3-shaded"
+
+ val lz4JavaVersion = "1.7.1"
+ val sparkProjectScalaVersion = "2.12.15"
+
+ val sparkVersion = "3.2.4"
+ val zstdJniVersion = "1.5.0-4"
+}
+
+object Spark33 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-3"
+ val sparkClientProjectName = "celeborn-client-spark-3"
+ val sparkClientShadeProjectPath = "client-spark/spark-3-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-3-shaded"
+
+ // val jacksonVersion = "2.13.4"
+ // val jacksonDatabindVersion = "2.13.4.2"
+ val lz4JavaVersion = "1.8.0"
+ val sparkProjectScalaVersion = "2.12.15"
+ // scalaBinaryVersion
+ // val scalaBinaryVersion = "2.12"
+ val sparkVersion = "3.3.2"
+ val zstdJniVersion = "1.5.2-1"
+}
+
+object Spark34 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-3"
+ val sparkClientProjectName = "celeborn-client-spark-3"
+ val sparkClientShadeProjectPath = "client-spark/spark-3-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-3-shaded"
+
+ val lz4JavaVersion = "1.8.0"
+ val sparkProjectScalaVersion = "2.12.17"
+
+ val sparkVersion = "3.4.1"
+ val zstdJniVersion = "1.5.2-5"
+
+ lazy val deps = Seq(
+ // Spark Use `log4j-slf4j2-impl` instead of `log4j-slf4j-impl` in SPARK-40511
+ // to fix the error:
+ // ```
+ // java.lang.NoSuchMethodError: org.apache.logging.slf4j.Log4jLoggerFactory.<init>(Lorg/apache/logging/slf4j/Log4jMarkerFactory;)V
+ // ```
+ "org.apache.logging.log4j" % "log4j-slf4j2-impl" % "2.19.0" % "test"
Review Comment:
does exclude `log4j-slf4j2-impl` and use `log4j-slf4j-impl` work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1274618227
##########
project/CelebornBuild.scala:
##########
@@ -0,0 +1,604 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.nio.file.Files
+import java.nio.charset.StandardCharsets.UTF_8
+import java.util.Locale
+
+import scala.util.Properties
+
+import sbtassembly.AssemblyPlugin.autoImport._
+import sbtprotoc.ProtocPlugin.autoImport._
+
+import sbt._
+import sbt.Keys._
+import Utils._
+import CelebornCommonSettings._
+// import sbt.Keys.streams
+
+object CelebornCommonSettings {
+
+ // Scala versions
+ val SCALA_2_11_12 = "2.11.12"
+ val SCALA_2_12_10 = "2.12.10"
+ val SCALA_2_12_15 = "2.12.15"
+ val SCALA_2_12_17 = "2.12.17"
+ val scala213 = "2.13.5"
+ val ALL_SCALA_VERSIONS = Seq(SCALA_2_11_12, SCALA_2_12_10, SCALA_2_12_15, SCALA_2_12_17, scala213)
+
+ val DEFAULT_SCALA_VERSION = SCALA_2_12_15
+
+ val projectScalaVersion = defaultScalaVersion()
+
+ val zstdJniVersion = sparkClientProjects.map(_.zstdJniVersion).getOrElse("1.5.2-1")
+ val lz4JavaVersion = sparkClientProjects.map(_.lz4JavaVersion).getOrElse("1.8.0")
+
+ // Dependent library versions
+ val commonsCryptoVersion = "1.0.0"
+ val commonsIoVersion = "2.13.0"
+ val commonsLang3Version = "3.12.0"
+ val findbugsVersion = "1.3.9"
+ val guavaVersion = "14.0.1"
+ val hadoopVersion = "3.2.4"
+ val javaxServletVersion = "3.1.0"
+ val leveldbJniVersion = "1.8"
+ val log4j2Version = "2.17.2"
+ val metricsVersion = "3.2.6"
+ val scalatestMockitoVersion = "1.17.14"
+ val nettyVersion = "4.1.93.Final"
+ val ratisVersion = "2.5.1"
+ val roaringBitmapVersion = "0.9.32"
+ val slf4jVersion = "1.7.36"
+ val snakeyamlVersion = "1.33"
+
+ // Versions for proto
+ val protocVersion = "3.19.2"
+ val protoVersion = "3.19.2"
+
+ scalaVersion := projectScalaVersion
+
+ autoScalaLibrary := false
+
+ // crossScalaVersions must be set to Nil on the root project
+ crossScalaVersions := Nil
+
+ lazy val commonSettings = Seq(
+ organization := "org.apache.celeborn",
+ scalaVersion := projectScalaVersion,
+ crossScalaVersions := ALL_SCALA_VERSIONS,
+ fork := true,
+ scalacOptions ++= Seq("-target:jvm-1.8"),
+ javacOptions ++= Seq("-encoding", UTF_8.name(), "-source", "1.8"),
+
+ // -target cannot be passed as a parameter to javadoc. See https://github.com/sbt/sbt/issues/355
+ Compile / compile / javacOptions ++= Seq("-target", "1.8"),
+
+ // Make sure any tests in any project that uses Spark is configured for running well locally
+ Test / javaOptions ++= Seq(
+ "-Xmx4g"
+ ),
+
+ testOptions += Tests.Argument("-oF"),
+
+ Test / testOptions += Tests.Argument("-oDF"),
+ Test / testOptions += Tests.Argument(TestFrameworks.JUnit, "-v", "-a"),
+
+ // Don't execute in parallel since we can't have multiple Sparks in the same JVM
+ Test / parallelExecution := false,
+
+ scalacOptions ++= Seq(
+ "-P:genjavadoc:strictVisibility=true" // hide package private types and methods in javadoc
+ ),
+
+ javaOptions += "-Xmx4g",
+
+ // Configurations to speed up tests and reduce memory footprint
+ Test / javaOptions ++= Seq(
+ "-Xmx4g"
+ ),
+
+ Test / envVars += ("IS_TESTING", "1")
+ )
+
+ lazy val protoSettings = Seq(
+ // Setting version for the protobuf compiler
+ PB.protocVersion := protocVersion,
+ // set proto sources path
+ Compile / PB.protoSources := Seq(sourceDirectory.value / "main" / "proto"),
+ Compile / PB.targets := Seq(PB.gens.java -> (Compile / sourceManaged).value)
+ )
+
+ lazy val commonUnitTestDependencies = Seq(
+ "org.mockito" % "mockito-core" % "4.11.0" % "test",
+ "org.scalatest" %% "scalatest" % "3.2.16" % "test",
+ "junit" % "junit" % "4.12" % "test",
+ // https://www.scala-sbt.org/1.x/docs/Testing.html
+ "com.github.sbt" % "junit-interface" % "0.13.3" % "test")
+}
+
+object CelebornBuild extends sbt.internal.BuildDef {
+ override def projectDefinitions(baseDirectory: File): Seq[Project] = {
+ Seq(
+ CelebornCommon.common,
+ CelebornClient.client,
+ CelebornService.service,
+ CelebornWorker.worker,
+ CelebornMaster.master) ++ maybeSparkClientModules
+ }
+
+ // ThisBuild / parallelExecution := false
+
+ // scalaVersion := "2.11.12"
+
+ // autoScalaLibrary := false
+
+ crossScalaVersions := Nil
+
+ // load user-defined Profiles
+ // loadProfiles()
+}
+
+object Utils {
+ val profiles = {
+ val profiles = Properties.envOrNone("SBT_MAVEN_PROFILES")
+ .orElse(Properties.propOrNone("sbt.maven.profiles")) match {
+ case None => Seq("sbt")
+ case Some(v) =>
+ v.split("(\\s+|,)").filterNot(_.isEmpty).map(_.trim.replaceAll("-P", "")).toSeq
+ }
+ if (profiles.contains("jdwp-test-debug")) {
+ sys.props.put("test.jdwp.enabled", "true")
+ }
+ profiles
+ }
+
+ val SPARK_VERSION = profiles.filter(_.startsWith("spark")).headOption
+
+ lazy val sparkClientProjects = SPARK_VERSION match {
+ case Some("spark-2.4") => Some(Spark24)
+ case Some("spark-3.0") => Some(Spark30)
+ case Some("spark-3.1") => Some(Spark31)
+ case Some("spark-3.2") => Some(Spark32)
+ case Some("spark-3.3") => Some(Spark33)
+ case Some("spark-3.4") => Some(Spark34)
+ case _ => None
+ }
+
+ lazy val maybeSparkClientModules: Seq[Project] = sparkClientProjects.map(_.modules).getOrElse(Seq.empty)
+
+ def defaultScalaVersion(): String = {
+ // 1. Inherit the scala version of the spark project
+ // 2. if the spark profile not specified, using the DEFAULT_SCALA_VERSION
+ val v = sparkClientProjects.map(_.sparkProjectScalaVersion).getOrElse(DEFAULT_SCALA_VERSION)
+ require(ALL_SCALA_VERSIONS.contains(v), s"found not allow scala version: $v")
+ v
+ }
+}
+
+object CelebornCommon {
+ lazy val common = (project in file("common"))
+ .settings (
+ name := "celeborn-common",
+ commonSettings,
+ protoSettings,
+ libraryDependencies ++= Seq(
+ "com.google.protobuf" % "protobuf-java" % protoVersion % "protobuf",
+ "com.google.code.findbugs" % "jsr305" % findbugsVersion,
+ "com.google.guava" % "guava" % guavaVersion,
+ "commons-io" % "commons-io" % commonsIoVersion,
+ "io.dropwizard.metrics" % "metrics-core" % metricsVersion,
+ "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion,
+ "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion,
+ "io.netty" % "netty-all" % nettyVersion,
+ "org.apache.commons" % "commons-crypto" % commonsCryptoVersion,
+ "org.apache.commons" % "commons-lang3" % commonsLang3Version,
+ "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion,
+ "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion,
+ "org.apache.ratis" % "ratis-client" % ratisVersion,
+ "org.apache.ratis" % "ratis-common" % ratisVersion,
+ "org.fusesource.leveldbjni" % "leveldbjni-all" % leveldbJniVersion,
+ "org.roaringbitmap" % "RoaringBitmap" % roaringBitmapVersion,
+ "org.scala-lang" % "scala-reflect" % scalaVersion.value,
+ "org.slf4j" % "jcl-over-slf4j" % slf4jVersion,
+ "org.slf4j" % "jul-to-slf4j" % slf4jVersion,
+ "org.slf4j" % "slf4j-api" % slf4jVersion,
+ "org.yaml" % "snakeyaml" % snakeyamlVersion,
+ "org.apache.logging.log4j" % "log4j-slf4j-impl" % log4j2Version % "test",
+ "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version % "test",
+
+ // Compiler plugins
+ // -- Bump up the genjavadoc version explicitly to 0.18 to work with Scala 2.12
+ compilerPlugin(
+ "com.typesafe.genjavadoc" %% "genjavadoc-plugin" % "0.18" cross CrossVersion.full)
+ ) ++ commonUnitTestDependencies,
+
+ Compile / sourceGenerators += Def.task {
+ val file = (Compile / sourceManaged).value / "org" / "apache" / "celeborn" / "package.scala"
+ streams.value.log.info("geneate version information file %s".format(file.toPath))
+ IO.write(file,
+ s"""package org.apache
+ |
+ |package object celeborn {
+ | val VERSION = "${version.value}"
+ |}
+ |""".stripMargin)
+ Seq(file)
+ // generate version task depends on PB generate to avoid concurrency generate source files
+ // otherwise we may encounter the error:
+ // ```
+ // [error] IO error while decoding ./celeborn/common/target/scala-2.12/src_managed/main/org/apache/celeborn/package.scala with UTF-8: ./celeborn/common/target/scala-2.12/src_managed/main/org/apache/celeborn/package.scala (No such file or directory)
+ // ```
+ }.dependsOn(Compile / PB.generate),
+
+ // a task to show current profiles
+ printProfiles := {
+ val message = profiles.mkString("", " ", "")
+ println("compile with profiles: %s".format(message))
+ }
+ )
+
+ lazy val printProfiles = taskKey[Unit]("Prints Profiles")
+}
+
+object CelebornClient {
+ lazy val client = (project in file("client"))
+ // ref: https://www.scala-sbt.org/1.x/docs/Multi-Project.html#Classpath+dependencies
+ .dependsOn(CelebornCommon.common % "test->test;compile->compile")
+ .settings (
+ name := "client",
+ commonSettings,
+ libraryDependencies ++= Seq(
+ "io.netty" % "netty-all" % nettyVersion,
+ "com.google.guava" % "guava" % guavaVersion,
+ "org.lz4" % "lz4-java" % lz4JavaVersion,
+ "com.github.luben" % "zstd-jni" % zstdJniVersion,
+ "org.apache.commons" % "commons-lang3" % commonsLang3Version,
+ "org.apache.logging.log4j" % "log4j-slf4j-impl" % log4j2Version % "test",
+ "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version % "test",
+
+ // Compiler plugins
+ // -- Bump up the genjavadoc version explicitly to 0.18 to work with Scala 2.12
+ compilerPlugin(
+ "com.typesafe.genjavadoc" %% "genjavadoc-plugin" % "0.18" cross CrossVersion.full)
+ ) ++ commonUnitTestDependencies
+ )
+}
+
+object CelebornService {
+ lazy val service = (project in file("service"))
+ .dependsOn(CelebornCommon.common)
+ .settings (
+ name := "service",
+ commonSettings,
+ libraryDependencies ++= Seq(
+ "com.google.code.findbugs" % "jsr305" % findbugsVersion,
+ "commons-io" % "commons-io" % commonsIoVersion,
+ "io.netty" % "netty-all" % nettyVersion,
+ "javax.servlet" % "javax.servlet-api" % javaxServletVersion,
+ "org.apache.commons" % "commons-crypto" % commonsCryptoVersion,
+ "org.slf4j" % "slf4j-api" % slf4jVersion,
+ "org.apache.logging.log4j" % "log4j-slf4j-impl" % log4j2Version % "test",
+ "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version % "test",
+
+ // Compiler plugins
+ // -- Bump up the genjavadoc version explicitly to 0.18 to work with Scala 2.12
+ compilerPlugin(
+ "com.typesafe.genjavadoc" %% "genjavadoc-plugin" % "0.18" cross CrossVersion.full)
+ ) ++ commonUnitTestDependencies
+ )
+}
+
+object CelebornMaster {
+ lazy val master = (project in file("master"))
+ .dependsOn(CelebornCommon.common, CelebornService.service)
+ .settings (
+ name := "master",
+ commonSettings,
+ protoSettings,
+ libraryDependencies ++= Seq(
+ "com.google.guava" % "guava" % guavaVersion,
+ "com.google.protobuf" % "protobuf-java" % protoVersion,
+ "io.netty" % "netty-all" % nettyVersion,
+ "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion,
+ "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version,
+ "org.apache.logging.log4j" % "log4j-slf4j-impl" % log4j2Version,
+ "org.apache.ratis" % "ratis-client" % ratisVersion,
+ "org.apache.ratis" % "ratis-common" % ratisVersion,
+ "org.apache.ratis" % "ratis-grpc" % ratisVersion,
+ "org.apache.ratis" % "ratis-netty" % ratisVersion,
+ "org.apache.ratis" % "ratis-server" % ratisVersion,
+ "org.apache.ratis" % "ratis-shell" % ratisVersion,
+
+ // Compiler plugins
+ // -- Bump up the genjavadoc version explicitly to 0.18 to work with Scala 2.12
+ compilerPlugin(
+ "com.typesafe.genjavadoc" %% "genjavadoc-plugin" % "0.18" cross CrossVersion.full)
+ ) ++ commonUnitTestDependencies
+ )
+}
+
+object CelebornWorker {
+ lazy val worker = (project in file("worker"))
+ .dependsOn(CelebornCommon.common, CelebornService.service)
+ .dependsOn(CelebornClient.client % "test->test;compile->compile")
+ .dependsOn(CelebornMaster.master % "test->test;compile->compile")
+ .settings (
+ name := "worker",
+ commonSettings,
+ libraryDependencies ++= Seq(
+ "com.google.guava" % "guava" % guavaVersion,
+ "commons-io" % "commons-io" % commonsIoVersion,
+ "io.netty" % "netty-all" % nettyVersion,
+ "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version,
+ "org.apache.logging.log4j" % "log4j-slf4j-impl" % log4j2Version,
+ "org.fusesource.leveldbjni" % "leveldbjni-all" % leveldbJniVersion,
+ "org.roaringbitmap" % "RoaringBitmap" % roaringBitmapVersion,
+ "org.mockito" %% "mockito-scala-scalatest" % scalatestMockitoVersion % "test",
+
+ // Compiler plugins
+ // -- Bump up the genjavadoc version explicitly to 0.18 to work with Scala 2.12
+ compilerPlugin(
+ "com.typesafe.genjavadoc" %% "genjavadoc-plugin" % "0.18" cross CrossVersion.full)
+ ) ++ commonUnitTestDependencies
+ )
+}
+
+////////////////////////////////////////////////////////
+// Spark Client //
+////////////////////////////////////////////////////////
+
+object Spark24 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-2"
+ val sparkClientProjectName = "celeborn-client-spark-2"
+ val sparkClientShadeProjectPath = "client-spark/spark-2-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-2-shaded"
+
+ // val jacksonVersion = "2.5.7"
+ // val jacksonDatabindVersion = "2.6.7.3"
+ val lz4JavaVersion = "1.4.0"
+ val sparkProjectScalaVersion = "2.11.12"
+ // scalaBinaryVersion
+ // val scalaBinaryVersion = "2.11"
+ val sparkVersion = "2.4.8"
+ val zstdJniVersion = "1.4.4-3"
+}
+
+object Spark30 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-3"
+ val sparkClientProjectName = "celeborn-client-spark-3"
+ val sparkClientShadeProjectPath = "client-spark/spark-3-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-3-shaded"
+
+ val lz4JavaVersion = "1.7.1"
+ val sparkProjectScalaVersion = "2.12.10"
+
+ val sparkVersion = "3.0.3"
+ val zstdJniVersion = "1.4.4-3"
+}
+
+object Spark31 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-3"
+ val sparkClientProjectName = "celeborn-client-spark-3"
+ val sparkClientShadeProjectPath = "client-spark/spark-3-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-3-shaded"
+
+ val lz4JavaVersion = "1.7.1"
+ val sparkProjectScalaVersion = "2.12.10"
+
+ val sparkVersion = "3.1.3"
+ val zstdJniVersion = "1.4.8-1"
+}
+
+object Spark32 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-3"
+ val sparkClientProjectName = "celeborn-client-spark-3"
+ val sparkClientShadeProjectPath = "client-spark/spark-3-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-3-shaded"
+
+ val lz4JavaVersion = "1.7.1"
+ val sparkProjectScalaVersion = "2.12.15"
+
+ val sparkVersion = "3.2.4"
+ val zstdJniVersion = "1.5.0-4"
+}
+
+object Spark33 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-3"
+ val sparkClientProjectName = "celeborn-client-spark-3"
+ val sparkClientShadeProjectPath = "client-spark/spark-3-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-3-shaded"
+
+ // val jacksonVersion = "2.13.4"
+ // val jacksonDatabindVersion = "2.13.4.2"
+ val lz4JavaVersion = "1.8.0"
+ val sparkProjectScalaVersion = "2.12.15"
+ // scalaBinaryVersion
+ // val scalaBinaryVersion = "2.12"
+ val sparkVersion = "3.3.2"
+ val zstdJniVersion = "1.5.2-1"
+}
+
+object Spark34 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-3"
+ val sparkClientProjectName = "celeborn-client-spark-3"
+ val sparkClientShadeProjectPath = "client-spark/spark-3-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-3-shaded"
+
+ val lz4JavaVersion = "1.8.0"
+ val sparkProjectScalaVersion = "2.12.17"
+
+ val sparkVersion = "3.4.1"
+ val zstdJniVersion = "1.5.2-5"
+
+ lazy val deps = Seq(
+ // Spark Use `log4j-slf4j2-impl` instead of `log4j-slf4j-impl` in SPARK-40511
+ // to fix the error:
+ // ```
+ // java.lang.NoSuchMethodError: org.apache.logging.slf4j.Log4jLoggerFactory.<init>(Lorg/apache/logging/slf4j/Log4jMarkerFactory;)V
+ // ```
+ "org.apache.logging.log4j" % "log4j-slf4j2-impl" % "2.19.0" % "test"
Review Comment:
if the error message is same, I think it may caused by
1. in Maven, slf4j-api 1.x takes effect
2. in SBT, slf4j-api 2.x takes effect
it may be caused by difference dependencies resolution strategy or configuration
##########
project/CelebornBuild.scala:
##########
@@ -0,0 +1,604 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.nio.file.Files
+import java.nio.charset.StandardCharsets.UTF_8
+import java.util.Locale
+
+import scala.util.Properties
+
+import sbtassembly.AssemblyPlugin.autoImport._
+import sbtprotoc.ProtocPlugin.autoImport._
+
+import sbt._
+import sbt.Keys._
+import Utils._
+import CelebornCommonSettings._
+// import sbt.Keys.streams
+
+object CelebornCommonSettings {
+
+ // Scala versions
+ val SCALA_2_11_12 = "2.11.12"
+ val SCALA_2_12_10 = "2.12.10"
+ val SCALA_2_12_15 = "2.12.15"
+ val SCALA_2_12_17 = "2.12.17"
+ val scala213 = "2.13.5"
+ val ALL_SCALA_VERSIONS = Seq(SCALA_2_11_12, SCALA_2_12_10, SCALA_2_12_15, SCALA_2_12_17, scala213)
+
+ val DEFAULT_SCALA_VERSION = SCALA_2_12_15
+
+ val projectScalaVersion = defaultScalaVersion()
+
+ val zstdJniVersion = sparkClientProjects.map(_.zstdJniVersion).getOrElse("1.5.2-1")
+ val lz4JavaVersion = sparkClientProjects.map(_.lz4JavaVersion).getOrElse("1.8.0")
+
+ // Dependent library versions
+ val commonsCryptoVersion = "1.0.0"
+ val commonsIoVersion = "2.13.0"
+ val commonsLang3Version = "3.12.0"
+ val findbugsVersion = "1.3.9"
+ val guavaVersion = "14.0.1"
+ val hadoopVersion = "3.2.4"
+ val javaxServletVersion = "3.1.0"
+ val leveldbJniVersion = "1.8"
+ val log4j2Version = "2.17.2"
+ val metricsVersion = "3.2.6"
+ val scalatestMockitoVersion = "1.17.14"
+ val nettyVersion = "4.1.93.Final"
+ val ratisVersion = "2.5.1"
+ val roaringBitmapVersion = "0.9.32"
+ val slf4jVersion = "1.7.36"
+ val snakeyamlVersion = "1.33"
+
+ // Versions for proto
+ val protocVersion = "3.19.2"
+ val protoVersion = "3.19.2"
+
+ scalaVersion := projectScalaVersion
+
+ autoScalaLibrary := false
+
+ // crossScalaVersions must be set to Nil on the root project
+ crossScalaVersions := Nil
+
+ lazy val commonSettings = Seq(
+ organization := "org.apache.celeborn",
+ scalaVersion := projectScalaVersion,
+ crossScalaVersions := ALL_SCALA_VERSIONS,
+ fork := true,
+ scalacOptions ++= Seq("-target:jvm-1.8"),
+ javacOptions ++= Seq("-encoding", UTF_8.name(), "-source", "1.8"),
+
+ // -target cannot be passed as a parameter to javadoc. See https://github.com/sbt/sbt/issues/355
+ Compile / compile / javacOptions ++= Seq("-target", "1.8"),
+
+ // Make sure any tests in any project that uses Spark is configured for running well locally
+ Test / javaOptions ++= Seq(
+ "-Xmx4g"
+ ),
+
+ testOptions += Tests.Argument("-oF"),
+
+ Test / testOptions += Tests.Argument("-oDF"),
+ Test / testOptions += Tests.Argument(TestFrameworks.JUnit, "-v", "-a"),
+
+ // Don't execute in parallel since we can't have multiple Sparks in the same JVM
+ Test / parallelExecution := false,
+
+ scalacOptions ++= Seq(
+ "-P:genjavadoc:strictVisibility=true" // hide package private types and methods in javadoc
+ ),
+
+ javaOptions += "-Xmx4g",
+
+ // Configurations to speed up tests and reduce memory footprint
+ Test / javaOptions ++= Seq(
+ "-Xmx4g"
+ ),
+
+ Test / envVars += ("IS_TESTING", "1")
+ )
+
+ lazy val protoSettings = Seq(
+ // Setting version for the protobuf compiler
+ PB.protocVersion := protocVersion,
+ // set proto sources path
+ Compile / PB.protoSources := Seq(sourceDirectory.value / "main" / "proto"),
+ Compile / PB.targets := Seq(PB.gens.java -> (Compile / sourceManaged).value)
+ )
+
+ lazy val commonUnitTestDependencies = Seq(
+ "org.mockito" % "mockito-core" % "4.11.0" % "test",
+ "org.scalatest" %% "scalatest" % "3.2.16" % "test",
+ "junit" % "junit" % "4.12" % "test",
+ // https://www.scala-sbt.org/1.x/docs/Testing.html
+ "com.github.sbt" % "junit-interface" % "0.13.3" % "test")
+}
+
+object CelebornBuild extends sbt.internal.BuildDef {
+ override def projectDefinitions(baseDirectory: File): Seq[Project] = {
+ Seq(
+ CelebornCommon.common,
+ CelebornClient.client,
+ CelebornService.service,
+ CelebornWorker.worker,
+ CelebornMaster.master) ++ maybeSparkClientModules
+ }
+
+ // ThisBuild / parallelExecution := false
+
+ // scalaVersion := "2.11.12"
+
+ // autoScalaLibrary := false
+
+ crossScalaVersions := Nil
+
+ // load user-defined Profiles
+ // loadProfiles()
+}
+
+object Utils {
+ val profiles = {
+ val profiles = Properties.envOrNone("SBT_MAVEN_PROFILES")
+ .orElse(Properties.propOrNone("sbt.maven.profiles")) match {
+ case None => Seq("sbt")
+ case Some(v) =>
+ v.split("(\\s+|,)").filterNot(_.isEmpty).map(_.trim.replaceAll("-P", "")).toSeq
+ }
+ if (profiles.contains("jdwp-test-debug")) {
+ sys.props.put("test.jdwp.enabled", "true")
+ }
+ profiles
+ }
+
+ val SPARK_VERSION = profiles.filter(_.startsWith("spark")).headOption
+
+ lazy val sparkClientProjects = SPARK_VERSION match {
+ case Some("spark-2.4") => Some(Spark24)
+ case Some("spark-3.0") => Some(Spark30)
+ case Some("spark-3.1") => Some(Spark31)
+ case Some("spark-3.2") => Some(Spark32)
+ case Some("spark-3.3") => Some(Spark33)
+ case Some("spark-3.4") => Some(Spark34)
+ case _ => None
+ }
+
+ lazy val maybeSparkClientModules: Seq[Project] = sparkClientProjects.map(_.modules).getOrElse(Seq.empty)
+
+ def defaultScalaVersion(): String = {
+ // 1. Inherit the scala version of the spark project
+ // 2. if the spark profile not specified, using the DEFAULT_SCALA_VERSION
+ val v = sparkClientProjects.map(_.sparkProjectScalaVersion).getOrElse(DEFAULT_SCALA_VERSION)
+ require(ALL_SCALA_VERSIONS.contains(v), s"found not allow scala version: $v")
+ v
+ }
+}
+
+object CelebornCommon {
+ lazy val common = (project in file("common"))
+ .settings (
+ name := "celeborn-common",
+ commonSettings,
+ protoSettings,
+ libraryDependencies ++= Seq(
+ "com.google.protobuf" % "protobuf-java" % protoVersion % "protobuf",
+ "com.google.code.findbugs" % "jsr305" % findbugsVersion,
+ "com.google.guava" % "guava" % guavaVersion,
+ "commons-io" % "commons-io" % commonsIoVersion,
+ "io.dropwizard.metrics" % "metrics-core" % metricsVersion,
+ "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion,
+ "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion,
+ "io.netty" % "netty-all" % nettyVersion,
+ "org.apache.commons" % "commons-crypto" % commonsCryptoVersion,
+ "org.apache.commons" % "commons-lang3" % commonsLang3Version,
+ "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion,
+ "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion,
+ "org.apache.ratis" % "ratis-client" % ratisVersion,
+ "org.apache.ratis" % "ratis-common" % ratisVersion,
+ "org.fusesource.leveldbjni" % "leveldbjni-all" % leveldbJniVersion,
+ "org.roaringbitmap" % "RoaringBitmap" % roaringBitmapVersion,
+ "org.scala-lang" % "scala-reflect" % scalaVersion.value,
+ "org.slf4j" % "jcl-over-slf4j" % slf4jVersion,
+ "org.slf4j" % "jul-to-slf4j" % slf4jVersion,
+ "org.slf4j" % "slf4j-api" % slf4jVersion,
+ "org.yaml" % "snakeyaml" % snakeyamlVersion,
+ "org.apache.logging.log4j" % "log4j-slf4j-impl" % log4j2Version % "test",
+ "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version % "test",
+
+ // Compiler plugins
+ // -- Bump up the genjavadoc version explicitly to 0.18 to work with Scala 2.12
+ compilerPlugin(
+ "com.typesafe.genjavadoc" %% "genjavadoc-plugin" % "0.18" cross CrossVersion.full)
+ ) ++ commonUnitTestDependencies,
+
+ Compile / sourceGenerators += Def.task {
+ val file = (Compile / sourceManaged).value / "org" / "apache" / "celeborn" / "package.scala"
+ streams.value.log.info("geneate version information file %s".format(file.toPath))
+ IO.write(file,
+ s"""package org.apache
+ |
+ |package object celeborn {
+ | val VERSION = "${version.value}"
+ |}
+ |""".stripMargin)
+ Seq(file)
+ // generate version task depends on PB generate to avoid concurrency generate source files
+ // otherwise we may encounter the error:
+ // ```
+ // [error] IO error while decoding ./celeborn/common/target/scala-2.12/src_managed/main/org/apache/celeborn/package.scala with UTF-8: ./celeborn/common/target/scala-2.12/src_managed/main/org/apache/celeborn/package.scala (No such file or directory)
+ // ```
+ }.dependsOn(Compile / PB.generate),
+
+ // a task to show current profiles
+ printProfiles := {
+ val message = profiles.mkString("", " ", "")
+ println("compile with profiles: %s".format(message))
+ }
+ )
+
+ lazy val printProfiles = taskKey[Unit]("Prints Profiles")
+}
+
+object CelebornClient {
+ lazy val client = (project in file("client"))
+ // ref: https://www.scala-sbt.org/1.x/docs/Multi-Project.html#Classpath+dependencies
+ .dependsOn(CelebornCommon.common % "test->test;compile->compile")
+ .settings (
+ name := "client",
+ commonSettings,
+ libraryDependencies ++= Seq(
+ "io.netty" % "netty-all" % nettyVersion,
+ "com.google.guava" % "guava" % guavaVersion,
+ "org.lz4" % "lz4-java" % lz4JavaVersion,
+ "com.github.luben" % "zstd-jni" % zstdJniVersion,
+ "org.apache.commons" % "commons-lang3" % commonsLang3Version,
+ "org.apache.logging.log4j" % "log4j-slf4j-impl" % log4j2Version % "test",
+ "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version % "test",
+
+ // Compiler plugins
+ // -- Bump up the genjavadoc version explicitly to 0.18 to work with Scala 2.12
+ compilerPlugin(
+ "com.typesafe.genjavadoc" %% "genjavadoc-plugin" % "0.18" cross CrossVersion.full)
+ ) ++ commonUnitTestDependencies
+ )
+}
+
+object CelebornService {
+ lazy val service = (project in file("service"))
+ .dependsOn(CelebornCommon.common)
+ .settings (
+ name := "service",
+ commonSettings,
+ libraryDependencies ++= Seq(
+ "com.google.code.findbugs" % "jsr305" % findbugsVersion,
+ "commons-io" % "commons-io" % commonsIoVersion,
+ "io.netty" % "netty-all" % nettyVersion,
+ "javax.servlet" % "javax.servlet-api" % javaxServletVersion,
+ "org.apache.commons" % "commons-crypto" % commonsCryptoVersion,
+ "org.slf4j" % "slf4j-api" % slf4jVersion,
+ "org.apache.logging.log4j" % "log4j-slf4j-impl" % log4j2Version % "test",
+ "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version % "test",
+
+ // Compiler plugins
+ // -- Bump up the genjavadoc version explicitly to 0.18 to work with Scala 2.12
+ compilerPlugin(
+ "com.typesafe.genjavadoc" %% "genjavadoc-plugin" % "0.18" cross CrossVersion.full)
+ ) ++ commonUnitTestDependencies
+ )
+}
+
+object CelebornMaster {
+ lazy val master = (project in file("master"))
+ .dependsOn(CelebornCommon.common, CelebornService.service)
+ .settings (
+ name := "master",
+ commonSettings,
+ protoSettings,
+ libraryDependencies ++= Seq(
+ "com.google.guava" % "guava" % guavaVersion,
+ "com.google.protobuf" % "protobuf-java" % protoVersion,
+ "io.netty" % "netty-all" % nettyVersion,
+ "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion,
+ "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version,
+ "org.apache.logging.log4j" % "log4j-slf4j-impl" % log4j2Version,
+ "org.apache.ratis" % "ratis-client" % ratisVersion,
+ "org.apache.ratis" % "ratis-common" % ratisVersion,
+ "org.apache.ratis" % "ratis-grpc" % ratisVersion,
+ "org.apache.ratis" % "ratis-netty" % ratisVersion,
+ "org.apache.ratis" % "ratis-server" % ratisVersion,
+ "org.apache.ratis" % "ratis-shell" % ratisVersion,
+
+ // Compiler plugins
+ // -- Bump up the genjavadoc version explicitly to 0.18 to work with Scala 2.12
+ compilerPlugin(
+ "com.typesafe.genjavadoc" %% "genjavadoc-plugin" % "0.18" cross CrossVersion.full)
+ ) ++ commonUnitTestDependencies
+ )
+}
+
+object CelebornWorker {
+ lazy val worker = (project in file("worker"))
+ .dependsOn(CelebornCommon.common, CelebornService.service)
+ .dependsOn(CelebornClient.client % "test->test;compile->compile")
+ .dependsOn(CelebornMaster.master % "test->test;compile->compile")
+ .settings (
+ name := "worker",
+ commonSettings,
+ libraryDependencies ++= Seq(
+ "com.google.guava" % "guava" % guavaVersion,
+ "commons-io" % "commons-io" % commonsIoVersion,
+ "io.netty" % "netty-all" % nettyVersion,
+ "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version,
+ "org.apache.logging.log4j" % "log4j-slf4j-impl" % log4j2Version,
+ "org.fusesource.leveldbjni" % "leveldbjni-all" % leveldbJniVersion,
+ "org.roaringbitmap" % "RoaringBitmap" % roaringBitmapVersion,
+ "org.mockito" %% "mockito-scala-scalatest" % scalatestMockitoVersion % "test",
+
+ // Compiler plugins
+ // -- Bump up the genjavadoc version explicitly to 0.18 to work with Scala 2.12
+ compilerPlugin(
+ "com.typesafe.genjavadoc" %% "genjavadoc-plugin" % "0.18" cross CrossVersion.full)
+ ) ++ commonUnitTestDependencies
+ )
+}
+
+////////////////////////////////////////////////////////
+// Spark Client //
+////////////////////////////////////////////////////////
+
+object Spark24 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-2"
+ val sparkClientProjectName = "celeborn-client-spark-2"
+ val sparkClientShadeProjectPath = "client-spark/spark-2-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-2-shaded"
+
+ // val jacksonVersion = "2.5.7"
+ // val jacksonDatabindVersion = "2.6.7.3"
+ val lz4JavaVersion = "1.4.0"
+ val sparkProjectScalaVersion = "2.11.12"
+ // scalaBinaryVersion
+ // val scalaBinaryVersion = "2.11"
+ val sparkVersion = "2.4.8"
+ val zstdJniVersion = "1.4.4-3"
+}
+
+object Spark30 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-3"
+ val sparkClientProjectName = "celeborn-client-spark-3"
+ val sparkClientShadeProjectPath = "client-spark/spark-3-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-3-shaded"
+
+ val lz4JavaVersion = "1.7.1"
+ val sparkProjectScalaVersion = "2.12.10"
+
+ val sparkVersion = "3.0.3"
+ val zstdJniVersion = "1.4.4-3"
+}
+
+object Spark31 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-3"
+ val sparkClientProjectName = "celeborn-client-spark-3"
+ val sparkClientShadeProjectPath = "client-spark/spark-3-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-3-shaded"
+
+ val lz4JavaVersion = "1.7.1"
+ val sparkProjectScalaVersion = "2.12.10"
+
+ val sparkVersion = "3.1.3"
+ val zstdJniVersion = "1.4.8-1"
+}
+
+object Spark32 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-3"
+ val sparkClientProjectName = "celeborn-client-spark-3"
+ val sparkClientShadeProjectPath = "client-spark/spark-3-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-3-shaded"
+
+ val lz4JavaVersion = "1.7.1"
+ val sparkProjectScalaVersion = "2.12.15"
+
+ val sparkVersion = "3.2.4"
+ val zstdJniVersion = "1.5.0-4"
+}
+
+object Spark33 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-3"
+ val sparkClientProjectName = "celeborn-client-spark-3"
+ val sparkClientShadeProjectPath = "client-spark/spark-3-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-3-shaded"
+
+ // val jacksonVersion = "2.13.4"
+ // val jacksonDatabindVersion = "2.13.4.2"
+ val lz4JavaVersion = "1.8.0"
+ val sparkProjectScalaVersion = "2.12.15"
+ // scalaBinaryVersion
+ // val scalaBinaryVersion = "2.12"
+ val sparkVersion = "3.3.2"
+ val zstdJniVersion = "1.5.2-1"
+}
+
+object Spark34 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-3"
+ val sparkClientProjectName = "celeborn-client-spark-3"
+ val sparkClientShadeProjectPath = "client-spark/spark-3-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-3-shaded"
+
+ val lz4JavaVersion = "1.8.0"
+ val sparkProjectScalaVersion = "2.12.17"
+
+ val sparkVersion = "3.4.1"
+ val zstdJniVersion = "1.5.2-5"
+
+ lazy val deps = Seq(
+ // Spark Use `log4j-slf4j2-impl` instead of `log4j-slf4j-impl` in SPARK-40511
+ // to fix the error:
+ // ```
+ // java.lang.NoSuchMethodError: org.apache.logging.slf4j.Log4jLoggerFactory.<init>(Lorg/apache/logging/slf4j/Log4jMarkerFactory;)V
+ // ```
+ "org.apache.logging.log4j" % "log4j-slf4j2-impl" % "2.19.0" % "test"
Review Comment:
if the error message is same, I think it may be caused by
1. in Maven, slf4j-api 1.x takes effect
2. in SBT, slf4j-api 2.x takes effect
it may be caused by difference dependencies resolution strategy or configuration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1273530282
##########
.gitignore:
##########
@@ -24,6 +24,7 @@
.settings
/lib/
build/apache-maven*
+build/sbt-launch-*.jar
Review Comment:
`.bsp/` should be excluded too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1274597211
##########
project/CelebornBuild.scala:
##########
@@ -0,0 +1,604 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.nio.file.Files
+import java.nio.charset.StandardCharsets.UTF_8
+import java.util.Locale
+
+import scala.util.Properties
+
+import sbtassembly.AssemblyPlugin.autoImport._
+import sbtprotoc.ProtocPlugin.autoImport._
+
+import sbt._
+import sbt.Keys._
+import Utils._
+import CelebornCommonSettings._
+// import sbt.Keys.streams
+
+object CelebornCommonSettings {
+
+ // Scala versions
+ val SCALA_2_11_12 = "2.11.12"
+ val SCALA_2_12_10 = "2.12.10"
+ val SCALA_2_12_15 = "2.12.15"
+ val SCALA_2_12_17 = "2.12.17"
+ val scala213 = "2.13.5"
+ val ALL_SCALA_VERSIONS = Seq(SCALA_2_11_12, SCALA_2_12_10, SCALA_2_12_15, SCALA_2_12_17, scala213)
+
+ val DEFAULT_SCALA_VERSION = SCALA_2_12_15
+
+ val projectScalaVersion = defaultScalaVersion()
+
+ val zstdJniVersion = sparkClientProjects.map(_.zstdJniVersion).getOrElse("1.5.2-1")
+ val lz4JavaVersion = sparkClientProjects.map(_.lz4JavaVersion).getOrElse("1.8.0")
+
+ // Dependent library versions
+ val commonsCryptoVersion = "1.0.0"
+ val commonsIoVersion = "2.13.0"
+ val commonsLang3Version = "3.12.0"
+ val findbugsVersion = "1.3.9"
+ val guavaVersion = "14.0.1"
+ val hadoopVersion = "3.2.4"
+ val javaxServletVersion = "3.1.0"
+ val leveldbJniVersion = "1.8"
+ val log4j2Version = "2.17.2"
+ val metricsVersion = "3.2.6"
+ val scalatestMockitoVersion = "1.17.14"
+ val nettyVersion = "4.1.93.Final"
+ val ratisVersion = "2.5.1"
+ val roaringBitmapVersion = "0.9.32"
+ val slf4jVersion = "1.7.36"
+ val snakeyamlVersion = "1.33"
+
+ // Versions for proto
+ val protocVersion = "3.19.2"
+ val protoVersion = "3.19.2"
+
+ scalaVersion := projectScalaVersion
+
+ autoScalaLibrary := false
+
+ // crossScalaVersions must be set to Nil on the root project
+ crossScalaVersions := Nil
+
+ lazy val commonSettings = Seq(
+ organization := "org.apache.celeborn",
+ scalaVersion := projectScalaVersion,
+ crossScalaVersions := ALL_SCALA_VERSIONS,
+ fork := true,
+ scalacOptions ++= Seq("-target:jvm-1.8"),
+ javacOptions ++= Seq("-encoding", UTF_8.name(), "-source", "1.8"),
+
+ // -target cannot be passed as a parameter to javadoc. See https://github.com/sbt/sbt/issues/355
+ Compile / compile / javacOptions ++= Seq("-target", "1.8"),
+
+ // Make sure any tests in any project that uses Spark is configured for running well locally
+ Test / javaOptions ++= Seq(
+ "-Xmx4g"
+ ),
+
+ testOptions += Tests.Argument("-oF"),
+
+ Test / testOptions += Tests.Argument("-oDF"),
+ Test / testOptions += Tests.Argument(TestFrameworks.JUnit, "-v", "-a"),
+
+ // Don't execute in parallel since we can't have multiple Sparks in the same JVM
+ Test / parallelExecution := false,
+
+ scalacOptions ++= Seq(
+ "-P:genjavadoc:strictVisibility=true" // hide package private types and methods in javadoc
+ ),
+
+ javaOptions += "-Xmx4g",
+
+ // Configurations to speed up tests and reduce memory footprint
+ Test / javaOptions ++= Seq(
+ "-Xmx4g"
+ ),
+
+ Test / envVars += ("IS_TESTING", "1")
+ )
+
+ lazy val protoSettings = Seq(
+ // Setting version for the protobuf compiler
+ PB.protocVersion := protocVersion,
+ // set proto sources path
+ Compile / PB.protoSources := Seq(sourceDirectory.value / "main" / "proto"),
+ Compile / PB.targets := Seq(PB.gens.java -> (Compile / sourceManaged).value)
+ )
+
+ lazy val commonUnitTestDependencies = Seq(
+ "org.mockito" % "mockito-core" % "4.11.0" % "test",
+ "org.scalatest" %% "scalatest" % "3.2.16" % "test",
+ "junit" % "junit" % "4.12" % "test",
+ // https://www.scala-sbt.org/1.x/docs/Testing.html
+ "com.github.sbt" % "junit-interface" % "0.13.3" % "test")
+}
+
+object CelebornBuild extends sbt.internal.BuildDef {
+ override def projectDefinitions(baseDirectory: File): Seq[Project] = {
+ Seq(
+ CelebornCommon.common,
+ CelebornClient.client,
+ CelebornService.service,
+ CelebornWorker.worker,
+ CelebornMaster.master) ++ maybeSparkClientModules
+ }
+
+ // ThisBuild / parallelExecution := false
+
+ // scalaVersion := "2.11.12"
+
+ // autoScalaLibrary := false
+
+ crossScalaVersions := Nil
+
+ // load user-defined Profiles
+ // loadProfiles()
+}
+
+object Utils {
+ val profiles = {
+ val profiles = Properties.envOrNone("SBT_MAVEN_PROFILES")
+ .orElse(Properties.propOrNone("sbt.maven.profiles")) match {
+ case None => Seq("sbt")
+ case Some(v) =>
+ v.split("(\\s+|,)").filterNot(_.isEmpty).map(_.trim.replaceAll("-P", "")).toSeq
+ }
+ if (profiles.contains("jdwp-test-debug")) {
+ sys.props.put("test.jdwp.enabled", "true")
+ }
+ profiles
+ }
+
+ val SPARK_VERSION = profiles.filter(_.startsWith("spark")).headOption
+
+ lazy val sparkClientProjects = SPARK_VERSION match {
+ case Some("spark-2.4") => Some(Spark24)
+ case Some("spark-3.0") => Some(Spark30)
+ case Some("spark-3.1") => Some(Spark31)
+ case Some("spark-3.2") => Some(Spark32)
+ case Some("spark-3.3") => Some(Spark33)
+ case Some("spark-3.4") => Some(Spark34)
+ case _ => None
+ }
+
+ lazy val maybeSparkClientModules: Seq[Project] = sparkClientProjects.map(_.modules).getOrElse(Seq.empty)
+
+ def defaultScalaVersion(): String = {
+ // 1. Inherit the scala version of the spark project
+ // 2. if the spark profile not specified, using the DEFAULT_SCALA_VERSION
+ val v = sparkClientProjects.map(_.sparkProjectScalaVersion).getOrElse(DEFAULT_SCALA_VERSION)
+ require(ALL_SCALA_VERSIONS.contains(v), s"found not allow scala version: $v")
+ v
+ }
+}
+
+object CelebornCommon {
+ lazy val common = (project in file("common"))
+ .settings (
+ name := "celeborn-common",
+ commonSettings,
+ protoSettings,
+ libraryDependencies ++= Seq(
+ "com.google.protobuf" % "protobuf-java" % protoVersion % "protobuf",
+ "com.google.code.findbugs" % "jsr305" % findbugsVersion,
+ "com.google.guava" % "guava" % guavaVersion,
+ "commons-io" % "commons-io" % commonsIoVersion,
+ "io.dropwizard.metrics" % "metrics-core" % metricsVersion,
+ "io.dropwizard.metrics" % "metrics-graphite" % metricsVersion,
+ "io.dropwizard.metrics" % "metrics-jvm" % metricsVersion,
+ "io.netty" % "netty-all" % nettyVersion,
+ "org.apache.commons" % "commons-crypto" % commonsCryptoVersion,
+ "org.apache.commons" % "commons-lang3" % commonsLang3Version,
+ "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion,
+ "org.apache.hadoop" % "hadoop-client-runtime" % hadoopVersion,
+ "org.apache.ratis" % "ratis-client" % ratisVersion,
+ "org.apache.ratis" % "ratis-common" % ratisVersion,
+ "org.fusesource.leveldbjni" % "leveldbjni-all" % leveldbJniVersion,
+ "org.roaringbitmap" % "RoaringBitmap" % roaringBitmapVersion,
+ "org.scala-lang" % "scala-reflect" % scalaVersion.value,
+ "org.slf4j" % "jcl-over-slf4j" % slf4jVersion,
+ "org.slf4j" % "jul-to-slf4j" % slf4jVersion,
+ "org.slf4j" % "slf4j-api" % slf4jVersion,
+ "org.yaml" % "snakeyaml" % snakeyamlVersion,
+ "org.apache.logging.log4j" % "log4j-slf4j-impl" % log4j2Version % "test",
+ "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version % "test",
+
+ // Compiler plugins
+ // -- Bump up the genjavadoc version explicitly to 0.18 to work with Scala 2.12
+ compilerPlugin(
+ "com.typesafe.genjavadoc" %% "genjavadoc-plugin" % "0.18" cross CrossVersion.full)
+ ) ++ commonUnitTestDependencies,
+
+ Compile / sourceGenerators += Def.task {
+ val file = (Compile / sourceManaged).value / "org" / "apache" / "celeborn" / "package.scala"
+ streams.value.log.info("geneate version information file %s".format(file.toPath))
+ IO.write(file,
+ s"""package org.apache
+ |
+ |package object celeborn {
+ | val VERSION = "${version.value}"
+ |}
+ |""".stripMargin)
+ Seq(file)
+ // generate version task depends on PB generate to avoid concurrency generate source files
+ // otherwise we may encounter the error:
+ // ```
+ // [error] IO error while decoding ./celeborn/common/target/scala-2.12/src_managed/main/org/apache/celeborn/package.scala with UTF-8: ./celeborn/common/target/scala-2.12/src_managed/main/org/apache/celeborn/package.scala (No such file or directory)
+ // ```
+ }.dependsOn(Compile / PB.generate),
+
+ // a task to show current profiles
+ printProfiles := {
+ val message = profiles.mkString("", " ", "")
+ println("compile with profiles: %s".format(message))
+ }
+ )
+
+ lazy val printProfiles = taskKey[Unit]("Prints Profiles")
+}
+
+object CelebornClient {
+ lazy val client = (project in file("client"))
+ // ref: https://www.scala-sbt.org/1.x/docs/Multi-Project.html#Classpath+dependencies
+ .dependsOn(CelebornCommon.common % "test->test;compile->compile")
+ .settings (
+ name := "client",
+ commonSettings,
+ libraryDependencies ++= Seq(
+ "io.netty" % "netty-all" % nettyVersion,
+ "com.google.guava" % "guava" % guavaVersion,
+ "org.lz4" % "lz4-java" % lz4JavaVersion,
+ "com.github.luben" % "zstd-jni" % zstdJniVersion,
+ "org.apache.commons" % "commons-lang3" % commonsLang3Version,
+ "org.apache.logging.log4j" % "log4j-slf4j-impl" % log4j2Version % "test",
+ "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version % "test",
+
+ // Compiler plugins
+ // -- Bump up the genjavadoc version explicitly to 0.18 to work with Scala 2.12
+ compilerPlugin(
+ "com.typesafe.genjavadoc" %% "genjavadoc-plugin" % "0.18" cross CrossVersion.full)
+ ) ++ commonUnitTestDependencies
+ )
+}
+
+object CelebornService {
+ lazy val service = (project in file("service"))
+ .dependsOn(CelebornCommon.common)
+ .settings (
+ name := "service",
+ commonSettings,
+ libraryDependencies ++= Seq(
+ "com.google.code.findbugs" % "jsr305" % findbugsVersion,
+ "commons-io" % "commons-io" % commonsIoVersion,
+ "io.netty" % "netty-all" % nettyVersion,
+ "javax.servlet" % "javax.servlet-api" % javaxServletVersion,
+ "org.apache.commons" % "commons-crypto" % commonsCryptoVersion,
+ "org.slf4j" % "slf4j-api" % slf4jVersion,
+ "org.apache.logging.log4j" % "log4j-slf4j-impl" % log4j2Version % "test",
+ "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version % "test",
+
+ // Compiler plugins
+ // -- Bump up the genjavadoc version explicitly to 0.18 to work with Scala 2.12
+ compilerPlugin(
+ "com.typesafe.genjavadoc" %% "genjavadoc-plugin" % "0.18" cross CrossVersion.full)
+ ) ++ commonUnitTestDependencies
+ )
+}
+
+object CelebornMaster {
+ lazy val master = (project in file("master"))
+ .dependsOn(CelebornCommon.common, CelebornService.service)
+ .settings (
+ name := "master",
+ commonSettings,
+ protoSettings,
+ libraryDependencies ++= Seq(
+ "com.google.guava" % "guava" % guavaVersion,
+ "com.google.protobuf" % "protobuf-java" % protoVersion,
+ "io.netty" % "netty-all" % nettyVersion,
+ "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion,
+ "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version,
+ "org.apache.logging.log4j" % "log4j-slf4j-impl" % log4j2Version,
+ "org.apache.ratis" % "ratis-client" % ratisVersion,
+ "org.apache.ratis" % "ratis-common" % ratisVersion,
+ "org.apache.ratis" % "ratis-grpc" % ratisVersion,
+ "org.apache.ratis" % "ratis-netty" % ratisVersion,
+ "org.apache.ratis" % "ratis-server" % ratisVersion,
+ "org.apache.ratis" % "ratis-shell" % ratisVersion,
+
+ // Compiler plugins
+ // -- Bump up the genjavadoc version explicitly to 0.18 to work with Scala 2.12
+ compilerPlugin(
+ "com.typesafe.genjavadoc" %% "genjavadoc-plugin" % "0.18" cross CrossVersion.full)
+ ) ++ commonUnitTestDependencies
+ )
+}
+
+object CelebornWorker {
+ lazy val worker = (project in file("worker"))
+ .dependsOn(CelebornCommon.common, CelebornService.service)
+ .dependsOn(CelebornClient.client % "test->test;compile->compile")
+ .dependsOn(CelebornMaster.master % "test->test;compile->compile")
+ .settings (
+ name := "worker",
+ commonSettings,
+ libraryDependencies ++= Seq(
+ "com.google.guava" % "guava" % guavaVersion,
+ "commons-io" % "commons-io" % commonsIoVersion,
+ "io.netty" % "netty-all" % nettyVersion,
+ "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version,
+ "org.apache.logging.log4j" % "log4j-slf4j-impl" % log4j2Version,
+ "org.fusesource.leveldbjni" % "leveldbjni-all" % leveldbJniVersion,
+ "org.roaringbitmap" % "RoaringBitmap" % roaringBitmapVersion,
+ "org.mockito" %% "mockito-scala-scalatest" % scalatestMockitoVersion % "test",
+
+ // Compiler plugins
+ // -- Bump up the genjavadoc version explicitly to 0.18 to work with Scala 2.12
+ compilerPlugin(
+ "com.typesafe.genjavadoc" %% "genjavadoc-plugin" % "0.18" cross CrossVersion.full)
+ ) ++ commonUnitTestDependencies
+ )
+}
+
+////////////////////////////////////////////////////////
+// Spark Client //
+////////////////////////////////////////////////////////
+
+object Spark24 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-2"
+ val sparkClientProjectName = "celeborn-client-spark-2"
+ val sparkClientShadeProjectPath = "client-spark/spark-2-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-2-shaded"
+
+ // val jacksonVersion = "2.5.7"
+ // val jacksonDatabindVersion = "2.6.7.3"
+ val lz4JavaVersion = "1.4.0"
+ val sparkProjectScalaVersion = "2.11.12"
+ // scalaBinaryVersion
+ // val scalaBinaryVersion = "2.11"
+ val sparkVersion = "2.4.8"
+ val zstdJniVersion = "1.4.4-3"
+}
+
+object Spark30 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-3"
+ val sparkClientProjectName = "celeborn-client-spark-3"
+ val sparkClientShadeProjectPath = "client-spark/spark-3-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-3-shaded"
+
+ val lz4JavaVersion = "1.7.1"
+ val sparkProjectScalaVersion = "2.12.10"
+
+ val sparkVersion = "3.0.3"
+ val zstdJniVersion = "1.4.4-3"
+}
+
+object Spark31 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-3"
+ val sparkClientProjectName = "celeborn-client-spark-3"
+ val sparkClientShadeProjectPath = "client-spark/spark-3-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-3-shaded"
+
+ val lz4JavaVersion = "1.7.1"
+ val sparkProjectScalaVersion = "2.12.10"
+
+ val sparkVersion = "3.1.3"
+ val zstdJniVersion = "1.4.8-1"
+}
+
+object Spark32 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-3"
+ val sparkClientProjectName = "celeborn-client-spark-3"
+ val sparkClientShadeProjectPath = "client-spark/spark-3-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-3-shaded"
+
+ val lz4JavaVersion = "1.7.1"
+ val sparkProjectScalaVersion = "2.12.15"
+
+ val sparkVersion = "3.2.4"
+ val zstdJniVersion = "1.5.0-4"
+}
+
+object Spark33 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-3"
+ val sparkClientProjectName = "celeborn-client-spark-3"
+ val sparkClientShadeProjectPath = "client-spark/spark-3-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-3-shaded"
+
+ // val jacksonVersion = "2.13.4"
+ // val jacksonDatabindVersion = "2.13.4.2"
+ val lz4JavaVersion = "1.8.0"
+ val sparkProjectScalaVersion = "2.12.15"
+ // scalaBinaryVersion
+ // val scalaBinaryVersion = "2.12"
+ val sparkVersion = "3.3.2"
+ val zstdJniVersion = "1.5.2-1"
+}
+
+object Spark34 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-3"
+ val sparkClientProjectName = "celeborn-client-spark-3"
+ val sparkClientShadeProjectPath = "client-spark/spark-3-shade"
+ val sparkClientShadeProjectName = "celeborn-client-spark-3-shaded"
+
+ val lz4JavaVersion = "1.8.0"
+ val sparkProjectScalaVersion = "2.12.17"
+
+ val sparkVersion = "3.4.1"
+ val zstdJniVersion = "1.5.2-5"
+
+ lazy val deps = Seq(
+ // Spark Use `log4j-slf4j2-impl` instead of `log4j-slf4j-impl` in SPARK-40511
+ // to fix the error:
+ // ```
+ // java.lang.NoSuchMethodError: org.apache.logging.slf4j.Log4jLoggerFactory.<init>(Lorg/apache/logging/slf4j/Log4jMarkerFactory;)V
+ // ```
+ "org.apache.logging.log4j" % "log4j-slf4j2-impl" % "2.19.0" % "test"
Review Comment:
it doesn't work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1273537374
##########
project/CelebornBuild.scala:
##########
@@ -0,0 +1,604 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.nio.file.Files
+import java.nio.charset.StandardCharsets.UTF_8
+import java.util.Locale
+
+import scala.util.Properties
+
+import sbtassembly.AssemblyPlugin.autoImport._
+import sbtprotoc.ProtocPlugin.autoImport._
+
+import sbt._
+import sbt.Keys._
+import Utils._
+import CelebornCommonSettings._
+// import sbt.Keys.streams
+
+object CelebornCommonSettings {
+
+ // Scala versions
+ val SCALA_2_11_12 = "2.11.12"
+ val SCALA_2_12_10 = "2.12.10"
+ val SCALA_2_12_15 = "2.12.15"
+ val SCALA_2_12_17 = "2.12.17"
+ val scala213 = "2.13.5"
+ val ALL_SCALA_VERSIONS = Seq(SCALA_2_11_12, SCALA_2_12_10, SCALA_2_12_15, SCALA_2_12_17, scala213)
+
+ val DEFAULT_SCALA_VERSION = SCALA_2_12_15
+
+ val projectScalaVersion = defaultScalaVersion()
+
+ val zstdJniVersion = sparkClientProjects.map(_.zstdJniVersion).getOrElse("1.5.2-1")
+ val lz4JavaVersion = sparkClientProjects.map(_.lz4JavaVersion).getOrElse("1.8.0")
+
+ // Dependent library versions
+ val commonsCryptoVersion = "1.0.0"
+ val commonsIoVersion = "2.13.0"
+ val commonsLang3Version = "3.12.0"
+ val findbugsVersion = "1.3.9"
+ val guavaVersion = "14.0.1"
+ val hadoopVersion = "3.2.4"
+ val javaxServletVersion = "3.1.0"
+ val leveldbjniAllVersion = "1.8"
+ val log4jVersion = "2.17.2"
Review Comment:
```suggestion
val log4j2Version = "2.17.2"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] codecov[bot] commented on pull request #1757: [WIP][CELEBORN-XXX][BUILD] Initial support sbt
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#issuecomment-1648918902
## [Codecov](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1757?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#1757](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1757?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (d3b3cfd) into [main](https://app.codecov.io/gh/apache/incubator-celeborn/commit/6427ed35cd63497191cfc13e7675710a4aa82d96?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (6427ed3) will **not change** coverage.
> The diff coverage is `n/a`.
> :exclamation: Current head d3b3cfd differs from pull request most recent head 19f9845. Consider uploading reports for the commit 19f9845 to get more accurate results
```diff
@@ Coverage Diff @@
## main #1757 +/- ##
=======================================
Coverage 46.72% 46.72%
=======================================
Files 161 161
Lines 10021 10021
Branches 925 925
=======================================
Hits 4681 4681
Misses 5021 5021
Partials 319 319
```
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#issuecomment-1649403613
cc @pan3793 @waitinfuture @AngersZhuuuu @FMX
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1273533179
##########
project/plugins.sbt:
##########
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "2.1.1")
+
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1274502450
##########
build/sbt:
##########
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+SELF=$(cd $(dirname $0) && pwd)
+. "$SELF/util.sh"
+
+# Check if repositories-local file exists, otherwise use repositories
+SBT_REPOSITORIES_LOCAL_CONFIG="$(dirname "$(realpath "$0")")/sbt-config/repositories-local"
+SBT_REPOSITORIES_CONFIG="$(dirname "$(realpath "$0")")/sbt-config/repositories"
+
+if [ -f "$SBT_REPOSITORIES_LOCAL_CONFIG" ]; then
+ SBT_OPTS="${SBT_OPTS:="-Dsbt.override.build.repos=true -Dsbt.repository.config=$SBT_REPOSITORIES_LOCAL_CONFIG"}"
+else
+ SBT_OPTS="${SBT_OPTS:="-Dsbt.override.build.repos=true -Dsbt.repository.config=$SBT_REPOSITORIES_CONFIG"}"
+fi
+
+export SBT_OPTS
+
+. "$(dirname "$(realpath "$0")")"/sbt-launch-lib.bash
+
+
+declare -r noshare_opts="-Dsbt.global.base=project/.sbtboot -Dsbt.boot.directory=project/.boot -Dsbt.ivy.home=project/.ivy"
+declare -r sbt_opts_file=".sbtopts"
+declare -r etc_sbt_opts_file="/etc/sbt/sbtopts"
+declare -r default_sbt_opts="-Xss64m"
+
+usage() {
+ cat <<EOM
+Usage: $script_name [options]
Review Comment:
nit: this is not the same as sbt official options, we may want to migrate it to the official one in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] waitinfuture commented on pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#issuecomment-1657402451
That's great! Thanks @cfmcgrady
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1273536939
##########
project/CelebornBuild.scala:
##########
@@ -0,0 +1,604 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.nio.file.Files
+import java.nio.charset.StandardCharsets.UTF_8
+import java.util.Locale
+
+import scala.util.Properties
+
+import sbtassembly.AssemblyPlugin.autoImport._
+import sbtprotoc.ProtocPlugin.autoImport._
+
+import sbt._
+import sbt.Keys._
+import Utils._
+import CelebornCommonSettings._
+// import sbt.Keys.streams
+
+object CelebornCommonSettings {
+
+ // Scala versions
+ val SCALA_2_11_12 = "2.11.12"
+ val SCALA_2_12_10 = "2.12.10"
+ val SCALA_2_12_15 = "2.12.15"
+ val SCALA_2_12_17 = "2.12.17"
+ val scala213 = "2.13.5"
+ val ALL_SCALA_VERSIONS = Seq(SCALA_2_11_12, SCALA_2_12_10, SCALA_2_12_15, SCALA_2_12_17, scala213)
+
+ val DEFAULT_SCALA_VERSION = SCALA_2_12_15
+
+ val projectScalaVersion = defaultScalaVersion()
+
+ val zstdJniVersion = sparkClientProjects.map(_.zstdJniVersion).getOrElse("1.5.2-1")
+ val lz4JavaVersion = sparkClientProjects.map(_.lz4JavaVersion).getOrElse("1.8.0")
+
+ // Dependent library versions
+ val commonsCryptoVersion = "1.0.0"
+ val commonsIoVersion = "2.13.0"
+ val commonsLang3Version = "3.12.0"
+ val findbugsVersion = "1.3.9"
+ val guavaVersion = "14.0.1"
+ val hadoopVersion = "3.2.4"
+ val javaxServletVersion = "3.1.0"
+ val leveldbjniAllVersion = "1.8"
+ val log4jVersion = "2.17.2"
+ val metricsVersion = "3.2.6"
+ val mockitoScalaScalatestVersion = "1.17.14"
Review Comment:
```suggestion
val scalatestMockitoVersion = "1.17.14"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1274595218
##########
build/sbt-config/repositories-cn.template:
##########
@@ -0,0 +1,11 @@
+# As a Chinese developer facing network issues, you can accelerate the download speed of bootstrap/plugin/dependencies jar packages by executing the following command to configure the mirror:
+#
+# ```
+# cp build/sbt-config/repositories-cn.template build/sbt-config/repositories-local
+# ```
+
+[repositories]
+ local
+ private: ${DEFAULT_ARTIFACT_REPOSITORY-file:///dev/null}
Review Comment:
the variable substitution required system property instead of the environment variable, updated in https://github.com/apache/incubator-celeborn/pull/1757/commits/48a1aa93801dedc3f2a4ed0b515efc24ae1e6662
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#issuecomment-1650161253
Also cc @LuciferYang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1274502479
##########
build/sbt-config/repositories-cn.template:
##########
@@ -0,0 +1,11 @@
+# As a Chinese developer facing network issues, you can accelerate the download speed of bootstrap/plugin/dependencies jar packages by executing the following command to configure the mirror:
+#
+# ```
+# cp build/sbt-config/repositories-cn.template build/sbt-config/repositories-local
+# ```
+
+[repositories]
+ local
+ private: ${DEFAULT_ARTIFACT_REPOSITORY-file:///dev/null}
Review Comment:
If the user has configured it, include the `DEFAULT_ARTIFACT_REPOSITORY` in the list of repositories. cc @pan3793
the variable substitution docs: https://www.scala-sbt.org/1.x/docs/Launcher-Configuration.html#Variable+Substitution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1274614738
##########
build/sbt-config/repositories-cn.template:
##########
@@ -0,0 +1,11 @@
+# As a Chinese developer facing network issues, you can accelerate the download speed of bootstrap/plugin/dependencies jar packages by executing the following command to configure the mirror:
+#
+# ```
+# cp build/sbt-config/repositories-cn.template build/sbt-config/repositories-local
+# ```
+
+[repositories]
+ local
+ private: ${DEFAULT_ARTIFACT_REPOSITORY-file:///dev/null}
Review Comment:
I see, thanks for fixing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1757: [WIP][CELEBORN-XXX][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1272938232
##########
build/sbt:
##########
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+SELF=$(cd $(dirname $0) && pwd)
+. "$SELF/util.sh"
+
+# When creating new tests for Spark SQL Hive, the HADOOP_CLASSPATH must contain the hive jars so
+# that we can run Hive to generate the golden answer. This is not required for normal development
+# or testing.
+if [ -n "$HIVE_HOME" ]; then
+ for i in "$HIVE_HOME"/lib/*
+ do HADOOP_CLASSPATH="$HADOOP_CLASSPATH:$i"
+ done
+ export HADOOP_CLASSPATH
+fi
+
+SBT_REPOSITORIES_CONFIG="$(dirname "$(realpath "$0")")/sbt-config/repositories"
+export SBT_OPTS="${SBT_OPTS:="-Dsbt.override.build.repos=true -Dsbt.repository.config=$SBT_REPOSITORIES_CONFIG"}"
+
+. "$(dirname "$(realpath "$0")")"/sbt-launch-lib.bash
+
+
+declare -r noshare_opts="-Dsbt.global.base=project/.sbtboot -Dsbt.boot.directory=project/.boot -Dsbt.ivy.home=project/.ivy"
+declare -r sbt_opts_file=".sbtopts"
+declare -r etc_sbt_opts_file="/etc/sbt/sbtopts"
+declare -r default_sbt_opts="-Xss64m"
+
+usage() {
+ cat <<EOM
+Usage: $script_name [options]
+
+ -h | -help print this message
+ -v | -verbose this runner is chattier
+ -d | -debug set sbt log level to debug
+ -no-colors disable ANSI color codes
+ -sbt-create start sbt even if current directory contains no sbt project
+ -sbt-dir <path> path to global settings/plugins directory (default: ~/.sbt)
+ -sbt-boot <path> path to shared boot directory (default: ~/.sbt/boot in 0.11 series)
+ -ivy <path> path to local Ivy repository (default: ~/.ivy2)
+ -mem <integer> set memory options (default: $sbt_default_mem, which is $(get_mem_opts $sbt_default_mem))
+ -no-share use all local caches; no sharing
+ -no-global uses global caches, but does not use global ~/.sbt directory.
+ -jvm-debug <port> Turn on JVM debugging, open at the given port.
+ -batch Disable interactive mode
+
+ # sbt version (default: from project/build.properties if present, else latest release)
+ -sbt-version <version> use the specified version of sbt
+ -sbt-jar <path> use the specified jar as the sbt launcher
+ -sbt-rc use an RC version of sbt
+ -sbt-snapshot use a snapshot version of sbt
+
+ # java version (default: java from PATH, currently $(java -version 2>&1 | grep version))
+ -java-home <path> alternate JAVA_HOME
+
+ # jvm options and output control
+ JAVA_OPTS environment variable, if unset uses "$java_opts"
+ SBT_OPTS environment variable, if unset uses "$default_sbt_opts"
+ .sbtopts if this file exists in the current directory, it is
+ prepended to the runner args
+ /etc/sbt/sbtopts if this file exists, it is prepended to the runner args
+ -Dkey=val pass -Dkey=val directly to the java runtime
+ -J-X pass option -X directly to the java runtime
+ (-J is stripped)
+ -S-X add -X to sbt's scalacOptions (-S is stripped)
+ -PmavenProfiles Enable a maven profile for the build.
+
+In the case of duplicated or conflicting options, the order above
+shows precedence: JAVA_OPTS lowest, command line options highest.
+EOM
+}
+
+process_my_args () {
+ while [[ $# -gt 0 ]]; do
+ case "$1" in
+ -no-colors) addJava "-Dsbt.log.noformat=true" && shift ;;
+ -no-share) addJava "$noshare_opts" && shift ;;
+ -no-global) addJava "-Dsbt.global.base=$(pwd)/project/.sbtboot" && shift ;;
+ -sbt-boot) require_arg path "$1" "$2" && addJava "-Dsbt.boot.directory=$2" && shift 2 ;;
+ -sbt-dir) require_arg path "$1" "$2" && addJava "-Dsbt.global.base=$2" && shift 2 ;;
+ -debug-inc) addJava "-Dxsbt.inc.debug=true" && shift ;;
+ -batch) exec </dev/null && shift ;;
+
+ -sbt-create) sbt_create=true && shift ;;
+
+ *) addResidual "$1" && shift ;;
+ esac
+ done
+
+ # Now, ensure sbt version is used.
+ [[ "${sbt_version}XXX" != "XXX" ]] && addJava "-Dsbt.version=$sbt_version"
+}
+
+loadConfigFile() {
+ cat "$1" | sed '/^\#/d'
+}
+
+# if sbtopts files exist, prepend their contents to $@ so it can be processed by this runner
+[[ -f "$etc_sbt_opts_file" ]] && set -- $(loadConfigFile "$etc_sbt_opts_file") "$@"
+[[ -f "$sbt_opts_file" ]] && set -- $(loadConfigFile "$sbt_opts_file") "$@"
+
+exit_status=127
+saved_stty=""
+
+restoreSttySettings() {
+ stty $saved_stty
+ saved_stty=""
+}
+
+onExit() {
+ if [[ "$saved_stty" != "" ]]; then
+ restoreSttySettings
+ fi
+ exit $exit_status
+}
+
+saveSttySettings() {
+ saved_stty=$(stty -g 2>/dev/null)
+ if [[ ! $? ]]; then
+ saved_stty=""
+ fi
+}
+
+if [ ! -z "${SPARK_LOCAL_HOSTNAME}" ]; then
+ echo "Using SPARK_LOCAL_HOSTNAME=$SPARK_LOCAL_HOSTNAME" 1>&2
Review Comment:
let's respect CELEBORN_* envs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1273560300
##########
project/build.properties:
##########
@@ -0,0 +1,17 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+sbt.version=1.9.2
Review Comment:
[sbt 1.9.3](https://github.com/sbt/sbt/releases/tag/v1.9.3) is available yesterday
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1274428402
##########
build/sbt-config/repositories-cn.template:
##########
@@ -0,0 +1,7 @@
+[repositories]
+ local
+ local-preloaded-ivy: file:///${sbt.preloaded-${sbt.global.base-${user.home}/.sbt}/preloaded/}, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
+ local-preloaded: file:///${sbt.preloaded-${sbt.global.base-${user.home}/.sbt}/preloaded/}
+ aliyun-maven: https://maven.aliyun.com/nexus/content/groups/public/
+ huawei-central: https://mirrors.huaweicloud.com/repository/maven/
+ maven-central
Review Comment:
seems we can drop it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1274425855
##########
.rat-excludes:
##########
@@ -34,4 +34,6 @@ NOTICE*
assets/**
build/apache-maven-*/**
build/scala-*/**
+build/sbt-config/repositories
+build/sbt-config/repositories-cn.template
Review Comment:
```suggestion
build/sbt-config/**
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1274428138
##########
build/sbt-config/repositories-cn.template:
##########
@@ -0,0 +1,7 @@
+[repositories]
Review Comment:
does the file support comments? we can add comments to guide user how to use this file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#issuecomment-1649861586
> Maybe we can refer to the Akka?
I believe we can list "the frequently used commands" in Akka way.
But for "how to set mirror", maybe another place. Especially, I believe there will be lots of people stuck on downloading `build/sbt` bootstrap jar, and the bootstrap phase because the network issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#issuecomment-1654890763
Thanks, merging to main
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1274467635
##########
build/sbt-config/repositories-cn.template:
##########
@@ -0,0 +1,7 @@
+[repositories]
Review Comment:
https://stackoverflow.com/questions/56343343/how-do-you-comment-in-the-sbt-repositories-file
The commenting style commences with the symbol `#`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1274501230
##########
build/sbt-config/repositories-cn.template:
##########
@@ -1,7 +1,11 @@
+# As a Chinese developer facing network issues, you can accelerate the download speed of bootstrap/plugin/dependencies jar packages by executing the following command to configure the mirror:
+#
+# ```
+# cp build/sbt-config/repositories-cn.template build/sbt-config/repositories-local
Review Comment:
```suggestion
# cp build/sbt-config/repositories-cn.template build/sbt-config/repositories-local
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 closed pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
URL: https://github.com/apache/incubator-celeborn/pull/1757
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] FMX commented on pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#issuecomment-1652963802
I tried assembly and I got errors. Is there anything that I missed?
>
> [info] 2 file(s) merged using strategy 'Rename' (Run the task at debug level to see the details)
> [info] 1 file(s) merged using strategy 'Discard' (Run the task at debug level to see the details)
> [info] Built: /Users/minxfeng/wcode/apache/incubator-celeborn/target/scala-2.12/incubator-celeborn-assembly-0.4.0-SNAPSHOT.jar
> [info] Jar hash: 6bec02bf548084ceb7c97f30458481d512ab2363
> [error] 9 error(s) were encountered during the merge:
> [error] 9 error(s) were encountered during the merge:
> [error] 9 error(s) were encountered during the merge:
> [error] 11 error(s) were encountered during the merge:
> [error] 11 error(s) were encountered during the merge:
> [error] stack trace is suppressed; run last common / assembly for the full output
> [error] stack trace is suppressed; run last client / assembly for the full output
> [error] stack trace is suppressed; run last worker / assembly for the full output
> [error] stack trace is suppressed; run last master / assembly for the full output
> [error] stack trace is suppressed; run last service / assembly for the full output
> [error] (common / assembly)
> [error] Deduplicate found different file contents in the following:
> [error] Jar name = commons-logging-1.1.3.jar, jar org = commons-logging, entry target = org/apache/commons/logging/impl/SimpleLog$1.class
> [error] Jar name = jcl-over-slf4j-1.7.36.jar, jar org = org.slf4j, entry target = org/apache/commons/logging/impl/SimpleLog$1.class
> [error] Deduplicate found different file contents in the following:
> [error] Jar name = commons-logging-1.1.3.jar, jar org = commons-logging, entry target = org/apache/commons/logging/Log.class
> [error] Jar name = jcl-over-slf4j-1.7.36.jar, jar org = org.slf4j, entry target = org/apache/commons/logging/Log.class
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1273536233
##########
project/CelebornBuild.scala:
##########
@@ -0,0 +1,604 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.nio.file.Files
+import java.nio.charset.StandardCharsets.UTF_8
+import java.util.Locale
+
+import scala.util.Properties
+
+import sbtassembly.AssemblyPlugin.autoImport._
+import sbtprotoc.ProtocPlugin.autoImport._
+
+import sbt._
+import sbt.Keys._
+import Utils._
+import CelebornCommonSettings._
+// import sbt.Keys.streams
+
+object CelebornCommonSettings {
+
+ // Scala versions
+ val SCALA_2_11_12 = "2.11.12"
+ val SCALA_2_12_10 = "2.12.10"
+ val SCALA_2_12_15 = "2.12.15"
+ val SCALA_2_12_17 = "2.12.17"
+ val scala213 = "2.13.5"
+ val ALL_SCALA_VERSIONS = Seq(SCALA_2_11_12, SCALA_2_12_10, SCALA_2_12_15, SCALA_2_12_17, scala213)
+
+ val DEFAULT_SCALA_VERSION = SCALA_2_12_15
+
+ val projectScalaVersion = defaultScalaVersion()
+
+ val zstdJniVersion = sparkClientProjects.map(_.zstdJniVersion).getOrElse("1.5.2-1")
+ val lz4JavaVersion = sparkClientProjects.map(_.lz4JavaVersion).getOrElse("1.8.0")
+
+ // Dependent library versions
+ val commonsCryptoVersion = "1.0.0"
+ val commonsIoVersion = "2.13.0"
+ val commonsLang3Version = "3.12.0"
+ val findbugsVersion = "1.3.9"
+ val guavaVersion = "14.0.1"
+ val hadoopVersion = "3.2.4"
+ val javaxServletVersion = "3.1.0"
+ val leveldbjniAllVersion = "1.8"
Review Comment:
```suggestion
val leveldbJniVersion = "1.8"
```
##########
project/CelebornBuild.scala:
##########
@@ -0,0 +1,604 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.nio.file.Files
+import java.nio.charset.StandardCharsets.UTF_8
+import java.util.Locale
+
+import scala.util.Properties
+
+import sbtassembly.AssemblyPlugin.autoImport._
+import sbtprotoc.ProtocPlugin.autoImport._
+
+import sbt._
+import sbt.Keys._
+import Utils._
+import CelebornCommonSettings._
+// import sbt.Keys.streams
+
+object CelebornCommonSettings {
+
+ // Scala versions
+ val SCALA_2_11_12 = "2.11.12"
+ val SCALA_2_12_10 = "2.12.10"
+ val SCALA_2_12_15 = "2.12.15"
+ val SCALA_2_12_17 = "2.12.17"
+ val scala213 = "2.13.5"
+ val ALL_SCALA_VERSIONS = Seq(SCALA_2_11_12, SCALA_2_12_10, SCALA_2_12_15, SCALA_2_12_17, scala213)
+
+ val DEFAULT_SCALA_VERSION = SCALA_2_12_15
+
+ val projectScalaVersion = defaultScalaVersion()
+
+ val zstdJniVersion = sparkClientProjects.map(_.zstdJniVersion).getOrElse("1.5.2-1")
+ val lz4JavaVersion = sparkClientProjects.map(_.lz4JavaVersion).getOrElse("1.8.0")
+
+ // Dependent library versions
+ val commonsCryptoVersion = "1.0.0"
+ val commonsIoVersion = "2.13.0"
+ val commonsLang3Version = "3.12.0"
+ val findbugsVersion = "1.3.9"
+ val guavaVersion = "14.0.1"
+ val hadoopVersion = "3.2.4"
+ val javaxServletVersion = "3.1.0"
+ val leveldbjniAllVersion = "1.8"
+ val log4jVersion = "2.17.2"
+ val metricsVersion = "3.2.6"
+ val mockitoScalaScalatestVersion = "1.17.14"
+ val nettyAllVersion = "4.1.93.Final"
Review Comment:
```suggestion
val nettyVersion = "4.1.93.Final"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#issuecomment-1649854446
> Can we add some instruction at someplace as well? e.g. how to set mirror, the frequently used commands
Maybe we can refer to the Akka?
https://github.com/akka/akka/blob/ca1341365cb4db8ba5f61699da4d90991a3d1d59/project/AkkaBuild.scala#L273-L309
```scala
lazy val welcomeSettings: Seq[Setting[_]] =
CliOptions.runningOnCi
.ifTrue(Seq())
.getOrElse(Seq(onLoadMessage :=
s"""
|
| .~!77!^ ::: :::
| .^!???????~ .5GP: .5GP:
| :~???????????! ... .PGP: . .5GP: . ...
| .~7??????????????! ~J5PPP5J!Y5Y. .5GP: .?557..5GP: .?557..!J5PPP5?!55J
| .^!??????????????????7. :5GGY!~~!JPGGP. .5GP:.7PGY^ .5GP:.7PGY^ ^5GGJ!~~!YGGG5
| :~7?????????????????????7. YGG! ^PGP. .5GP?PG5^ .5GP?PG5^ .PGP^ ~GGY
| .^7?????????????JJYY55555YYJ?. .5GG^ .PGP. .5GGG5GP7. .5GGG5GP7. :PGP: :PGY
| :!????????????JY5PPGGGGGGGGGGGP5^ ~PGP7:..:!5GGP: .PGG~ ~5BP! .5GG~ ~5BP! !GG5!:..:7PGG5.
| .~7??????????JY5PGGGGGGGGGGGGGGGGGGP^ :?5GGPPPPJJPGG7 5GP: .7PGY^.5GP: 7PGY^^JPGGPPPP?JPGP!
| :??????????JYPGGGGPPPPPGGGGGGGGGGGGGGJ .:^^^:. :^^: ::: .::: ::: .::: .:^^^:. :^^.
| ^????????7???7~~^::::::^^~!?Y5GGGGGGG7
| ^!7??7~:. .^7YPP57
| ... .:.
|
| ${version.value}
|
|
| Useful sbt tasks:
| > testOnly *.AnySpec Only run a selected test
| > verifyCodeStyle Verify code style
| > applyCodeStyle Apply code style
| > sortImports Sort the imports
| > mimaReportBinaryIssues Check binary issues
| > validatePullRequest Validate pull request
| > akka-docs/paradox Build documentation
| > akka-docs/paradoxBrowse Browse the generated documentation
|
| Contributing guide: https://github.com/akka/akka/blob/main/CONTRIBUTING.md
| tips: prefix commands with + to run against cross Scala versions.
|
|""".stripMargin))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#issuecomment-1649845956
Can we add some instruction at someplace as well? e.g. how to set mirror, the frequently used commands
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1757: [WIP][CELEBORN-XXX][BUILD] Initial support sbt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#discussion_r1272938080
##########
build/sbt:
##########
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+SELF=$(cd $(dirname $0) && pwd)
+. "$SELF/util.sh"
+
+# When creating new tests for Spark SQL Hive, the HADOOP_CLASSPATH must contain the hive jars so
+# that we can run Hive to generate the golden answer. This is not required for normal development
+# or testing.
+if [ -n "$HIVE_HOME" ]; then
Review Comment:
seems not needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] waitinfuture commented on pull request #1757: [CELEBORN-836][BUILD] Initial support sbt
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on PR #1757:
URL: https://github.com/apache/incubator-celeborn/pull/1757#issuecomment-1657394590
Thanks @cfmcgrady for this PR! I tested and it really speeded up build time. Just wonder do you have a plan to add a doc describing building Celeborn?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org