You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "yew1eb (via GitHub)" <gi...@apache.org> on 2026/10/13 11:37:01 UTC
[PR] [CELEBORN-2478] Add celeborn-client build information to Spark UI Celeborn tab [celeborn]
yew1eb opened a new pull request, #3870:
URL: https://github.com/apache/celeborn/pull/3870
### Why are the changes needed?
The Celeborn tab in the Spark UI does not show which celeborn-client version the application is using, which makes version-mismatch debugging between a Spark job and the cluster unnecessarily hard. See CELEBORN-2478.
### What changes were included?
Mirrors Uniffle's "Uniffle Build Information" pattern — the info is baked into the client at compile time and rendered by the Spark UI plugin, with **no reporting to the master**:
- **Compile-time constants** (`org.apache.celeborn.ProjectConstants`, `VERSION` / `REVISION` / `getGitCommitId()`) generated from both build systems:
- Maven: `templating-maven-plugin` + `buildnumber-maven-plugin` + `properties-maven-plugin` on the new `common/src/main/java-templates` (same recipe as Uniffle)
- SBT: the existing version source generator in `project/CelebornBuild.scala` is extended to also emit the same Java class using `gitHeadCommit`
- **Persistence**: `CelebornDriverPlugin.init` writes a `CelebornBuildInfoUIData` entry into the Spark KVStore, so it survives application end and shows up on History Server replay
- **UI**: a collapsible "Celeborn Client Build Information" table rendered in `CelebornShufflePage`, ahead of "Celeborn Properties"
- Tests: `CelebornBuildInfoSuite` covering KVStore round-trip and constant sanity
### How was this patch tested?
- `./build/mvn -Pspark-3.5 -pl common,client-spark/spark-3-ui -am clean install` — build success; the generated `ProjectConstants.java` contains the correct version and current git revision
- `./build/mvn -Pspark-3.5 -pl client-spark/spark-3-ui clean test` — 8/8 tests pass (5 existing + 3 new)
- `./build/sbt "celeborn-common/Compile/compile"` — SBT generates identical constants
- Spotless formatting applied
End-to-end verification against a live cluster / History Server was not run in this change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2478] Add celeborn-client build information to Spark UI Celeborn tab [celeborn]
Posted by "zaynt4606 (via GitHub)" <gi...@apache.org>.
zaynt4606 closed pull request #3870: [CELEBORN-2478] Add celeborn-client build information to Spark UI Celeborn tab
URL: https://github.com/apache/celeborn/pull/3870
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2478] Add celeborn-client build information to Spark UI Celeborn tab [celeborn]
Posted by "yew1eb (via GitHub)" <gi...@apache.org>.
yew1eb commented on code in PR #3870:
URL: https://github.com/apache/celeborn/pull/3870#discussion_r4267653792
##########
common/pom.xml:
##########
@@ -192,6 +192,56 @@
<groupId>org.xolstice.maven.plugins</groupId>
<artifactId>protobuf-maven-plugin</artifactId>
</plugin>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>buildnumber-maven-plugin</artifactId>
+ <configuration>
+ <outputDirectory>${project.build.directory}/generated/build-metadata</outputDirectory>
+ <outputName>build.properties</outputName>
+ <revisionPropertyName>git.revision</revisionPropertyName>
Review Comment:
Done in ea7162a: `revisionOnScmFailure` set to `N/A` (matching the SBT fallback), plus a `source-archive` CI job building from a `git archive` export.
##########
client-spark/spark-3-ui/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornPlugin.scala:
##########
@@ -49,10 +50,29 @@ private class CelebornDriverPlugin extends DriverPlugin with Logging {
logInfo("Initializing CelebornDriverPlugin...")
this.sc = sc
val kvStore = sc.statusStore.store
+ writeBuildInfo(kvStore)
new CelebornListener(kvStore, sc.conf).register(sc)
java.util.Collections.emptyMap[String, String]()
}
+ /**
+ * Persists the build info of the celeborn-client bundled with this application into the
+ * KVStore so that it survives application end and is available on History Server replay,
+ * where the Spark UI "Celeborn" tab renders it as "Celeborn Client Build Information".
+ */
+ private def writeBuildInfo(kvStore: KVStore): Unit = {
+ val buildInfo = new CelebornBuildInfoUIData(Seq(
+ "Version" -> org.apache.celeborn.ProjectConstants.VERSION,
+ "Commit Id" -> org.apache.celeborn.ProjectConstants.getGitCommitId,
+ "Revision" -> org.apache.celeborn.ProjectConstants.REVISION))
+ try {
+ kvStore.write(buildInfo)
Review Comment:
Done in ea7162a: the plugin now posts a `CelebornBuildInfoEvent` (same pattern as gluten's `GlutenBuildInfoEvent`), and the listener restores the build info from the same event both live and on replay. Covered in `CelebornListenerSuite`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2478] Add celeborn-client build information to Spark UI Celeborn tab [celeborn]
Posted by "zaynt4606 (via GitHub)" <gi...@apache.org>.
zaynt4606 commented on code in PR #3870:
URL: https://github.com/apache/celeborn/pull/3870#discussion_r4266170067
##########
common/pom.xml:
##########
@@ -192,6 +192,56 @@
<groupId>org.xolstice.maven.plugins</groupId>
<artifactId>protobuf-maven-plugin</artifactId>
</plugin>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>buildnumber-maven-plugin</artifactId>
+ <configuration>
+ <outputDirectory>${project.build.directory}/generated/build-metadata</outputDirectory>
+ <outputName>build.properties</outputName>
+ <revisionPropertyName>git.revision</revisionPropertyName>
Review Comment:
[P1] Allow builds when Git revision metadata is unavailable
This new goal makes Git metadata mandatory for building `celeborn-common`. Celeborn's source release is produced with `git archive`, so its extracted sources do not contain `.git`. I reproduced a build failure from an archive of this revision with:
```text
mvn -pl common -am -Dmaven.test.skip=true package
```
The build fails in `buildnumber-maven-plugin:1.4:create-metadata` with `NullPointerException` at `CreateMetadataMojo.java:188`. The currently failing [C++ integration CI](https://github.com/apache/celeborn/actions/runs/38397241592/job/115295484286) reports the same exception in this newly added goal.
Could we configure `revisionOnScmFailure` (for example, `N/A`, matching the SBT fallback) and add validation for a source archive without Git metadata? Adding `-Dmaven.buildNumber.revisionOnScmFailure=N/A` made the same archived-source package build pass.
##########
client-spark/spark-3-ui/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornPlugin.scala:
##########
@@ -49,10 +50,29 @@ private class CelebornDriverPlugin extends DriverPlugin with Logging {
logInfo("Initializing CelebornDriverPlugin...")
this.sc = sc
val kvStore = sc.statusStore.store
+ writeBuildInfo(kvStore)
new CelebornListener(kvStore, sc.conf).register(sc)
java.util.Collections.emptyMap[String, String]()
}
+ /**
+ * Persists the build info of the celeborn-client bundled with this application into the
+ * KVStore so that it survives application end and is available on History Server replay,
+ * where the Spark UI "Celeborn" tab renders it as "Celeborn Client Build Information".
+ */
+ private def writeBuildInfo(kvStore: KVStore): Unit = {
+ val buildInfo = new CelebornBuildInfoUIData(Seq(
+ "Version" -> org.apache.celeborn.ProjectConstants.VERSION,
+ "Commit Id" -> org.apache.celeborn.ProjectConstants.getGitCommitId,
+ "Revision" -> org.apache.celeborn.ProjectConstants.REVISION))
+ try {
+ kvStore.write(buildInfo)
Review Comment:
[P2] Record build information in the event log so History Server can restore it
Writing to the Driver's KVStore does not persist this entry in the application's event log. History Server constructs a separate store during replay, and neither `CelebornHistoryServerPlugin` nor `CelebornListener` restores `CelebornBuildInfoUIData`, so the new table is empty there despite the documented replay support.
I verified this with a real Spark 3.5.1 application using `CelebornPlugin` and event logging: the live store contained all 3 build-info entries. Replaying that application's log with `ReplayListenerBus` and the actual `CelebornHistoryServerPlugin` listeners produced `extensionEnabled=true` but 0 build-info entries.
Could we record the application's build information in a replayable event or environment properties, reconstruct it in the replay listener, and add an event-log round-trip test? The restored values need to come from the application, rather than the client artifact installed on History Server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2478] Add celeborn-client build information to Spark UI Celeborn tab [celeborn]
Posted by "zaynt4606 (via GitHub)" <gi...@apache.org>.
zaynt4606 commented on PR #3870:
URL: https://github.com/apache/celeborn/pull/3870#issuecomment-6306996389
Thanks @yew1eb for the contribution!
Merged to main(v1.0.0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2478] Add celeborn-client build information to Spark UI Celeborn tab [celeborn]
Posted by "yew1eb (via GitHub)" <gi...@apache.org>.
yew1eb commented on PR #3870:
URL: https://github.com/apache/celeborn/pull/3870#issuecomment-6287934115
@zaynt4606
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2478] Add celeborn-client build information to Spark UI Celeborn tab [celeborn]
Posted by "zaynt4606 (via GitHub)" <gi...@apache.org>.
zaynt4606 commented on PR #3870:
URL: https://github.com/apache/celeborn/pull/3870#issuecomment-6306977318
**Merge Summary:**
- merged into main https://github.com/apache/celeborn/commit/34f1ab5a
*Posted by `merge_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org