You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@celeborn.apache.org by Ethan Feng <et...@apache.org> on 2024/06/19 04:45:52 UTC
[VOTE] Release Apache Celeborn 0.5.0-rc3
Hello, Celeborn community,
This is a call for a vote to release Apache Celeborn
0.5.0-rc3
The git tag to be voted upon:
https://github.com/apache/celeborn/releases/tag/v0.5.0-rc3
Source and binary artifacts can be found at:
https://dist.apache.org/repos/dist/dev/celeborn/v0.5.0-rc3
The git commit hash:
048ef207359113247bff05dcc203c70021ccfa10
The staging repo:
https://repository.apache.org/content/repositories/orgapacheceleborn-1076/
The fingerprint of the PGP key release artifacts is signed with:
FCF20BB29C7BEFDF58F998F76392F71F37356FA0
My public key to verify signatures can be found in:
https://dist.apache.org/repos/dist/release/celeborn/KEYS
The vote will be open for at least 72 hours or until the necessary
number of votes are reached.
Please vote accordingly:
[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove (and the reason)
Steps to validate the release:
https://www.apache.org/info/verification.html
* Download links, checksums, and PGP signatures are valid.
* Source code distributions have correct names matching the current release.
* LICENSE and NOTICE files are correct.
* All files have license headers if necessary.
* No unlicensed compiled archives bundled in the source archive.
* The source tarball matches the git tag.
* Build from source is successful.
There are additional tests:
* Performance test no regression
1 TB TPC-DS, 0.5.0 VS 0.4.1 : 2042(s) VS 2050(s)
1.1 TB pure shuffle, 0.5.0 VS 0.4.1 : 11.8min vs 11.8min
* Result correctness test passed
1TB TPC-DS runs concurrently, the results are identical.
* Usability test passed
Rolling upgrade from version 0.4.1 to 0.5.0 succeed.
The metrics system works as expected.
* Stability test passed
Random worker failures, Celeborn works as expected.
Random master failures, Celeborn works as expected.
Master meta corrupted, Celeborn works as expected.
* Compatibility test passed
The Celeborn server version of 0.5.0 works fine with the Celeborn client 0.4.1.
* Grafana dashboard layout checked
Regards,
Ethan Feng
Re: [VOTE] Release Apache Celeborn 0.5.0-rc3
Posted by Ethan Feng <et...@apache.org>.
Hi Mridul,
The Celeborn dashboard will not be included in this version
release. Therefore, it should not pose any issues for us at this time.
After some investigation, I discovered that our CI tests are based
on a fixed list of projects, which is why this problem was not
detected earlier.
Thank you for bringing this to our attention. To bypass the lint
exceptions for unit tests, please use the following command:
```
SPARK_BINARY_VERSION=3.1
SPARK_MAJOR_VERSION=${SPARK_BINARY_VERSION%%.*}
PROFILES="-Pgoogle-mirror,spark-3.1"
TEST_MODULES="client-spark/common,client-spark/spark-${SPARK_MAJOR_VERSION},client-spark/spark-${SPARK_MAJOR_VERSION}-shaded,tests/spark-it"
build/mvn $PROFILES -pl $TEST_MODULES -am clean install -DskipTests
build/mvn $PROFILES -pl $TEST_MODULES
-Dspark.shuffle.sort.io.plugin.class=org.apache.spark.shuffle.sort.io.LocalDiskShuffleDataIO
test
```
Additionally, I have created a Jira ticket[0] to track this lint exception.
Thanks,
Ethan Feng
[0] https://issues.apache.org/jira/browse/CELEBORN-1480
Mridul Muralidharan <mr...@gmail.com> 于2024年6月24日周一 15:35写道：
>
> Forgot to update here.
>
> Signatures, digests, etc check out fine.
> Checked out tag and build/tested with "-Pspark3.1"
>
> I keep getting the following error:
>
> - metrics/prometheus *** FAILED ***
> 200 did not equal 404 (ApiBaseResourceSuite.scala:90)
> - metrics/json *** FAILED ***
> 200 did not equal 404 (ApiBaseResourceSuite.scala:96)
>
> In unit-tests.log, I have this [1]
>
> After explicitly ignoring these two tests, I then ran into lint issues with
> "web" submodule - and could not get around to debugging the issue (I
> installed pnpm - which does not seem to be called out in readme).
>
> Thanks,
> Mridul
>
> [1]
> 24/06/24 01:57:33,545 ERROR [ScalaTest-main-running-DiscoverySuite]
> MetricsConfig: Error loading configuration file
> file:/home/mridul/work/apache/vote/celeborn/service/target/celeborn-service_2.12-0.5.0-tests.jar!/metrics-api.properties
> java.io.FileNotFoundException:
> file:/home/mridul/work/apache/vote/celeborn/service/target/celeborn-service_2.12-0.5.0-tests.jar!/metrics-api.properties
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.<init>(FileInputStream.java:138)
> at java.io.FileInputStream.<init>(FileInputStream.java:93)
> at
> org.apache.celeborn.common.metrics.MetricsConfig.loadPropertiesFromFile(MetricsConfig.scala:95)
> at
> org.apache.celeborn.common.metrics.MetricsConfig.initialize(MetricsConfig.scala:50)
> at
> org.apache.celeborn.common.metrics.MetricsSystem.<init>(MetricsSystem.scala:53)
> at
> org.apache.celeborn.common.metrics.MetricsSystem$.createMetricsSystem(MetricsSystem.scala:197)
> at
> org.apache.celeborn.service.deploy.master.Master.<init>(Master.scala:66)
> at
> org.apache.celeborn.service.deploy.master.http.api.ApiMasterResourceSuite.beforeAll(ApiMasterResourceSuite.scala:54)
> at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
> at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
> at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
> at
> org.apache.celeborn.server.common.http.ApiBaseResourceSuite.run(ApiBaseResourceSuite.scala:23)
> at org.scalatest.Suite.callExecuteOnSuite$1(Suite.scala:1178)
> at org.scalatest.Suite.$anonfun$runNestedSuites$1(Suite.scala:1225)
> at
> scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
> at
> scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
> at
> scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
> at org.scalatest.Suite.runNestedSuites(Suite.scala:1223)
> at org.scalatest.Suite.runNestedSuites$(Suite.scala:1156)
> at
> org.scalatest.tools.DiscoverySuite.runNestedSuites(DiscoverySuite.scala:30)
> at org.scalatest.Suite.run(Suite.scala:1111)
> at org.scalatest.Suite.run$(Suite.scala:1096)
> at org.scalatest.tools.DiscoverySuite.run(DiscoverySuite.scala:30)
> at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:47)
> at
> org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1321)
> at
> org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1315)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at
> org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1315)
> at
> org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:992)
> at
> org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:970)
> at
> org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1481)
> at
> org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:970)
> at org.scalatest.tools.Runner$.main(Runner.scala:775)
> at org.scalatest.tools.Runner.main(Runner.scala)
>
>
> On Mon, Jun 24, 2024 at 2:22 AM Ethan Feng <et...@apache.org> wrote:
>
> > Thanks for your feedback, I will close this vote
> > thread and announce the results soon since 72 hours have passed.
> >
> >
> > Ethan Feng.
> >
> > kerwin zhang <ke...@gmail.com> 于2024年6月24日周一 15:18写道：
> > >
> > > +1 (binding)
> > >
> > > I checked
> > > - git commit hash is correct.
> > > - links are valid.
> > > - signatures are good.
> > > ```
> > > gpg --verify apache-celeborn-0.5.0-bin.tgz.asc
> > apache-celeborn-0.5.0-bin.tgz
> > > gpg --verify apache-celeborn-0.5.0-source.tgz.asc
> > > apache-celeborn-0.5.0-source.tgz
> > > ```
> > > - checksums are good.
> > > ```
> > > shasum -a 512 apache-celeborn-0.5.0-bin.tgz
> > > shasum -a 512 apache-celeborn-0.5.0-source.tgz
> > > ```
> > >
> > > - build success from source code (macOS).
> > >
> > > ```
> > > ./build/make-distribution.sh -Pspark-3.4
> > > ```
> > >
> > > Thanks,
> > > Kerwin Zhang
> > >
> > > Fu Chen <cf...@gmail.com> 于2024年6月24日周一 14:46写道：
> > > >
> > > > +1
> > > >
> > > > I checked
> > > > - download links are valid.
> > > > - git commit hash is correct.
> > > > - no binary files in the source release.
> > > > - build success from source code (JDK 17).
> > > > ```
> > > > ./build/make-distribution.sh --sbt-enabled -Pspark-3.5
> > > > ```
> > > >
> > > > Cheng Pan <pa...@gmail.com> 于2024年6月24日周一 14:09写道：
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > - All links are valid.
> > > > > - LICENSE/NOTICE looks fine.
> > > > > - CI result looks great.
> > > > > - No known blockers.
> > > > >
> > > > > Thanks,
> > > > > Cheng Pan
> > > > >
> > > > >
> > > > > > On Jun 19, 2024, at 12:45, Ethan Feng <et...@apache.org>
> > wrote:
> > > > > >
> > > > > > Hello, Celeborn community,
> > > > > >
> > > > > > This is a call for a vote to release Apache Celeborn
> > > > > > 0.5.0-rc3
> > > > > >
> > > > > > The git tag to be voted upon:
> > > > > > https://github.com/apache/celeborn/releases/tag/v0.5.0-rc3
> > > > > >
> > > > > > Source and binary artifacts can be found at:
> > > > > > https://dist.apache.org/repos/dist/dev/celeborn/v0.5.0-rc3
> > > > > >
> > > > > > The git commit hash:
> > > > > > 048ef207359113247bff05dcc203c70021ccfa10
> > > > > >
> > > > > > The staging repo:
> > > > > >
> > https://repository.apache.org/content/repositories/orgapacheceleborn-1076/
> > > > > >
> > > > > > The fingerprint of the PGP key release artifacts is signed with:
> > > > > > FCF20BB29C7BEFDF58F998F76392F71F37356FA0
> > > > > >
> > > > > > My public key to verify signatures can be found in:
> > > > > > https://dist.apache.org/repos/dist/release/celeborn/KEYS
> > > > > >
> > > > > > The vote will be open for at least 72 hours or until the necessary
> > > > > > number of votes are reached.
> > > > > >
> > > > > > Please vote accordingly:
> > > > > >
> > > > > > [ ] +1 approve
> > > > > > [ ] +0 no opinion
> > > > > > [ ] -1 disapprove (and the reason)
> > > > > >
> > > > > > Steps to validate the release:
> > > > > > https://www.apache.org/info/verification.html
> > > > > >
> > > > > > * Download links, checksums, and PGP signatures are valid.
> > > > > > * Source code distributions have correct names matching the
> > current release.
> > > > > > * LICENSE and NOTICE files are correct.
> > > > > > * All files have license headers if necessary.
> > > > > > * No unlicensed compiled archives bundled in the source archive.
> > > > > > * The source tarball matches the git tag.
> > > > > > * Build from source is successful.
> > > > > >
> > > > > > There are additional tests:
> > > > > > * Performance test no regression
> > > > > > 1 TB TPC-DS, 0.5.0 VS 0.4.1 : 2042(s) VS 2050(s)
> > > > > > 1.1 TB pure shuffle, 0.5.0 VS 0.4.1 : 11.8min vs 11.8min
> > > > > >
> > > > > > * Result correctness test passed
> > > > > > 1TB TPC-DS runs concurrently, the results are identical.
> > > > > >
> > > > > > * Usability test passed
> > > > > > Rolling upgrade from version 0.4.1 to 0.5.0 succeed.
> > > > > > The metrics system works as expected.
> > > > > >
> > > > > > * Stability test passed
> > > > > > Random worker failures, Celeborn works as expected.
> > > > > > Random master failures, Celeborn works as expected.
> > > > > > Master meta corrupted, Celeborn works as expected.
> > > > > >
> > > > > > * Compatibility test passed
> > > > > > The Celeborn server version of 0.5.0 works fine with the Celeborn
> > client 0.4.1.
> > > > > >
> > > > > > * Grafana dashboard layout checked
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > > Ethan Feng
> > > > >
> >
Re: [VOTE] Release Apache Celeborn 0.5.0-rc3
Posted by Mridul Muralidharan <mr...@gmail.com>.
Forgot to update here.
Signatures, digests, etc check out fine.
Checked out tag and build/tested with "-Pspark3.1"
I keep getting the following error:
- metrics/prometheus *** FAILED ***
200 did not equal 404 (ApiBaseResourceSuite.scala:90)
- metrics/json *** FAILED ***
200 did not equal 404 (ApiBaseResourceSuite.scala:96)
In unit-tests.log, I have this [1]
After explicitly ignoring these two tests, I then ran into lint issues with
"web" submodule - and could not get around to debugging the issue (I
installed pnpm - which does not seem to be called out in readme).
Thanks,
Mridul
[1]
24/06/24 01:57:33,545 ERROR [ScalaTest-main-running-DiscoverySuite]
MetricsConfig: Error loading configuration file
file:/home/mridul/work/apache/vote/celeborn/service/target/celeborn-service_2.12-0.5.0-tests.jar!/metrics-api.properties
java.io.FileNotFoundException:
file:/home/mridul/work/apache/vote/celeborn/service/target/celeborn-service_2.12-0.5.0-tests.jar!/metrics-api.properties
(No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at
org.apache.celeborn.common.metrics.MetricsConfig.loadPropertiesFromFile(MetricsConfig.scala:95)
at
org.apache.celeborn.common.metrics.MetricsConfig.initialize(MetricsConfig.scala:50)
at
org.apache.celeborn.common.metrics.MetricsSystem.<init>(MetricsSystem.scala:53)
at
org.apache.celeborn.common.metrics.MetricsSystem$.createMetricsSystem(MetricsSystem.scala:197)
at
org.apache.celeborn.service.deploy.master.Master.<init>(Master.scala:66)
at
org.apache.celeborn.service.deploy.master.http.api.ApiMasterResourceSuite.beforeAll(ApiMasterResourceSuite.scala:54)
at
org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
at
org.apache.celeborn.server.common.http.ApiBaseResourceSuite.run(ApiBaseResourceSuite.scala:23)
at org.scalatest.Suite.callExecuteOnSuite$1(Suite.scala:1178)
at org.scalatest.Suite.$anonfun$runNestedSuites$1(Suite.scala:1225)
at
scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at
scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at org.scalatest.Suite.runNestedSuites(Suite.scala:1223)
at org.scalatest.Suite.runNestedSuites$(Suite.scala:1156)
at
org.scalatest.tools.DiscoverySuite.runNestedSuites(DiscoverySuite.scala:30)
at org.scalatest.Suite.run(Suite.scala:1111)
at org.scalatest.Suite.run$(Suite.scala:1096)
at org.scalatest.tools.DiscoverySuite.run(DiscoverySuite.scala:30)
at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:47)
at
org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1321)
at
org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1315)
at scala.collection.immutable.List.foreach(List.scala:392)
at
org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1315)
at
org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:992)
at
org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:970)
at
org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1481)
at
org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:970)
at org.scalatest.tools.Runner$.main(Runner.scala:775)
at org.scalatest.tools.Runner.main(Runner.scala)
On Mon, Jun 24, 2024 at 2:22 AM Ethan Feng <et...@apache.org> wrote:
> Thanks for your feedback, I will close this vote
> thread and announce the results soon since 72 hours have passed.
>
>
> Ethan Feng.
>
> kerwin zhang <ke...@gmail.com> 于2024年6月24日周一 15:18写道：
> >
> > +1 (binding)
> >
> > I checked
> > - git commit hash is correct.
> > - links are valid.
> > - signatures are good.
> > ```
> > gpg --verify apache-celeborn-0.5.0-bin.tgz.asc
> apache-celeborn-0.5.0-bin.tgz
> > gpg --verify apache-celeborn-0.5.0-source.tgz.asc
> > apache-celeborn-0.5.0-source.tgz
> > ```
> > - checksums are good.
> > ```
> > shasum -a 512 apache-celeborn-0.5.0-bin.tgz
> > shasum -a 512 apache-celeborn-0.5.0-source.tgz
> > ```
> >
> > - build success from source code (macOS).
> >
> > ```
> > ./build/make-distribution.sh -Pspark-3.4
> > ```
> >
> > Thanks,
> > Kerwin Zhang
> >
> > Fu Chen <cf...@gmail.com> 于2024年6月24日周一 14:46写道：
> > >
> > > +1
> > >
> > > I checked
> > > - download links are valid.
> > > - git commit hash is correct.
> > > - no binary files in the source release.
> > > - build success from source code (JDK 17).
> > > ```
> > > ./build/make-distribution.sh --sbt-enabled -Pspark-3.5
> > > ```
> > >
> > > Cheng Pan <pa...@gmail.com> 于2024年6月24日周一 14:09写道：
> > > >
> > > > +1 (binding)
> > > >
> > > > - All links are valid.
> > > > - LICENSE/NOTICE looks fine.
> > > > - CI result looks great.
> > > > - No known blockers.
> > > >
> > > > Thanks,
> > > > Cheng Pan
> > > >
> > > >
> > > > > On Jun 19, 2024, at 12:45, Ethan Feng <et...@apache.org>
> wrote:
> > > > >
> > > > > Hello, Celeborn community,
> > > > >
> > > > > This is a call for a vote to release Apache Celeborn
> > > > > 0.5.0-rc3
> > > > >
> > > > > The git tag to be voted upon:
> > > > > https://github.com/apache/celeborn/releases/tag/v0.5.0-rc3
> > > > >
> > > > > Source and binary artifacts can be found at:
> > > > > https://dist.apache.org/repos/dist/dev/celeborn/v0.5.0-rc3
> > > > >
> > > > > The git commit hash:
> > > > > 048ef207359113247bff05dcc203c70021ccfa10
> > > > >
> > > > > The staging repo:
> > > > >
> https://repository.apache.org/content/repositories/orgapacheceleborn-1076/
> > > > >
> > > > > The fingerprint of the PGP key release artifacts is signed with:
> > > > > FCF20BB29C7BEFDF58F998F76392F71F37356FA0
> > > > >
> > > > > My public key to verify signatures can be found in:
> > > > > https://dist.apache.org/repos/dist/release/celeborn/KEYS
> > > > >
> > > > > The vote will be open for at least 72 hours or until the necessary
> > > > > number of votes are reached.
> > > > >
> > > > > Please vote accordingly:
> > > > >
> > > > > [ ] +1 approve
> > > > > [ ] +0 no opinion
> > > > > [ ] -1 disapprove (and the reason)
> > > > >
> > > > > Steps to validate the release:
> > > > > https://www.apache.org/info/verification.html
> > > > >
> > > > > * Download links, checksums, and PGP signatures are valid.
> > > > > * Source code distributions have correct names matching the
> current release.
> > > > > * LICENSE and NOTICE files are correct.
> > > > > * All files have license headers if necessary.
> > > > > * No unlicensed compiled archives bundled in the source archive.
> > > > > * The source tarball matches the git tag.
> > > > > * Build from source is successful.
> > > > >
> > > > > There are additional tests:
> > > > > * Performance test no regression
> > > > > 1 TB TPC-DS, 0.5.0 VS 0.4.1 : 2042(s) VS 2050(s)
> > > > > 1.1 TB pure shuffle, 0.5.0 VS 0.4.1 : 11.8min vs 11.8min
> > > > >
> > > > > * Result correctness test passed
> > > > > 1TB TPC-DS runs concurrently, the results are identical.
> > > > >
> > > > > * Usability test passed
> > > > > Rolling upgrade from version 0.4.1 to 0.5.0 succeed.
> > > > > The metrics system works as expected.
> > > > >
> > > > > * Stability test passed
> > > > > Random worker failures, Celeborn works as expected.
> > > > > Random master failures, Celeborn works as expected.
> > > > > Master meta corrupted, Celeborn works as expected.
> > > > >
> > > > > * Compatibility test passed
> > > > > The Celeborn server version of 0.5.0 works fine with the Celeborn
> client 0.4.1.
> > > > >
> > > > > * Grafana dashboard layout checked
> > > > >
> > > > >
> > > > > Regards,
> > > > > Ethan Feng
> > > >
>
Re: [VOTE] Release Apache Celeborn 0.5.0-rc3
Posted by Ethan Feng <et...@apache.org>.
Thanks for your feedback, I will close this vote
thread and announce the results soon since 72 hours have passed.
Ethan Feng.
kerwin zhang <ke...@gmail.com> 于2024年6月24日周一 15:18写道：
>
> +1 (binding)
>
> I checked
> - git commit hash is correct.
> - links are valid.
> - signatures are good.
> ```
> gpg --verify apache-celeborn-0.5.0-bin.tgz.asc apache-celeborn-0.5.0-bin.tgz
> gpg --verify apache-celeborn-0.5.0-source.tgz.asc
> apache-celeborn-0.5.0-source.tgz
> ```
> - checksums are good.
> ```
> shasum -a 512 apache-celeborn-0.5.0-bin.tgz
> shasum -a 512 apache-celeborn-0.5.0-source.tgz
> ```
>
> - build success from source code (macOS).
>
> ```
> ./build/make-distribution.sh -Pspark-3.4
> ```
>
> Thanks,
> Kerwin Zhang
>
> Fu Chen <cf...@gmail.com> 于2024年6月24日周一 14:46写道：
> >
> > +1
> >
> > I checked
> > - download links are valid.
> > - git commit hash is correct.
> > - no binary files in the source release.
> > - build success from source code (JDK 17).
> > ```
> > ./build/make-distribution.sh --sbt-enabled -Pspark-3.5
> > ```
> >
> > Cheng Pan <pa...@gmail.com> 于2024年6月24日周一 14:09写道：
> > >
> > > +1 (binding)
> > >
> > > - All links are valid.
> > > - LICENSE/NOTICE looks fine.
> > > - CI result looks great.
> > > - No known blockers.
> > >
> > > Thanks,
> > > Cheng Pan
> > >
> > >
> > > > On Jun 19, 2024, at 12:45, Ethan Feng <et...@apache.org> wrote:
> > > >
> > > > Hello, Celeborn community,
> > > >
> > > > This is a call for a vote to release Apache Celeborn
> > > > 0.5.0-rc3
> > > >
> > > > The git tag to be voted upon:
> > > > https://github.com/apache/celeborn/releases/tag/v0.5.0-rc3
> > > >
> > > > Source and binary artifacts can be found at:
> > > > https://dist.apache.org/repos/dist/dev/celeborn/v0.5.0-rc3
> > > >
> > > > The git commit hash:
> > > > 048ef207359113247bff05dcc203c70021ccfa10
> > > >
> > > > The staging repo:
> > > > https://repository.apache.org/content/repositories/orgapacheceleborn-1076/
> > > >
> > > > The fingerprint of the PGP key release artifacts is signed with:
> > > > FCF20BB29C7BEFDF58F998F76392F71F37356FA0
> > > >
> > > > My public key to verify signatures can be found in:
> > > > https://dist.apache.org/repos/dist/release/celeborn/KEYS
> > > >
> > > > The vote will be open for at least 72 hours or until the necessary
> > > > number of votes are reached.
> > > >
> > > > Please vote accordingly:
> > > >
> > > > [ ] +1 approve
> > > > [ ] +0 no opinion
> > > > [ ] -1 disapprove (and the reason)
> > > >
> > > > Steps to validate the release:
> > > > https://www.apache.org/info/verification.html
> > > >
> > > > * Download links, checksums, and PGP signatures are valid.
> > > > * Source code distributions have correct names matching the current release.
> > > > * LICENSE and NOTICE files are correct.
> > > > * All files have license headers if necessary.
> > > > * No unlicensed compiled archives bundled in the source archive.
> > > > * The source tarball matches the git tag.
> > > > * Build from source is successful.
> > > >
> > > > There are additional tests:
> > > > * Performance test no regression
> > > > 1 TB TPC-DS, 0.5.0 VS 0.4.1 : 2042(s) VS 2050(s)
> > > > 1.1 TB pure shuffle, 0.5.0 VS 0.4.1 : 11.8min vs 11.8min
> > > >
> > > > * Result correctness test passed
> > > > 1TB TPC-DS runs concurrently, the results are identical.
> > > >
> > > > * Usability test passed
> > > > Rolling upgrade from version 0.4.1 to 0.5.0 succeed.
> > > > The metrics system works as expected.
> > > >
> > > > * Stability test passed
> > > > Random worker failures, Celeborn works as expected.
> > > > Random master failures, Celeborn works as expected.
> > > > Master meta corrupted, Celeborn works as expected.
> > > >
> > > > * Compatibility test passed
> > > > The Celeborn server version of 0.5.0 works fine with the Celeborn client 0.4.1.
> > > >
> > > > * Grafana dashboard layout checked
> > > >
> > > >
> > > > Regards,
> > > > Ethan Feng
> > >
Re: [VOTE] Release Apache Celeborn 0.5.0-rc3
Posted by kerwin zhang <ke...@gmail.com>.
+1 (binding)
I checked
- git commit hash is correct.
- links are valid.
- signatures are good.
```
gpg --verify apache-celeborn-0.5.0-bin.tgz.asc apache-celeborn-0.5.0-bin.tgz
gpg --verify apache-celeborn-0.5.0-source.tgz.asc
apache-celeborn-0.5.0-source.tgz
```
- checksums are good.
```
shasum -a 512 apache-celeborn-0.5.0-bin.tgz
shasum -a 512 apache-celeborn-0.5.0-source.tgz
```
- build success from source code (macOS).
```
./build/make-distribution.sh -Pspark-3.4
```
Thanks,
Kerwin Zhang
Fu Chen <cf...@gmail.com> 于2024年6月24日周一 14:46写道：
>
> +1
>
> I checked
> - download links are valid.
> - git commit hash is correct.
> - no binary files in the source release.
> - build success from source code (JDK 17).
> ```
> ./build/make-distribution.sh --sbt-enabled -Pspark-3.5
> ```
>
> Cheng Pan <pa...@gmail.com> 于2024年6月24日周一 14:09写道：
> >
> > +1 (binding)
> >
> > - All links are valid.
> > - LICENSE/NOTICE looks fine.
> > - CI result looks great.
> > - No known blockers.
> >
> > Thanks,
> > Cheng Pan
> >
> >
> > > On Jun 19, 2024, at 12:45, Ethan Feng <et...@apache.org> wrote:
> > >
> > > Hello, Celeborn community,
> > >
> > > This is a call for a vote to release Apache Celeborn
> > > 0.5.0-rc3
> > >
> > > The git tag to be voted upon:
> > > https://github.com/apache/celeborn/releases/tag/v0.5.0-rc3
> > >
> > > Source and binary artifacts can be found at:
> > > https://dist.apache.org/repos/dist/dev/celeborn/v0.5.0-rc3
> > >
> > > The git commit hash:
> > > 048ef207359113247bff05dcc203c70021ccfa10
> > >
> > > The staging repo:
> > > https://repository.apache.org/content/repositories/orgapacheceleborn-1076/
> > >
> > > The fingerprint of the PGP key release artifacts is signed with:
> > > FCF20BB29C7BEFDF58F998F76392F71F37356FA0
> > >
> > > My public key to verify signatures can be found in:
> > > https://dist.apache.org/repos/dist/release/celeborn/KEYS
> > >
> > > The vote will be open for at least 72 hours or until the necessary
> > > number of votes are reached.
> > >
> > > Please vote accordingly:
> > >
> > > [ ] +1 approve
> > > [ ] +0 no opinion
> > > [ ] -1 disapprove (and the reason)
> > >
> > > Steps to validate the release:
> > > https://www.apache.org/info/verification.html
> > >
> > > * Download links, checksums, and PGP signatures are valid.
> > > * Source code distributions have correct names matching the current release.
> > > * LICENSE and NOTICE files are correct.
> > > * All files have license headers if necessary.
> > > * No unlicensed compiled archives bundled in the source archive.
> > > * The source tarball matches the git tag.
> > > * Build from source is successful.
> > >
> > > There are additional tests:
> > > * Performance test no regression
> > > 1 TB TPC-DS, 0.5.0 VS 0.4.1 : 2042(s) VS 2050(s)
> > > 1.1 TB pure shuffle, 0.5.0 VS 0.4.1 : 11.8min vs 11.8min
> > >
> > > * Result correctness test passed
> > > 1TB TPC-DS runs concurrently, the results are identical.
> > >
> > > * Usability test passed
> > > Rolling upgrade from version 0.4.1 to 0.5.0 succeed.
> > > The metrics system works as expected.
> > >
> > > * Stability test passed
> > > Random worker failures, Celeborn works as expected.
> > > Random master failures, Celeborn works as expected.
> > > Master meta corrupted, Celeborn works as expected.
> > >
> > > * Compatibility test passed
> > > The Celeborn server version of 0.5.0 works fine with the Celeborn client 0.4.1.
> > >
> > > * Grafana dashboard layout checked
> > >
> > >
> > > Regards,
> > > Ethan Feng
> >
Re: [VOTE] Release Apache Celeborn 0.5.0-rc3
Posted by Fu Chen <cf...@gmail.com>.
+1
I checked
- download links are valid.
- git commit hash is correct.
- no binary files in the source release.
- build success from source code (JDK 17).
```
./build/make-distribution.sh --sbt-enabled -Pspark-3.5
```
Cheng Pan <pa...@gmail.com> 于2024年6月24日周一 14:09写道：
>
> +1 (binding)
>
> - All links are valid.
> - LICENSE/NOTICE looks fine.
> - CI result looks great.
> - No known blockers.
>
> Thanks,
> Cheng Pan
>
>
> > On Jun 19, 2024, at 12:45, Ethan Feng <et...@apache.org> wrote:
> >
> > Hello, Celeborn community,
> >
> > This is a call for a vote to release Apache Celeborn
> > 0.5.0-rc3
> >
> > The git tag to be voted upon:
> > https://github.com/apache/celeborn/releases/tag/v0.5.0-rc3
> >
> > Source and binary artifacts can be found at:
> > https://dist.apache.org/repos/dist/dev/celeborn/v0.5.0-rc3
> >
> > The git commit hash:
> > 048ef207359113247bff05dcc203c70021ccfa10
> >
> > The staging repo:
> > https://repository.apache.org/content/repositories/orgapacheceleborn-1076/
> >
> > The fingerprint of the PGP key release artifacts is signed with:
> > FCF20BB29C7BEFDF58F998F76392F71F37356FA0
> >
> > My public key to verify signatures can be found in:
> > https://dist.apache.org/repos/dist/release/celeborn/KEYS
> >
> > The vote will be open for at least 72 hours or until the necessary
> > number of votes are reached.
> >
> > Please vote accordingly:
> >
> > [ ] +1 approve
> > [ ] +0 no opinion
> > [ ] -1 disapprove (and the reason)
> >
> > Steps to validate the release:
> > https://www.apache.org/info/verification.html
> >
> > * Download links, checksums, and PGP signatures are valid.
> > * Source code distributions have correct names matching the current release.
> > * LICENSE and NOTICE files are correct.
> > * All files have license headers if necessary.
> > * No unlicensed compiled archives bundled in the source archive.
> > * The source tarball matches the git tag.
> > * Build from source is successful.
> >
> > There are additional tests:
> > * Performance test no regression
> > 1 TB TPC-DS, 0.5.0 VS 0.4.1 : 2042(s) VS 2050(s)
> > 1.1 TB pure shuffle, 0.5.0 VS 0.4.1 : 11.8min vs 11.8min
> >
> > * Result correctness test passed
> > 1TB TPC-DS runs concurrently, the results are identical.
> >
> > * Usability test passed
> > Rolling upgrade from version 0.4.1 to 0.5.0 succeed.
> > The metrics system works as expected.
> >
> > * Stability test passed
> > Random worker failures, Celeborn works as expected.
> > Random master failures, Celeborn works as expected.
> > Master meta corrupted, Celeborn works as expected.
> >
> > * Compatibility test passed
> > The Celeborn server version of 0.5.0 works fine with the Celeborn client 0.4.1.
> >
> > * Grafana dashboard layout checked
> >
> >
> > Regards,
> > Ethan Feng
>
Re: [VOTE] Release Apache Celeborn 0.5.0-rc3
Posted by Cheng Pan <pa...@gmail.com>.
+1 (binding)
- All links are valid.
- LICENSE/NOTICE looks fine.
- CI result looks great.
- No known blockers.
Thanks,
Cheng Pan
> On Jun 19, 2024, at 12:45, Ethan Feng <et...@apache.org> wrote:
>
> Hello, Celeborn community,
>
> This is a call for a vote to release Apache Celeborn
> 0.5.0-rc3
>
> The git tag to be voted upon:
> https://github.com/apache/celeborn/releases/tag/v0.5.0-rc3
>
> Source and binary artifacts can be found at:
> https://dist.apache.org/repos/dist/dev/celeborn/v0.5.0-rc3
>
> The git commit hash:
> 048ef207359113247bff05dcc203c70021ccfa10
>
> The staging repo:
> https://repository.apache.org/content/repositories/orgapacheceleborn-1076/
>
> The fingerprint of the PGP key release artifacts is signed with:
> FCF20BB29C7BEFDF58F998F76392F71F37356FA0
>
> My public key to verify signatures can be found in:
> https://dist.apache.org/repos/dist/release/celeborn/KEYS
>
> The vote will be open for at least 72 hours or until the necessary
> number of votes are reached.
>
> Please vote accordingly:
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove (and the reason)
>
> Steps to validate the release:
> https://www.apache.org/info/verification.html
>
> * Download links, checksums, and PGP signatures are valid.
> * Source code distributions have correct names matching the current release.
> * LICENSE and NOTICE files are correct.
> * All files have license headers if necessary.
> * No unlicensed compiled archives bundled in the source archive.
> * The source tarball matches the git tag.
> * Build from source is successful.
>
> There are additional tests:
> * Performance test no regression
> 1 TB TPC-DS, 0.5.0 VS 0.4.1 : 2042(s) VS 2050(s)
> 1.1 TB pure shuffle, 0.5.0 VS 0.4.1 : 11.8min vs 11.8min
>
> * Result correctness test passed
> 1TB TPC-DS runs concurrently, the results are identical.
>
> * Usability test passed
> Rolling upgrade from version 0.4.1 to 0.5.0 succeed.
> The metrics system works as expected.
>
> * Stability test passed
> Random worker failures, Celeborn works as expected.
> Random master failures, Celeborn works as expected.
> Master meta corrupted, Celeborn works as expected.
>
> * Compatibility test passed
> The Celeborn server version of 0.5.0 works fine with the Celeborn client 0.4.1.
>
> * Grafana dashboard layout checked
>
>
> Regards,
> Ethan Feng
Re: [VOTE] Release Apache Celeborn 0.5.0-rc3
Posted by Keyong Zhou <zh...@apache.org>.
+1 (binding)
I checked
- git commit hash is correct.
- links are valid.
- signatures are good.
```
gpg --import KEYS
gpg --verify apache-celeborn-0.5.0-bin.tgz.asc
gpg --verify apache-celeborn-0.5.0-source.tgz.asc
```
- checksums are good.
```
sha512sum --check apache-celeborn-0.5.0-bin.tgz.sha512
sha512sum --check apache-celeborn-0.5.0-source.tgz.sha512
```
- LICENSE looks good.
- NOTICE looks good.
- build success from source code (macOS).
```
./build/make-distribution.sh --sbt-enabled --release
```
BTW, thanks for the rich tests!
Regards,
Keyong Zhou
Ethan Feng <et...@apache.org> 于2024年6月19日周三 12:47写道：
> Hello, Celeborn community,
>
> This is a call for a vote to release Apache Celeborn
> 0.5.0-rc3
>
> The git tag to be voted upon:
> https://github.com/apache/celeborn/releases/tag/v0.5.0-rc3
>
> Source and binary artifacts can be found at:
> https://dist.apache.org/repos/dist/dev/celeborn/v0.5.0-rc3
>
> The git commit hash:
> 048ef207359113247bff05dcc203c70021ccfa10
>
> The staging repo:
> https://repository.apache.org/content/repositories/orgapacheceleborn-1076/
>
> The fingerprint of the PGP key release artifacts is signed with:
> FCF20BB29C7BEFDF58F998F76392F71F37356FA0
>
> My public key to verify signatures can be found in:
> https://dist.apache.org/repos/dist/release/celeborn/KEYS
>
> The vote will be open for at least 72 hours or until the necessary
> number of votes are reached.
>
> Please vote accordingly:
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove (and the reason)
>
> Steps to validate the release:
> https://www.apache.org/info/verification.html
>
> * Download links, checksums, and PGP signatures are valid.
> * Source code distributions have correct names matching the current
> release.
> * LICENSE and NOTICE files are correct.
> * All files have license headers if necessary.
> * No unlicensed compiled archives bundled in the source archive.
> * The source tarball matches the git tag.
> * Build from source is successful.
>
> There are additional tests:
> * Performance test no regression
> 1 TB TPC-DS, 0.5.0 VS 0.4.1 : 2042(s) VS 2050(s)
> 1.1 TB pure shuffle, 0.5.0 VS 0.4.1 : 11.8min vs 11.8min
>
> * Result correctness test passed
> 1TB TPC-DS runs concurrently, the results are identical.
>
> * Usability test passed
> Rolling upgrade from version 0.4.1 to 0.5.0 succeed.
> The metrics system works as expected.
>
> * Stability test passed
> Random worker failures, Celeborn works as expected.
> Random master failures, Celeborn works as expected.
> Master meta corrupted, Celeborn works as expected.
>
> * Compatibility test passed
> The Celeborn server version of 0.5.0 works fine with the Celeborn client
> 0.4.1.
>
> * Grafana dashboard layout checked
>
>
> Regards,
> Ethan Feng
>
Re: [VOTE] Release Apache Celeborn 0.5.0-rc3
Posted by rexxiong <re...@apache.org>.
+1 (binding)
I checked
- Download links are valid.
- git commit hash is correct
- Checksums and signatures are valid.
- No binary files in the source release
- Successfully built the binary from the source on MacOs with Command:
./build/make-distribution.sh -Pspark-3.3
BTW，The testing work of this version seems to be very nice! Thanks!
Thanks,
Jiashu Xiong
Nicholas Jiang <ni...@apache.org> 于2024年6月22日周六 19:32写道：
> +1 (non-binding)
>
> I have checked:
>
> - Download links are valid.
> - git commit hash is correct
> - Checksums and signatures are valid.
> - No binary files in the source release
> - Successfully built binary from source via ./build/make-distribution.sh
> -Pspark-3.5/flink-1.19/mr
>
> BTW, I have tested the clean performance of Worker for 0.5.0-rc3. It works
> well and there is no backlog for clean task queue.
>
> Regards,
> Nicholas Jiang
>
> On 2024/06/19 04:45:52 Ethan Feng wrote:
> > Hello, Celeborn community,
> >
> > This is a call for a vote to release Apache Celeborn
> > 0.5.0-rc3
> >
> > The git tag to be voted upon:
> > https://github.com/apache/celeborn/releases/tag/v0.5.0-rc3
> >
> > Source and binary artifacts can be found at:
> > https://dist.apache.org/repos/dist/dev/celeborn/v0.5.0-rc3
> >
> > The git commit hash:
> > 048ef207359113247bff05dcc203c70021ccfa10
> >
> > The staging repo:
> >
> https://repository.apache.org/content/repositories/orgapacheceleborn-1076/
> >
> > The fingerprint of the PGP key release artifacts is signed with:
> > FCF20BB29C7BEFDF58F998F76392F71F37356FA0
> >
> > My public key to verify signatures can be found in:
> > https://dist.apache.org/repos/dist/release/celeborn/KEYS
> >
> > The vote will be open for at least 72 hours or until the necessary
> > number of votes are reached.
> >
> > Please vote accordingly:
> >
> > [ ] +1 approve
> > [ ] +0 no opinion
> > [ ] -1 disapprove (and the reason)
> >
> > Steps to validate the release:
> > https://www.apache.org/info/verification.html
> >
> > * Download links, checksums, and PGP signatures are valid.
> > * Source code distributions have correct names matching the current
> release.
> > * LICENSE and NOTICE files are correct.
> > * All files have license headers if necessary.
> > * No unlicensed compiled archives bundled in the source archive.
> > * The source tarball matches the git tag.
> > * Build from source is successful.
> >
> > There are additional tests:
> > * Performance test no regression
> > 1 TB TPC-DS, 0.5.0 VS 0.4.1 : 2042(s) VS 2050(s)
> > 1.1 TB pure shuffle, 0.5.0 VS 0.4.1 : 11.8min vs 11.8min
> >
> > * Result correctness test passed
> > 1TB TPC-DS runs concurrently, the results are identical.
> >
> > * Usability test passed
> > Rolling upgrade from version 0.4.1 to 0.5.0 succeed.
> > The metrics system works as expected.
> >
> > * Stability test passed
> > Random worker failures, Celeborn works as expected.
> > Random master failures, Celeborn works as expected.
> > Master meta corrupted, Celeborn works as expected.
> >
> > * Compatibility test passed
> > The Celeborn server version of 0.5.0 works fine with the Celeborn client
> 0.4.1.
> >
> > * Grafana dashboard layout checked
> >
> >
> > Regards,
> > Ethan Feng
> >
>
Re: [VOTE] Release Apache Celeborn 0.5.0-rc3
Posted by Nicholas Jiang <ni...@apache.org>.
+1 (non-binding)
I have checked:
- Download links are valid.
- git commit hash is correct
- Checksums and signatures are valid.
- No binary files in the source release
- Successfully built binary from source via ./build/make-distribution.sh -Pspark-3.5/flink-1.19/mr
BTW, I have tested the clean performance of Worker for 0.5.0-rc3. It works well and there is no backlog for clean task queue.
Regards,
Nicholas Jiang
On 2024/06/19 04:45:52 Ethan Feng wrote:
> Hello, Celeborn community,
>
> This is a call for a vote to release Apache Celeborn
> 0.5.0-rc3
>
> The git tag to be voted upon:
> https://github.com/apache/celeborn/releases/tag/v0.5.0-rc3
>
> Source and binary artifacts can be found at:
> https://dist.apache.org/repos/dist/dev/celeborn/v0.5.0-rc3
>
> The git commit hash:
> 048ef207359113247bff05dcc203c70021ccfa10
>
> The staging repo:
> https://repository.apache.org/content/repositories/orgapacheceleborn-1076/
>
> The fingerprint of the PGP key release artifacts is signed with:
> FCF20BB29C7BEFDF58F998F76392F71F37356FA0
>
> My public key to verify signatures can be found in:
> https://dist.apache.org/repos/dist/release/celeborn/KEYS
>
> The vote will be open for at least 72 hours or until the necessary
> number of votes are reached.
>
> Please vote accordingly:
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove (and the reason)
>
> Steps to validate the release:
> https://www.apache.org/info/verification.html
>
> * Download links, checksums, and PGP signatures are valid.
> * Source code distributions have correct names matching the current release.
> * LICENSE and NOTICE files are correct.
> * All files have license headers if necessary.
> * No unlicensed compiled archives bundled in the source archive.
> * The source tarball matches the git tag.
> * Build from source is successful.
>
> There are additional tests:
> * Performance test no regression
> 1 TB TPC-DS, 0.5.0 VS 0.4.1 : 2042(s) VS 2050(s)
> 1.1 TB pure shuffle, 0.5.0 VS 0.4.1 : 11.8min vs 11.8min
>
> * Result correctness test passed
> 1TB TPC-DS runs concurrently, the results are identical.
>
> * Usability test passed
> Rolling upgrade from version 0.4.1 to 0.5.0 succeed.
> The metrics system works as expected.
>
> * Stability test passed
> Random worker failures, Celeborn works as expected.
> Random master failures, Celeborn works as expected.
> Master meta corrupted, Celeborn works as expected.
>
> * Compatibility test passed
> The Celeborn server version of 0.5.0 works fine with the Celeborn client 0.4.1.
>
> * Grafana dashboard layout checked
>
>
> Regards,
> Ethan Feng
>