You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@celeborn.apache.org by Zhentao Shuai <za...@apache.org> on 2026/07/22 08:02:15 UTC
[DISCUSS] Time for 0.7.0 release and looking ahead to 1.0.0
Hi Celeborn community,
I'd like to kick off the 0.7.0 release now that we're at the end of July.
It has been about 10 months since 0.6.0, and master has accumulated a substantial set of new features, improvements, and fixes since 0.6.3 alone. Highlights include:
● Engine support. Spark 4.1/4.2 and Flink 2.1/2.2/2.3 support; Spark 4.0 and Flink 2.0 client maturity. Removed Spark 2.4, Flink 1.16/1.17, and Hadoop 2 client support.
● C++ native client (CIP-14). Full push/merge data path, revive, LZ4/ZSTD compression, retry logic, heartbeat decoding, native transport, and C++-write/Java-read hybrid integration tests — a major push toward feature parity with the Java client.
● Encryption at Rest (CIP-22). Spark-side implementation landed.
● Shuffle integrity. End-to-end integrity check for Flink, iterator fully-consumed validation, and extended E2E checked zone.
● Remote storage (S3/OSS/HDFS). S3 client caching, credentials provider support, DFS replication factor configuration, MapPartitionData on DFS, and multiple robustness fixes.
● Observability. Flush latency per storage tier, eviction counts, read buffer availability, metadata operation metrics, and more — all reflected in the Grafana dashboards.
● Performance. Zero-copy sendfile in Netty native transports, parallelized open stream and commit paths, and reduced lock contention on hot paths.
● Durability & reliability. fsync on commit, automatic RocksDB restoration, disk-full rejection, Ratis stepdown and outbox retry, and numerous race-condition and resource-leak fixes.
● Security. RPC authorization, Java deserialization filter, and configurable HTTP auth bypass paths.
● Standalone LifecycleManager and Rust SDK, plus Helm chart improvements.
Looking ahead: 1.0.0
With 0.7.0 we are essentially completing the feature surface for a 1.0. I'd like to propose that the next major release after 0.7.0 will be 1.0.0 — marking Celeborn's technical maturity and production-grade stability, reflecting the collective work since graduation: Spark stage rerun, MapReduce and Flink support, memory and remote storage, Flink hybrid shuffle, HARD_SPLIT in PushMergedData, skew partition optimization, the C++ client, encryption at rest, and broad real-world deployment feedback.
I'm volunteering to serve as the release manager for 0.7.0. If there are no objections, I'll start preparing the release branch and the first RC after this thread concludes (72 hours, per convention).
Regards,
Zhentao Shuai
Re: [DISCUSS] Time for 0.7.0 release and looking ahead to 1.0.0
Posted by rexxiong <re...@apache.org>.
Hi Zhentao,
Thanks for driving this! The 0.7.0 release looks like a huge milestone— the
C++ client, encryption at rest, and all the new engine support are really
impressive.
+1 on the 1.0.0 proposal, it feels like the right time. Looking forward to
the RC.
Regards,
Jiashu Xiong
Nicholas Jiang <ni...@apache.org> 于2026年7月22日周三 17:54写道：
> Hi Zhentao,
>
> Thanks for initiating the discussion and for volunteering to serve as the
> release manager.
> +1 on starting the Celeborn 0.7.0 release. Given the substantial features,
> improvements, and fixes accumulated since 0.6.x, now is a good time to cut
> the release branch and begin the RC process.
>
> I also support the proposal to target 1.0.0 as the next major release
> after 0.7.0. The project has developed a mature feature set and has seen
> broad production adoption. After 0.7.0, it would be helpful to align on
> explicit 1.0.0 readiness criteria, particularly around API and protocol
> compatibility, upgrade paths, documentation, integration coverage, and
> production validation.
>
> Thanks again for driving the release.
>
> Best regards,
> Nicholas Jiang
>
> On 2026/07/22 08:02:15 Zhentao Shuai wrote:
> > Hi Celeborn community,
> > I'd like to kick off the 0.7.0 release now that we're at the end of
> July.
> > It has been about 10 months since 0.6.0, and master has accumulated a
> substantial set of new features, improvements, and fixes since 0.6.3 alone.
> Highlights include:
> > ● Engine support. Spark 4.1/4.2 and Flink 2.1/2.2/2.3 support; Spark 4.0
> and Flink 2.0 client maturity. Removed Spark 2.4, Flink 1.16/1.17, and
> Hadoop 2 client support.
> > ● C++ native client (CIP-14). Full push/merge data path, revive,
> LZ4/ZSTD compression, retry logic, heartbeat decoding, native transport,
> and C++-write/Java-read hybrid integration tests — a major push toward
> feature parity with the Java client.
> > ● Encryption at Rest (CIP-22). Spark-side implementation landed.
> > ● Shuffle integrity. End-to-end integrity check for Flink, iterator
> fully-consumed validation, and extended E2E checked zone.
> > ● Remote storage (S3/OSS/HDFS). S3 client caching, credentials provider
> support, DFS replication factor configuration, MapPartitionData on DFS, and
> multiple robustness fixes.
> > ● Observability. Flush latency per storage tier, eviction counts, read
> buffer availability, metadata operation metrics, and more — all reflected
> in the Grafana dashboards.
> > ● Performance. Zero-copy sendfile in Netty native transports,
> parallelized open stream and commit paths, and reduced lock contention on
> hot paths.
> > ● Durability & reliability. fsync on commit, automatic RocksDB
> restoration, disk-full rejection, Ratis stepdown and outbox retry, and
> numerous race-condition and resource-leak fixes.
> > ● Security. RPC authorization, Java deserialization filter, and
> configurable HTTP auth bypass paths.
> > ● Standalone LifecycleManager and Rust SDK, plus Helm chart improvements.
> > Looking ahead: 1.0.0
> > With 0.7.0 we are essentially completing the feature surface for a
> 1.0. I'd like to propose that the next major release after 0.7.0 will be
> 1.0.0 — marking Celeborn's technical maturity and production-grade
> stability, reflecting the collective work since graduation: Spark stage
> rerun, MapReduce and Flink support, memory and remote storage, Flink hybrid
> shuffle, HARD_SPLIT in PushMergedData, skew partition optimization, the C++
> client, encryption at rest, and broad real-world deployment feedback.
> > I'm volunteering to serve as the release manager for 0.7.0. If there
> are no objections, I'll start preparing the release branch and the first RC
> after this thread concludes (72 hours, per convention).
> > Regards,
> > Zhentao Shuai
> >
>
Re: [DISCUSS] Time for 0.7.0 release and looking ahead to 1.0.0
Posted by Nicholas Jiang <ni...@apache.org>.
Hi Zhentao,
Thanks for initiating the discussion and for volunteering to serve as the release manager.
+1 on starting the Celeborn 0.7.0 release. Given the substantial features, improvements, and fixes accumulated since 0.6.x, now is a good time to cut the release branch and begin the RC process.
I also support the proposal to target 1.0.0 as the next major release after 0.7.0. The project has developed a mature feature set and has seen broad production adoption. After 0.7.0, it would be helpful to align on explicit 1.0.0 readiness criteria, particularly around API and protocol compatibility, upgrade paths, documentation, integration coverage, and production validation.
Thanks again for driving the release.
Best regards,
Nicholas Jiang
On 2026/07/22 08:02:15 Zhentao Shuai wrote:
> Hi Celeborn community,
> I'd like to kick off the 0.7.0 release now that we're at the end of July.
> It has been about 10 months since 0.6.0, and master has accumulated a substantial set of new features, improvements, and fixes since 0.6.3 alone. Highlights include:
> ● Engine support. Spark 4.1/4.2 and Flink 2.1/2.2/2.3 support; Spark 4.0 and Flink 2.0 client maturity. Removed Spark 2.4, Flink 1.16/1.17, and Hadoop 2 client support.
> ● C++ native client (CIP-14). Full push/merge data path, revive, LZ4/ZSTD compression, retry logic, heartbeat decoding, native transport, and C++-write/Java-read hybrid integration tests — a major push toward feature parity with the Java client.
> ● Encryption at Rest (CIP-22). Spark-side implementation landed.
> ● Shuffle integrity. End-to-end integrity check for Flink, iterator fully-consumed validation, and extended E2E checked zone.
> ● Remote storage (S3/OSS/HDFS). S3 client caching, credentials provider support, DFS replication factor configuration, MapPartitionData on DFS, and multiple robustness fixes.
> ● Observability. Flush latency per storage tier, eviction counts, read buffer availability, metadata operation metrics, and more — all reflected in the Grafana dashboards.
> ● Performance. Zero-copy sendfile in Netty native transports, parallelized open stream and commit paths, and reduced lock contention on hot paths.
> ● Durability & reliability. fsync on commit, automatic RocksDB restoration, disk-full rejection, Ratis stepdown and outbox retry, and numerous race-condition and resource-leak fixes.
> ● Security. RPC authorization, Java deserialization filter, and configurable HTTP auth bypass paths.
> ● Standalone LifecycleManager and Rust SDK, plus Helm chart improvements.
> Looking ahead: 1.0.0
> With 0.7.0 we are essentially completing the feature surface for a 1.0. I'd like to propose that the next major release after 0.7.0 will be 1.0.0 — marking Celeborn's technical maturity and production-grade stability, reflecting the collective work since graduation: Spark stage rerun, MapReduce and Flink support, memory and remote storage, Flink hybrid shuffle, HARD_SPLIT in PushMergedData, skew partition optimization, the C++ client, encryption at rest, and broad real-world deployment feedback.
> I'm volunteering to serve as the release manager for 0.7.0. If there are no objections, I'll start preparing the release branch and the first RC after this thread concludes (72 hours, per convention).
> Regards,
> Zhentao Shuai
>
Re: [DISCUSS] Time for 0.7.0 release and looking ahead to 1.0.0
Posted by Venkatakrishnan Sowrirajan <vs...@asu.edu>.
Hi Celeborn devs,
We found a concurrency bug with GetReducerFileGroup in ShuffleClientImpl in
our internal workloads. The blocking RPC runs inside
ConcurrentHashMap.compute, so it holds the per-key bin lock across the
whole round-trip and queues behind every reduce task of the shuffle. At
scale this wedged a stage for over 20 minutes with no fetch activity. I
filed https://github.com/apache/celeborn/pull/3776 to fix it (isolated
change, has a regression test).
After filing it, I noticed @Chao Sun <su...@apache.org> already had
https://github.com/apache/celeborn/pull/3687 open for the same issue, and
later superseded it with https://github.com/apache/celeborn/pull/3745,
which scopes the reduceFileGroups fetch to the reader's partition range
instead of shipping the full map to every executor.
Both are solid improvements: the concurrency fix and the fetch
optimization. I would like to get both into 0.7.0. #3776 is small and ready
for review; happy to rebase it onto the 0.7.0 branch. Would appreciate
thoughts on whether #3745 can make 0.7.0 as well.
Thanks
Venkat
On Mon, Jul 27, 2026 at 4:27 AM keyong zhou <wa...@gmail.com> wrote:
> +1, thanks Zhentao for volunteering!
>
> Regards,
> Keyong Zhou
>
> Zhentao Shuai <za...@apache.org> 于2026年7月22日周三 16:02写道：
>
> > Hi Celeborn community,
> > I'd like to kick off the 0.7.0 release now that we're at the end of
> July.
> > It has been about 10 months since 0.6.0, and master has accumulated a
> > substantial set of new features, improvements, and fixes since 0.6.3
> alone.
> > Highlights include:
> > ● Engine support. Spark 4.1/4.2 and Flink 2.1/2.2/2.3 support; Spark 4.0
> > and Flink 2.0 client maturity. Removed Spark 2.4, Flink 1.16/1.17, and
> > Hadoop 2 client support.
> > ● C++ native client (CIP-14). Full push/merge data path, revive, LZ4/ZSTD
> > compression, retry logic, heartbeat decoding, native transport, and
> > C++-write/Java-read hybrid integration tests — a major push toward
> feature
> > parity with the Java client.
> > ● Encryption at Rest (CIP-22). Spark-side implementation landed.
> > ● Shuffle integrity. End-to-end integrity check for Flink, iterator
> > fully-consumed validation, and extended E2E checked zone.
> > ● Remote storage (S3/OSS/HDFS). S3 client caching, credentials provider
> > support, DFS replication factor configuration, MapPartitionData on DFS,
> and
> > multiple robustness fixes.
> > ● Observability. Flush latency per storage tier, eviction counts, read
> > buffer availability, metadata operation metrics, and more — all reflected
> > in the Grafana dashboards.
> > ● Performance. Zero-copy sendfile in Netty native transports,
> parallelized
> > open stream and commit paths, and reduced lock contention on hot paths.
> > ● Durability & reliability. fsync on commit, automatic RocksDB
> > restoration, disk-full rejection, Ratis stepdown and outbox retry, and
> > numerous race-condition and resource-leak fixes.
> > ● Security. RPC authorization, Java deserialization filter, and
> > configurable HTTP auth bypass paths.
> > ● Standalone LifecycleManager and Rust SDK, plus Helm chart improvements.
> > Looking ahead: 1.0.0
> > With 0.7.0 we are essentially completing the feature surface for a 1.0.
> > I'd like to propose that the next major release after 0.7.0 will be
> 1.0.0 —
> > marking Celeborn's technical maturity and production-grade stability,
> > reflecting the collective work since graduation: Spark stage rerun,
> > MapReduce and Flink support, memory and remote storage, Flink hybrid
> > shuffle, HARD_SPLIT in PushMergedData, skew partition optimization, the
> C++
> > client, encryption at rest, and broad real-world deployment feedback.
> > I'm volunteering to serve as the release manager for 0.7.0. If there
> are
> > no objections, I'll start preparing the release branch and the first RC
> > after this thread concludes (72 hours, per convention).
> > Regards,
> > Zhentao Shuai
> >
>
Re: [DISCUSS] Time for 0.7.0 release and looking ahead to 1.0.0
Posted by keyong zhou <wa...@gmail.com>.
+1, thanks Zhentao for volunteering!
Regards,
Keyong Zhou
Zhentao Shuai <za...@apache.org> 于2026年7月22日周三 16:02写道：
> Hi Celeborn community,
> I'd like to kick off the 0.7.0 release now that we're at the end of July.
> It has been about 10 months since 0.6.0, and master has accumulated a
> substantial set of new features, improvements, and fixes since 0.6.3 alone.
> Highlights include:
> ● Engine support. Spark 4.1/4.2 and Flink 2.1/2.2/2.3 support; Spark 4.0
> and Flink 2.0 client maturity. Removed Spark 2.4, Flink 1.16/1.17, and
> Hadoop 2 client support.
> ● C++ native client (CIP-14). Full push/merge data path, revive, LZ4/ZSTD
> compression, retry logic, heartbeat decoding, native transport, and
> C++-write/Java-read hybrid integration tests — a major push toward feature
> parity with the Java client.
> ● Encryption at Rest (CIP-22). Spark-side implementation landed.
> ● Shuffle integrity. End-to-end integrity check for Flink, iterator
> fully-consumed validation, and extended E2E checked zone.
> ● Remote storage (S3/OSS/HDFS). S3 client caching, credentials provider
> support, DFS replication factor configuration, MapPartitionData on DFS, and
> multiple robustness fixes.
> ● Observability. Flush latency per storage tier, eviction counts, read
> buffer availability, metadata operation metrics, and more — all reflected
> in the Grafana dashboards.
> ● Performance. Zero-copy sendfile in Netty native transports, parallelized
> open stream and commit paths, and reduced lock contention on hot paths.
> ● Durability & reliability. fsync on commit, automatic RocksDB
> restoration, disk-full rejection, Ratis stepdown and outbox retry, and
> numerous race-condition and resource-leak fixes.
> ● Security. RPC authorization, Java deserialization filter, and
> configurable HTTP auth bypass paths.
> ● Standalone LifecycleManager and Rust SDK, plus Helm chart improvements.
> Looking ahead: 1.0.0
> With 0.7.0 we are essentially completing the feature surface for a 1.0.
> I'd like to propose that the next major release after 0.7.0 will be 1.0.0 —
> marking Celeborn's technical maturity and production-grade stability,
> reflecting the collective work since graduation: Spark stage rerun,
> MapReduce and Flink support, memory and remote storage, Flink hybrid
> shuffle, HARD_SPLIT in PushMergedData, skew partition optimization, the C++
> client, encryption at rest, and broad real-world deployment feedback.
> I'm volunteering to serve as the release manager for 0.7.0. If there are
> no objections, I'll start preparing the release branch and the first RC
> after this thread concludes (72 hours, per convention).
> Regards,
> Zhentao Shuai
>