You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@celeborn.apache.org by Karthik Prabhakar <kn...@gmail.com> on 2026/04/19 05:24:03 UTC
Fwd: [DISCUSS] At-rest encryption for shuffle data on workers and tiered storage
Hi dev@,
I’d like to propose adding at-rest encryption for shuffle data in Celeborn
and would appreciate the community’s input before writing a full
implementation.
cURRENT gap
Celeborn encrypts data in transit (TLS, SASL) but not at rest. When a
worker flushes shuffle data to local disk, HDFS, S3, or OSS, the bytes land
as plaintext.
The only write site for local disk is LocalFlushTask.flush() in
FlushTask.scala (L66, L71 at commit a56f69a), which calls
fileChannel.write(buffer) with no cipher transform. The tiered-storage
paths (HdfsFlushTask, S3FlushTask, OssFlushTask) are the same — raw bytes
to the underlying store.
Verified with:
grep -rnE 'cipher|\.encrypt|aes|envelope' worker/src/main/
grep -rn 'javax\.crypto' worker/src/main/
(both zero matches)
This matters because spark.io.encryption.enabled does *not* cover the
Celeborn path. When Celeborn’s ShuffleManager replaces Spark’s shuffle
writer, Spark’s encryption key is never consulted — confirmed by grepping
client-spark/ for IOEncryptionKey (zero matches).
Teams adopting Celeborn for performance silently lose shuffle-encryption
guarantees their compliance posture may assume.
Who Needs This
- Regulated industries (healthcare, finance, public sector) whose
auditors require application-layer encryption independent of disk/volume
encryption.
- Multi-tenant platforms needing cryptographic isolation between tenants
on shared workers.
- Teams using object-store tiering who want encryption before offload.
Proposed Approach (High Level)
1. A *StreamCipher SPI* in common/ for wrapping WritableByteChannel /
ReadableByteChannel with encrypt/decrypt. No KMS SDK in core.
2. A *KeyService SPI* for envelope encryption — generate/unwrap DEKs
using a KMS-held KEK. Implementations live in separate optional modules (
aws-kms, gcp-kms, azure-kv, vault, static for dev/PoC).
3. Wire into the worker write path: LocalFlushTask wraps fileChannel
with StreamCipher.wrapForWrite(). Same for HDFS/S3/OSS flush tasks.
4. Wire into the reader path: LocalPartitionDataReader detects a 16-byte
encrypted-file header, unwraps the DEK (cached per worker+shuffle), wraps
the channel with StreamCipher.wrapForRead().
5. Opt-in via celeborn.shuffle.io.encryption.enabled=true. Default off.
Unencrypted deployments are byte-identical to today, zero overhead.
6. Per-shuffle DEKs by default (one KMS call per shuffle reservation,
amortized). Per-application DEK scope as an option.
Interaction with Recent Work
CELEBORN-2301 (commit 95419e1) recently landed enhanced zero-copy sendfile
for FileRegion on native transports — a nice throughput win for the fetch
path.
Encryption and sendfile are fundamentally incompatible: sendfile(2) cannot
transform bytes, so encrypted partitions must use a buffered read path.
This is only relevant for encrypted workloads; unencrypted workloads on the
same cluster keep the full CELEBORN-2301 benefit. Per-application
encryption flags (not per-cluster) would let encrypted and unencrypted apps
coexist without regressing the latter.
Questions for the Community
Trimming to three since these are the ones I’d need opinions on before
writing code. Happy to take the rest up in follow-ups.
- Any prior design work or internal discussion on this topic I should
know about before proceeding?
- *Per-shuffle vs. per-application DEK scope* as the default?
Per-shuffle gives smaller blast radius and simpler lifecycle;
per-application amortizes KMS round-trips and is friendlier for
long-running jobs.
- *Key distribution path:* wrapped DEKs flow through Master metadata
(simpler, one KMS-aware role) vs. workers unwrap directly from KMS (removes
Master from the key path, but every worker needs KMS credentials).
Preference?
Tracking
JIRA: CELEBORN-2311 <https://issues.apache.org/jira/browse/CELEBORN-2311>
I have a detailed design document with source citations, threat model,
performance analysis, and phased implementation plan. Happy to share
on-list or off-list if there’s interest.
- Karthik
Re: [DISCUSS] At-rest encryption for shuffle data on workers and tiered storage
Posted by Aravind Patnam <ak...@gmail.com>.
Hey folks,
If there is no further discussion or alignment needed, I will start a vote
thread next week. I am out on vacation this week.
Thanks,
Aravind
Aravind K. Patnam
On Wed, Apr 22, 2026 at 1:59 PM Aravind Patnam <ak...@gmail.com> wrote:
> Hi Folks,
>
> As mentioned in the thread, here is the CIP22: Encryption at Rest for
> Celeborn Shuffle Data
> <https://docs.google.com/document/d/1xBrLtpb8bk8CdJENiM3aLJCbFThRKLor8uAjSsfXoxE/edit?usp=sharing>
> .
> Pls let me know if there are any concerns, and feel free to comment on the
> doc itself or reply in this thread.
>
> Otherwise, I will plan to start a vote thread early next week.
>
>
> Thanks,
> Aravind
>
> On Mon, Apr 20, 2026 at 6:07 PM Karthik Prabhakar <kn...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> Great to see the convergence; client-side encryption is a clean approach
>> and aligns well with the original intent. Looking forward to the CIP
>> thread, Aravind. Happy to contribute.
>>
>> - Karthik
>>
>> On Mon, Apr 20, 2026 at 1:17 PM Aravind Patnam <ak...@gmail.com>
>> wrote:
>>
>> > Let me start a thread for a CIP later this week.
>> >
>> > Aravind K. Patnam
>> >
>> >
>> > On Mon, Apr 20, 2026 at 8:37 AM Mridul Muralidharan <mr...@gmail.com>
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > This is pretty much what Aravind has implemented internally :-)
>> > >
>> > > Regards,
>> > > Mridul
>> > >
>> > > On Mon, Apr 20, 2026 at 8:34 AM rexxiong <re...@apache.org> wrote:
>> > >
>> > > > Hi Karthik,
>> > > >
>> > > > Thanks for the thorough proposal. I'd like to suggest an alternative
>> > > > approach: client-side encryption, where encryption/decryption
>> happens
>> > > > entirely in the Spark
>> > > > client — similar to how compression works today. Celeborn workers
>> and
>> > > > master would only ever see ciphertext.
>> > > >
>> > > > Rationale:
>> > > >
>> > > > 1. Simpler architecture — no worker write/read path changes.
>> Celeborn
>> > > > remains a stateless byte pipe, just like it is for compression.
>> > > > 2. Stronger security — Plaintext and encryption keys never reach
>> > workers
>> > > or
>> > > > master. The trust boundary stays within the client, eliminating the
>> > need
>> > > > for KMS
>> > > > credentials on the server side.
>> > > > 3. No sendfile regression — Since workers store ciphertext natively,
>> > > > CELEBORN-2301's zero-copy sendfile works unchanged for all
>> workloads,
>> > > > encrypted or not.
>> > > > 4. Aligns with Spark — This naturally respects
>> > > spark.io.encryption.enabled
>> > > > and we can reuse Spark's existing key distribution via
>> IOEncryptionKey.
>> > > >
>> > > > Happy to discuss further.
>> > > >
>> > > > Regards,
>> > > > Jiashu Xiong
>> > > >
>> > > > Aravind Patnam <ak...@gmail.com> 于2026年4月20日周一 05:08写道：
>> > > >
>> > > > > Hi,
>> > > > >
>> > > > > We already have EAR for celeborn shuffle data internally at
>> LinkedIn,
>> > > > where
>> > > > > we have added this support to respect the existing
>> > > > > spark.io.encryption.enabled config in Spark on the client side.
>> > > > >
>> > > > > I am happy to contribute this back and start a CIP for this next
>> > week.
>> > > > >
>> > > > > Thanks,
>> > > > > Aravind
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Sat, Apr 18, 2026 at 10:24 PM Karthik Prabhakar <
>> > > > knkarthik01@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi dev@,
>> > > > > >
>> > > > > > I’d like to propose adding at-rest encryption for shuffle data
>> in
>> > > > > Celeborn
>> > > > > > and would appreciate the community’s input before writing a full
>> > > > > > implementation.
>> > > > > > cURRENT gap
>> > > > > >
>> > > > > > Celeborn encrypts data in transit (TLS, SASL) but not at rest.
>> > When a
>> > > > > > worker flushes shuffle data to local disk, HDFS, S3, or OSS, the
>> > > bytes
>> > > > > land
>> > > > > > as plaintext.
>> > > > > >
>> > > > > > The only write site for local disk is LocalFlushTask.flush() in
>> > > > > > FlushTask.scala (L66, L71 at commit a56f69a), which calls
>> > > > > > fileChannel.write(buffer) with no cipher transform. The
>> > > tiered-storage
>> > > > > > paths (HdfsFlushTask, S3FlushTask, OssFlushTask) are the same —
>> raw
>> > > > bytes
>> > > > > > to the underlying store.
>> > > > > >
>> > > > > > Verified with:
>> > > > > >
>> > > > > > grep -rnE 'cipher|\.encrypt|aes|envelope' worker/src/main/
>> > > > > > grep -rn 'javax\.crypto' worker/src/main/
>> > > > > > (both zero matches)
>> > > > > >
>> > > > > > This matters because spark.io.encryption.enabled does *not*
>> cover
>> > the
>> > > > > > Celeborn path. When Celeborn’s ShuffleManager replaces Spark’s
>> > > shuffle
>> > > > > > writer, Spark’s encryption key is never consulted — confirmed by
>> > > > grepping
>> > > > > > client-spark/ for IOEncryptionKey (zero matches).
>> > > > > >
>> > > > > > Teams adopting Celeborn for performance silently lose
>> > > > shuffle-encryption
>> > > > > > guarantees their compliance posture may assume.
>> > > > > > Who Needs This
>> > > > > >
>> > > > > > - Regulated industries (healthcare, finance, public sector)
>> > whose
>> > > > > > auditors require application-layer encryption independent of
>> > > > > disk/volume
>> > > > > > encryption.
>> > > > > > - Multi-tenant platforms needing cryptographic isolation
>> between
>> > > > > tenants
>> > > > > > on shared workers.
>> > > > > > - Teams using object-store tiering who want encryption before
>> > > > offload.
>> > > > > >
>> > > > > > Proposed Approach (High Level)
>> > > > > >
>> > > > > > 1. A *StreamCipher SPI* in common/ for wrapping
>> > > WritableByteChannel
>> > > > /
>> > > > > > ReadableByteChannel with encrypt/decrypt. No KMS SDK in core.
>> > > > > > 2. A *KeyService SPI* for envelope encryption —
>> generate/unwrap
>> > > DEKs
>> > > > > > using a KMS-held KEK. Implementations live in separate
>> optional
>> > > > > modules
>> > > > > > (
>> > > > > > aws-kms, gcp-kms, azure-kv, vault, static for dev/PoC).
>> > > > > > 3. Wire into the worker write path: LocalFlushTask wraps
>> > > fileChannel
>> > > > > > with StreamCipher.wrapForWrite(). Same for HDFS/S3/OSS flush
>> > > tasks.
>> > > > > > 4. Wire into the reader path: LocalPartitionDataReader
>> detects a
>> > > > > 16-byte
>> > > > > > encrypted-file header, unwraps the DEK (cached per
>> > > worker+shuffle),
>> > > > > > wraps
>> > > > > > the channel with StreamCipher.wrapForRead().
>> > > > > > 5. Opt-in via celeborn.shuffle.io.encryption.enabled=true.
>> > > Default
>> > > > > off.
>> > > > > > Unencrypted deployments are byte-identical to today, zero
>> > > overhead.
>> > > > > > 6. Per-shuffle DEKs by default (one KMS call per shuffle
>> > > > reservation,
>> > > > > > amortized). Per-application DEK scope as an option.
>> > > > > >
>> > > > > > Interaction with Recent Work
>> > > > > >
>> > > > > > CELEBORN-2301 (commit 95419e1) recently landed enhanced
>> zero-copy
>> > > > > sendfile
>> > > > > > for FileRegion on native transports — a nice throughput win for
>> the
>> > > > fetch
>> > > > > > path.
>> > > > > >
>> > > > > > Encryption and sendfile are fundamentally incompatible:
>> sendfile(2)
>> > > > > cannot
>> > > > > > transform bytes, so encrypted partitions must use a buffered
>> read
>> > > path.
>> > > > > > This is only relevant for encrypted workloads; unencrypted
>> > workloads
>> > > on
>> > > > > the
>> > > > > > same cluster keep the full CELEBORN-2301 benefit.
>> Per-application
>> > > > > > encryption flags (not per-cluster) would let encrypted and
>> > > unencrypted
>> > > > > apps
>> > > > > > coexist without regressing the latter.
>> > > > > > Questions for the Community
>> > > > > >
>> > > > > > Trimming to three since these are the ones I’d need opinions on
>> > > before
>> > > > > > writing code. Happy to take the rest up in follow-ups.
>> > > > > >
>> > > > > > - Any prior design work or internal discussion on this topic
>> I
>> > > > should
>> > > > > > know about before proceeding?
>> > > > > > - *Per-shuffle vs. per-application DEK scope* as the default?
>> > > > > > Per-shuffle gives smaller blast radius and simpler lifecycle;
>> > > > > > per-application amortizes KMS round-trips and is friendlier
>> for
>> > > > > > long-running jobs.
>> > > > > > - *Key distribution path:* wrapped DEKs flow through Master
>> > > metadata
>> > > > > > (simpler, one KMS-aware role) vs. workers unwrap directly
>> from
>> > KMS
>> > > > > > (removes
>> > > > > > Master from the key path, but every worker needs KMS
>> > credentials).
>> > > > > > Preference?
>> > > > > >
>> > > > > > Tracking
>> > > > > >
>> > > > > > JIRA: CELEBORN-2311 <
>> > > > https://issues.apache.org/jira/browse/CELEBORN-2311
>> > > > > >
>> > > > > >
>> > > > > > I have a detailed design document with source citations, threat
>> > > model,
>> > > > > > performance analysis, and phased implementation plan. Happy to
>> > share
>> > > > > > on-list or off-list if there’s interest.
>> > > > > >
>> > > > > > - Karthik
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Aravind K. Patnam
>> > > > >
>> > > >
>> > >
>> >
>>
>>
>> --
>> Thanks,
>> Karthik
>>
>
>
> --
> Aravind K. Patnam
>
Re: [DISCUSS] At-rest encryption for shuffle data on workers and tiered storage
Posted by Aravind Patnam <ak...@gmail.com>.
Hi Folks,
As mentioned in the thread, here is the CIP22: Encryption at Rest for
Celeborn Shuffle Data
<https://docs.google.com/document/d/1xBrLtpb8bk8CdJENiM3aLJCbFThRKLor8uAjSsfXoxE/edit?usp=sharing>
.
Pls let me know if there are any concerns, and feel free to comment on the
doc itself or reply in this thread.
Otherwise, I will plan to start a vote thread early next week.
Thanks,
Aravind
On Mon, Apr 20, 2026 at 6:07 PM Karthik Prabhakar <kn...@gmail.com>
wrote:
> Hi all,
>
> Great to see the convergence; client-side encryption is a clean approach
> and aligns well with the original intent. Looking forward to the CIP
> thread, Aravind. Happy to contribute.
>
> - Karthik
>
> On Mon, Apr 20, 2026 at 1:17 PM Aravind Patnam <ak...@gmail.com>
> wrote:
>
> > Let me start a thread for a CIP later this week.
> >
> > Aravind K. Patnam
> >
> >
> > On Mon, Apr 20, 2026 at 8:37 AM Mridul Muralidharan <mr...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > This is pretty much what Aravind has implemented internally :-)
> > >
> > > Regards,
> > > Mridul
> > >
> > > On Mon, Apr 20, 2026 at 8:34 AM rexxiong <re...@apache.org> wrote:
> > >
> > > > Hi Karthik,
> > > >
> > > > Thanks for the thorough proposal. I'd like to suggest an alternative
> > > > approach: client-side encryption, where encryption/decryption happens
> > > > entirely in the Spark
> > > > client — similar to how compression works today. Celeborn workers and
> > > > master would only ever see ciphertext.
> > > >
> > > > Rationale:
> > > >
> > > > 1. Simpler architecture — no worker write/read path changes. Celeborn
> > > > remains a stateless byte pipe, just like it is for compression.
> > > > 2. Stronger security — Plaintext and encryption keys never reach
> > workers
> > > or
> > > > master. The trust boundary stays within the client, eliminating the
> > need
> > > > for KMS
> > > > credentials on the server side.
> > > > 3. No sendfile regression — Since workers store ciphertext natively,
> > > > CELEBORN-2301's zero-copy sendfile works unchanged for all workloads,
> > > > encrypted or not.
> > > > 4. Aligns with Spark — This naturally respects
> > > spark.io.encryption.enabled
> > > > and we can reuse Spark's existing key distribution via
> IOEncryptionKey.
> > > >
> > > > Happy to discuss further.
> > > >
> > > > Regards,
> > > > Jiashu Xiong
> > > >
> > > > Aravind Patnam <ak...@gmail.com> 于2026年4月20日周一 05:08写道：
> > > >
> > > > > Hi,
> > > > >
> > > > > We already have EAR for celeborn shuffle data internally at
> LinkedIn,
> > > > where
> > > > > we have added this support to respect the existing
> > > > > spark.io.encryption.enabled config in Spark on the client side.
> > > > >
> > > > > I am happy to contribute this back and start a CIP for this next
> > week.
> > > > >
> > > > > Thanks,
> > > > > Aravind
> > > > >
> > > > >
> > > > >
> > > > > On Sat, Apr 18, 2026 at 10:24 PM Karthik Prabhakar <
> > > > knkarthik01@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi dev@,
> > > > > >
> > > > > > I’d like to propose adding at-rest encryption for shuffle data in
> > > > > Celeborn
> > > > > > and would appreciate the community’s input before writing a full
> > > > > > implementation.
> > > > > > cURRENT gap
> > > > > >
> > > > > > Celeborn encrypts data in transit (TLS, SASL) but not at rest.
> > When a
> > > > > > worker flushes shuffle data to local disk, HDFS, S3, or OSS, the
> > > bytes
> > > > > land
> > > > > > as plaintext.
> > > > > >
> > > > > > The only write site for local disk is LocalFlushTask.flush() in
> > > > > > FlushTask.scala (L66, L71 at commit a56f69a), which calls
> > > > > > fileChannel.write(buffer) with no cipher transform. The
> > > tiered-storage
> > > > > > paths (HdfsFlushTask, S3FlushTask, OssFlushTask) are the same —
> raw
> > > > bytes
> > > > > > to the underlying store.
> > > > > >
> > > > > > Verified with:
> > > > > >
> > > > > > grep -rnE 'cipher|\.encrypt|aes|envelope' worker/src/main/
> > > > > > grep -rn 'javax\.crypto' worker/src/main/
> > > > > > (both zero matches)
> > > > > >
> > > > > > This matters because spark.io.encryption.enabled does *not* cover
> > the
> > > > > > Celeborn path. When Celeborn’s ShuffleManager replaces Spark’s
> > > shuffle
> > > > > > writer, Spark’s encryption key is never consulted — confirmed by
> > > > grepping
> > > > > > client-spark/ for IOEncryptionKey (zero matches).
> > > > > >
> > > > > > Teams adopting Celeborn for performance silently lose
> > > > shuffle-encryption
> > > > > > guarantees their compliance posture may assume.
> > > > > > Who Needs This
> > > > > >
> > > > > > - Regulated industries (healthcare, finance, public sector)
> > whose
> > > > > > auditors require application-layer encryption independent of
> > > > > disk/volume
> > > > > > encryption.
> > > > > > - Multi-tenant platforms needing cryptographic isolation
> between
> > > > > tenants
> > > > > > on shared workers.
> > > > > > - Teams using object-store tiering who want encryption before
> > > > offload.
> > > > > >
> > > > > > Proposed Approach (High Level)
> > > > > >
> > > > > > 1. A *StreamCipher SPI* in common/ for wrapping
> > > WritableByteChannel
> > > > /
> > > > > > ReadableByteChannel with encrypt/decrypt. No KMS SDK in core.
> > > > > > 2. A *KeyService SPI* for envelope encryption —
> generate/unwrap
> > > DEKs
> > > > > > using a KMS-held KEK. Implementations live in separate
> optional
> > > > > modules
> > > > > > (
> > > > > > aws-kms, gcp-kms, azure-kv, vault, static for dev/PoC).
> > > > > > 3. Wire into the worker write path: LocalFlushTask wraps
> > > fileChannel
> > > > > > with StreamCipher.wrapForWrite(). Same for HDFS/S3/OSS flush
> > > tasks.
> > > > > > 4. Wire into the reader path: LocalPartitionDataReader
> detects a
> > > > > 16-byte
> > > > > > encrypted-file header, unwraps the DEK (cached per
> > > worker+shuffle),
> > > > > > wraps
> > > > > > the channel with StreamCipher.wrapForRead().
> > > > > > 5. Opt-in via celeborn.shuffle.io.encryption.enabled=true.
> > > Default
> > > > > off.
> > > > > > Unencrypted deployments are byte-identical to today, zero
> > > overhead.
> > > > > > 6. Per-shuffle DEKs by default (one KMS call per shuffle
> > > > reservation,
> > > > > > amortized). Per-application DEK scope as an option.
> > > > > >
> > > > > > Interaction with Recent Work
> > > > > >
> > > > > > CELEBORN-2301 (commit 95419e1) recently landed enhanced zero-copy
> > > > > sendfile
> > > > > > for FileRegion on native transports — a nice throughput win for
> the
> > > > fetch
> > > > > > path.
> > > > > >
> > > > > > Encryption and sendfile are fundamentally incompatible:
> sendfile(2)
> > > > > cannot
> > > > > > transform bytes, so encrypted partitions must use a buffered read
> > > path.
> > > > > > This is only relevant for encrypted workloads; unencrypted
> > workloads
> > > on
> > > > > the
> > > > > > same cluster keep the full CELEBORN-2301 benefit. Per-application
> > > > > > encryption flags (not per-cluster) would let encrypted and
> > > unencrypted
> > > > > apps
> > > > > > coexist without regressing the latter.
> > > > > > Questions for the Community
> > > > > >
> > > > > > Trimming to three since these are the ones I’d need opinions on
> > > before
> > > > > > writing code. Happy to take the rest up in follow-ups.
> > > > > >
> > > > > > - Any prior design work or internal discussion on this topic I
> > > > should
> > > > > > know about before proceeding?
> > > > > > - *Per-shuffle vs. per-application DEK scope* as the default?
> > > > > > Per-shuffle gives smaller blast radius and simpler lifecycle;
> > > > > > per-application amortizes KMS round-trips and is friendlier
> for
> > > > > > long-running jobs.
> > > > > > - *Key distribution path:* wrapped DEKs flow through Master
> > > metadata
> > > > > > (simpler, one KMS-aware role) vs. workers unwrap directly from
> > KMS
> > > > > > (removes
> > > > > > Master from the key path, but every worker needs KMS
> > credentials).
> > > > > > Preference?
> > > > > >
> > > > > > Tracking
> > > > > >
> > > > > > JIRA: CELEBORN-2311 <
> > > > https://issues.apache.org/jira/browse/CELEBORN-2311
> > > > > >
> > > > > >
> > > > > > I have a detailed design document with source citations, threat
> > > model,
> > > > > > performance analysis, and phased implementation plan. Happy to
> > share
> > > > > > on-list or off-list if there’s interest.
> > > > > >
> > > > > > - Karthik
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Aravind K. Patnam
> > > > >
> > > >
> > >
> >
>
>
> --
> Thanks,
> Karthik
>
--
Aravind K. Patnam
Re: [DISCUSS] At-rest encryption for shuffle data on workers and tiered storage
Posted by Karthik Prabhakar <kn...@gmail.com>.
Hi all,
Great to see the convergence; client-side encryption is a clean approach
and aligns well with the original intent. Looking forward to the CIP
thread, Aravind. Happy to contribute.
- Karthik
On Mon, Apr 20, 2026 at 1:17 PM Aravind Patnam <ak...@gmail.com> wrote:
> Let me start a thread for a CIP later this week.
>
> Aravind K. Patnam
>
>
> On Mon, Apr 20, 2026 at 8:37 AM Mridul Muralidharan <mr...@gmail.com>
> wrote:
>
> > Hi,
> >
> > This is pretty much what Aravind has implemented internally :-)
> >
> > Regards,
> > Mridul
> >
> > On Mon, Apr 20, 2026 at 8:34 AM rexxiong <re...@apache.org> wrote:
> >
> > > Hi Karthik,
> > >
> > > Thanks for the thorough proposal. I'd like to suggest an alternative
> > > approach: client-side encryption, where encryption/decryption happens
> > > entirely in the Spark
> > > client — similar to how compression works today. Celeborn workers and
> > > master would only ever see ciphertext.
> > >
> > > Rationale:
> > >
> > > 1. Simpler architecture — no worker write/read path changes. Celeborn
> > > remains a stateless byte pipe, just like it is for compression.
> > > 2. Stronger security — Plaintext and encryption keys never reach
> workers
> > or
> > > master. The trust boundary stays within the client, eliminating the
> need
> > > for KMS
> > > credentials on the server side.
> > > 3. No sendfile regression — Since workers store ciphertext natively,
> > > CELEBORN-2301's zero-copy sendfile works unchanged for all workloads,
> > > encrypted or not.
> > > 4. Aligns with Spark — This naturally respects
> > spark.io.encryption.enabled
> > > and we can reuse Spark's existing key distribution via IOEncryptionKey.
> > >
> > > Happy to discuss further.
> > >
> > > Regards,
> > > Jiashu Xiong
> > >
> > > Aravind Patnam <ak...@gmail.com> 于2026年4月20日周一 05:08写道：
> > >
> > > > Hi,
> > > >
> > > > We already have EAR for celeborn shuffle data internally at LinkedIn,
> > > where
> > > > we have added this support to respect the existing
> > > > spark.io.encryption.enabled config in Spark on the client side.
> > > >
> > > > I am happy to contribute this back and start a CIP for this next
> week.
> > > >
> > > > Thanks,
> > > > Aravind
> > > >
> > > >
> > > >
> > > > On Sat, Apr 18, 2026 at 10:24 PM Karthik Prabhakar <
> > > knkarthik01@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi dev@,
> > > > >
> > > > > I’d like to propose adding at-rest encryption for shuffle data in
> > > > Celeborn
> > > > > and would appreciate the community’s input before writing a full
> > > > > implementation.
> > > > > cURRENT gap
> > > > >
> > > > > Celeborn encrypts data in transit (TLS, SASL) but not at rest.
> When a
> > > > > worker flushes shuffle data to local disk, HDFS, S3, or OSS, the
> > bytes
> > > > land
> > > > > as plaintext.
> > > > >
> > > > > The only write site for local disk is LocalFlushTask.flush() in
> > > > > FlushTask.scala (L66, L71 at commit a56f69a), which calls
> > > > > fileChannel.write(buffer) with no cipher transform. The
> > tiered-storage
> > > > > paths (HdfsFlushTask, S3FlushTask, OssFlushTask) are the same — raw
> > > bytes
> > > > > to the underlying store.
> > > > >
> > > > > Verified with:
> > > > >
> > > > > grep -rnE 'cipher|\.encrypt|aes|envelope' worker/src/main/
> > > > > grep -rn 'javax\.crypto' worker/src/main/
> > > > > (both zero matches)
> > > > >
> > > > > This matters because spark.io.encryption.enabled does *not* cover
> the
> > > > > Celeborn path. When Celeborn’s ShuffleManager replaces Spark’s
> > shuffle
> > > > > writer, Spark’s encryption key is never consulted — confirmed by
> > > grepping
> > > > > client-spark/ for IOEncryptionKey (zero matches).
> > > > >
> > > > > Teams adopting Celeborn for performance silently lose
> > > shuffle-encryption
> > > > > guarantees their compliance posture may assume.
> > > > > Who Needs This
> > > > >
> > > > > - Regulated industries (healthcare, finance, public sector)
> whose
> > > > > auditors require application-layer encryption independent of
> > > > disk/volume
> > > > > encryption.
> > > > > - Multi-tenant platforms needing cryptographic isolation between
> > > > tenants
> > > > > on shared workers.
> > > > > - Teams using object-store tiering who want encryption before
> > > offload.
> > > > >
> > > > > Proposed Approach (High Level)
> > > > >
> > > > > 1. A *StreamCipher SPI* in common/ for wrapping
> > WritableByteChannel
> > > /
> > > > > ReadableByteChannel with encrypt/decrypt. No KMS SDK in core.
> > > > > 2. A *KeyService SPI* for envelope encryption — generate/unwrap
> > DEKs
> > > > > using a KMS-held KEK. Implementations live in separate optional
> > > > modules
> > > > > (
> > > > > aws-kms, gcp-kms, azure-kv, vault, static for dev/PoC).
> > > > > 3. Wire into the worker write path: LocalFlushTask wraps
> > fileChannel
> > > > > with StreamCipher.wrapForWrite(). Same for HDFS/S3/OSS flush
> > tasks.
> > > > > 4. Wire into the reader path: LocalPartitionDataReader detects a
> > > > 16-byte
> > > > > encrypted-file header, unwraps the DEK (cached per
> > worker+shuffle),
> > > > > wraps
> > > > > the channel with StreamCipher.wrapForRead().
> > > > > 5. Opt-in via celeborn.shuffle.io.encryption.enabled=true.
> > Default
> > > > off.
> > > > > Unencrypted deployments are byte-identical to today, zero
> > overhead.
> > > > > 6. Per-shuffle DEKs by default (one KMS call per shuffle
> > > reservation,
> > > > > amortized). Per-application DEK scope as an option.
> > > > >
> > > > > Interaction with Recent Work
> > > > >
> > > > > CELEBORN-2301 (commit 95419e1) recently landed enhanced zero-copy
> > > > sendfile
> > > > > for FileRegion on native transports — a nice throughput win for the
> > > fetch
> > > > > path.
> > > > >
> > > > > Encryption and sendfile are fundamentally incompatible: sendfile(2)
> > > > cannot
> > > > > transform bytes, so encrypted partitions must use a buffered read
> > path.
> > > > > This is only relevant for encrypted workloads; unencrypted
> workloads
> > on
> > > > the
> > > > > same cluster keep the full CELEBORN-2301 benefit. Per-application
> > > > > encryption flags (not per-cluster) would let encrypted and
> > unencrypted
> > > > apps
> > > > > coexist without regressing the latter.
> > > > > Questions for the Community
> > > > >
> > > > > Trimming to three since these are the ones I’d need opinions on
> > before
> > > > > writing code. Happy to take the rest up in follow-ups.
> > > > >
> > > > > - Any prior design work or internal discussion on this topic I
> > > should
> > > > > know about before proceeding?
> > > > > - *Per-shuffle vs. per-application DEK scope* as the default?
> > > > > Per-shuffle gives smaller blast radius and simpler lifecycle;
> > > > > per-application amortizes KMS round-trips and is friendlier for
> > > > > long-running jobs.
> > > > > - *Key distribution path:* wrapped DEKs flow through Master
> > metadata
> > > > > (simpler, one KMS-aware role) vs. workers unwrap directly from
> KMS
> > > > > (removes
> > > > > Master from the key path, but every worker needs KMS
> credentials).
> > > > > Preference?
> > > > >
> > > > > Tracking
> > > > >
> > > > > JIRA: CELEBORN-2311 <
> > > https://issues.apache.org/jira/browse/CELEBORN-2311
> > > > >
> > > > >
> > > > > I have a detailed design document with source citations, threat
> > model,
> > > > > performance analysis, and phased implementation plan. Happy to
> share
> > > > > on-list or off-list if there’s interest.
> > > > >
> > > > > - Karthik
> > > > >
> > > >
> > > >
> > > > --
> > > > Aravind K. Patnam
> > > >
> > >
> >
>
--
Thanks,
Karthik
Re: [DISCUSS] At-rest encryption for shuffle data on workers and tiered storage
Posted by Aravind Patnam <ak...@gmail.com>.
Let me start a thread for a CIP later this week.
Aravind K. Patnam
On Mon, Apr 20, 2026 at 8:37 AM Mridul Muralidharan <mr...@gmail.com>
wrote:
> Hi,
>
> This is pretty much what Aravind has implemented internally :-)
>
> Regards,
> Mridul
>
> On Mon, Apr 20, 2026 at 8:34 AM rexxiong <re...@apache.org> wrote:
>
> > Hi Karthik,
> >
> > Thanks for the thorough proposal. I'd like to suggest an alternative
> > approach: client-side encryption, where encryption/decryption happens
> > entirely in the Spark
> > client — similar to how compression works today. Celeborn workers and
> > master would only ever see ciphertext.
> >
> > Rationale:
> >
> > 1. Simpler architecture — no worker write/read path changes. Celeborn
> > remains a stateless byte pipe, just like it is for compression.
> > 2. Stronger security — Plaintext and encryption keys never reach workers
> or
> > master. The trust boundary stays within the client, eliminating the need
> > for KMS
> > credentials on the server side.
> > 3. No sendfile regression — Since workers store ciphertext natively,
> > CELEBORN-2301's zero-copy sendfile works unchanged for all workloads,
> > encrypted or not.
> > 4. Aligns with Spark — This naturally respects
> spark.io.encryption.enabled
> > and we can reuse Spark's existing key distribution via IOEncryptionKey.
> >
> > Happy to discuss further.
> >
> > Regards,
> > Jiashu Xiong
> >
> > Aravind Patnam <ak...@gmail.com> 于2026年4月20日周一 05:08写道：
> >
> > > Hi,
> > >
> > > We already have EAR for celeborn shuffle data internally at LinkedIn,
> > where
> > > we have added this support to respect the existing
> > > spark.io.encryption.enabled config in Spark on the client side.
> > >
> > > I am happy to contribute this back and start a CIP for this next week.
> > >
> > > Thanks,
> > > Aravind
> > >
> > >
> > >
> > > On Sat, Apr 18, 2026 at 10:24 PM Karthik Prabhakar <
> > knkarthik01@gmail.com>
> > > wrote:
> > >
> > > > Hi dev@,
> > > >
> > > > I’d like to propose adding at-rest encryption for shuffle data in
> > > Celeborn
> > > > and would appreciate the community’s input before writing a full
> > > > implementation.
> > > > cURRENT gap
> > > >
> > > > Celeborn encrypts data in transit (TLS, SASL) but not at rest. When a
> > > > worker flushes shuffle data to local disk, HDFS, S3, or OSS, the
> bytes
> > > land
> > > > as plaintext.
> > > >
> > > > The only write site for local disk is LocalFlushTask.flush() in
> > > > FlushTask.scala (L66, L71 at commit a56f69a), which calls
> > > > fileChannel.write(buffer) with no cipher transform. The
> tiered-storage
> > > > paths (HdfsFlushTask, S3FlushTask, OssFlushTask) are the same — raw
> > bytes
> > > > to the underlying store.
> > > >
> > > > Verified with:
> > > >
> > > > grep -rnE 'cipher|\.encrypt|aes|envelope' worker/src/main/
> > > > grep -rn 'javax\.crypto' worker/src/main/
> > > > (both zero matches)
> > > >
> > > > This matters because spark.io.encryption.enabled does *not* cover the
> > > > Celeborn path. When Celeborn’s ShuffleManager replaces Spark’s
> shuffle
> > > > writer, Spark’s encryption key is never consulted — confirmed by
> > grepping
> > > > client-spark/ for IOEncryptionKey (zero matches).
> > > >
> > > > Teams adopting Celeborn for performance silently lose
> > shuffle-encryption
> > > > guarantees their compliance posture may assume.
> > > > Who Needs This
> > > >
> > > > - Regulated industries (healthcare, finance, public sector) whose
> > > > auditors require application-layer encryption independent of
> > > disk/volume
> > > > encryption.
> > > > - Multi-tenant platforms needing cryptographic isolation between
> > > tenants
> > > > on shared workers.
> > > > - Teams using object-store tiering who want encryption before
> > offload.
> > > >
> > > > Proposed Approach (High Level)
> > > >
> > > > 1. A *StreamCipher SPI* in common/ for wrapping
> WritableByteChannel
> > /
> > > > ReadableByteChannel with encrypt/decrypt. No KMS SDK in core.
> > > > 2. A *KeyService SPI* for envelope encryption — generate/unwrap
> DEKs
> > > > using a KMS-held KEK. Implementations live in separate optional
> > > modules
> > > > (
> > > > aws-kms, gcp-kms, azure-kv, vault, static for dev/PoC).
> > > > 3. Wire into the worker write path: LocalFlushTask wraps
> fileChannel
> > > > with StreamCipher.wrapForWrite(). Same for HDFS/S3/OSS flush
> tasks.
> > > > 4. Wire into the reader path: LocalPartitionDataReader detects a
> > > 16-byte
> > > > encrypted-file header, unwraps the DEK (cached per
> worker+shuffle),
> > > > wraps
> > > > the channel with StreamCipher.wrapForRead().
> > > > 5. Opt-in via celeborn.shuffle.io.encryption.enabled=true.
> Default
> > > off.
> > > > Unencrypted deployments are byte-identical to today, zero
> overhead.
> > > > 6. Per-shuffle DEKs by default (one KMS call per shuffle
> > reservation,
> > > > amortized). Per-application DEK scope as an option.
> > > >
> > > > Interaction with Recent Work
> > > >
> > > > CELEBORN-2301 (commit 95419e1) recently landed enhanced zero-copy
> > > sendfile
> > > > for FileRegion on native transports — a nice throughput win for the
> > fetch
> > > > path.
> > > >
> > > > Encryption and sendfile are fundamentally incompatible: sendfile(2)
> > > cannot
> > > > transform bytes, so encrypted partitions must use a buffered read
> path.
> > > > This is only relevant for encrypted workloads; unencrypted workloads
> on
> > > the
> > > > same cluster keep the full CELEBORN-2301 benefit. Per-application
> > > > encryption flags (not per-cluster) would let encrypted and
> unencrypted
> > > apps
> > > > coexist without regressing the latter.
> > > > Questions for the Community
> > > >
> > > > Trimming to three since these are the ones I’d need opinions on
> before
> > > > writing code. Happy to take the rest up in follow-ups.
> > > >
> > > > - Any prior design work or internal discussion on this topic I
> > should
> > > > know about before proceeding?
> > > > - *Per-shuffle vs. per-application DEK scope* as the default?
> > > > Per-shuffle gives smaller blast radius and simpler lifecycle;
> > > > per-application amortizes KMS round-trips and is friendlier for
> > > > long-running jobs.
> > > > - *Key distribution path:* wrapped DEKs flow through Master
> metadata
> > > > (simpler, one KMS-aware role) vs. workers unwrap directly from KMS
> > > > (removes
> > > > Master from the key path, but every worker needs KMS credentials).
> > > > Preference?
> > > >
> > > > Tracking
> > > >
> > > > JIRA: CELEBORN-2311 <
> > https://issues.apache.org/jira/browse/CELEBORN-2311
> > > >
> > > >
> > > > I have a detailed design document with source citations, threat
> model,
> > > > performance analysis, and phased implementation plan. Happy to share
> > > > on-list or off-list if there’s interest.
> > > >
> > > > - Karthik
> > > >
> > >
> > >
> > > --
> > > Aravind K. Patnam
> > >
> >
>
Re: [DISCUSS] At-rest encryption for shuffle data on workers and tiered storage
Posted by Mridul Muralidharan <mr...@gmail.com>.
Hi,
This is pretty much what Aravind has implemented internally :-)
Regards,
Mridul
On Mon, Apr 20, 2026 at 8:34 AM rexxiong <re...@apache.org> wrote:
> Hi Karthik,
>
> Thanks for the thorough proposal. I'd like to suggest an alternative
> approach: client-side encryption, where encryption/decryption happens
> entirely in the Spark
> client — similar to how compression works today. Celeborn workers and
> master would only ever see ciphertext.
>
> Rationale:
>
> 1. Simpler architecture — no worker write/read path changes. Celeborn
> remains a stateless byte pipe, just like it is for compression.
> 2. Stronger security — Plaintext and encryption keys never reach workers or
> master. The trust boundary stays within the client, eliminating the need
> for KMS
> credentials on the server side.
> 3. No sendfile regression — Since workers store ciphertext natively,
> CELEBORN-2301's zero-copy sendfile works unchanged for all workloads,
> encrypted or not.
> 4. Aligns with Spark — This naturally respects spark.io.encryption.enabled
> and we can reuse Spark's existing key distribution via IOEncryptionKey.
>
> Happy to discuss further.
>
> Regards,
> Jiashu Xiong
>
> Aravind Patnam <ak...@gmail.com> 于2026年4月20日周一 05:08写道：
>
> > Hi,
> >
> > We already have EAR for celeborn shuffle data internally at LinkedIn,
> where
> > we have added this support to respect the existing
> > spark.io.encryption.enabled config in Spark on the client side.
> >
> > I am happy to contribute this back and start a CIP for this next week.
> >
> > Thanks,
> > Aravind
> >
> >
> >
> > On Sat, Apr 18, 2026 at 10:24 PM Karthik Prabhakar <
> knkarthik01@gmail.com>
> > wrote:
> >
> > > Hi dev@,
> > >
> > > I’d like to propose adding at-rest encryption for shuffle data in
> > Celeborn
> > > and would appreciate the community’s input before writing a full
> > > implementation.
> > > cURRENT gap
> > >
> > > Celeborn encrypts data in transit (TLS, SASL) but not at rest. When a
> > > worker flushes shuffle data to local disk, HDFS, S3, or OSS, the bytes
> > land
> > > as plaintext.
> > >
> > > The only write site for local disk is LocalFlushTask.flush() in
> > > FlushTask.scala (L66, L71 at commit a56f69a), which calls
> > > fileChannel.write(buffer) with no cipher transform. The tiered-storage
> > > paths (HdfsFlushTask, S3FlushTask, OssFlushTask) are the same — raw
> bytes
> > > to the underlying store.
> > >
> > > Verified with:
> > >
> > > grep -rnE 'cipher|\.encrypt|aes|envelope' worker/src/main/
> > > grep -rn 'javax\.crypto' worker/src/main/
> > > (both zero matches)
> > >
> > > This matters because spark.io.encryption.enabled does *not* cover the
> > > Celeborn path. When Celeborn’s ShuffleManager replaces Spark’s shuffle
> > > writer, Spark’s encryption key is never consulted — confirmed by
> grepping
> > > client-spark/ for IOEncryptionKey (zero matches).
> > >
> > > Teams adopting Celeborn for performance silently lose
> shuffle-encryption
> > > guarantees their compliance posture may assume.
> > > Who Needs This
> > >
> > > - Regulated industries (healthcare, finance, public sector) whose
> > > auditors require application-layer encryption independent of
> > disk/volume
> > > encryption.
> > > - Multi-tenant platforms needing cryptographic isolation between
> > tenants
> > > on shared workers.
> > > - Teams using object-store tiering who want encryption before
> offload.
> > >
> > > Proposed Approach (High Level)
> > >
> > > 1. A *StreamCipher SPI* in common/ for wrapping WritableByteChannel
> /
> > > ReadableByteChannel with encrypt/decrypt. No KMS SDK in core.
> > > 2. A *KeyService SPI* for envelope encryption — generate/unwrap DEKs
> > > using a KMS-held KEK. Implementations live in separate optional
> > modules
> > > (
> > > aws-kms, gcp-kms, azure-kv, vault, static for dev/PoC).
> > > 3. Wire into the worker write path: LocalFlushTask wraps fileChannel
> > > with StreamCipher.wrapForWrite(). Same for HDFS/S3/OSS flush tasks.
> > > 4. Wire into the reader path: LocalPartitionDataReader detects a
> > 16-byte
> > > encrypted-file header, unwraps the DEK (cached per worker+shuffle),
> > > wraps
> > > the channel with StreamCipher.wrapForRead().
> > > 5. Opt-in via celeborn.shuffle.io.encryption.enabled=true. Default
> > off.
> > > Unencrypted deployments are byte-identical to today, zero overhead.
> > > 6. Per-shuffle DEKs by default (one KMS call per shuffle
> reservation,
> > > amortized). Per-application DEK scope as an option.
> > >
> > > Interaction with Recent Work
> > >
> > > CELEBORN-2301 (commit 95419e1) recently landed enhanced zero-copy
> > sendfile
> > > for FileRegion on native transports — a nice throughput win for the
> fetch
> > > path.
> > >
> > > Encryption and sendfile are fundamentally incompatible: sendfile(2)
> > cannot
> > > transform bytes, so encrypted partitions must use a buffered read path.
> > > This is only relevant for encrypted workloads; unencrypted workloads on
> > the
> > > same cluster keep the full CELEBORN-2301 benefit. Per-application
> > > encryption flags (not per-cluster) would let encrypted and unencrypted
> > apps
> > > coexist without regressing the latter.
> > > Questions for the Community
> > >
> > > Trimming to three since these are the ones I’d need opinions on before
> > > writing code. Happy to take the rest up in follow-ups.
> > >
> > > - Any prior design work or internal discussion on this topic I
> should
> > > know about before proceeding?
> > > - *Per-shuffle vs. per-application DEK scope* as the default?
> > > Per-shuffle gives smaller blast radius and simpler lifecycle;
> > > per-application amortizes KMS round-trips and is friendlier for
> > > long-running jobs.
> > > - *Key distribution path:* wrapped DEKs flow through Master metadata
> > > (simpler, one KMS-aware role) vs. workers unwrap directly from KMS
> > > (removes
> > > Master from the key path, but every worker needs KMS credentials).
> > > Preference?
> > >
> > > Tracking
> > >
> > > JIRA: CELEBORN-2311 <
> https://issues.apache.org/jira/browse/CELEBORN-2311
> > >
> > >
> > > I have a detailed design document with source citations, threat model,
> > > performance analysis, and phased implementation plan. Happy to share
> > > on-list or off-list if there’s interest.
> > >
> > > - Karthik
> > >
> >
> >
> > --
> > Aravind K. Patnam
> >
>
Re: [DISCUSS] At-rest encryption for shuffle data on workers and tiered storage
Posted by rexxiong <re...@apache.org>.
Hi Karthik,
Thanks for the thorough proposal. I'd like to suggest an alternative
approach: client-side encryption, where encryption/decryption happens
entirely in the Spark
client — similar to how compression works today. Celeborn workers and
master would only ever see ciphertext.
Rationale:
1. Simpler architecture — no worker write/read path changes. Celeborn
remains a stateless byte pipe, just like it is for compression.
2. Stronger security — Plaintext and encryption keys never reach workers or
master. The trust boundary stays within the client, eliminating the need
for KMS
credentials on the server side.
3. No sendfile regression — Since workers store ciphertext natively,
CELEBORN-2301's zero-copy sendfile works unchanged for all workloads,
encrypted or not.
4. Aligns with Spark — This naturally respects spark.io.encryption.enabled
and we can reuse Spark's existing key distribution via IOEncryptionKey.
Happy to discuss further.
Regards,
Jiashu Xiong
Aravind Patnam <ak...@gmail.com> 于2026年4月20日周一 05:08写道：
> Hi,
>
> We already have EAR for celeborn shuffle data internally at LinkedIn, where
> we have added this support to respect the existing
> spark.io.encryption.enabled config in Spark on the client side.
>
> I am happy to contribute this back and start a CIP for this next week.
>
> Thanks,
> Aravind
>
>
>
> On Sat, Apr 18, 2026 at 10:24 PM Karthik Prabhakar <kn...@gmail.com>
> wrote:
>
> > Hi dev@,
> >
> > I’d like to propose adding at-rest encryption for shuffle data in
> Celeborn
> > and would appreciate the community’s input before writing a full
> > implementation.
> > cURRENT gap
> >
> > Celeborn encrypts data in transit (TLS, SASL) but not at rest. When a
> > worker flushes shuffle data to local disk, HDFS, S3, or OSS, the bytes
> land
> > as plaintext.
> >
> > The only write site for local disk is LocalFlushTask.flush() in
> > FlushTask.scala (L66, L71 at commit a56f69a), which calls
> > fileChannel.write(buffer) with no cipher transform. The tiered-storage
> > paths (HdfsFlushTask, S3FlushTask, OssFlushTask) are the same — raw bytes
> > to the underlying store.
> >
> > Verified with:
> >
> > grep -rnE 'cipher|\.encrypt|aes|envelope' worker/src/main/
> > grep -rn 'javax\.crypto' worker/src/main/
> > (both zero matches)
> >
> > This matters because spark.io.encryption.enabled does *not* cover the
> > Celeborn path. When Celeborn’s ShuffleManager replaces Spark’s shuffle
> > writer, Spark’s encryption key is never consulted — confirmed by grepping
> > client-spark/ for IOEncryptionKey (zero matches).
> >
> > Teams adopting Celeborn for performance silently lose shuffle-encryption
> > guarantees their compliance posture may assume.
> > Who Needs This
> >
> > - Regulated industries (healthcare, finance, public sector) whose
> > auditors require application-layer encryption independent of
> disk/volume
> > encryption.
> > - Multi-tenant platforms needing cryptographic isolation between
> tenants
> > on shared workers.
> > - Teams using object-store tiering who want encryption before offload.
> >
> > Proposed Approach (High Level)
> >
> > 1. A *StreamCipher SPI* in common/ for wrapping WritableByteChannel /
> > ReadableByteChannel with encrypt/decrypt. No KMS SDK in core.
> > 2. A *KeyService SPI* for envelope encryption — generate/unwrap DEKs
> > using a KMS-held KEK. Implementations live in separate optional
> modules
> > (
> > aws-kms, gcp-kms, azure-kv, vault, static for dev/PoC).
> > 3. Wire into the worker write path: LocalFlushTask wraps fileChannel
> > with StreamCipher.wrapForWrite(). Same for HDFS/S3/OSS flush tasks.
> > 4. Wire into the reader path: LocalPartitionDataReader detects a
> 16-byte
> > encrypted-file header, unwraps the DEK (cached per worker+shuffle),
> > wraps
> > the channel with StreamCipher.wrapForRead().
> > 5. Opt-in via celeborn.shuffle.io.encryption.enabled=true. Default
> off.
> > Unencrypted deployments are byte-identical to today, zero overhead.
> > 6. Per-shuffle DEKs by default (one KMS call per shuffle reservation,
> > amortized). Per-application DEK scope as an option.
> >
> > Interaction with Recent Work
> >
> > CELEBORN-2301 (commit 95419e1) recently landed enhanced zero-copy
> sendfile
> > for FileRegion on native transports — a nice throughput win for the fetch
> > path.
> >
> > Encryption and sendfile are fundamentally incompatible: sendfile(2)
> cannot
> > transform bytes, so encrypted partitions must use a buffered read path.
> > This is only relevant for encrypted workloads; unencrypted workloads on
> the
> > same cluster keep the full CELEBORN-2301 benefit. Per-application
> > encryption flags (not per-cluster) would let encrypted and unencrypted
> apps
> > coexist without regressing the latter.
> > Questions for the Community
> >
> > Trimming to three since these are the ones I’d need opinions on before
> > writing code. Happy to take the rest up in follow-ups.
> >
> > - Any prior design work or internal discussion on this topic I should
> > know about before proceeding?
> > - *Per-shuffle vs. per-application DEK scope* as the default?
> > Per-shuffle gives smaller blast radius and simpler lifecycle;
> > per-application amortizes KMS round-trips and is friendlier for
> > long-running jobs.
> > - *Key distribution path:* wrapped DEKs flow through Master metadata
> > (simpler, one KMS-aware role) vs. workers unwrap directly from KMS
> > (removes
> > Master from the key path, but every worker needs KMS credentials).
> > Preference?
> >
> > Tracking
> >
> > JIRA: CELEBORN-2311 <https://issues.apache.org/jira/browse/CELEBORN-2311
> >
> >
> > I have a detailed design document with source citations, threat model,
> > performance analysis, and phased implementation plan. Happy to share
> > on-list or off-list if there’s interest.
> >
> > - Karthik
> >
>
>
> --
> Aravind K. Patnam
>
Re: [DISCUSS] At-rest encryption for shuffle data on workers and tiered storage
Posted by Aravind Patnam <ak...@gmail.com>.
Hi,
We already have EAR for celeborn shuffle data internally at LinkedIn, where
we have added this support to respect the existing
spark.io.encryption.enabled config in Spark on the client side.
I am happy to contribute this back and start a CIP for this next week.
Thanks,
Aravind
On Sat, Apr 18, 2026 at 10:24 PM Karthik Prabhakar <kn...@gmail.com>
wrote:
> Hi dev@,
>
> I’d like to propose adding at-rest encryption for shuffle data in Celeborn
> and would appreciate the community’s input before writing a full
> implementation.
> cURRENT gap
>
> Celeborn encrypts data in transit (TLS, SASL) but not at rest. When a
> worker flushes shuffle data to local disk, HDFS, S3, or OSS, the bytes land
> as plaintext.
>
> The only write site for local disk is LocalFlushTask.flush() in
> FlushTask.scala (L66, L71 at commit a56f69a), which calls
> fileChannel.write(buffer) with no cipher transform. The tiered-storage
> paths (HdfsFlushTask, S3FlushTask, OssFlushTask) are the same — raw bytes
> to the underlying store.
>
> Verified with:
>
> grep -rnE 'cipher|\.encrypt|aes|envelope' worker/src/main/
> grep -rn 'javax\.crypto' worker/src/main/
> (both zero matches)
>
> This matters because spark.io.encryption.enabled does *not* cover the
> Celeborn path. When Celeborn’s ShuffleManager replaces Spark’s shuffle
> writer, Spark’s encryption key is never consulted — confirmed by grepping
> client-spark/ for IOEncryptionKey (zero matches).
>
> Teams adopting Celeborn for performance silently lose shuffle-encryption
> guarantees their compliance posture may assume.
> Who Needs This
>
> - Regulated industries (healthcare, finance, public sector) whose
> auditors require application-layer encryption independent of disk/volume
> encryption.
> - Multi-tenant platforms needing cryptographic isolation between tenants
> on shared workers.
> - Teams using object-store tiering who want encryption before offload.
>
> Proposed Approach (High Level)
>
> 1. A *StreamCipher SPI* in common/ for wrapping WritableByteChannel /
> ReadableByteChannel with encrypt/decrypt. No KMS SDK in core.
> 2. A *KeyService SPI* for envelope encryption — generate/unwrap DEKs
> using a KMS-held KEK. Implementations live in separate optional modules
> (
> aws-kms, gcp-kms, azure-kv, vault, static for dev/PoC).
> 3. Wire into the worker write path: LocalFlushTask wraps fileChannel
> with StreamCipher.wrapForWrite(). Same for HDFS/S3/OSS flush tasks.
> 4. Wire into the reader path: LocalPartitionDataReader detects a 16-byte
> encrypted-file header, unwraps the DEK (cached per worker+shuffle),
> wraps
> the channel with StreamCipher.wrapForRead().
> 5. Opt-in via celeborn.shuffle.io.encryption.enabled=true. Default off.
> Unencrypted deployments are byte-identical to today, zero overhead.
> 6. Per-shuffle DEKs by default (one KMS call per shuffle reservation,
> amortized). Per-application DEK scope as an option.
>
> Interaction with Recent Work
>
> CELEBORN-2301 (commit 95419e1) recently landed enhanced zero-copy sendfile
> for FileRegion on native transports — a nice throughput win for the fetch
> path.
>
> Encryption and sendfile are fundamentally incompatible: sendfile(2) cannot
> transform bytes, so encrypted partitions must use a buffered read path.
> This is only relevant for encrypted workloads; unencrypted workloads on the
> same cluster keep the full CELEBORN-2301 benefit. Per-application
> encryption flags (not per-cluster) would let encrypted and unencrypted apps
> coexist without regressing the latter.
> Questions for the Community
>
> Trimming to three since these are the ones I’d need opinions on before
> writing code. Happy to take the rest up in follow-ups.
>
> - Any prior design work or internal discussion on this topic I should
> know about before proceeding?
> - *Per-shuffle vs. per-application DEK scope* as the default?
> Per-shuffle gives smaller blast radius and simpler lifecycle;
> per-application amortizes KMS round-trips and is friendlier for
> long-running jobs.
> - *Key distribution path:* wrapped DEKs flow through Master metadata
> (simpler, one KMS-aware role) vs. workers unwrap directly from KMS
> (removes
> Master from the key path, but every worker needs KMS credentials).
> Preference?
>
> Tracking
>
> JIRA: CELEBORN-2311 <https://issues.apache.org/jira/browse/CELEBORN-2311>
>
> I have a detailed design document with source citations, threat model,
> performance analysis, and phased implementation plan. Happy to share
> on-list or off-list if there’s interest.
>
> - Karthik
>
--
Aravind K. Patnam