You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "aajisaka (via GitHub)" <gi...@apache.org> on 2026/03/26 01:44:02 UTC
[PR] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
aajisaka opened a new pull request, #55028:
URL: https://github.com/apache/spark/pull/55028
### What changes were proposed in this pull request?
This fixes two bugs in `GcmTransportCipher` introduced by SPARK-47172.
Bug 1: `DecryptionHandler` silently drops every message after the first.
`AesGcmHkdfStreaming` is a one-shot streaming primitive — each independently encrypted message carries its own random IV and requires a fresh `StreamSegmentDecrypter`. The `DecryptionHandler` never reset its per-message state (`completed`, `decrypterInit`, `expectedLength`, `segmentNumber`, etc.) nor replaced the single `final StreamSegmentDecrypter` instance between messages. After the first message was decoded, `completed = true` permanently, and all subsequent messages were silently dropped because both `initalizeExpectedLength()` and `initalizeDecrypter()` returned early as no-ops and the inner while loop never ran.
Fix: add `resetForNextMessage()` which clears all per-message fields and allocates a new `StreamSegmentDecrypter`; call it after each fully decoded message.
Bug 2: `DecryptionHandler` discards bytes from messages batched in the same `channelRead()` call.
Under shuffle load, TCP coalesces multiple encrypted messages into a single `ByteBuf`. The original code exited the decryption loop as soon as one message completed and released the buffer — including any trailing bytes belonging to subsequent messages. The next `channelRead()` then received bytes starting mid-stream of the second message, interpreted them as an 8-byte length header, and threw: `IllegalStateException: Invalid expected ciphertext length.`
Fix: wrap the decryption logic in an outer loop that continues consuming messages from the same buffer until either the buffer is exhausted or a partial message is encountered. `resetForNextMessage()` is called inside the loop immediately after each complete message while the buffer is still held.
Bug 3 (minor): `EncryptionHandler` shares working buffers across concurrent `GcmEncryptedMessage` instances.
`plaintextBuffer` and `ciphertextBuffer` were fields of `EncryptionHandler` passed into every `GcmEncryptedMessage`. The constructor's `ciphertextBuffer.limit(0)` call could corrupt an in-flight message's buffer state if Netty batched writes. Fix: move buffer ownership into `GcmEncryptedMessage` so each message allocates its own working buffers.
Without these fixes, enabling `AES/GCM/NoPadding` RPC encryption causes YARN executor containers to fail: the auth handshake succeeds but all post-auth RPC messages are dropped or corrupted, leaving the channel hung until YARN kills the container.
### Why are the changes needed?
To successfully run Spark jobs on YARN with `spark.network.crypto.cipher="AES/GCM/NoPadding"`
Fixes #54999
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added unit tests:
- `testMultipleMessages`: encrypts and decrypts two independent messages through the same handler pair with separate `channelRead()` calls.
- `testBatchedMessages`: concatenates two ciphertexts into one `ByteBuf` and delivers them in a single `channelRead()` call, verifying both are decoded correctly.
Ported these changes to our Spark 3.4.x-based internal branch and ran multiple jobs in YARN cluster successfully.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4233304396
The test failure is caused by https://github.com/apache/spark/pull/55071
I'll rebase the PR to the latest master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4884018717
Ah fantastic. That does make me feel better about this change. Let me take a closer look after the holiday weekend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "asf-gitbox-commits (via GitHub)" <gi...@apache.org>.
asf-gitbox-commits closed pull request #55028: [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel
URL: https://github.com/apache/spark/pull/55028
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4937253158
Thank you @holdenk, I've updated these backport PRs:
- 4.0: https://github.com/apache/spark/pull/57186
- 3.5: https://github.com/apache/spark/pull/55621
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on code in PR #55028:
URL: https://github.com/apache/spark/pull/55028#discussion_r3533042821
##########
common/network-common/src/main/java/org/apache/spark/network/crypto/GcmTransportCipher.java:
##########
@@ -354,60 +410,116 @@ public void channelRead(ChannelHandlerContext ctx, Object ciphertextMessage)
"Unrecognized message type: %s",
ciphertextMessage.getClass().getName());
ByteBuf ciphertextNettyBuf = (ByteBuf) ciphertextMessage;
- // The format of the output is:
+ // The format of each message is:
// [8 byte length][Internal IV and header][Ciphertext][Auth Tag]
+ //
+ // A single channelRead() call may deliver bytes from multiple back-to-back
+ // GCM messages (common under shuffle load when TCP coalesces writes). The
+ // outer loop processes as many complete messages as possible from the buffer
+ // before releasing it, so that bytes belonging to the next message are never
+ // discarded mid-stream.
try {
- if (!initalizeExpectedLength(ciphertextNettyBuf)) {
- // We have not read enough bytes to initialize the expected length.
- return;
- }
- if (!initalizeDecrypter(ciphertextNettyBuf)) {
- // We have not read enough bytes to initialize a header, needed to
- // initialize a decrypter.
- return;
- }
- int nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
- while (nettyBufReadableBytes > 0 && !completed) {
- // Read the ciphertext into the local buffer
- int readableBytes = Integer.min(
- nettyBufReadableBytes,
- ciphertextBuffer.remaining());
- int expectedRemaining = (int) (expectedLength - ciphertextRead);
- int bytesToRead = Integer.min(readableBytes, expectedRemaining);
- // The smallest ciphertext size is 16 bytes for the auth tag
- ciphertextBuffer.limit(ciphertextBuffer.position() + bytesToRead);
- ciphertextNettyBuf.readBytes(ciphertextBuffer);
- ciphertextRead += bytesToRead;
- // Check if this is the last segment
- if (ciphertextRead == expectedLength) {
- completed = true;
- } else if (ciphertextRead > expectedLength) {
- throw new IllegalStateException("Read more ciphertext than expected.");
+ while (true) {
+ if (!initializeExpectedLength(ciphertextNettyBuf)) {
+ // We have not read enough bytes to initialize the expected length.
+ break;
+ }
+ if (!initializeDecrypter(ciphertextNettyBuf)) {
+ // We have not read enough bytes to initialize a header, needed to
+ // initialize a decrypter.
+ break;
+ }
+ int nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
+ while (nettyBufReadableBytes > 0 && !completed) {
+ // Read the ciphertext into the local buffer
+ int readableBytes = Math.min(
+ nettyBufReadableBytes,
+ ciphertextBuffer.remaining());
+ int expectedRemaining = (int) (expectedLength - ciphertextRead);
+ int bytesToRead = Math.min(readableBytes, expectedRemaining);
+ // The smallest ciphertext size is 16 bytes for the auth tag
+ ciphertextBuffer.limit(ciphertextBuffer.position() + bytesToRead);
+ ciphertextNettyBuf.readBytes(ciphertextBuffer);
+ ciphertextRead += bytesToRead;
+ // Check if this is the last segment
+ if (ciphertextRead == expectedLength) {
+ completed = true;
+ } else if (ciphertextRead > expectedLength) {
+ throw new IllegalStateException("Read more ciphertext than expected.");
+ }
+ // If the ciphertext buffer is full, or this is the last segment,
+ // then decrypt it and fire a read.
+ if (ciphertextBuffer.limit() == ciphertextBuffer.capacity() || completed) {
+ ByteBuffer plaintextBuffer = ByteBuffer.allocate(plaintextSegmentSize);
+ ciphertextBuffer.flip();
+ decrypter.decryptSegment(
+ ciphertextBuffer,
+ segmentNumber,
+ completed,
+ plaintextBuffer);
+ segmentNumber++;
+ // Clear the ciphertext buffer because it's been read
+ ciphertextBuffer.clear();
+ plaintextBuffer.flip();
+ if (plaintextAccumulator == null) {
+ // Integer.MAX_VALUE disables consolidation entirely.
+ // CompositeByteBuf.newCompArray() always initialises the
+ // backing array to min(16, maxNumComponents) regardless of
+ // this value, so there is no upfront memory cost.
+ plaintextAccumulator =
+ Unpooled.compositeBuffer(Integer.MAX_VALUE);
+ }
+ // Zero-copy append: addComponent(true, ...) increases writerIndex
+ // so the component is immediately readable from the composite.
+ plaintextAccumulator.addComponent(
+ true, Unpooled.wrappedBuffer(plaintextBuffer));
+ } else {
+ // Set the ciphertext buffer up to read the next chunk
+ ciphertextBuffer.limit(ciphertextBuffer.capacity());
+ }
+ nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
}
- // If the ciphertext buffer is full, or this is the last segment,
- // then decrypt it and fire a read.
- if (ciphertextBuffer.limit() == ciphertextBuffer.capacity() || completed) {
- ByteBuffer plaintextBuffer = ByteBuffer.allocate(plaintextSegmentSize);
- ciphertextBuffer.flip();
- decrypter.decryptSegment(
- ciphertextBuffer,
- segmentNumber,
- completed,
- plaintextBuffer);
- segmentNumber++;
- // Clear the ciphertext buffer because it's been read
- ciphertextBuffer.clear();
- plaintextBuffer.flip();
- ctx.fireChannelRead(Unpooled.wrappedBuffer(plaintextBuffer));
- } else {
- // Set the ciphertext buffer up to read the next chunk
- ciphertextBuffer.limit(ciphertextBuffer.capacity());
+ if (!completed) {
+ // Partial message: more bytes needed from the next channelRead() call.
+ break;
}
- nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
+ // Fire the entire plaintext as a single event so that downstream
+ // handlers receive one callback per Spark message instead of one per
+ // 32 KB segment.
+ if (plaintextAccumulator != null) {
+ ctx.fireChannelRead(plaintextAccumulator);
Review Comment:
Fixed by https://github.com/apache/spark/pull/55028/changes/8c8ae520535ffd76f7d5419357f892d4b016bda9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #55028:
URL: https://github.com/apache/spark/pull/55028#discussion_r3532829465
##########
common/network-common/src/main/java/org/apache/spark/network/crypto/GcmTransportCipher.java:
##########
@@ -116,23 +108,37 @@ static class GcmEncryptedMessage extends AbstractFileRegion {
private final long encryptedCount;
GcmEncryptedMessage(AesGcmHkdfStreaming aesGcmHkdfStreaming,
- Object plaintextMessage,
- ByteBuffer plaintextBuffer,
- ByteBuffer ciphertextBuffer) throws GeneralSecurityException {
+ Object plaintextMessage) throws GeneralSecurityException {
JavaUtils.checkArgument(
plaintextMessage instanceof ByteBuf || plaintextMessage instanceof FileRegion,
"Unrecognized message type: %s", plaintextMessage.getClass().getName());
this.plaintextMessage = plaintextMessage;
- this.plaintextBuffer = plaintextBuffer;
- this.ciphertextBuffer = ciphertextBuffer;
+ this.plaintextBuffer =
+ ByteBuffer.allocate(aesGcmHkdfStreaming.getPlaintextSegmentSize());
+ this.ciphertextBuffer =
+ ByteBuffer.allocate(aesGcmHkdfStreaming.getCiphertextSegmentSize());
// If the ciphertext buffer cannot be fully written the target, transferTo may
// return with it containing some unwritten data. The initial call we'll explicitly
// set its limit to 0 to indicate the first call to transferTo.
this.ciphertextBuffer.limit(0);
this.bytesToRead = getReadableBytes();
- this.encryptedCount =
- LENGTH_HEADER_BYTES + aesGcmHkdfStreaming.expectedCiphertextSize(bytesToRead);
+ int plaintextSegmentSize = aesGcmHkdfStreaming.getPlaintextSegmentSize();
+ int ciphertextSegmentSize = aesGcmHkdfStreaming.getCiphertextSegmentSize();
+ int tagSize = ciphertextSegmentSize - plaintextSegmentSize;
+ long fullSegments = bytesToRead / plaintextSegmentSize;
+ long partialBytes = bytesToRead % plaintextSegmentSize;
+ // encryptedCount covers all bytes written to the wire:
+ // LENGTH_HEADER_BYTES – Spark's 8-byte big-endian framing prefix, read by the
Review Comment:
These added comments use non-ASCII typographic dashes (en-dash U+2013 here and at lines 134, 136, 311; em-dash U+2014 at `GcmAuthEngineSuite.java:293` and `:610`). Spark's convention is ASCII in code and comments — replace with `-` (or ` -- `). Easy to catch project-wide with `grep -nP "[^\x00-\x7F]"`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4927792110
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4233316003
We are still benchmarking the patch and we found performance issue with shuffle spills. Now we are investigating the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4131599868
Converted to draft. We are seeing job failures in benchmarking test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on code in PR #55028:
URL: https://github.com/apache/spark/pull/55028#discussion_r3469586016
##########
common/network-common/src/main/java/org/apache/spark/network/crypto/GcmTransportCipher.java:
##########
@@ -116,23 +108,40 @@ static class GcmEncryptedMessage extends AbstractFileRegion {
private final long encryptedCount;
GcmEncryptedMessage(AesGcmHkdfStreaming aesGcmHkdfStreaming,
- Object plaintextMessage,
- ByteBuffer plaintextBuffer,
- ByteBuffer ciphertextBuffer) throws GeneralSecurityException {
+ Object plaintextMessage) throws GeneralSecurityException {
JavaUtils.checkArgument(
plaintextMessage instanceof ByteBuf || plaintextMessage instanceof FileRegion,
"Unrecognized message type: %s", plaintextMessage.getClass().getName());
this.plaintextMessage = plaintextMessage;
- this.plaintextBuffer = plaintextBuffer;
- this.ciphertextBuffer = ciphertextBuffer;
+ this.plaintextBuffer =
+ ByteBuffer.allocate(aesGcmHkdfStreaming.getPlaintextSegmentSize());
+ this.ciphertextBuffer =
+ ByteBuffer.allocate(aesGcmHkdfStreaming.getCiphertextSegmentSize());
// If the ciphertext buffer cannot be fully written the target, transferTo may
// return with it containing some unwritten data. The initial call we'll explicitly
// set its limit to 0 to indicate the first call to transferTo.
this.ciphertextBuffer.limit(0);
this.bytesToRead = getReadableBytes();
- this.encryptedCount =
- LENGTH_HEADER_BYTES + aesGcmHkdfStreaming.expectedCiphertextSize(bytesToRead);
+ // Tink's expectedCiphertextSize(P) internally adds getCiphertextOffset() (the 24-byte
+ // streaming header) to P before computing the segment count:
+ // fullSegments = (P + getCiphertextOffset()) / plaintextSegmentSize
+ // This formula counts the header as occupying capacity in the first segment, so the
+ // effective plaintext capacity of segment 0 is (plaintextSegmentSize -
+ // getCiphertextOffset()) = 32728 bytes rather than 32752.
+ //
+ // However, transferTo() writes the streaming header separately (via headerByteBuffer)
+ // and passes all P bytes to encryptSegment() calls. For P in (32728, 32752], Tink's
+ // formula predicts two ciphertext segments but transferTo() produces only one,
+ // inflating encryptedCount by TAG_SIZE_IN_BYTES (16 bytes). The receiver then waits
+ // indefinitely for 16 bytes that were never written, causing a shuffle fetch stall.
+ //
+ // Fix: subtract getCiphertextOffset() before calling expectedCiphertextSize(), then
+ // add getHeaderLength() explicitly to account for the separately-written header.
+ this.encryptedCount = LENGTH_HEADER_BYTES
+ + aesGcmHkdfStreaming.getHeaderLength()
+ + aesGcmHkdfStreaming.expectedCiphertextSize(
+ bytesToRead - aesGcmHkdfStreaming.getCiphertextOffset());
Review Comment:
This seems suspcious to me we have header bytes that we're adding but were saying that we don't need those here because they're written out seperately and infact we're adding LENGTH_HEADER_BYTES and getHeaderLength.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4927945571
@aajisaka There is a merge conflict in 4.0, so i didn't merge in 4.0. If we need a fix in 4.0, please submit a backport PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on code in PR #55028:
URL: https://github.com/apache/spark/pull/55028#discussion_r3527519606
##########
common/network-common/src/main/java/org/apache/spark/network/crypto/GcmTransportCipher.java:
##########
@@ -354,60 +410,116 @@ public void channelRead(ChannelHandlerContext ctx, Object ciphertextMessage)
"Unrecognized message type: %s",
ciphertextMessage.getClass().getName());
ByteBuf ciphertextNettyBuf = (ByteBuf) ciphertextMessage;
- // The format of the output is:
+ // The format of each message is:
// [8 byte length][Internal IV and header][Ciphertext][Auth Tag]
+ //
+ // A single channelRead() call may deliver bytes from multiple back-to-back
+ // GCM messages (common under shuffle load when TCP coalesces writes). The
+ // outer loop processes as many complete messages as possible from the buffer
+ // before releasing it, so that bytes belonging to the next message are never
+ // discarded mid-stream.
try {
- if (!initalizeExpectedLength(ciphertextNettyBuf)) {
- // We have not read enough bytes to initialize the expected length.
- return;
- }
- if (!initalizeDecrypter(ciphertextNettyBuf)) {
- // We have not read enough bytes to initialize a header, needed to
- // initialize a decrypter.
- return;
- }
- int nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
- while (nettyBufReadableBytes > 0 && !completed) {
- // Read the ciphertext into the local buffer
- int readableBytes = Integer.min(
- nettyBufReadableBytes,
- ciphertextBuffer.remaining());
- int expectedRemaining = (int) (expectedLength - ciphertextRead);
- int bytesToRead = Integer.min(readableBytes, expectedRemaining);
- // The smallest ciphertext size is 16 bytes for the auth tag
- ciphertextBuffer.limit(ciphertextBuffer.position() + bytesToRead);
- ciphertextNettyBuf.readBytes(ciphertextBuffer);
- ciphertextRead += bytesToRead;
- // Check if this is the last segment
- if (ciphertextRead == expectedLength) {
- completed = true;
- } else if (ciphertextRead > expectedLength) {
- throw new IllegalStateException("Read more ciphertext than expected.");
+ while (true) {
+ if (!initializeExpectedLength(ciphertextNettyBuf)) {
+ // We have not read enough bytes to initialize the expected length.
+ break;
+ }
+ if (!initializeDecrypter(ciphertextNettyBuf)) {
+ // We have not read enough bytes to initialize a header, needed to
+ // initialize a decrypter.
+ break;
+ }
+ int nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
+ while (nettyBufReadableBytes > 0 && !completed) {
+ // Read the ciphertext into the local buffer
+ int readableBytes = Math.min(
+ nettyBufReadableBytes,
+ ciphertextBuffer.remaining());
+ int expectedRemaining = (int) (expectedLength - ciphertextRead);
+ int bytesToRead = Math.min(readableBytes, expectedRemaining);
+ // The smallest ciphertext size is 16 bytes for the auth tag
+ ciphertextBuffer.limit(ciphertextBuffer.position() + bytesToRead);
+ ciphertextNettyBuf.readBytes(ciphertextBuffer);
+ ciphertextRead += bytesToRead;
+ // Check if this is the last segment
+ if (ciphertextRead == expectedLength) {
+ completed = true;
+ } else if (ciphertextRead > expectedLength) {
+ throw new IllegalStateException("Read more ciphertext than expected.");
+ }
+ // If the ciphertext buffer is full, or this is the last segment,
+ // then decrypt it and fire a read.
+ if (ciphertextBuffer.limit() == ciphertextBuffer.capacity() || completed) {
+ ByteBuffer plaintextBuffer = ByteBuffer.allocate(plaintextSegmentSize);
+ ciphertextBuffer.flip();
+ decrypter.decryptSegment(
+ ciphertextBuffer,
+ segmentNumber,
+ completed,
+ plaintextBuffer);
+ segmentNumber++;
+ // Clear the ciphertext buffer because it's been read
+ ciphertextBuffer.clear();
+ plaintextBuffer.flip();
+ if (plaintextAccumulator == null) {
+ // Integer.MAX_VALUE disables consolidation entirely.
+ // CompositeByteBuf.newCompArray() always initialises the
+ // backing array to min(16, maxNumComponents) regardless of
+ // this value, so there is no upfront memory cost.
+ plaintextAccumulator =
+ Unpooled.compositeBuffer(Integer.MAX_VALUE);
+ }
+ // Zero-copy append: addComponent(true, ...) increases writerIndex
+ // so the component is immediately readable from the composite.
+ plaintextAccumulator.addComponent(
+ true, Unpooled.wrappedBuffer(plaintextBuffer));
+ } else {
+ // Set the ciphertext buffer up to read the next chunk
+ ciphertextBuffer.limit(ciphertextBuffer.capacity());
+ }
+ nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
}
- // If the ciphertext buffer is full, or this is the last segment,
- // then decrypt it and fire a read.
- if (ciphertextBuffer.limit() == ciphertextBuffer.capacity() || completed) {
- ByteBuffer plaintextBuffer = ByteBuffer.allocate(plaintextSegmentSize);
- ciphertextBuffer.flip();
- decrypter.decryptSegment(
- ciphertextBuffer,
- segmentNumber,
- completed,
- plaintextBuffer);
- segmentNumber++;
- // Clear the ciphertext buffer because it's been read
- ciphertextBuffer.clear();
- plaintextBuffer.flip();
- ctx.fireChannelRead(Unpooled.wrappedBuffer(plaintextBuffer));
- } else {
- // Set the ciphertext buffer up to read the next chunk
- ciphertextBuffer.limit(ciphertextBuffer.capacity());
+ if (!completed) {
+ // Partial message: more bytes needed from the next channelRead() call.
+ break;
}
- nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
+ // Fire the entire plaintext as a single event so that downstream
+ // handlers receive one callback per Spark message instead of one per
+ // 32 KB segment.
+ if (plaintextAccumulator != null) {
+ ctx.fireChannelRead(plaintextAccumulator);
+ plaintextAccumulator = null; // ownership transferred to downstream
+ }
+ // Current message is fully decoded. Reset state so the handler can
+ // decode the next independent GCM message on the same channel.
+ resetForNextMessage();
+ if (ciphertextNettyBuf.readableBytes() == 0) {
+ break;
+ }
+ // Remaining bytes may belong to another message; loop to process them.
}
} finally {
ciphertextNettyBuf.release();
}
}
+
+ @Override
+ public void channelInactive(ChannelHandlerContext ctx) throws Exception {
+ releaseAccumulator();
+ ctx.fireChannelInactive();
+ }
+
+ @Override
+ public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
+ releaseAccumulator();
+ ctx.close();
+ }
Review Comment:
Small thing, but I think this quietly changes the error-handling behaviour. Before this PR `DecryptionHandler` didn't override `exceptionCaught`, so a failure propagated down to `TransportChannelHandler`, which logs `Exception in connection from ...` and fails the outstanding RPCs with the actual cause. Closing the channel directly here means we lose that log line, and the pending RPCs only get failed later via `channelInactive` — without the cause.
Releasing the accumulator is definitely right; could we just propagate afterwards so the existing reporting still fires? Something like:
```suggestion
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
releaseAccumulator();
super.exceptionCaught(ctx, cause);
}
```
If the direct close was intentional (e.g. to keep expected disconnects quiet) feel free to ignore — I mostly wanted to make sure it wasn't accidental, since `channelInactive` right above does propagate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "roczei (via GitHub)" <gi...@apache.org>.
roczei commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4334739151
Hi @aajisaka,
I've verified that the latest version works correctly in local mode using Spark 4. However, after attempting a backport to Spark 3.5.4 for my YARN cluster, it is failing in both local and YARN modes. It's possible I incorrectly resolved a merge conflict during the backport. Would it be possible for you to provide a version compatible with this branch: https://github.com/apache/spark/tree/branch-3.5?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4891256187
Thank you for the detailed review. I really appreciate it.
> `testSplitHeader` exercises a split inside the Tink header nicely. The new partial-fill path for the 8-byte length prefix itself (say a split at byte 4) isn't directly covered, cheap to add a case for if you want the symmetry.
Added testSplitLengthPrefix to cover. Also, renamed `testSplitHeader` to `testSplitGcmHeader`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on code in PR #55028:
URL: https://github.com/apache/spark/pull/55028#discussion_r3533042039
##########
common/network-common/src/main/java/org/apache/spark/network/crypto/GcmTransportCipher.java:
##########
@@ -354,60 +410,116 @@ public void channelRead(ChannelHandlerContext ctx, Object ciphertextMessage)
"Unrecognized message type: %s",
ciphertextMessage.getClass().getName());
ByteBuf ciphertextNettyBuf = (ByteBuf) ciphertextMessage;
- // The format of the output is:
+ // The format of each message is:
// [8 byte length][Internal IV and header][Ciphertext][Auth Tag]
+ //
+ // A single channelRead() call may deliver bytes from multiple back-to-back
+ // GCM messages (common under shuffle load when TCP coalesces writes). The
+ // outer loop processes as many complete messages as possible from the buffer
+ // before releasing it, so that bytes belonging to the next message are never
+ // discarded mid-stream.
try {
- if (!initalizeExpectedLength(ciphertextNettyBuf)) {
- // We have not read enough bytes to initialize the expected length.
- return;
- }
- if (!initalizeDecrypter(ciphertextNettyBuf)) {
- // We have not read enough bytes to initialize a header, needed to
- // initialize a decrypter.
- return;
- }
- int nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
- while (nettyBufReadableBytes > 0 && !completed) {
- // Read the ciphertext into the local buffer
- int readableBytes = Integer.min(
- nettyBufReadableBytes,
- ciphertextBuffer.remaining());
- int expectedRemaining = (int) (expectedLength - ciphertextRead);
- int bytesToRead = Integer.min(readableBytes, expectedRemaining);
- // The smallest ciphertext size is 16 bytes for the auth tag
- ciphertextBuffer.limit(ciphertextBuffer.position() + bytesToRead);
- ciphertextNettyBuf.readBytes(ciphertextBuffer);
- ciphertextRead += bytesToRead;
- // Check if this is the last segment
- if (ciphertextRead == expectedLength) {
- completed = true;
- } else if (ciphertextRead > expectedLength) {
- throw new IllegalStateException("Read more ciphertext than expected.");
+ while (true) {
+ if (!initializeExpectedLength(ciphertextNettyBuf)) {
+ // We have not read enough bytes to initialize the expected length.
+ break;
+ }
+ if (!initializeDecrypter(ciphertextNettyBuf)) {
+ // We have not read enough bytes to initialize a header, needed to
+ // initialize a decrypter.
+ break;
+ }
+ int nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
+ while (nettyBufReadableBytes > 0 && !completed) {
+ // Read the ciphertext into the local buffer
+ int readableBytes = Math.min(
+ nettyBufReadableBytes,
+ ciphertextBuffer.remaining());
+ int expectedRemaining = (int) (expectedLength - ciphertextRead);
+ int bytesToRead = Math.min(readableBytes, expectedRemaining);
+ // The smallest ciphertext size is 16 bytes for the auth tag
+ ciphertextBuffer.limit(ciphertextBuffer.position() + bytesToRead);
+ ciphertextNettyBuf.readBytes(ciphertextBuffer);
+ ciphertextRead += bytesToRead;
+ // Check if this is the last segment
+ if (ciphertextRead == expectedLength) {
+ completed = true;
+ } else if (ciphertextRead > expectedLength) {
+ throw new IllegalStateException("Read more ciphertext than expected.");
+ }
+ // If the ciphertext buffer is full, or this is the last segment,
+ // then decrypt it and fire a read.
+ if (ciphertextBuffer.limit() == ciphertextBuffer.capacity() || completed) {
+ ByteBuffer plaintextBuffer = ByteBuffer.allocate(plaintextSegmentSize);
+ ciphertextBuffer.flip();
+ decrypter.decryptSegment(
+ ciphertextBuffer,
+ segmentNumber,
+ completed,
+ plaintextBuffer);
+ segmentNumber++;
+ // Clear the ciphertext buffer because it's been read
+ ciphertextBuffer.clear();
+ plaintextBuffer.flip();
+ if (plaintextAccumulator == null) {
+ // Integer.MAX_VALUE disables consolidation entirely.
+ // CompositeByteBuf.newCompArray() always initialises the
+ // backing array to min(16, maxNumComponents) regardless of
+ // this value, so there is no upfront memory cost.
+ plaintextAccumulator =
+ Unpooled.compositeBuffer(Integer.MAX_VALUE);
+ }
+ // Zero-copy append: addComponent(true, ...) increases writerIndex
+ // so the component is immediately readable from the composite.
+ plaintextAccumulator.addComponent(
Review Comment:
Thank you! Flush when the size become >= 1MB: https://github.com/apache/spark/pull/55028/commits/e10d2dce1898cbf7ffaeb36359e97166f5563298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on code in PR #55028:
URL: https://github.com/apache/spark/pull/55028#discussion_r3533039168
##########
common/network-common/src/main/java/org/apache/spark/network/crypto/GcmTransportCipher.java:
##########
@@ -116,23 +108,37 @@ static class GcmEncryptedMessage extends AbstractFileRegion {
private final long encryptedCount;
GcmEncryptedMessage(AesGcmHkdfStreaming aesGcmHkdfStreaming,
- Object plaintextMessage,
- ByteBuffer plaintextBuffer,
- ByteBuffer ciphertextBuffer) throws GeneralSecurityException {
+ Object plaintextMessage) throws GeneralSecurityException {
JavaUtils.checkArgument(
plaintextMessage instanceof ByteBuf || plaintextMessage instanceof FileRegion,
"Unrecognized message type: %s", plaintextMessage.getClass().getName());
this.plaintextMessage = plaintextMessage;
- this.plaintextBuffer = plaintextBuffer;
- this.ciphertextBuffer = ciphertextBuffer;
+ this.plaintextBuffer =
+ ByteBuffer.allocate(aesGcmHkdfStreaming.getPlaintextSegmentSize());
+ this.ciphertextBuffer =
+ ByteBuffer.allocate(aesGcmHkdfStreaming.getCiphertextSegmentSize());
// If the ciphertext buffer cannot be fully written the target, transferTo may
// return with it containing some unwritten data. The initial call we'll explicitly
// set its limit to 0 to indicate the first call to transferTo.
this.ciphertextBuffer.limit(0);
this.bytesToRead = getReadableBytes();
- this.encryptedCount =
- LENGTH_HEADER_BYTES + aesGcmHkdfStreaming.expectedCiphertextSize(bytesToRead);
+ int plaintextSegmentSize = aesGcmHkdfStreaming.getPlaintextSegmentSize();
+ int ciphertextSegmentSize = aesGcmHkdfStreaming.getCiphertextSegmentSize();
+ int tagSize = ciphertextSegmentSize - plaintextSegmentSize;
+ long fullSegments = bytesToRead / plaintextSegmentSize;
+ long partialBytes = bytesToRead % plaintextSegmentSize;
+ // encryptedCount covers all bytes written to the wire:
+ // LENGTH_HEADER_BYTES – Spark's 8-byte big-endian framing prefix, read by the
Review Comment:
Thanks. Fixed in https://github.com/apache/spark/pull/55028/commits/e10d2dce1898cbf7ffaeb36359e97166f5563298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on code in PR #55028:
URL: https://github.com/apache/spark/pull/55028#discussion_r3527519606
##########
common/network-common/src/main/java/org/apache/spark/network/crypto/GcmTransportCipher.java:
##########
@@ -354,60 +410,116 @@ public void channelRead(ChannelHandlerContext ctx, Object ciphertextMessage)
"Unrecognized message type: %s",
ciphertextMessage.getClass().getName());
ByteBuf ciphertextNettyBuf = (ByteBuf) ciphertextMessage;
- // The format of the output is:
+ // The format of each message is:
// [8 byte length][Internal IV and header][Ciphertext][Auth Tag]
+ //
+ // A single channelRead() call may deliver bytes from multiple back-to-back
+ // GCM messages (common under shuffle load when TCP coalesces writes). The
+ // outer loop processes as many complete messages as possible from the buffer
+ // before releasing it, so that bytes belonging to the next message are never
+ // discarded mid-stream.
try {
- if (!initalizeExpectedLength(ciphertextNettyBuf)) {
- // We have not read enough bytes to initialize the expected length.
- return;
- }
- if (!initalizeDecrypter(ciphertextNettyBuf)) {
- // We have not read enough bytes to initialize a header, needed to
- // initialize a decrypter.
- return;
- }
- int nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
- while (nettyBufReadableBytes > 0 && !completed) {
- // Read the ciphertext into the local buffer
- int readableBytes = Integer.min(
- nettyBufReadableBytes,
- ciphertextBuffer.remaining());
- int expectedRemaining = (int) (expectedLength - ciphertextRead);
- int bytesToRead = Integer.min(readableBytes, expectedRemaining);
- // The smallest ciphertext size is 16 bytes for the auth tag
- ciphertextBuffer.limit(ciphertextBuffer.position() + bytesToRead);
- ciphertextNettyBuf.readBytes(ciphertextBuffer);
- ciphertextRead += bytesToRead;
- // Check if this is the last segment
- if (ciphertextRead == expectedLength) {
- completed = true;
- } else if (ciphertextRead > expectedLength) {
- throw new IllegalStateException("Read more ciphertext than expected.");
+ while (true) {
+ if (!initializeExpectedLength(ciphertextNettyBuf)) {
+ // We have not read enough bytes to initialize the expected length.
+ break;
+ }
+ if (!initializeDecrypter(ciphertextNettyBuf)) {
+ // We have not read enough bytes to initialize a header, needed to
+ // initialize a decrypter.
+ break;
+ }
+ int nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
+ while (nettyBufReadableBytes > 0 && !completed) {
+ // Read the ciphertext into the local buffer
+ int readableBytes = Math.min(
+ nettyBufReadableBytes,
+ ciphertextBuffer.remaining());
+ int expectedRemaining = (int) (expectedLength - ciphertextRead);
+ int bytesToRead = Math.min(readableBytes, expectedRemaining);
+ // The smallest ciphertext size is 16 bytes for the auth tag
+ ciphertextBuffer.limit(ciphertextBuffer.position() + bytesToRead);
+ ciphertextNettyBuf.readBytes(ciphertextBuffer);
+ ciphertextRead += bytesToRead;
+ // Check if this is the last segment
+ if (ciphertextRead == expectedLength) {
+ completed = true;
+ } else if (ciphertextRead > expectedLength) {
+ throw new IllegalStateException("Read more ciphertext than expected.");
+ }
+ // If the ciphertext buffer is full, or this is the last segment,
+ // then decrypt it and fire a read.
+ if (ciphertextBuffer.limit() == ciphertextBuffer.capacity() || completed) {
+ ByteBuffer plaintextBuffer = ByteBuffer.allocate(plaintextSegmentSize);
+ ciphertextBuffer.flip();
+ decrypter.decryptSegment(
+ ciphertextBuffer,
+ segmentNumber,
+ completed,
+ plaintextBuffer);
+ segmentNumber++;
+ // Clear the ciphertext buffer because it's been read
+ ciphertextBuffer.clear();
+ plaintextBuffer.flip();
+ if (plaintextAccumulator == null) {
+ // Integer.MAX_VALUE disables consolidation entirely.
+ // CompositeByteBuf.newCompArray() always initialises the
+ // backing array to min(16, maxNumComponents) regardless of
+ // this value, so there is no upfront memory cost.
+ plaintextAccumulator =
+ Unpooled.compositeBuffer(Integer.MAX_VALUE);
+ }
+ // Zero-copy append: addComponent(true, ...) increases writerIndex
+ // so the component is immediately readable from the composite.
+ plaintextAccumulator.addComponent(
+ true, Unpooled.wrappedBuffer(plaintextBuffer));
+ } else {
+ // Set the ciphertext buffer up to read the next chunk
+ ciphertextBuffer.limit(ciphertextBuffer.capacity());
+ }
+ nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
}
- // If the ciphertext buffer is full, or this is the last segment,
- // then decrypt it and fire a read.
- if (ciphertextBuffer.limit() == ciphertextBuffer.capacity() || completed) {
- ByteBuffer plaintextBuffer = ByteBuffer.allocate(plaintextSegmentSize);
- ciphertextBuffer.flip();
- decrypter.decryptSegment(
- ciphertextBuffer,
- segmentNumber,
- completed,
- plaintextBuffer);
- segmentNumber++;
- // Clear the ciphertext buffer because it's been read
- ciphertextBuffer.clear();
- plaintextBuffer.flip();
- ctx.fireChannelRead(Unpooled.wrappedBuffer(plaintextBuffer));
- } else {
- // Set the ciphertext buffer up to read the next chunk
- ciphertextBuffer.limit(ciphertextBuffer.capacity());
+ if (!completed) {
+ // Partial message: more bytes needed from the next channelRead() call.
+ break;
}
- nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
+ // Fire the entire plaintext as a single event so that downstream
+ // handlers receive one callback per Spark message instead of one per
+ // 32 KB segment.
+ if (plaintextAccumulator != null) {
+ ctx.fireChannelRead(plaintextAccumulator);
+ plaintextAccumulator = null; // ownership transferred to downstream
+ }
+ // Current message is fully decoded. Reset state so the handler can
+ // decode the next independent GCM message on the same channel.
+ resetForNextMessage();
+ if (ciphertextNettyBuf.readableBytes() == 0) {
+ break;
+ }
+ // Remaining bytes may belong to another message; loop to process them.
}
} finally {
ciphertextNettyBuf.release();
}
}
+
+ @Override
+ public void channelInactive(ChannelHandlerContext ctx) throws Exception {
+ releaseAccumulator();
+ ctx.fireChannelInactive();
+ }
+
+ @Override
+ public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
+ releaseAccumulator();
+ ctx.close();
+ }
Review Comment:
Small thing, but I think this quietly changes the error-handling behaviour. Before this PR `DecryptionHandler` didn't override `exceptionCaught`, so a failure propagated down to `TransportChannelHandler`, which logs `Exception in connection from ...` and fails the outstanding RPCs with the actual cause. Closing the channel directly here means we lose that log line, and the pending RPCs only get failed later via `channelInactive`, without the cause.
Releasing the accumulator is definitely right; could we just propagate afterwards so the existing reporting still fires? Something like:
```suggestion
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
releaseAccumulator();
super.exceptionCaught(ctx, cause);
}
```
If the direct close was intentional (e.g. to keep expected disconnects quiet) feel free to ignore, I mostly wanted to make sure it wasn't accidental, since `channelInactive` right above does propagate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "roczei (via GitHub)" <gi...@apache.org>.
roczei commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4152765446
> @roczei Would you take a look?
@aajisaka
Yes, I'm going to try it this week both on YARN and in local mode as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "roczei (via GitHub)" <gi...@apache.org>.
roczei commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4226256759
@aajisaka,
There are two test failures:
On pull request update / Build (pull_request_target)Failing after 131m — Test results
Related CI links:
https://github.com/aajisaka/spark/actions/runs/23640067622/job/68858408200
https://productionresultssa19.blob.core.windows.net/actions-results/8ec6461f-81e9-4425-965c-961ce8c47c3f/workflow-job-run-eab23aab-74a3-5d68-b724-adc65d3ff4b9/logs/job/job-logs.txt?rsct=text%2Fplain&se=2026-04-10T18%3A03%3A06Z&sig=U2uDsNYF4X0M%2B0GqcFGYCSCVozlrSokhKaxPeLAabsE%3D&ske=2026-04-10T18%3A59%3A15Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2026-04-10T14%3A59%3A15Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-11-05&sp=r&spr=https&sr=b&st=2026-04-10T17%3A53%3A01Z&sv=2025-11-05
```
2026-03-27T10:52:26.2393208Z [0m[[0m[0minfo[0m] [0m[0m[31m- read stream-stream join v4 state change feed *** FAILED *** (2 seconds, 389 milliseconds)[0m[0m
2026-03-27T11:24:10.8625996Z [0m[[0m[0minfo[0m] [0m[0m[31m- read stream-stream join v4 state change feed *** FAILED *** (2 seconds, 924 milliseconds)[0m[0m
```
I've checked this locally and I believe it's just an intermittent failure. Could you please rerun the failed tests to get them passing? I'd like to ensure we have a completely green build for the committer/PMC review to keep things moving smoothly. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on code in PR #55028:
URL: https://github.com/apache/spark/pull/55028#discussion_r3493366606
##########
common/network-common/src/main/java/org/apache/spark/network/crypto/GcmTransportCipher.java:
##########
@@ -116,23 +108,40 @@ static class GcmEncryptedMessage extends AbstractFileRegion {
private final long encryptedCount;
GcmEncryptedMessage(AesGcmHkdfStreaming aesGcmHkdfStreaming,
- Object plaintextMessage,
- ByteBuffer plaintextBuffer,
- ByteBuffer ciphertextBuffer) throws GeneralSecurityException {
+ Object plaintextMessage) throws GeneralSecurityException {
JavaUtils.checkArgument(
plaintextMessage instanceof ByteBuf || plaintextMessage instanceof FileRegion,
"Unrecognized message type: %s", plaintextMessage.getClass().getName());
this.plaintextMessage = plaintextMessage;
- this.plaintextBuffer = plaintextBuffer;
- this.ciphertextBuffer = ciphertextBuffer;
+ this.plaintextBuffer =
+ ByteBuffer.allocate(aesGcmHkdfStreaming.getPlaintextSegmentSize());
+ this.ciphertextBuffer =
+ ByteBuffer.allocate(aesGcmHkdfStreaming.getCiphertextSegmentSize());
// If the ciphertext buffer cannot be fully written the target, transferTo may
// return with it containing some unwritten data. The initial call we'll explicitly
// set its limit to 0 to indicate the first call to transferTo.
this.ciphertextBuffer.limit(0);
this.bytesToRead = getReadableBytes();
- this.encryptedCount =
- LENGTH_HEADER_BYTES + aesGcmHkdfStreaming.expectedCiphertextSize(bytesToRead);
+ // Tink's expectedCiphertextSize(P) internally adds getCiphertextOffset() (the 24-byte
+ // streaming header) to P before computing the segment count:
+ // fullSegments = (P + getCiphertextOffset()) / plaintextSegmentSize
+ // This formula counts the header as occupying capacity in the first segment, so the
+ // effective plaintext capacity of segment 0 is (plaintextSegmentSize -
+ // getCiphertextOffset()) = 32728 bytes rather than 32752.
+ //
+ // However, transferTo() writes the streaming header separately (via headerByteBuffer)
+ // and passes all P bytes to encryptSegment() calls. For P in (32728, 32752], Tink's
+ // formula predicts two ciphertext segments but transferTo() produces only one,
+ // inflating encryptedCount by TAG_SIZE_IN_BYTES (16 bytes). The receiver then waits
+ // indefinitely for 16 bytes that were never written, causing a shuffle fetch stall.
+ //
+ // Fix: subtract getCiphertextOffset() before calling expectedCiphertextSize(), then
+ // add getHeaderLength() explicitly to account for the separately-written header.
+ this.encryptedCount = LENGTH_HEADER_BYTES
+ + aesGcmHkdfStreaming.getHeaderLength()
+ + aesGcmHkdfStreaming.expectedCiphertextSize(
+ bytesToRead - aesGcmHkdfStreaming.getCiphertextOffset());
Review Comment:
what is the legnth header bytes for then? I am confused why we're adding two different headers and its cryptography so I want to be more sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4404078383
Hello @Ngone51, would you take a look at this PR? I found your name at AES-GCM support PR: https://github.com/apache/spark/pull/46515. This fix is to support AES-GCM on YARN cluster with external shuffle service enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4899998636
Thank you @viirya! Removed non-ascii chars and added a new test case to test 1MB bounded flush.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4358277679
@roczei Thank you for the testing. Did you update the jar in both NodeManager (for external shuffle service) and Spark classpath (both driver and executor)?
The error message seems related to the Bug 1 in the description:
> Bug 1: DecryptionHandler silently drops every message after the first.
> In your setup, do you have the external shuffle service enabled?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4141381731
Our internal benchmark tests passed in YARN cluster. This patch is ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4152169216
@roczei Would you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4792555207
CC @mridulm & @Ngone51 & @sweisdb who were all involved in the original PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4927940144
Merged to master/4.x/4.2/4.1. Thanks every one!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4899835196
Thanks for working through all the rounds on this — the fix looks solid to me. Two minor things worth cleaning up before merge, neither a correctness blocker:
**Leftover non-ASCII in the test file.** Commit e10d2dce ("Remove non-ASCII characters") only touched `GcmTransportCipher.java`, so the two em-dashes I believe @HyukjinKwon flagged are still present at head:
- `GcmAuthEngineSuite.java:293` — `...silently dropped — which manifested as...`
- `GcmAuthEngineSuite.java:610` — `...(32728, 32752] — test lower boundary...`
CI won't catch these (the `nonascii` scalastyle rule only runs on Scala sources, not Java), but they go against the ASCII-in-comments convention. Replacing both `—` with `--` (or `-`) is all that's needed.
**The 1 MB bounded flush is untested, and one docstring now overstates the guarantee.** The `plaintextAccumulator.readableBytes() >= MAX_PLAINTEXT_BATCH_BYTES` flush branch added in 8c8ae52 is the subtlest new piece of logic here, but nothing exercises it — crossing 1 MB needs ~32 segments and the largest existing case (`testSingleFirePerMessage`) is 5 segments (~160 KB), staying under the threshold. A regression test with a > 1 MB plaintext, asserting multiple `fireChannelRead` calls whose concatenation round-trips to the original bytes, would lock in the batching behaviour. Relatedly, `testSingleFirePerMessage`'s docstring says it fires "exactly one `fireChannelRead` call regardless of the segment count" — that's no longer true above 1 MB; worth rewording to scope it to the small-message case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on code in PR #55028:
URL: https://github.com/apache/spark/pull/55028#discussion_r3533042821
##########
common/network-common/src/main/java/org/apache/spark/network/crypto/GcmTransportCipher.java:
##########
@@ -354,60 +410,116 @@ public void channelRead(ChannelHandlerContext ctx, Object ciphertextMessage)
"Unrecognized message type: %s",
ciphertextMessage.getClass().getName());
ByteBuf ciphertextNettyBuf = (ByteBuf) ciphertextMessage;
- // The format of the output is:
+ // The format of each message is:
// [8 byte length][Internal IV and header][Ciphertext][Auth Tag]
+ //
+ // A single channelRead() call may deliver bytes from multiple back-to-back
+ // GCM messages (common under shuffle load when TCP coalesces writes). The
+ // outer loop processes as many complete messages as possible from the buffer
+ // before releasing it, so that bytes belonging to the next message are never
+ // discarded mid-stream.
try {
- if (!initalizeExpectedLength(ciphertextNettyBuf)) {
- // We have not read enough bytes to initialize the expected length.
- return;
- }
- if (!initalizeDecrypter(ciphertextNettyBuf)) {
- // We have not read enough bytes to initialize a header, needed to
- // initialize a decrypter.
- return;
- }
- int nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
- while (nettyBufReadableBytes > 0 && !completed) {
- // Read the ciphertext into the local buffer
- int readableBytes = Integer.min(
- nettyBufReadableBytes,
- ciphertextBuffer.remaining());
- int expectedRemaining = (int) (expectedLength - ciphertextRead);
- int bytesToRead = Integer.min(readableBytes, expectedRemaining);
- // The smallest ciphertext size is 16 bytes for the auth tag
- ciphertextBuffer.limit(ciphertextBuffer.position() + bytesToRead);
- ciphertextNettyBuf.readBytes(ciphertextBuffer);
- ciphertextRead += bytesToRead;
- // Check if this is the last segment
- if (ciphertextRead == expectedLength) {
- completed = true;
- } else if (ciphertextRead > expectedLength) {
- throw new IllegalStateException("Read more ciphertext than expected.");
+ while (true) {
+ if (!initializeExpectedLength(ciphertextNettyBuf)) {
+ // We have not read enough bytes to initialize the expected length.
+ break;
+ }
+ if (!initializeDecrypter(ciphertextNettyBuf)) {
+ // We have not read enough bytes to initialize a header, needed to
+ // initialize a decrypter.
+ break;
+ }
+ int nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
+ while (nettyBufReadableBytes > 0 && !completed) {
+ // Read the ciphertext into the local buffer
+ int readableBytes = Math.min(
+ nettyBufReadableBytes,
+ ciphertextBuffer.remaining());
+ int expectedRemaining = (int) (expectedLength - ciphertextRead);
+ int bytesToRead = Math.min(readableBytes, expectedRemaining);
+ // The smallest ciphertext size is 16 bytes for the auth tag
+ ciphertextBuffer.limit(ciphertextBuffer.position() + bytesToRead);
+ ciphertextNettyBuf.readBytes(ciphertextBuffer);
+ ciphertextRead += bytesToRead;
+ // Check if this is the last segment
+ if (ciphertextRead == expectedLength) {
+ completed = true;
+ } else if (ciphertextRead > expectedLength) {
+ throw new IllegalStateException("Read more ciphertext than expected.");
+ }
+ // If the ciphertext buffer is full, or this is the last segment,
+ // then decrypt it and fire a read.
+ if (ciphertextBuffer.limit() == ciphertextBuffer.capacity() || completed) {
+ ByteBuffer plaintextBuffer = ByteBuffer.allocate(plaintextSegmentSize);
+ ciphertextBuffer.flip();
+ decrypter.decryptSegment(
+ ciphertextBuffer,
+ segmentNumber,
+ completed,
+ plaintextBuffer);
+ segmentNumber++;
+ // Clear the ciphertext buffer because it's been read
+ ciphertextBuffer.clear();
+ plaintextBuffer.flip();
+ if (plaintextAccumulator == null) {
+ // Integer.MAX_VALUE disables consolidation entirely.
+ // CompositeByteBuf.newCompArray() always initialises the
+ // backing array to min(16, maxNumComponents) regardless of
+ // this value, so there is no upfront memory cost.
+ plaintextAccumulator =
+ Unpooled.compositeBuffer(Integer.MAX_VALUE);
+ }
+ // Zero-copy append: addComponent(true, ...) increases writerIndex
+ // so the component is immediately readable from the composite.
+ plaintextAccumulator.addComponent(
+ true, Unpooled.wrappedBuffer(plaintextBuffer));
+ } else {
+ // Set the ciphertext buffer up to read the next chunk
+ ciphertextBuffer.limit(ciphertextBuffer.capacity());
+ }
+ nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
}
- // If the ciphertext buffer is full, or this is the last segment,
- // then decrypt it and fire a read.
- if (ciphertextBuffer.limit() == ciphertextBuffer.capacity() || completed) {
- ByteBuffer plaintextBuffer = ByteBuffer.allocate(plaintextSegmentSize);
- ciphertextBuffer.flip();
- decrypter.decryptSegment(
- ciphertextBuffer,
- segmentNumber,
- completed,
- plaintextBuffer);
- segmentNumber++;
- // Clear the ciphertext buffer because it's been read
- ciphertextBuffer.clear();
- plaintextBuffer.flip();
- ctx.fireChannelRead(Unpooled.wrappedBuffer(plaintextBuffer));
- } else {
- // Set the ciphertext buffer up to read the next chunk
- ciphertextBuffer.limit(ciphertextBuffer.capacity());
+ if (!completed) {
+ // Partial message: more bytes needed from the next channelRead() call.
+ break;
}
- nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
+ // Fire the entire plaintext as a single event so that downstream
+ // handlers receive one callback per Spark message instead of one per
+ // 32 KB segment.
+ if (plaintextAccumulator != null) {
+ ctx.fireChannelRead(plaintextAccumulator);
Review Comment:
Fixed by https://github.com/apache/spark/pull/55028/commits/e10d2dce1898cbf7ffaeb36359e97166f5563298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "dbtsai (via GitHub)" <gi...@apache.org>.
dbtsai commented on code in PR #55028:
URL: https://github.com/apache/spark/pull/55028#discussion_r3532888978
##########
common/network-common/src/main/java/org/apache/spark/network/crypto/GcmTransportCipher.java:
##########
@@ -116,23 +108,37 @@ static class GcmEncryptedMessage extends AbstractFileRegion {
private final long encryptedCount;
GcmEncryptedMessage(AesGcmHkdfStreaming aesGcmHkdfStreaming,
- Object plaintextMessage,
- ByteBuffer plaintextBuffer,
- ByteBuffer ciphertextBuffer) throws GeneralSecurityException {
+ Object plaintextMessage) throws GeneralSecurityException {
JavaUtils.checkArgument(
plaintextMessage instanceof ByteBuf || plaintextMessage instanceof FileRegion,
"Unrecognized message type: %s", plaintextMessage.getClass().getName());
this.plaintextMessage = plaintextMessage;
- this.plaintextBuffer = plaintextBuffer;
- this.ciphertextBuffer = ciphertextBuffer;
+ this.plaintextBuffer =
+ ByteBuffer.allocate(aesGcmHkdfStreaming.getPlaintextSegmentSize());
+ this.ciphertextBuffer =
+ ByteBuffer.allocate(aesGcmHkdfStreaming.getCiphertextSegmentSize());
// If the ciphertext buffer cannot be fully written the target, transferTo may
// return with it containing some unwritten data. The initial call we'll explicitly
// set its limit to 0 to indicate the first call to transferTo.
this.ciphertextBuffer.limit(0);
this.bytesToRead = getReadableBytes();
- this.encryptedCount =
- LENGTH_HEADER_BYTES + aesGcmHkdfStreaming.expectedCiphertextSize(bytesToRead);
+ int plaintextSegmentSize = aesGcmHkdfStreaming.getPlaintextSegmentSize();
+ int ciphertextSegmentSize = aesGcmHkdfStreaming.getCiphertextSegmentSize();
+ int tagSize = ciphertextSegmentSize - plaintextSegmentSize;
+ long fullSegments = bytesToRead / plaintextSegmentSize;
+ long partialBytes = bytesToRead % plaintextSegmentSize;
+ // encryptedCount covers all bytes written to the wire:
+ // LENGTH_HEADER_BYTES – Spark's 8-byte big-endian framing prefix, read by the
+ // receiver to know where this message ends in the TCP stream.
+ // getHeaderLength() – Tink's streaming header (salt + nonce prefix), written by
+ // the encrypter before any ciphertext segments.
+ // segment bytes – full segments each padded with a 16-byte GCM auth tag,
+ // plus any partial final segment similarly padded.
+ this.encryptedCount = LENGTH_HEADER_BYTES
+ + aesGcmHkdfStreaming.getHeaderLength()
+ + fullSegments * ciphertextSegmentSize
+ + (partialBytes > 0 ? partialBytes + tagSize : 0);
Review Comment:
**Minor -- `encryptedCount` hard-codes Tink's segmentation layout.**
The manual formula is correct today and matches the `transferTo` encrypt loop by construction, but it silently couples to Tink internals (segment sizes, 16-byte tag, header written separately). A future Tink streaming-layout change would resurface as a silent stall rather than a loud failure. `testEncryptedCountBoundary` guards current behavior; a one-line invariant note pointing at the coupling would help future maintainers.
##########
common/network-common/src/main/java/org/apache/spark/network/crypto/GcmTransportCipher.java:
##########
@@ -354,60 +410,116 @@ public void channelRead(ChannelHandlerContext ctx, Object ciphertextMessage)
"Unrecognized message type: %s",
ciphertextMessage.getClass().getName());
ByteBuf ciphertextNettyBuf = (ByteBuf) ciphertextMessage;
- // The format of the output is:
+ // The format of each message is:
// [8 byte length][Internal IV and header][Ciphertext][Auth Tag]
+ //
+ // A single channelRead() call may deliver bytes from multiple back-to-back
+ // GCM messages (common under shuffle load when TCP coalesces writes). The
+ // outer loop processes as many complete messages as possible from the buffer
+ // before releasing it, so that bytes belonging to the next message are never
+ // discarded mid-stream.
try {
- if (!initalizeExpectedLength(ciphertextNettyBuf)) {
- // We have not read enough bytes to initialize the expected length.
- return;
- }
- if (!initalizeDecrypter(ciphertextNettyBuf)) {
- // We have not read enough bytes to initialize a header, needed to
- // initialize a decrypter.
- return;
- }
- int nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
- while (nettyBufReadableBytes > 0 && !completed) {
- // Read the ciphertext into the local buffer
- int readableBytes = Integer.min(
- nettyBufReadableBytes,
- ciphertextBuffer.remaining());
- int expectedRemaining = (int) (expectedLength - ciphertextRead);
- int bytesToRead = Integer.min(readableBytes, expectedRemaining);
- // The smallest ciphertext size is 16 bytes for the auth tag
- ciphertextBuffer.limit(ciphertextBuffer.position() + bytesToRead);
- ciphertextNettyBuf.readBytes(ciphertextBuffer);
- ciphertextRead += bytesToRead;
- // Check if this is the last segment
- if (ciphertextRead == expectedLength) {
- completed = true;
- } else if (ciphertextRead > expectedLength) {
- throw new IllegalStateException("Read more ciphertext than expected.");
+ while (true) {
+ if (!initializeExpectedLength(ciphertextNettyBuf)) {
+ // We have not read enough bytes to initialize the expected length.
+ break;
+ }
+ if (!initializeDecrypter(ciphertextNettyBuf)) {
+ // We have not read enough bytes to initialize a header, needed to
+ // initialize a decrypter.
+ break;
+ }
+ int nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
+ while (nettyBufReadableBytes > 0 && !completed) {
+ // Read the ciphertext into the local buffer
+ int readableBytes = Math.min(
+ nettyBufReadableBytes,
+ ciphertextBuffer.remaining());
+ int expectedRemaining = (int) (expectedLength - ciphertextRead);
+ int bytesToRead = Math.min(readableBytes, expectedRemaining);
+ // The smallest ciphertext size is 16 bytes for the auth tag
+ ciphertextBuffer.limit(ciphertextBuffer.position() + bytesToRead);
+ ciphertextNettyBuf.readBytes(ciphertextBuffer);
+ ciphertextRead += bytesToRead;
+ // Check if this is the last segment
+ if (ciphertextRead == expectedLength) {
+ completed = true;
+ } else if (ciphertextRead > expectedLength) {
+ throw new IllegalStateException("Read more ciphertext than expected.");
+ }
+ // If the ciphertext buffer is full, or this is the last segment,
+ // then decrypt it and fire a read.
+ if (ciphertextBuffer.limit() == ciphertextBuffer.capacity() || completed) {
+ ByteBuffer plaintextBuffer = ByteBuffer.allocate(plaintextSegmentSize);
+ ciphertextBuffer.flip();
+ decrypter.decryptSegment(
+ ciphertextBuffer,
+ segmentNumber,
+ completed,
+ plaintextBuffer);
+ segmentNumber++;
+ // Clear the ciphertext buffer because it's been read
+ ciphertextBuffer.clear();
+ plaintextBuffer.flip();
+ if (plaintextAccumulator == null) {
+ // Integer.MAX_VALUE disables consolidation entirely.
+ // CompositeByteBuf.newCompArray() always initialises the
+ // backing array to min(16, maxNumComponents) regardless of
+ // this value, so there is no upfront memory cost.
+ plaintextAccumulator =
+ Unpooled.compositeBuffer(Integer.MAX_VALUE);
+ }
+ // Zero-copy append: addComponent(true, ...) increases writerIndex
+ // so the component is immediately readable from the composite.
+ plaintextAccumulator.addComponent(
+ true, Unpooled.wrappedBuffer(plaintextBuffer));
+ } else {
+ // Set the ciphertext buffer up to read the next chunk
+ ciphertextBuffer.limit(ciphertextBuffer.capacity());
+ }
+ nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
}
- // If the ciphertext buffer is full, or this is the last segment,
- // then decrypt it and fire a read.
- if (ciphertextBuffer.limit() == ciphertextBuffer.capacity() || completed) {
- ByteBuffer plaintextBuffer = ByteBuffer.allocate(plaintextSegmentSize);
- ciphertextBuffer.flip();
- decrypter.decryptSegment(
- ciphertextBuffer,
- segmentNumber,
- completed,
- plaintextBuffer);
- segmentNumber++;
- // Clear the ciphertext buffer because it's been read
- ciphertextBuffer.clear();
- plaintextBuffer.flip();
- ctx.fireChannelRead(Unpooled.wrappedBuffer(plaintextBuffer));
- } else {
- // Set the ciphertext buffer up to read the next chunk
- ciphertextBuffer.limit(ciphertextBuffer.capacity());
+ if (!completed) {
+ // Partial message: more bytes needed from the next channelRead() call.
+ break;
}
- nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
+ // Fire the entire plaintext as a single event so that downstream
+ // handlers receive one callback per Spark message instead of one per
+ // 32 KB segment.
+ if (plaintextAccumulator != null) {
+ ctx.fireChannelRead(plaintextAccumulator);
Review Comment:
**Nit -- null the accumulator before firing.**
Not a live bug in a standard Netty pipeline (a downstream `fireChannelRead` throw becomes an `exceptionCaught` event propagating downstream, not a synchronous re-entry here), but the idiomatic ordering removes reliance on that invariant:
```java
CompositeByteBuf out = plaintextAccumulator;
plaintextAccumulator = null;
ctx.fireChannelRead(out);
```
##########
common/network-common/src/main/java/org/apache/spark/network/crypto/GcmTransportCipher.java:
##########
@@ -354,60 +410,116 @@ public void channelRead(ChannelHandlerContext ctx, Object ciphertextMessage)
"Unrecognized message type: %s",
ciphertextMessage.getClass().getName());
ByteBuf ciphertextNettyBuf = (ByteBuf) ciphertextMessage;
- // The format of the output is:
+ // The format of each message is:
// [8 byte length][Internal IV and header][Ciphertext][Auth Tag]
+ //
+ // A single channelRead() call may deliver bytes from multiple back-to-back
+ // GCM messages (common under shuffle load when TCP coalesces writes). The
+ // outer loop processes as many complete messages as possible from the buffer
+ // before releasing it, so that bytes belonging to the next message are never
+ // discarded mid-stream.
try {
- if (!initalizeExpectedLength(ciphertextNettyBuf)) {
- // We have not read enough bytes to initialize the expected length.
- return;
- }
- if (!initalizeDecrypter(ciphertextNettyBuf)) {
- // We have not read enough bytes to initialize a header, needed to
- // initialize a decrypter.
- return;
- }
- int nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
- while (nettyBufReadableBytes > 0 && !completed) {
- // Read the ciphertext into the local buffer
- int readableBytes = Integer.min(
- nettyBufReadableBytes,
- ciphertextBuffer.remaining());
- int expectedRemaining = (int) (expectedLength - ciphertextRead);
- int bytesToRead = Integer.min(readableBytes, expectedRemaining);
- // The smallest ciphertext size is 16 bytes for the auth tag
- ciphertextBuffer.limit(ciphertextBuffer.position() + bytesToRead);
- ciphertextNettyBuf.readBytes(ciphertextBuffer);
- ciphertextRead += bytesToRead;
- // Check if this is the last segment
- if (ciphertextRead == expectedLength) {
- completed = true;
- } else if (ciphertextRead > expectedLength) {
- throw new IllegalStateException("Read more ciphertext than expected.");
+ while (true) {
+ if (!initializeExpectedLength(ciphertextNettyBuf)) {
+ // We have not read enough bytes to initialize the expected length.
+ break;
+ }
+ if (!initializeDecrypter(ciphertextNettyBuf)) {
+ // We have not read enough bytes to initialize a header, needed to
+ // initialize a decrypter.
+ break;
+ }
+ int nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
+ while (nettyBufReadableBytes > 0 && !completed) {
+ // Read the ciphertext into the local buffer
+ int readableBytes = Math.min(
+ nettyBufReadableBytes,
+ ciphertextBuffer.remaining());
+ int expectedRemaining = (int) (expectedLength - ciphertextRead);
+ int bytesToRead = Math.min(readableBytes, expectedRemaining);
+ // The smallest ciphertext size is 16 bytes for the auth tag
+ ciphertextBuffer.limit(ciphertextBuffer.position() + bytesToRead);
+ ciphertextNettyBuf.readBytes(ciphertextBuffer);
+ ciphertextRead += bytesToRead;
+ // Check if this is the last segment
+ if (ciphertextRead == expectedLength) {
+ completed = true;
+ } else if (ciphertextRead > expectedLength) {
+ throw new IllegalStateException("Read more ciphertext than expected.");
+ }
+ // If the ciphertext buffer is full, or this is the last segment,
+ // then decrypt it and fire a read.
+ if (ciphertextBuffer.limit() == ciphertextBuffer.capacity() || completed) {
+ ByteBuffer plaintextBuffer = ByteBuffer.allocate(plaintextSegmentSize);
+ ciphertextBuffer.flip();
+ decrypter.decryptSegment(
+ ciphertextBuffer,
+ segmentNumber,
+ completed,
+ plaintextBuffer);
+ segmentNumber++;
+ // Clear the ciphertext buffer because it's been read
+ ciphertextBuffer.clear();
+ plaintextBuffer.flip();
+ if (plaintextAccumulator == null) {
+ // Integer.MAX_VALUE disables consolidation entirely.
+ // CompositeByteBuf.newCompArray() always initialises the
+ // backing array to min(16, maxNumComponents) regardless of
+ // this value, so there is no upfront memory cost.
+ plaintextAccumulator =
+ Unpooled.compositeBuffer(Integer.MAX_VALUE);
+ }
+ // Zero-copy append: addComponent(true, ...) increases writerIndex
+ // so the component is immediately readable from the composite.
+ plaintextAccumulator.addComponent(
Review Comment:
**Major -- unbounded plaintext accumulation may regress the fetch-to-disk path.**
This handler is added `addFirst`, upstream of `TransportFrameDecoder`, whose interceptor is designed to consume bytes incrementally and explicitly must not retain the buffers passed to `handle()`. The previous per-segment `fireChannelRead` streamed ~32 KB at a time, keeping heap flat for large blocks. This code instead accumulates every segment of a message into the `CompositeByteBuf` and fires once per message.
For a large block routed to disk (above `spark.maxRemoteBlockSizeFetchToMem`), the entire block is now fully materialized on-heap before the frame decoder / `StreamInterceptor` can drain a single byte. On the send side a `FileRegion`/large body is written as a single object (`MessageEncoder`), so it becomes one `GcmEncryptedMessage` = one GCM stream = one accumulated buffer on receive. Under concurrent shuffle with encryption enabled this looks like an OOM vector and partially defeats fetch-to-disk.
Suggestion: bounded batching -- flush the accumulator when it exceeds a byte/segment threshold **or** the message completes. That keeps the "1 callback instead of N" EventLoop-fairness win for the common small-message case while capping transient memory for large ones.
Separately, the comment just above about "no upfront memory cost" is about the component *array*, not the retained data (the whole message); worth rewording so it doesn't read as reassurance about total memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on code in PR #55028:
URL: https://github.com/apache/spark/pull/55028#discussion_r3533042039
##########
common/network-common/src/main/java/org/apache/spark/network/crypto/GcmTransportCipher.java:
##########
@@ -354,60 +410,116 @@ public void channelRead(ChannelHandlerContext ctx, Object ciphertextMessage)
"Unrecognized message type: %s",
ciphertextMessage.getClass().getName());
ByteBuf ciphertextNettyBuf = (ByteBuf) ciphertextMessage;
- // The format of the output is:
+ // The format of each message is:
// [8 byte length][Internal IV and header][Ciphertext][Auth Tag]
+ //
+ // A single channelRead() call may deliver bytes from multiple back-to-back
+ // GCM messages (common under shuffle load when TCP coalesces writes). The
+ // outer loop processes as many complete messages as possible from the buffer
+ // before releasing it, so that bytes belonging to the next message are never
+ // discarded mid-stream.
try {
- if (!initalizeExpectedLength(ciphertextNettyBuf)) {
- // We have not read enough bytes to initialize the expected length.
- return;
- }
- if (!initalizeDecrypter(ciphertextNettyBuf)) {
- // We have not read enough bytes to initialize a header, needed to
- // initialize a decrypter.
- return;
- }
- int nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
- while (nettyBufReadableBytes > 0 && !completed) {
- // Read the ciphertext into the local buffer
- int readableBytes = Integer.min(
- nettyBufReadableBytes,
- ciphertextBuffer.remaining());
- int expectedRemaining = (int) (expectedLength - ciphertextRead);
- int bytesToRead = Integer.min(readableBytes, expectedRemaining);
- // The smallest ciphertext size is 16 bytes for the auth tag
- ciphertextBuffer.limit(ciphertextBuffer.position() + bytesToRead);
- ciphertextNettyBuf.readBytes(ciphertextBuffer);
- ciphertextRead += bytesToRead;
- // Check if this is the last segment
- if (ciphertextRead == expectedLength) {
- completed = true;
- } else if (ciphertextRead > expectedLength) {
- throw new IllegalStateException("Read more ciphertext than expected.");
+ while (true) {
+ if (!initializeExpectedLength(ciphertextNettyBuf)) {
+ // We have not read enough bytes to initialize the expected length.
+ break;
+ }
+ if (!initializeDecrypter(ciphertextNettyBuf)) {
+ // We have not read enough bytes to initialize a header, needed to
+ // initialize a decrypter.
+ break;
+ }
+ int nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
+ while (nettyBufReadableBytes > 0 && !completed) {
+ // Read the ciphertext into the local buffer
+ int readableBytes = Math.min(
+ nettyBufReadableBytes,
+ ciphertextBuffer.remaining());
+ int expectedRemaining = (int) (expectedLength - ciphertextRead);
+ int bytesToRead = Math.min(readableBytes, expectedRemaining);
+ // The smallest ciphertext size is 16 bytes for the auth tag
+ ciphertextBuffer.limit(ciphertextBuffer.position() + bytesToRead);
+ ciphertextNettyBuf.readBytes(ciphertextBuffer);
+ ciphertextRead += bytesToRead;
+ // Check if this is the last segment
+ if (ciphertextRead == expectedLength) {
+ completed = true;
+ } else if (ciphertextRead > expectedLength) {
+ throw new IllegalStateException("Read more ciphertext than expected.");
+ }
+ // If the ciphertext buffer is full, or this is the last segment,
+ // then decrypt it and fire a read.
+ if (ciphertextBuffer.limit() == ciphertextBuffer.capacity() || completed) {
+ ByteBuffer plaintextBuffer = ByteBuffer.allocate(plaintextSegmentSize);
+ ciphertextBuffer.flip();
+ decrypter.decryptSegment(
+ ciphertextBuffer,
+ segmentNumber,
+ completed,
+ plaintextBuffer);
+ segmentNumber++;
+ // Clear the ciphertext buffer because it's been read
+ ciphertextBuffer.clear();
+ plaintextBuffer.flip();
+ if (plaintextAccumulator == null) {
+ // Integer.MAX_VALUE disables consolidation entirely.
+ // CompositeByteBuf.newCompArray() always initialises the
+ // backing array to min(16, maxNumComponents) regardless of
+ // this value, so there is no upfront memory cost.
+ plaintextAccumulator =
+ Unpooled.compositeBuffer(Integer.MAX_VALUE);
+ }
+ // Zero-copy append: addComponent(true, ...) increases writerIndex
+ // so the component is immediately readable from the composite.
+ plaintextAccumulator.addComponent(
Review Comment:
Thank you! Flush when the size become >= 1MB: https://github.com/apache/spark/pull/55028/changes/8c8ae520535ffd76f7d5419357f892d4b016bda9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on code in PR #55028:
URL: https://github.com/apache/spark/pull/55028#discussion_r3478639693
##########
common/network-common/src/main/java/org/apache/spark/network/crypto/GcmTransportCipher.java:
##########
@@ -116,23 +108,40 @@ static class GcmEncryptedMessage extends AbstractFileRegion {
private final long encryptedCount;
GcmEncryptedMessage(AesGcmHkdfStreaming aesGcmHkdfStreaming,
- Object plaintextMessage,
- ByteBuffer plaintextBuffer,
- ByteBuffer ciphertextBuffer) throws GeneralSecurityException {
+ Object plaintextMessage) throws GeneralSecurityException {
JavaUtils.checkArgument(
plaintextMessage instanceof ByteBuf || plaintextMessage instanceof FileRegion,
"Unrecognized message type: %s", plaintextMessage.getClass().getName());
this.plaintextMessage = plaintextMessage;
- this.plaintextBuffer = plaintextBuffer;
- this.ciphertextBuffer = ciphertextBuffer;
+ this.plaintextBuffer =
+ ByteBuffer.allocate(aesGcmHkdfStreaming.getPlaintextSegmentSize());
+ this.ciphertextBuffer =
+ ByteBuffer.allocate(aesGcmHkdfStreaming.getCiphertextSegmentSize());
// If the ciphertext buffer cannot be fully written the target, transferTo may
// return with it containing some unwritten data. The initial call we'll explicitly
// set its limit to 0 to indicate the first call to transferTo.
this.ciphertextBuffer.limit(0);
this.bytesToRead = getReadableBytes();
- this.encryptedCount =
- LENGTH_HEADER_BYTES + aesGcmHkdfStreaming.expectedCiphertextSize(bytesToRead);
+ // Tink's expectedCiphertextSize(P) internally adds getCiphertextOffset() (the 24-byte
+ // streaming header) to P before computing the segment count:
+ // fullSegments = (P + getCiphertextOffset()) / plaintextSegmentSize
+ // This formula counts the header as occupying capacity in the first segment, so the
+ // effective plaintext capacity of segment 0 is (plaintextSegmentSize -
+ // getCiphertextOffset()) = 32728 bytes rather than 32752.
+ //
+ // However, transferTo() writes the streaming header separately (via headerByteBuffer)
+ // and passes all P bytes to encryptSegment() calls. For P in (32728, 32752], Tink's
+ // formula predicts two ciphertext segments but transferTo() produces only one,
+ // inflating encryptedCount by TAG_SIZE_IN_BYTES (16 bytes). The receiver then waits
+ // indefinitely for 16 bytes that were never written, causing a shuffle fetch stall.
+ //
+ // Fix: subtract getCiphertextOffset() before calling expectedCiphertextSize(), then
+ // add getHeaderLength() explicitly to account for the separately-written header.
+ this.encryptedCount = LENGTH_HEADER_BYTES
+ + aesGcmHkdfStreaming.getHeaderLength()
+ + aesGcmHkdfStreaming.expectedCiphertextSize(
+ bytesToRead - aesGcmHkdfStreaming.getCiphertextOffset());
Review Comment:
If I'm going to implement the logic without `expectedCiphertextSize()`, the calculation is like below:
```
int plaintextSegmentSize = aesGcmHkdfStreaming.getPlaintextSegmentSize();
int ciphertextSegmentSize = aesGcmHkdfStreaming.getCiphertextSegmentSize();
int tagSize = ciphertextSegmentSize - plaintextSegmentSize;
long fullSegments = bytesToRead / plaintextSegmentSize;
long partialBytes = bytesToRead % plaintextSegmentSize;
this.encryptedCount = LENGTH_HEADER_BYTES
+ aesGcmHkdfStreaming.getHeaderLength()
+ fullSegments * ciphertextSegmentSize
+ (partialBytes > 0 ? partialBytes + tagSize : 0);
```
which seems better?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4358317837
Added 1 commit for Java 8 compatibility in #55621. If you run the cluster with Java 8, would you try the latest one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4890842561
I've been asked to review this from a Security point of view and it looks good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on code in PR #55028:
URL: https://github.com/apache/spark/pull/55028#discussion_r3527755483
##########
common/network-common/src/main/java/org/apache/spark/network/crypto/GcmTransportCipher.java:
##########
@@ -354,60 +410,116 @@ public void channelRead(ChannelHandlerContext ctx, Object ciphertextMessage)
"Unrecognized message type: %s",
ciphertextMessage.getClass().getName());
ByteBuf ciphertextNettyBuf = (ByteBuf) ciphertextMessage;
- // The format of the output is:
+ // The format of each message is:
// [8 byte length][Internal IV and header][Ciphertext][Auth Tag]
+ //
+ // A single channelRead() call may deliver bytes from multiple back-to-back
+ // GCM messages (common under shuffle load when TCP coalesces writes). The
+ // outer loop processes as many complete messages as possible from the buffer
+ // before releasing it, so that bytes belonging to the next message are never
+ // discarded mid-stream.
try {
- if (!initalizeExpectedLength(ciphertextNettyBuf)) {
- // We have not read enough bytes to initialize the expected length.
- return;
- }
- if (!initalizeDecrypter(ciphertextNettyBuf)) {
- // We have not read enough bytes to initialize a header, needed to
- // initialize a decrypter.
- return;
- }
- int nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
- while (nettyBufReadableBytes > 0 && !completed) {
- // Read the ciphertext into the local buffer
- int readableBytes = Integer.min(
- nettyBufReadableBytes,
- ciphertextBuffer.remaining());
- int expectedRemaining = (int) (expectedLength - ciphertextRead);
- int bytesToRead = Integer.min(readableBytes, expectedRemaining);
- // The smallest ciphertext size is 16 bytes for the auth tag
- ciphertextBuffer.limit(ciphertextBuffer.position() + bytesToRead);
- ciphertextNettyBuf.readBytes(ciphertextBuffer);
- ciphertextRead += bytesToRead;
- // Check if this is the last segment
- if (ciphertextRead == expectedLength) {
- completed = true;
- } else if (ciphertextRead > expectedLength) {
- throw new IllegalStateException("Read more ciphertext than expected.");
+ while (true) {
+ if (!initializeExpectedLength(ciphertextNettyBuf)) {
+ // We have not read enough bytes to initialize the expected length.
+ break;
+ }
+ if (!initializeDecrypter(ciphertextNettyBuf)) {
+ // We have not read enough bytes to initialize a header, needed to
+ // initialize a decrypter.
+ break;
+ }
+ int nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
+ while (nettyBufReadableBytes > 0 && !completed) {
+ // Read the ciphertext into the local buffer
+ int readableBytes = Math.min(
+ nettyBufReadableBytes,
+ ciphertextBuffer.remaining());
+ int expectedRemaining = (int) (expectedLength - ciphertextRead);
+ int bytesToRead = Math.min(readableBytes, expectedRemaining);
+ // The smallest ciphertext size is 16 bytes for the auth tag
+ ciphertextBuffer.limit(ciphertextBuffer.position() + bytesToRead);
+ ciphertextNettyBuf.readBytes(ciphertextBuffer);
+ ciphertextRead += bytesToRead;
+ // Check if this is the last segment
+ if (ciphertextRead == expectedLength) {
+ completed = true;
+ } else if (ciphertextRead > expectedLength) {
+ throw new IllegalStateException("Read more ciphertext than expected.");
+ }
+ // If the ciphertext buffer is full, or this is the last segment,
+ // then decrypt it and fire a read.
+ if (ciphertextBuffer.limit() == ciphertextBuffer.capacity() || completed) {
+ ByteBuffer plaintextBuffer = ByteBuffer.allocate(plaintextSegmentSize);
+ ciphertextBuffer.flip();
+ decrypter.decryptSegment(
+ ciphertextBuffer,
+ segmentNumber,
+ completed,
+ plaintextBuffer);
+ segmentNumber++;
+ // Clear the ciphertext buffer because it's been read
+ ciphertextBuffer.clear();
+ plaintextBuffer.flip();
+ if (plaintextAccumulator == null) {
+ // Integer.MAX_VALUE disables consolidation entirely.
+ // CompositeByteBuf.newCompArray() always initialises the
+ // backing array to min(16, maxNumComponents) regardless of
+ // this value, so there is no upfront memory cost.
+ plaintextAccumulator =
+ Unpooled.compositeBuffer(Integer.MAX_VALUE);
+ }
+ // Zero-copy append: addComponent(true, ...) increases writerIndex
+ // so the component is immediately readable from the composite.
+ plaintextAccumulator.addComponent(
+ true, Unpooled.wrappedBuffer(plaintextBuffer));
+ } else {
+ // Set the ciphertext buffer up to read the next chunk
+ ciphertextBuffer.limit(ciphertextBuffer.capacity());
+ }
+ nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
}
- // If the ciphertext buffer is full, or this is the last segment,
- // then decrypt it and fire a read.
- if (ciphertextBuffer.limit() == ciphertextBuffer.capacity() || completed) {
- ByteBuffer plaintextBuffer = ByteBuffer.allocate(plaintextSegmentSize);
- ciphertextBuffer.flip();
- decrypter.decryptSegment(
- ciphertextBuffer,
- segmentNumber,
- completed,
- plaintextBuffer);
- segmentNumber++;
- // Clear the ciphertext buffer because it's been read
- ciphertextBuffer.clear();
- plaintextBuffer.flip();
- ctx.fireChannelRead(Unpooled.wrappedBuffer(plaintextBuffer));
- } else {
- // Set the ciphertext buffer up to read the next chunk
- ciphertextBuffer.limit(ciphertextBuffer.capacity());
+ if (!completed) {
+ // Partial message: more bytes needed from the next channelRead() call.
+ break;
}
- nettyBufReadableBytes = ciphertextNettyBuf.readableBytes();
+ // Fire the entire plaintext as a single event so that downstream
+ // handlers receive one callback per Spark message instead of one per
+ // 32 KB segment.
+ if (plaintextAccumulator != null) {
+ ctx.fireChannelRead(plaintextAccumulator);
+ plaintextAccumulator = null; // ownership transferred to downstream
+ }
+ // Current message is fully decoded. Reset state so the handler can
+ // decode the next independent GCM message on the same channel.
+ resetForNextMessage();
+ if (ciphertextNettyBuf.readableBytes() == 0) {
+ break;
+ }
+ // Remaining bytes may belong to another message; loop to process them.
}
} finally {
ciphertextNettyBuf.release();
}
}
+
+ @Override
+ public void channelInactive(ChannelHandlerContext ctx) throws Exception {
+ releaseAccumulator();
+ ctx.fireChannelInactive();
+ }
+
+ @Override
+ public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
+ releaseAccumulator();
+ ctx.close();
+ }
Review Comment:
Thank you! Fixed in https://github.com/apache/spark/pull/55028/commits/64c28209294e3c73c8c9aa53bf46a5e0fef428a8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "roczei (via GitHub)" <gi...@apache.org>.
roczei commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4371644332
@aajisaka
> Did you update the jar in both NodeManager (for external shuffle service) and Spark classpath (both driver and executor)?
I've identified the cause of the issue. The property spark.network.crypto.cipher="AES/GCM/NoPadding" needs to be added to yarn-site.xml to ensure the external shuffle service utilizes it. The following error will appear in the NodeManager log without this configuration:
```
2026-05-04 13:10:27,652 WARN org.apache.spark.network.server.TransportChannelHandler: Exception in connection from /10.140.126.143:33358
java.lang.IllegalArgumentException: Too large frame: 4683959760308586577
at org.sparkproject.guava.base.Preconditions.checkArgument(Preconditions.java:203)
at org.apache.spark.network.util.TransportFrameDecoder.decodeNext(TransportFrameDecoder.java:148)
at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:98)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at org.apache.spark.network.crypto.CtrTransportCipher$DecryptionHandler.channelRead(CtrTransportCipher.java:195)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at org.sparkproject.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at org.sparkproject.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
at org.sparkproject.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:171)
at org.sparkproject.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796)
at org.sparkproject.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732)
at org.sparkproject.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658)
at org.sparkproject.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at org.sparkproject.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
at org.sparkproject.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.sparkproject.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)
```
It works perfectly after setting the above.
> Added 1 commit for Java 8 compatibility in https://github.com/apache/spark/pull/55621. If you run the cluster with Java 8, would you try the latest one?
I've verified this with both JDK 17 and JDK 8. It functions correctly in both environments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on code in PR #55028:
URL: https://github.com/apache/spark/pull/55028#discussion_r3478616344
##########
common/network-common/src/main/java/org/apache/spark/network/crypto/GcmTransportCipher.java:
##########
@@ -116,23 +108,40 @@ static class GcmEncryptedMessage extends AbstractFileRegion {
private final long encryptedCount;
GcmEncryptedMessage(AesGcmHkdfStreaming aesGcmHkdfStreaming,
- Object plaintextMessage,
- ByteBuffer plaintextBuffer,
- ByteBuffer ciphertextBuffer) throws GeneralSecurityException {
+ Object plaintextMessage) throws GeneralSecurityException {
JavaUtils.checkArgument(
plaintextMessage instanceof ByteBuf || plaintextMessage instanceof FileRegion,
"Unrecognized message type: %s", plaintextMessage.getClass().getName());
this.plaintextMessage = plaintextMessage;
- this.plaintextBuffer = plaintextBuffer;
- this.ciphertextBuffer = ciphertextBuffer;
+ this.plaintextBuffer =
+ ByteBuffer.allocate(aesGcmHkdfStreaming.getPlaintextSegmentSize());
+ this.ciphertextBuffer =
+ ByteBuffer.allocate(aesGcmHkdfStreaming.getCiphertextSegmentSize());
// If the ciphertext buffer cannot be fully written the target, transferTo may
// return with it containing some unwritten data. The initial call we'll explicitly
// set its limit to 0 to indicate the first call to transferTo.
this.ciphertextBuffer.limit(0);
this.bytesToRead = getReadableBytes();
- this.encryptedCount =
- LENGTH_HEADER_BYTES + aesGcmHkdfStreaming.expectedCiphertextSize(bytesToRead);
+ // Tink's expectedCiphertextSize(P) internally adds getCiphertextOffset() (the 24-byte
+ // streaming header) to P before computing the segment count:
+ // fullSegments = (P + getCiphertextOffset()) / plaintextSegmentSize
+ // This formula counts the header as occupying capacity in the first segment, so the
+ // effective plaintext capacity of segment 0 is (plaintextSegmentSize -
+ // getCiphertextOffset()) = 32728 bytes rather than 32752.
+ //
+ // However, transferTo() writes the streaming header separately (via headerByteBuffer)
+ // and passes all P bytes to encryptSegment() calls. For P in (32728, 32752], Tink's
+ // formula predicts two ciphertext segments but transferTo() produces only one,
+ // inflating encryptedCount by TAG_SIZE_IN_BYTES (16 bytes). The receiver then waits
+ // indefinitely for 16 bytes that were never written, causing a shuffle fetch stall.
+ //
+ // Fix: subtract getCiphertextOffset() before calling expectedCiphertextSize(), then
+ // add getHeaderLength() explicitly to account for the separately-written header.
+ this.encryptedCount = LENGTH_HEADER_BYTES
+ + aesGcmHkdfStreaming.getHeaderLength()
+ + aesGcmHkdfStreaming.expectedCiphertextSize(
+ bytesToRead - aesGcmHkdfStreaming.getCiphertextOffset());
Review Comment:
Thank you for your review!
We need to subtract `getCiphertextOffset()` because it cancels the internal addition that Tink uses when counting the number of segments, correcting for the fact that our code writes the header separately. And we need to add `getHeaderLength()` back because the function's return value excludes the header.
Updated the code comment in https://github.com/apache/spark/pull/55028/commits/23a9622f96d4ce2073bebf8bcafaacdee3a87d71
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4351012724
Sure. Backported to branch-3.5 in #55621
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4932275084
Thank you all! I'll create a backport PR for 4.0 and update the branch-3.5 backport PR https://github.com/apache/spark/pull/55621
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4932828409
Awesome, thanks @aajisaka if you can ping me once you've got those updated I'd love to get this included in the next RC of 3.5 and 4.0 (going to try and roll it tomorrow).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on code in PR #55028:
URL: https://github.com/apache/spark/pull/55028#discussion_r3533044431
##########
common/network-common/src/main/java/org/apache/spark/network/crypto/GcmTransportCipher.java:
##########
@@ -116,23 +108,37 @@ static class GcmEncryptedMessage extends AbstractFileRegion {
private final long encryptedCount;
GcmEncryptedMessage(AesGcmHkdfStreaming aesGcmHkdfStreaming,
- Object plaintextMessage,
- ByteBuffer plaintextBuffer,
- ByteBuffer ciphertextBuffer) throws GeneralSecurityException {
+ Object plaintextMessage) throws GeneralSecurityException {
JavaUtils.checkArgument(
plaintextMessage instanceof ByteBuf || plaintextMessage instanceof FileRegion,
"Unrecognized message type: %s", plaintextMessage.getClass().getName());
this.plaintextMessage = plaintextMessage;
- this.plaintextBuffer = plaintextBuffer;
- this.ciphertextBuffer = ciphertextBuffer;
+ this.plaintextBuffer =
+ ByteBuffer.allocate(aesGcmHkdfStreaming.getPlaintextSegmentSize());
+ this.ciphertextBuffer =
+ ByteBuffer.allocate(aesGcmHkdfStreaming.getCiphertextSegmentSize());
// If the ciphertext buffer cannot be fully written the target, transferTo may
// return with it containing some unwritten data. The initial call we'll explicitly
// set its limit to 0 to indicate the first call to transferTo.
this.ciphertextBuffer.limit(0);
this.bytesToRead = getReadableBytes();
- this.encryptedCount =
- LENGTH_HEADER_BYTES + aesGcmHkdfStreaming.expectedCiphertextSize(bytesToRead);
+ int plaintextSegmentSize = aesGcmHkdfStreaming.getPlaintextSegmentSize();
+ int ciphertextSegmentSize = aesGcmHkdfStreaming.getCiphertextSegmentSize();
+ int tagSize = ciphertextSegmentSize - plaintextSegmentSize;
+ long fullSegments = bytesToRead / plaintextSegmentSize;
+ long partialBytes = bytesToRead % plaintextSegmentSize;
+ // encryptedCount covers all bytes written to the wire:
+ // LENGTH_HEADER_BYTES – Spark's 8-byte big-endian framing prefix, read by the
+ // receiver to know where this message ends in the TCP stream.
+ // getHeaderLength() – Tink's streaming header (salt + nonce prefix), written by
+ // the encrypter before any ciphertext segments.
+ // segment bytes – full segments each padded with a 16-byte GCM auth tag,
+ // plus any partial final segment similarly padded.
+ this.encryptedCount = LENGTH_HEADER_BYTES
+ + aesGcmHkdfStreaming.getHeaderLength()
+ + fullSegments * ciphertextSegmentSize
+ + (partialBytes > 0 ? partialBytes + tagSize : 0);
Review Comment:
Thanks. Added a note: https://github.com/apache/spark/pull/55028/commits/b451d691d2307192b2af6365e9a1d5ef7035f623
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4324642851
Updated. The additional commit contains 2 fixes:
1. Fix `GcmEncryptedMessage.encryptedCount` miscalculation, which caused Spark executor to stall if the message size is in a specific range.
2. Optimize `DecryptionHandler` to avoid heartbeat lost issue between Spark driver and executor.
Thanks @LamiumAmplexicaule for the stress tests and validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel [spark]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #55028:
URL: https://github.com/apache/spark/pull/55028#issuecomment-4332395570
@roczei Would you take a look again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org