You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "Kalvin2077 (via GitHub)" <gi...@apache.org> on 2026/09/23 07:59:52 UTC

[PR] [CELEBORN-2468] Add transport security extension points for custom authn and authz [celeborn]

Kalvin2077 opened a new pull request, #3857:
URL: https://github.com/apache/celeborn/pull/3857

   ## What changes were proposed in this pull request?
   
   This PR adds transport security extension points so downstream deployments can plug custom authentication and authorization into Celeborn without forking the transport layer.
   
   - Add configurable `TransportClientBootstrap` and `TransportServerBootstrap` loading per transport module.
   - Add connection-local `ApplicationAuthorizer` and `ServiceAuthorizer`.
   - Propagate verified `UserIdentifier` into Master quota and Worker resource ownership.
   - Adjust client-side transport initialization to wait for LifecycleManager credentials.
   
   ## Why are the changes needed?
   
   Current authentication is tightly coupled to the native SASL/application-ID model. Downstream systems need to compose their own credential verification, tenant policy, and service-role policy with Celeborn without modifying core transport code.
   
   ## Does this PR resolve a correctness bug?
   
   - [ ] Yes
   
   
   ## Does this PR introduce any user-facing change?
   
   - [x] Yes
   
   Adds new extension interfaces and configuration keys. Default behavior remains unchanged when no external bootstrap is configured.
   
   ## How was this patch tested?
   
   - `TransportBootstrapSuiteJ`
   - `ExternalAuthenticationSuiteJ`
   - `RpcEndpointSecuritySuite`
   - `MasterSecuritySuite`
   - `ControllerSecuritySuite`
   - `AuthenticatedShuffleSuite`
   - Flink direct JUnit tests
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2468] Support pluggable transport authentication and authorization [celeborn]

Posted by "gauravkm (via GitHub)" <gi...@apache.org>.
gauravkm commented on PR #3857:
URL: https://github.com/apache/celeborn/pull/3857#issuecomment-5905743824

   CI update for `ef31bbb`: six additional SBT service checks have now failed: Java 8 / Scala [2.12.18](https://github.com/apache/celeborn/actions/runs/36674059363/job/109755070272) and [2.13.16](https://github.com/apache/celeborn/actions/runs/36674059363/job/109755070383), Java 11 / Scala [2.12.18](https://github.com/apache/celeborn/actions/runs/36674059363/job/109755070385) and [2.13.16](https://github.com/apache/celeborn/actions/runs/36674059363/job/109755070282), and Java 17 / Scala [2.12.18](https://github.com/apache/celeborn/actions/runs/36674059363/job/109755070438) and [2.13.16](https://github.com/apache/celeborn/actions/runs/36674059363/job/109755070356). The three Maven service failures in my previous summary remain. I have not established the causes of these additional failures; the earlier Java 11 Maven log diagnosis does not establish their cause.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2468] Support pluggable transport authentication and authorization [celeborn]

Posted by "Kalvin2077 (via GitHub)" <gi...@apache.org>.
Kalvin2077 closed pull request #3857: [CELEBORN-2468] Support pluggable transport authentication and authorization
URL: https://github.com/apache/celeborn/pull/3857


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2468] Add transport security extension points for custom authn and authz [celeborn]

Posted by "gauravkm (via GitHub)" <gi...@apache.org>.
gauravkm commented on PR #3857:
URL: https://github.com/apache/celeborn/pull/3857#issuecomment-5799088219

   CI has six failing service checks that need investigation: Java 8 / Scala [2.12.18](https://github.com/apache/celeborn/actions/runs/35834648560/job/107095096678) and [2.13.16](https://github.com/apache/celeborn/actions/runs/35834648560/job/107095096683), Java 11 / Scala [2.12.18](https://github.com/apache/celeborn/actions/runs/35834648560/job/107095096563) and [2.13.16](https://github.com/apache/celeborn/actions/runs/35834648560/job/107095096806), and Java 17 / Scala [2.12.18](https://github.com/apache/celeborn/actions/runs/35834648560/job/107095096497) and [2.13.16](https://github.com/apache/celeborn/actions/runs/35834648560/job/107095096697). The Java 11 / Scala 2.13 log reports a `StoragePolicyCase1` failure.
   
   The code review looks good; no introduced P0/P1 issues found. I checked bootstrap ordering and failure handling, native SASL compatibility, request/reply authorization, and verified quota/resource ownership. Added tests cover authentication rejection, cross-application access, reconnects, and Flink context cleanup.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2468] Add transport security extension points for custom authn and authz [celeborn]

Posted by "gauravkm (via GitHub)" <gi...@apache.org>.
gauravkm commented on code in PR #3857:
URL: https://github.com/apache/celeborn/pull/3857#discussion_r4129321822


##########
common/src/main/java/org/apache/celeborn/common/network/client/TransportClient.java:
##########
@@ -401,6 +406,58 @@ public void setClientId(String id) {
     this.clientId = id;
   }
 
+  /** Installs a connection-bound policy once, after the plugin has authenticated its peer. */
+  public synchronized void setSecurityContext(ConnectionSecurityContext context) {
+    Preconditions.checkNotNull(context, "securityContext");
+    Preconditions.checkState(securityContext == null, "Security context has already been set.");
+    securityContext = context;

Review Comment:
   CI has failing checks; I listed the current failures in the [review summary](https://github.com/apache/celeborn/pull/3857#issuecomment-5883036944).
   
   [P1] Wire the installed security context into RPC authorization. With a custom bootstrap authenticating a connection and installing a context here, but native SASL disabled, `clientId` remains null. A `PbUnregisterShuffle` request for another application then goes through `NettyRpcHandler` to `Master.receiveAndReply`; `RpcEndpoint.checkAuth` checks only the native client ID, so it permits the request without invoking the installed policy. Master can consequently remove another application’s shuffle even when that context would deny it. The new direct `client.authorize(...)` tests do not exercise this dispatch path. Please route RPC operations through the connection policy before state changes, and add a regression test that sends a cross-application unregister RPC through an authenticated connection and verifies both rejection and unchanged shuffle state.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2468] Support pluggable transport authentication and authorization [celeborn]

Posted by "Kalvin2077 (via GitHub)" <gi...@apache.org>.
Kalvin2077 commented on PR #3857:
URL: https://github.com/apache/celeborn/pull/3857#issuecomment-6058198098

   PTAL. @SteNicholas @zaynt4606 @RexXiong 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2468] Support pluggable transport authentication and authorization [celeborn]

Posted by "gauravkm (via GitHub)" <gi...@apache.org>.
gauravkm commented on PR #3857:
URL: https://github.com/apache/celeborn/pull/3857#issuecomment-5944479202

   CI on `a3e730a` has two failing checks: [flink2 (17, 2.2)](https://github.com/apache/celeborn/actions/runs/36954394027/job/110674032674) and [flink2 (11, 2.0)](https://github.com/apache/celeborn/actions/runs/36954394027/job/110674032638). The Java 17/Flink 2.2 log reports `RemoteShuffleMasterSuiteJV22.testRegisterPartitionWithProducer` failing its counter assertion at line 145: expected 2, observed 1. I have not established the root causes of these failures. Other checks are still running.
   
   Looks good from my code review; no P0/P1 issues found. Production authorization code is unchanged from `28d44d5`, and the prior dispatch fixes remain intact. I checked the child-JVM module flags, bounded listener-rebind assertion and per-test `MemoryManager` cleanup; the startup-failure, close-count and storage-tier assertions remain. This code-review conclusion does not clear the outstanding CI failures.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2468] Add transport security extension points for custom authn and authz [celeborn]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3857:
URL: https://github.com/apache/celeborn/pull/3857#discussion_r4091647676


##########
common/src/main/java/org/apache/celeborn/common/network/security/ServiceAuthorizer.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.common.network.security;
+
+import javax.annotation.Nullable;
+
+/** Connection-local policy for the Master's service request/reply operations. */
+@FunctionalInterface
+public interface ServiceAuthorizer {
+  enum Operation {
+    REGISTER_WORKER,
+    WORKER_HEARTBEAT,
+    REPORT_WORKER_UNAVAILABLE,
+    REPORT_WORKER_DECOMMISSION,
+    EXCLUDE_WORKERS,
+    WORKER_LOST,
+    WORKER_EVENT,
+    GET_APPLICATION_META,
+    REMOVE_WORKERS_UNAVAILABLE_INFO

Review Comment:
   `PbCheckWorkersAvailable` is not represented here and `Master.receiveAndReply` dispatches that RPC without calling `checkServiceAuth`. Since `LifecycleManager.checkWorkersAvailable()` uses this endpoint, a connection accepted by a custom service bootstrap can invoke it regardless of its service role. Add an operation and guard this case (with a regression test), or explicitly route it through the intended application policy.



##########
common/src/main/java/org/apache/celeborn/common/network/security/ServiceAuthorizer.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.common.network.security;
+
+import javax.annotation.Nullable;
+
+/** Connection-local policy for the Master's service request/reply operations. */
+@FunctionalInterface
+public interface ServiceAuthorizer {
+  enum Operation {
+    REGISTER_WORKER,
+    WORKER_HEARTBEAT,
+    REPORT_WORKER_UNAVAILABLE,
+    REPORT_WORKER_DECOMMISSION,
+    EXCLUDE_WORKERS,
+    WORKER_LOST,
+    WORKER_EVENT,
+    GET_APPLICATION_META,
+    REMOVE_WORKERS_UNAVAILABLE_INFO
+  }
+
+  /**
+   * Throws if this connection cannot perform the operation. {@code appId} identifies the target
+   * application for {@link Operation#GET_APPLICATION_META}; it is null for other operations. This
+   * policy does not cover one-way messages, which have no RPC call context.

Review Comment:
   This exemption leaves the one-way `PbWorkerLost` path outside the new authorization boundary: `Master.receive` handles the wire `WORKER_LOST` message without an `RpcCallContext`, while `Operation.WORKER_LOST` is checked only in `receiveAndReply`. A remotely authenticated connection can therefore send a one-way worker-loss message and mutate master state without its service policy being consulted. Route this operation through an authorization-bearing RPC or otherwise restrict the one-way path to trusted local messages instead of making the bypass part of the public contract.



##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala:
##########
@@ -621,6 +627,7 @@ private[celeborn] class Master(
           pb.getRequestId))
 
     case pb: PbWorkerLost =>
+      checkServiceAuth(context, Operation.WORKER_LOST)

Review Comment:
   This adds the service check only to the request/reply branch, but `Master.receive` still accepts `PbWorkerLost` as a one-way message and calls `handleWorkerLost` without any `ServiceAuthorizer` check. A remote authenticated client can therefore send the same worker-loss operation via `RpcEndpointRef.send` and bypass the new service-role policy. Remove the remote one-way handling or route it through an authenticated context (while preserving the internal timeout path).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2468] Add transport security extension points for custom authn and authz [celeborn]

Posted by "gauravkm (via GitHub)" <gi...@apache.org>.
gauravkm commented on PR #3857:
URL: https://github.com/apache/celeborn/pull/3857#issuecomment-5883036944

   CI currently has failing checks: [flink2 (11, 2.2)](https://github.com/apache/celeborn/actions/runs/36515990446/job/109238336028); [service (11)](https://github.com/apache/celeborn/actions/runs/36515990446/job/109238336010); [service (8)](https://github.com/apache/celeborn/actions/runs/36515990446/job/109238335905); [service (17)](https://github.com/apache/celeborn/actions/runs/36515990446/job/109238335702). Other checks are still running.
   
   I revisited the current revision and am withholding a positive code-review conclusion. The earlier “Looks good” comment should not be treated as clearance for this revision.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2468] Support pluggable transport authentication and authorization [celeborn]

Posted by "gauravkm (via GitHub)" <gi...@apache.org>.
gauravkm commented on PR #3857:
URL: https://github.com/apache/celeborn/pull/3857#issuecomment-5945045056

   CI update for `a3e730a`: [SBT service (Java 8 / Scala 2.12.18)](https://github.com/apache/celeborn/actions/runs/36954393843/job/110674031832) has also failed. Its log lists `GrpcRatisMasterStatusSystemSuiteJ` under failed tests. I have not established the cause. The two Flink failures in my [previous summary](https://github.com/apache/celeborn/pull/3857#issuecomment-5944479202) remain, bringing the current total to three failing checks; one Spark check is still running.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2468] Support pluggable transport authentication and authorization [celeborn]

Posted by "gauravkm (via GitHub)" <gi...@apache.org>.
gauravkm commented on PR #3857:
URL: https://github.com/apache/celeborn/pull/3857#issuecomment-5909712192

   CI update for `28d44d5`: [Spark 3.5 / Java 8 / LocalDiskShuffleDataIO](https://github.com/apache/celeborn/actions/runs/36685924172/job/109791656422) has also failed. The run now has nine failing checks, including the eight listed above, and no checks still running. I have not established the cause of this additional failure.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2468] Support pluggable transport authentication and authorization [celeborn]

Posted by "gauravkm (via GitHub)" <gi...@apache.org>.
gauravkm commented on PR #3857:
URL: https://github.com/apache/celeborn/pull/3857#issuecomment-5907687939

   CI on `28d44d5` has eight failing checks: Maven service Java [8](https://github.com/apache/celeborn/actions/runs/36685924172/job/109791656544), [11](https://github.com/apache/celeborn/actions/runs/36685924172/job/109791656091), [17](https://github.com/apache/celeborn/actions/runs/36685924172/job/109791656067); SBT service Java 11 / Scala [2.12.18](https://github.com/apache/celeborn/actions/runs/36685924160/job/109791656246), [2.13.16](https://github.com/apache/celeborn/actions/runs/36685924160/job/109791656186), Java 17 / Scala [2.12.18](https://github.com/apache/celeborn/actions/runs/36685924160/job/109791656106), [2.13.16](https://github.com/apache/celeborn/actions/runs/36685924160/job/109791656358); and [Spark 3.4 / Java 8 / LocalDiskShuffleDataIO](https://github.com/apache/celeborn/actions/runs/36685924172/job/109791656351). Their causes remain unverified; the previous revision’s whitespace diagnosis does not explain these failures.
   
   Looks good from my code review; no P0/P1 issues found. I verified that this revision adds only two trailing spaces to the generated configuration table, with all production and test files unchanged. The previously checked RPC authorization, identity propagation, replication and stream controls—and their test coverage—remain intact. CI investigation is still outstanding.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2468] Support pluggable transport authentication and authorization [celeborn]

Posted by "gauravkm (via GitHub)" <gi...@apache.org>.
gauravkm commented on code in PR #3857:
URL: https://github.com/apache/celeborn/pull/3857#discussion_r4141242400


##########
common/src/main/java/org/apache/celeborn/common/network/client/TransportClient.java:
##########
@@ -401,6 +406,58 @@ public void setClientId(String id) {
     this.clientId = id;
   }
 
+  /** Installs a connection-bound policy once, after the plugin has authenticated its peer. */
+  public synchronized void setSecurityContext(ConnectionSecurityContext context) {
+    Preconditions.checkNotNull(context, "securityContext");
+    Preconditions.checkState(securityContext == null, "Security context has already been set.");
+    securityContext = context;

Review Comment:
   CI still has the three Maven service failures linked in my [current summary](https://github.com/apache/celeborn/pull/3857#issuecomment-5905045796).
   
   I verified this P1 is fixed in `ef31bbb`: network dispatch preserves the receiving connection, `Inbox` authorizes before both handlers, and Master’s internal endpoint forwards the same context. Unregister now consults the installed policy before mutation even with a null native client ID. The new external/internal RPC denial tests exercise this boundary. Their unregister matrix starts empty, so it does not specifically assert preservation of a seeded victim shuffle; the dispatch ordering establishes the fix in source.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2468] Support pluggable transport authentication and authorization [celeborn]

Posted by "gauravkm (via GitHub)" <gi...@apache.org>.
gauravkm commented on PR #3857:
URL: https://github.com/apache/celeborn/pull/3857#issuecomment-5905045796

   CI is still failing: Maven [service (8)](https://github.com/apache/celeborn/actions/runs/36674059346/job/109755070246), [service (11)](https://github.com/apache/celeborn/actions/runs/36674059346/job/109755070240), and [service (17)](https://github.com/apache/celeborn/actions/runs/36674059346/job/109755070969). The Java 11 log shows a ConfigurationSuite golden-file mismatch for `docs/configuration/network.md` (trailing whitespace); I have not established the causes of the other two failures. Other checks are still running.
   
   Looks good from my code review of `ef31bbb`; no P0/P1 issues found. I checked remote versus local dispatch, internal forwarding, native/custom authentication composition, effective quota/ownership identity, replica permissions, stream controls and bootstrap cleanup. The shared dispatch hook applies policy before both request/reply and one-way processing. Added test source covers real RPC denial, worker-loss state preservation, reconnects and replicated shuffle data/ownership. This code-review conclusion does not clear the outstanding CI failures.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org