You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "otterc (via GitHub)" <gi...@apache.org> on 2024/01/24 04:20:36 UTC

[PR] [CELEBORN-1251] Connect the server and client bootstraps to RpcEnv [incubator-celeborn]

otterc opened a new pull request, #2257:
URL: https://github.com/apache/incubator-celeborn/pull/2257

   ### What changes were proposed in this pull request?
   This connects client/server bootstraps to the RpcEnv in Celeborn.
   It is part of the epic: https://issues.apache.org/jira/browse/CELEBORN-1011.
   
   ### Why are the changes needed?
   The changes are needed for adding authentication to Celeborn. See [CELEBORN-1011](https://issues.apache.org/jira/browse/CELEBORN-1011).
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   ### How was this patch tested?
   Added some UTs
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1251] Connect the server and client bootstraps to RpcEnv [incubator-celeborn]

Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong closed pull request #2257: [CELEBORN-1251] Connect the server and client bootstraps to RpcEnv
URL: https://github.com/apache/incubator-celeborn/pull/2257


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1251] Connect the server and client bootstraps to RpcEnv [incubator-celeborn]

Posted by "otterc (via GitHub)" <gi...@apache.org>.
otterc commented on PR #2257:
URL: https://github.com/apache/incubator-celeborn/pull/2257#issuecomment-1909204040

   @waitinfuture @RexXiong @FMX @pan3793 Please help review
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1251] Connect the server and client bootstraps to RpcEnv [incubator-celeborn]

Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on PR #2257:
URL: https://github.com/apache/incubator-celeborn/pull/2257#issuecomment-1909744362

   Thanks, merge to main(v0.5.0)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1251] Connect the server and client bootstraps to RpcEnv [incubator-celeborn]

Posted by "otterc (via GitHub)" <gi...@apache.org>.
otterc commented on code in PR #2257:
URL: https://github.com/apache/incubator-celeborn/pull/2257#discussion_r1465420304


##########
common/src/test/scala/org/apache/celeborn/common/rpc/RpcSecurityContextSuite.scala:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.common.rpc
+
+import org.scalatest.matchers.must.Matchers.be
+import org.scalatest.matchers.should.Matchers.{an, convertToAnyShouldWrapper}
+
+import org.apache.celeborn.CelebornFunSuite
+import org.apache.celeborn.common.exception.CelebornException
+import org.apache.celeborn.common.network.sasl.{SaslCredentials, SecretRegistryImpl}
+import org.apache.celeborn.common.network.sasl.registration.RegistrationInfo
+import org.apache.celeborn.common.security.{ClientSaslContext, ClientSaslContextBuilder, RpcSecurityContextBuilder, ServerSaslContext, ServerSaslContextBuilder}
+
+class RpcSecurityContextSuite extends CelebornFunSuite {
+
+  test("RpcSecurityContext should be created with client and server sasl contexts") {
+    val clientContext = ClientSaslContext(
+      "clientAppId",
+      new SaslCredentials("user", "password"),
+      addRegistrationBootstrap = true)
+    val serverContext = ServerSaslContext(new SecretRegistryImpl())
+
+    val rpcSecurityContext = new RpcSecurityContextBuilder()
+      .withClientSaslContext(clientContext)
+      .withServerSaslContext(serverContext)
+      .build()
+
+    rpcSecurityContext.clientSaslContext shouldBe Some(clientContext)
+    rpcSecurityContext.serverSaslContext shouldBe Some(serverContext)

Review Comment:
   Yes, it will just one. I'll change it to reflect that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1251] Connect the server and client bootstraps to RpcEnv [incubator-celeborn]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #2257:
URL: https://github.com/apache/incubator-celeborn/pull/2257#issuecomment-1908969820

   ## [Codecov](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2257?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: `22 lines` in your changes are missing coverage. Please review.
   > Comparison is base [(`67e6cbf`)](https://app.codecov.io/gh/apache/incubator-celeborn/commit/67e6cbfb517ec192a048faede7e478eb42a3bd10?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 48.49% compared to head [(`519f21f`)](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2257?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 48.55%.
   > Report is 3 commits behind head on main.
   
   | [Files](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2257?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...apache/celeborn/common/rpc/netty/NettyRpcEnv.scala](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2257?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvY2VsZWJvcm4vY29tbW9uL3JwYy9uZXR0eS9OZXR0eVJwY0Vudi5zY2FsYQ==) | 25.00% | [21 Missing :warning: ](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2257?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [.../scala/org/apache/celeborn/common/rpc/RpcEnv.scala](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2257?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvY2VsZWJvcm4vY29tbW9uL3JwYy9ScGNFbnYuc2NhbGE=) | 80.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2257?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main    #2257      +/-   ##
   ==========================================
   + Coverage   48.49%   48.55%   +0.06%     
   ==========================================
     Files         196      196              
     Lines       12142    12173      +31     
     Branches     1069     1069              
   ==========================================
   + Hits         5887     5909      +22     
   - Misses       5868     5879      +11     
   + Partials      387      385       -2     
   ```
   
   
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2257?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1251] Connect the server and client bootstraps to RpcEnv [incubator-celeborn]

Posted by "otterc (via GitHub)" <gi...@apache.org>.
otterc commented on code in PR #2257:
URL: https://github.com/apache/incubator-celeborn/pull/2257#discussion_r1465568573


##########
common/src/main/scala/org/apache/celeborn/common/rpc/netty/NettyRpcEnv.scala:
##########
@@ -61,7 +65,31 @@ class NettyRpcEnv(
   private val transportContext =
     new TransportContext(transportConf, new NettyRpcHandler(dispatcher, this))
 
-  val clientFactory = transportContext.createClientFactory()
+  private def createClientBootstraps(): java.util.List[TransportClientBootstrap] = {
+    val bootstraps = new java.util.ArrayList[TransportClientBootstrap]()
+    for {
+      secContext <- securityContext
+      clientSaslContext <- secContext.clientSaslContext
+    } {
+      if (clientSaslContext.addRegistrationBootstrap) {
+        logInfo("Add registration client bootstrap")
+        bootstraps.add(new RegistrationClientBootstrap(
+          transportConf,
+          clientSaslContext.appId,
+          clientSaslContext.saslCredentials,
+          clientSaslContext.registrationInfo))
+      } else {
+        logInfo("Add sasl client bootstrap")
+        bootstraps.add(new SaslClientBootstrap(
+          transportConf,
+          clientSaslContext.appId,
+          clientSaslContext.saslCredentials))
+      }
+    }
+    bootstraps

Review Comment:
   I made the change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1251] Connect the server and client bootstraps to RpcEnv [incubator-celeborn]

Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on code in PR #2257:
URL: https://github.com/apache/incubator-celeborn/pull/2257#discussion_r1464318798


##########
common/src/main/scala/org/apache/celeborn/common/rpc/netty/NettyRpcEnv.scala:
##########
@@ -61,7 +65,31 @@ class NettyRpcEnv(
   private val transportContext =
     new TransportContext(transportConf, new NettyRpcHandler(dispatcher, this))
 
-  val clientFactory = transportContext.createClientFactory()
+  private def createClientBootstraps(): java.util.List[TransportClientBootstrap] = {
+    val bootstraps = new java.util.ArrayList[TransportClientBootstrap]()
+    for {
+      secContext <- securityContext
+      clientSaslContext <- secContext.clientSaslContext
+    } {
+      if (clientSaslContext.addRegistrationBootstrap) {
+        logInfo("Add registration client bootstrap")
+        bootstraps.add(new RegistrationClientBootstrap(
+          transportConf,
+          clientSaslContext.appId,
+          clientSaslContext.saslCredentials,
+          clientSaslContext.registrationInfo))
+      } else {
+        logInfo("Add sasl client bootstrap")
+        bootstraps.add(new SaslClientBootstrap(
+          transportConf,
+          clientSaslContext.appId,
+          clientSaslContext.saslCredentials))
+      }
+    }
+    bootstraps

Review Comment:
   Nit:
   
   Both in `createClientBootstraps` and in `createServerBootstraps`, we have either 0 or 1 bootstrap (essentially an `Option`) - while the code gives the impression it can be multivalued (use of `ArrayList` for `bootstraps`, etc).
   
   
   I am wondering if we want to make this more explicit.
   Something like:
   
   ```
   private def createClientBootstraps(): JList[TransportClientBootstrap] = {
     val bootstrapOpt = securityContext.flatMap(_.clientSaslContext.map { clientSaslContext =>
       <existing code within for>
     })
     bootstrapOpt.toList.asJava
   }
   ```
   
   
   This is a bit aesthetic change though, so I am going to leave it as a nit :-) 
   



##########
common/src/test/scala/org/apache/celeborn/common/rpc/RpcSecurityContextSuite.scala:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.common.rpc
+
+import org.scalatest.matchers.must.Matchers.be
+import org.scalatest.matchers.should.Matchers.{an, convertToAnyShouldWrapper}
+
+import org.apache.celeborn.CelebornFunSuite
+import org.apache.celeborn.common.exception.CelebornException
+import org.apache.celeborn.common.network.sasl.{SaslCredentials, SecretRegistryImpl}
+import org.apache.celeborn.common.network.sasl.registration.RegistrationInfo
+import org.apache.celeborn.common.security.{ClientSaslContext, ClientSaslContextBuilder, RpcSecurityContextBuilder, ServerSaslContext, ServerSaslContextBuilder}
+
+class RpcSecurityContextSuite extends CelebornFunSuite {
+
+  test("RpcSecurityContext should be created with client and server sasl contexts") {
+    val clientContext = ClientSaslContext(
+      "clientAppId",
+      new SaslCredentials("user", "password"),
+      addRegistrationBootstrap = true)
+    val serverContext = ServerSaslContext(new SecretRegistryImpl())
+
+    val rpcSecurityContext = new RpcSecurityContextBuilder()
+      .withClientSaslContext(clientContext)
+      .withServerSaslContext(serverContext)
+      .build()
+
+    rpcSecurityContext.clientSaslContext shouldBe Some(clientContext)
+    rpcSecurityContext.serverSaslContext shouldBe Some(serverContext)

Review Comment:
   Practically, we would be having utmost one in an `RpcSecurityContext` right ? Do we have cases where both are required ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org