You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "otterc (via GitHub)" <gi...@apache.org> on 2024/01/24 04:20:36 UTC
[PR] [CELEBORN-1251] Connect the server and client bootstraps to RpcEnv [incubator-celeborn]
otterc opened a new pull request, #2257:
URL: https://github.com/apache/incubator-celeborn/pull/2257
### What changes were proposed in this pull request?
This connects client/server bootstraps to the RpcEnv in Celeborn.
It is part of the epic: https://issues.apache.org/jira/browse/CELEBORN-1011.
### Why are the changes needed?
The changes are needed for adding authentication to Celeborn. See [CELEBORN-1011](https://issues.apache.org/jira/browse/CELEBORN-1011).
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added some UTs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1251] Connect the server and client bootstraps to RpcEnv [incubator-celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong closed pull request #2257: [CELEBORN-1251] Connect the server and client bootstraps to RpcEnv
URL: https://github.com/apache/incubator-celeborn/pull/2257
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1251] Connect the server and client bootstraps to RpcEnv [incubator-celeborn]
Posted by "otterc (via GitHub)" <gi...@apache.org>.
otterc commented on PR #2257:
URL: https://github.com/apache/incubator-celeborn/pull/2257#issuecomment-1909204040
@waitinfuture @RexXiong @FMX @pan3793 Please help review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1251] Connect the server and client bootstraps to RpcEnv [incubator-celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on PR #2257:
URL: https://github.com/apache/incubator-celeborn/pull/2257#issuecomment-1909744362
Thanks, merge to main(v0.5.0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1251] Connect the server and client bootstraps to RpcEnv [incubator-celeborn]
Posted by "otterc (via GitHub)" <gi...@apache.org>.
otterc commented on code in PR #2257:
URL: https://github.com/apache/incubator-celeborn/pull/2257#discussion_r1465420304
##########
common/src/test/scala/org/apache/celeborn/common/rpc/RpcSecurityContextSuite.scala:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.common.rpc
+
+import org.scalatest.matchers.must.Matchers.be
+import org.scalatest.matchers.should.Matchers.{an, convertToAnyShouldWrapper}
+
+import org.apache.celeborn.CelebornFunSuite
+import org.apache.celeborn.common.exception.CelebornException
+import org.apache.celeborn.common.network.sasl.{SaslCredentials, SecretRegistryImpl}
+import org.apache.celeborn.common.network.sasl.registration.RegistrationInfo
+import org.apache.celeborn.common.security.{ClientSaslContext, ClientSaslContextBuilder, RpcSecurityContextBuilder, ServerSaslContext, ServerSaslContextBuilder}
+
+class RpcSecurityContextSuite extends CelebornFunSuite {
+
+ test("RpcSecurityContext should be created with client and server sasl contexts") {
+ val clientContext = ClientSaslContext(
+ "clientAppId",
+ new SaslCredentials("user", "password"),
+ addRegistrationBootstrap = true)
+ val serverContext = ServerSaslContext(new SecretRegistryImpl())
+
+ val rpcSecurityContext = new RpcSecurityContextBuilder()
+ .withClientSaslContext(clientContext)
+ .withServerSaslContext(serverContext)
+ .build()
+
+ rpcSecurityContext.clientSaslContext shouldBe Some(clientContext)
+ rpcSecurityContext.serverSaslContext shouldBe Some(serverContext)
Review Comment:
Yes, it will just one. I'll change it to reflect that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1251] Connect the server and client bootstraps to RpcEnv [incubator-celeborn]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #2257:
URL: https://github.com/apache/incubator-celeborn/pull/2257#issuecomment-1908969820
## [Codecov](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2257?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: `22 lines` in your changes are missing coverage. Please review.
> Comparison is base [(`67e6cbf`)](https://app.codecov.io/gh/apache/incubator-celeborn/commit/67e6cbfb517ec192a048faede7e478eb42a3bd10?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 48.49% compared to head [(`519f21f`)](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2257?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 48.55%.
> Report is 3 commits behind head on main.
| [Files](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2257?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...apache/celeborn/common/rpc/netty/NettyRpcEnv.scala](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2257?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvY2VsZWJvcm4vY29tbW9uL3JwYy9uZXR0eS9OZXR0eVJwY0Vudi5zY2FsYQ==) | 25.00% | [21 Missing :warning: ](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2257?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../scala/org/apache/celeborn/common/rpc/RpcEnv.scala](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2257?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvY2VsZWJvcm4vY29tbW9uL3JwYy9ScGNFbnYuc2NhbGE=) | 80.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2257?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #2257 +/- ##
==========================================
+ Coverage 48.49% 48.55% +0.06%
==========================================
Files 196 196
Lines 12142 12173 +31
Branches 1069 1069
==========================================
+ Hits 5887 5909 +22
- Misses 5868 5879 +11
+ Partials 387 385 -2
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2257?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1251] Connect the server and client bootstraps to RpcEnv [incubator-celeborn]
Posted by "otterc (via GitHub)" <gi...@apache.org>.
otterc commented on code in PR #2257:
URL: https://github.com/apache/incubator-celeborn/pull/2257#discussion_r1465568573
##########
common/src/main/scala/org/apache/celeborn/common/rpc/netty/NettyRpcEnv.scala:
##########
@@ -61,7 +65,31 @@ class NettyRpcEnv(
private val transportContext =
new TransportContext(transportConf, new NettyRpcHandler(dispatcher, this))
- val clientFactory = transportContext.createClientFactory()
+ private def createClientBootstraps(): java.util.List[TransportClientBootstrap] = {
+ val bootstraps = new java.util.ArrayList[TransportClientBootstrap]()
+ for {
+ secContext <- securityContext
+ clientSaslContext <- secContext.clientSaslContext
+ } {
+ if (clientSaslContext.addRegistrationBootstrap) {
+ logInfo("Add registration client bootstrap")
+ bootstraps.add(new RegistrationClientBootstrap(
+ transportConf,
+ clientSaslContext.appId,
+ clientSaslContext.saslCredentials,
+ clientSaslContext.registrationInfo))
+ } else {
+ logInfo("Add sasl client bootstrap")
+ bootstraps.add(new SaslClientBootstrap(
+ transportConf,
+ clientSaslContext.appId,
+ clientSaslContext.saslCredentials))
+ }
+ }
+ bootstraps
Review Comment:
I made the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1251] Connect the server and client bootstraps to RpcEnv [incubator-celeborn]
Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on code in PR #2257:
URL: https://github.com/apache/incubator-celeborn/pull/2257#discussion_r1464318798
##########
common/src/main/scala/org/apache/celeborn/common/rpc/netty/NettyRpcEnv.scala:
##########
@@ -61,7 +65,31 @@ class NettyRpcEnv(
private val transportContext =
new TransportContext(transportConf, new NettyRpcHandler(dispatcher, this))
- val clientFactory = transportContext.createClientFactory()
+ private def createClientBootstraps(): java.util.List[TransportClientBootstrap] = {
+ val bootstraps = new java.util.ArrayList[TransportClientBootstrap]()
+ for {
+ secContext <- securityContext
+ clientSaslContext <- secContext.clientSaslContext
+ } {
+ if (clientSaslContext.addRegistrationBootstrap) {
+ logInfo("Add registration client bootstrap")
+ bootstraps.add(new RegistrationClientBootstrap(
+ transportConf,
+ clientSaslContext.appId,
+ clientSaslContext.saslCredentials,
+ clientSaslContext.registrationInfo))
+ } else {
+ logInfo("Add sasl client bootstrap")
+ bootstraps.add(new SaslClientBootstrap(
+ transportConf,
+ clientSaslContext.appId,
+ clientSaslContext.saslCredentials))
+ }
+ }
+ bootstraps
Review Comment:
Nit:
Both in `createClientBootstraps` and in `createServerBootstraps`, we have either 0 or 1 bootstrap (essentially an `Option`) - while the code gives the impression it can be multivalued (use of `ArrayList` for `bootstraps`, etc).
I am wondering if we want to make this more explicit.
Something like:
```
private def createClientBootstraps(): JList[TransportClientBootstrap] = {
val bootstrapOpt = securityContext.flatMap(_.clientSaslContext.map { clientSaslContext =>
<existing code within for>
})
bootstrapOpt.toList.asJava
}
```
This is a bit aesthetic change though, so I am going to leave it as a nit :-)
##########
common/src/test/scala/org/apache/celeborn/common/rpc/RpcSecurityContextSuite.scala:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.common.rpc
+
+import org.scalatest.matchers.must.Matchers.be
+import org.scalatest.matchers.should.Matchers.{an, convertToAnyShouldWrapper}
+
+import org.apache.celeborn.CelebornFunSuite
+import org.apache.celeborn.common.exception.CelebornException
+import org.apache.celeborn.common.network.sasl.{SaslCredentials, SecretRegistryImpl}
+import org.apache.celeborn.common.network.sasl.registration.RegistrationInfo
+import org.apache.celeborn.common.security.{ClientSaslContext, ClientSaslContextBuilder, RpcSecurityContextBuilder, ServerSaslContext, ServerSaslContextBuilder}
+
+class RpcSecurityContextSuite extends CelebornFunSuite {
+
+ test("RpcSecurityContext should be created with client and server sasl contexts") {
+ val clientContext = ClientSaslContext(
+ "clientAppId",
+ new SaslCredentials("user", "password"),
+ addRegistrationBootstrap = true)
+ val serverContext = ServerSaslContext(new SecretRegistryImpl())
+
+ val rpcSecurityContext = new RpcSecurityContextBuilder()
+ .withClientSaslContext(clientContext)
+ .withServerSaslContext(serverContext)
+ .build()
+
+ rpcSecurityContext.clientSaslContext shouldBe Some(clientContext)
+ rpcSecurityContext.serverSaslContext shouldBe Some(serverContext)
Review Comment:
Practically, we would be having utmost one in an `RpcSecurityContext` right ? Do we have cases where both are required ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org