You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "gaborgsomogyi (via GitHub)" <gi...@apache.org> on 2023/03/29 11:52:08 UTC
[GitHub] [flink] gaborgsomogyi opened a new pull request, #22298: [FLINK-31656][runtime][security] Obtain delegation tokens early to support external file system usage in blob server
gaborgsomogyi opened a new pull request, #22298:
URL: https://github.com/apache/flink/pull/22298
## What is the purpose of the change
At the moment there are no delegation tokens available when blob server is starting. If the blob server uses an external file system where the authentication type is delegation token based then it throws and exception since there are no credentials.
In this PR I've moved the delegation token manager initialization before blob server and trigger a manual token obtain + local JVM receiver propagation.
## Brief change log
* The delegation token manager initialization moved before blob server and trigger a manual token obtain + local JVM receiver propagation
* Changed the `DelegationTokenManager` API documentation for better clarity
* Changed a log message for better clarity
## Verifying this change
Manually on cluster.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: yes
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on a diff in pull request #22298: [FLINK-31656][runtime][security] Obtain delegation tokens early to support external file system usage in HA services
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #22298:
URL: https://github.com/apache/flink/pull/22298#discussion_r1154229815
##########
flink-runtime/src/main/java/org/apache/flink/runtime/security/token/DefaultDelegationTokenManager.java:
##########
@@ -213,6 +213,20 @@ public void obtainDelegationTokens(DelegationTokenContainer container) throws Ex
LOG.info("Delegation tokens obtained successfully");
}
+ @Override
+ public void obtainDelegationTokens() throws Exception {
Review Comment:
@HuangZhenQiu please confirm back that you see working cluster tests on your side just to double check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #22298: [FLINK-31656][runtime][security] Obtain delegation tokens early to support external file system usage in blob server
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #22298:
URL: https://github.com/apache/flink/pull/22298#issuecomment-1488464180
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "683fb9b5639ea8d1893b50bb8902f5b9873eb142",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "683fb9b5639ea8d1893b50bb8902f5b9873eb142",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 683fb9b5639ea8d1893b50bb8902f5b9873eb142 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a diff in pull request #22298: [FLINK-31656][runtime][security] Obtain delegation tokens early to support external file system usage in blob server
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on code in PR #22298:
URL: https://github.com/apache/flink/pull/22298#discussion_r1152336876
##########
flink-runtime/src/main/java/org/apache/flink/runtime/security/token/DefaultDelegationTokenManager.java:
##########
@@ -213,6 +213,20 @@ public void obtainDelegationTokens(DelegationTokenContainer container) throws Ex
LOG.info("Delegation tokens obtained successfully");
}
+ @Override
+ public void obtainDelegationTokens() throws Exception {
Review Comment:
We need to the immediately propagate the existing tokens (fetched here) to Task managers once start function is called. Otherwise, I see No credential error in task manager. I guess it is due to the latency of additional token fetching in startTokenUpdate function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on a diff in pull request #22298: [FLINK-31656][runtime][security] Obtain delegation tokens early to support external file system usage in HA services
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #22298:
URL: https://github.com/apache/flink/pull/22298#discussion_r1154481057
##########
flink-runtime/src/main/java/org/apache/flink/runtime/minicluster/MiniCluster.java:
##########
@@ -418,6 +418,21 @@ public void start() throws Exception {
ClusterEntrypointUtils.getPoolSize(configuration),
new ExecutorThreadFactory("mini-cluster-io"));
+ delegationTokenManager =
+ DefaultDelegationTokenManagerFactory.create(
+ configuration,
+ miniClusterConfiguration.getPluginManager(),
+ commonRpcService.getScheduledExecutor(),
+ ioExecutor);
+ // Obtaining delegation tokens and propagating them to the local JVM receivers in a
+ // one-time fashion is required because BlobServer may connect to external file
+ // systems
+ delegationTokenManager.obtainDelegationTokens();
Review Comment:
Added that we obtain tokens in `ClusterEntrypointTest` but it would be overkill to check that token obtain happens before HA services.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gyfora commented on a diff in pull request #22298: [FLINK-31656][runtime][security] Obtain delegation tokens early to support external file system usage in HA services
Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora commented on code in PR #22298:
URL: https://github.com/apache/flink/pull/22298#discussion_r1154386632
##########
flink-runtime/src/main/java/org/apache/flink/runtime/minicluster/MiniCluster.java:
##########
@@ -418,6 +418,21 @@ public void start() throws Exception {
ClusterEntrypointUtils.getPoolSize(configuration),
new ExecutorThreadFactory("mini-cluster-io"));
+ delegationTokenManager =
+ DefaultDelegationTokenManagerFactory.create(
+ configuration,
+ miniClusterConfiguration.getPluginManager(),
+ commonRpcService.getScheduledExecutor(),
+ ioExecutor);
+ // Obtaining delegation tokens and propagating them to the local JVM receivers in a
+ // one-time fashion is required because BlobServer may connect to external file
+ // systems
+ delegationTokenManager.obtainDelegationTokens();
Review Comment:
I don't seem to find any test for this new behaviour, would be good to add something to guard against accidental regressions in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on pull request #22298: [FLINK-31656][runtime][security] Obtain delegation tokens early to support external file system usage in HA services
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on PR #22298:
URL: https://github.com/apache/flink/pull/22298#issuecomment-1491595469
I've just fixed the unit tests + changed the title and description. The cluster tests are green.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on a diff in pull request #22298: [FLINK-31656][runtime][security] Obtain delegation tokens early to support external file system usage in blob server
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #22298:
URL: https://github.com/apache/flink/pull/22298#discussion_r1153337140
##########
flink-runtime/src/main/java/org/apache/flink/runtime/security/token/DefaultDelegationTokenManager.java:
##########
@@ -213,6 +213,20 @@ public void obtainDelegationTokens(DelegationTokenContainer container) throws Ex
LOG.info("Delegation tokens obtained successfully");
}
+ @Override
+ public void obtainDelegationTokens() throws Exception {
Review Comment:
I've just added further changes to solve the TM issue. We should test it on cluster in-depth because that's modifying Flink's critical path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on a diff in pull request #22298: [FLINK-31656][runtime][security] Obtain delegation tokens early to support external file system usage in blob server
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #22298:
URL: https://github.com/apache/flink/pull/22298#discussion_r1152340630
##########
flink-runtime/src/main/java/org/apache/flink/runtime/security/token/DefaultDelegationTokenManager.java:
##########
@@ -213,6 +213,20 @@ public void obtainDelegationTokens(DelegationTokenContainer container) throws Ex
LOG.info("Delegation tokens obtained successfully");
}
+ @Override
+ public void obtainDelegationTokens() throws Exception {
Review Comment:
At the one-time token obtain stage there are no task managers but I see that there is an issue so I'll take a look...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on pull request #22298: [FLINK-31656][runtime][security] Obtain delegation tokens early to support external file system usage in HA services
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on PR #22298:
URL: https://github.com/apache/flink/pull/22298#issuecomment-1491595867
cc @gyfora @mbalassi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a diff in pull request #22298: [FLINK-31656][runtime][security] Obtain delegation tokens early to support external file system usage in HA services
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on code in PR #22298:
URL: https://github.com/apache/flink/pull/22298#discussion_r1160103711
##########
flink-runtime/src/main/java/org/apache/flink/runtime/security/token/DefaultDelegationTokenManager.java:
##########
@@ -213,6 +213,20 @@ public void obtainDelegationTokens(DelegationTokenContainer container) throws Ex
LOG.info("Delegation tokens obtained successfully");
}
+ @Override
+ public void obtainDelegationTokens() throws Exception {
Review Comment:
@gaborgsomogyi Sorry for late rely. The feature is fully tested end to end.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/security/token/DefaultDelegationTokenManager.java:
##########
@@ -213,6 +213,20 @@ public void obtainDelegationTokens(DelegationTokenContainer container) throws Ex
LOG.info("Delegation tokens obtained successfully");
}
+ @Override
+ public void obtainDelegationTokens() throws Exception {
Review Comment:
@gaborgsomogyi Sorry for late reply. The feature is fully tested end to end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gyfora merged pull request #22298: [FLINK-31656][runtime][security] Obtain delegation tokens early to support external file system usage in HA services
Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora merged PR #22298:
URL: https://github.com/apache/flink/pull/22298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org