You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/07/19 08:11:28 UTC
[GitHub] [flink] gaborgsomogyi opened a new pull request, #20307: [FLINK-28608][runtime][security]Make Hadoop FS token renewer configurable
gaborgsomogyi opened a new pull request, #20307:
URL: https://github.com/apache/flink/pull/20307
## What is the purpose of the change
YARN is trying to renew the Hadoop FS tokens at the initial job submit phase. The successful of this renew is depending on YARN configuration. Namely token must be obtained with the appropriate renewer which is not yet possible to configure. In this PR I've added `security.kerberos.token.provider.hadoofs.renewer` config possiblity.
## Brief change log
Added new config possibility for token renewer.
## Verifying this change
Manually on cluster.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? All documentation is intended to be added in [FLINK-25911](https://issues.apache.org/jira/browse/FLINK-25911) when everything works as a whole
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on pull request #20307: [FLINK-28608][runtime][security]Make Hadoop FS token renewer configurable
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on PR #20307:
URL: https://github.com/apache/flink/pull/20307#issuecomment-1190374301
@JackWangCS thanks for your effort in finding the bug and testing the proposal!
@mbalassi I think we're ready to go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on pull request #20307: [FLINK-28608][runtime][security]Make Hadoop FS token renewer configurable
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on PR #20307:
URL: https://github.com/apache/flink/pull/20307#issuecomment-1190451385
I've considered more bulletproof testing via calling `obtainDelegationTokens` but 3 static mocking, mock Hadoop FS implementation + registration would be an overkill for a single config param.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on pull request #20307: [FLINK-28608][runtime][security]Make Hadoop FS token renewer configurable
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on PR #20307:
URL: https://github.com/apache/flink/pull/20307#issuecomment-1190010617
@JackWangCS any update on this? Since this is a real issue it would be good to solve it quickly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] JackWangCS commented on pull request #20307: [FLINK-28608][runtime][security]Make Hadoop FS token renewer configurable
Posted by GitBox <gi...@apache.org>.
JackWangCS commented on PR #20307:
URL: https://github.com/apache/flink/pull/20307#issuecomment-1190357468
Hi @gaborgsomogyi , it works after I add this configuration for hadoofs delegation token provider. Here is the related log:
```log
2022-07-20 14:22:03,080 INFO org.apache.flink.runtime.security.token.HBaseDelegationTokenProvider [] - Added HBase Kerberos security token to credentials.
2022-07-20 14:22:03,080 INFO org.apache.zookeeper.ClientCnxn [] - EventThread shut down for session: 0x1000003c15a000c
2022-07-20 14:22:03,080 DEBUG org.apache.flink.runtime.security.token.KerberosDelegationTokenManager [] - Obtained delegation token for service hbase successfully
2022-07-20 14:22:03,085 DEBUG org.apache.flink.runtime.security.token.KerberosDelegationTokenManager [] - Kind: HBASE_AUTH_TOKEN, Service: 17785c00-d402-492d-ba4f-e62ebf5acc25, Ident: ((username=hadoop/ip-10-101-113-223.ec2.internal@EC2.INTERNAL, keyId=3, issueDate=1658326923059, expirationDate=1658931723059, sequenceNumber=0))
2022-07-20 14:22:03,086 DEBUG org.apache.flink.runtime.security.token.KerberosDelegationTokenManager [] - Kind: HDFS_DELEGATION_TOKEN, Service: 10.101.113.223:8020, Ident: (HDFS_DELEGATION_TOKEN token 1 for hadoop)
2022-07-20 14:22:03,086 DEBUG org.apache.flink.runtime.security.token.KerberosDelegationTokenManager [] - Kind: kms-dt, Service: 10.101.113.223:9700, Ident: (owner=hadoop, renewer=yarn, realUser=, issueDate=1658326922230, maxDate=1658931722230, sequenceNumber=1, masterKeyId=2)
2022-07-20 14:22:03,086 INFO org.apache.flink.runtime.security.token.KerberosDelegationTokenManager [] - Delegation tokens obtained successfully
2022-07-20 14:22:03,086 DEBUG org.apache.flink.yarn.YarnClusterDescriptor [] - Tokens Obtained by DelegationTokenManager
2022-07-20 14:22:03,086 DEBUG org.apache.flink.yarn.YarnClusterDescriptor [] - Kind: HBASE_AUTH_TOKEN, Service: 17785c00-d402-492d-ba4f-e62ebf5acc25, Ident: ((username=hadoop/ip-10-101-113-223.ec2.internal@EC2.INTERNAL, keyId=3, issueDate=1658326923059, expirationDate=1658931723059, sequenceNumber=0))
2022-07-20 14:22:03,087 DEBUG org.apache.flink.yarn.YarnClusterDescriptor [] - Kind: HDFS_DELEGATION_TOKEN, Service: 10.101.113.223:8020, Ident: (HDFS_DELEGATION_TOKEN token 1 for hadoop)
2022-07-20 14:22:03,087 DEBUG org.apache.flink.yarn.YarnClusterDescriptor [] - Kind: kms-dt, Service: 10.101.113.223:9700, Ident: (owner=hadoop, renewer=yarn, realUser=, issueDate=1658326922230, maxDate=1658931722230, sequenceNumber=1, masterKeyId=2)
2022-07-20 14:22:03,087 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Delegation tokens added to the AM container.
2022-07-20 14:22:03,092 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Submitting application master application_1658306268998_0001
2022-07-20 14:22:03,103 INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl [] - Timeline service address: ip-10-101-113-223.ec2.internal:8188
2022-07-20 14:22:04,704 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [] - Submitted application application_1658306268998_0001
2022-07-20 14:22:04,704 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Waiting for the cluster to be allocated
2022-07-20 14:22:04,706 DEBUG org.apache.flink.yarn.YarnClusterDescriptor [] - Application State: ACCEPTED
```
As you can see, the renewer is set to yarn for kms-dt token.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] mbalassi commented on pull request #20307: [FLINK-28608][runtime][security]Make Hadoop FS token renewer configurable
Posted by GitBox <gi...@apache.org>.
mbalassi commented on PR #20307:
URL: https://github.com/apache/flink/pull/20307#issuecomment-1191455923
Thanks, @gaborgsomogyi. This will do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #20307: [FLINK-28608][runtime][security]Make Hadoop FS token renewer configurable
Posted by GitBox <gi...@apache.org>.
flinkbot commented on PR #20307:
URL: https://github.com/apache/flink/pull/20307#issuecomment-1188747207
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1e50674e613729242b37eaa848a94a07721d8598",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1e50674e613729242b37eaa848a94a07721d8598",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1e50674e613729242b37eaa848a94a07721d8598 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on pull request #20307: [FLINK-28608][runtime][security]Make Hadoop FS token renewer configurable
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on PR #20307:
URL: https://github.com/apache/flink/pull/20307#issuecomment-1190446388
Introduced some tests to add coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] mbalassi merged pull request #20307: [FLINK-28608][runtime][security]Make Hadoop FS token renewer configurable
Posted by GitBox <gi...@apache.org>.
mbalassi merged PR #20307:
URL: https://github.com/apache/flink/pull/20307
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on pull request #20307: [FLINK-28608][runtime][security]Make Hadoop FS token renewer configurable
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on PR #20307:
URL: https://github.com/apache/flink/pull/20307#issuecomment-1188740172
@JackWangCS can you retest it in your environment to double check that it works?
If I understand your setup correctly then `security.kerberos.token.provider.hadoofs.renewer=yarn` config is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on a diff in pull request #20307: [FLINK-28608][runtime][security]Make Hadoop FS token renewer configurable
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on code in PR #20307:
URL: https://github.com/apache/flink/pull/20307#discussion_r925767556
##########
flink-runtime/src/main/java/org/apache/flink/runtime/security/token/HadoopFSDelegationTokenProvider.java:
##########
@@ -155,6 +162,7 @@ protected void obtainDelegationTokens(
});
}
+ @VisibleForTesting
Review Comment:
This is not related just a quick fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] JackWangCS commented on pull request #20307: [FLINK-28608][runtime][security]Make Hadoop FS token renewer configurable
Posted by GitBox <gi...@apache.org>.
JackWangCS commented on PR #20307:
URL: https://github.com/apache/flink/pull/20307#issuecomment-1188784871
>
Sure, I will get back to you after I finish the tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-28608][runtime][security]Make Hadoop FS token renewer configurable [flink]
Posted by "hidataplus (via GitHub)" <gi...@apache.org>.
hidataplus commented on PR #20307:
URL: https://github.com/apache/flink/pull/20307#issuecomment-2060234261
> @JackWangCS can you retest it in your environment to double check that it works? If I understand your setup correctly then `security.kerberos.token.provider.hadoofs.renewer=yarn` config is needed.
there is spelling errors, should be
'security.kerberos.token.provider.hadoopfs.renewer: yarn' in flink-conf.yaml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org