You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Theofilos Kakantousis <tk...@kth.se> on 2016/11/08 09:25:50 UTC
Flink on Yarn delegation token renewal
Hi everyone,
I'm using Flink 1.1.3 with Hadoop 2.7.3 and was wondering about
delegation token renewal when running Flink on Yarn. Yarn demands
services to renew delegation tokens on their own and if not, Yarn jobs
will fail after one week.
https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html
For example, Spark has implemented a solution for that
https://github.com/apache/spark/blob/v2.0.1/yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala
and people had reported issues running Flink in the past,
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-job-on-secure-Yarn-fails-after-many-hours-td3856.html
What is the status of this issue? How can Flink on Yarn handle this
issue so that jobs do not fail due to token expiration?
Cheers,
Theo
Re: Flink on Yarn delegation token renewal
Posted by Theofilos Kakantousis <tk...@kth.se>.
Thank you for the prompt reply Stefan!
Cheers,
Theo
On 2016-11-08 11:29, Stefan Richter wrote:
> Hi,
>
> I think this problem tracked in this issue:
> https://issues.apache.org/jira/browse/FLINK-3670 . This means that the
> current master and upcoming release 1.2 should work correctly.
>
> Best,
> Stefan
>
>> Am 08.11.2016 um 10:25 schrieb Theofilos Kakantousis <tkak@kth.se
>> <ma...@kth.se>>:
>>
>> Hi everyone,
>>
>> I'm using Flink 1.1.3 with Hadoop 2.7.3 and was wondering about
>> delegation token renewal when running Flink on Yarn. Yarn demands
>> services to renew delegation tokens on their own and if not, Yarn
>> jobs will fail after one week.
>> https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html
>>
>> For example, Spark has implemented a solution for that
>> https://github.com/apache/spark/blob/v2.0.1/yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala
>>
>> and people had reported issues running Flink in the past,
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-job-on-secure-Yarn-fails-after-many-hours-td3856.html
>>
>> What is the status of this issue? How can Flink on Yarn handle this
>> issue so that jobs do not fail due to token expiration?
>>
>> Cheers,
>> Theo
>>
>>
>>
>
Re: Flink on Yarn delegation token renewal
Posted by Stefan Richter <s....@data-artisans.com>.
Hi,
I think this problem tracked in this issue: https://issues.apache.org/jira/browse/FLINK-3670 <https://issues.apache.org/jira/browse/FLINK-3670> . This means that the current master and upcoming release 1.2 should work correctly.
Best,
Stefan
> Am 08.11.2016 um 10:25 schrieb Theofilos Kakantousis <tk...@kth.se>:
>
> Hi everyone,
>
> I'm using Flink 1.1.3 with Hadoop 2.7.3 and was wondering about delegation token renewal when running Flink on Yarn. Yarn demands services to renew delegation tokens on their own and if not, Yarn jobs will fail after one week.
> https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html
>
> For example, Spark has implemented a solution for that
> https://github.com/apache/spark/blob/v2.0.1/yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala
>
> and people had reported issues running Flink in the past,
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-job-on-secure-Yarn-fails-after-many-hours-td3856.html
>
> What is the status of this issue? How can Flink on Yarn handle this issue so that jobs do not fail due to token expiration?
>
> Cheers,
> Theo
>
>
>