You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by Eron Wright <ew...@live.com> on 2016/03/25 00:10:01 UTC

Kerberos for Streaming & Kafka

Hi,
Given the other thread about per-job Kerberos identity, now's a good time to discuss some problems with the current delegation-token approach, since the answer could bear on the per-job enhancement.
Two problems:Delegation tokens expire.  For a continuous streaming job to survive, the original keytab is needed to re-authenticate.   Spark Streaming solved this problem with `--keytab` on spark-submit (see AMDelegationTokenRenewer.scala).Kafka doesn't support delegation tokens yet (see KIP-48 and KAFKA-1696).
Thoughts?  Thanks!
- Eron Wright

Re: Kerberos for Streaming & Kafka

Posted by Maximilian Michels <mx...@apache.org>.

Hi Eron,

Thank you for your feedback! Indeed, we have seen in the past, that
Hadoop's Delegation Tokens are not meant to renewed over a long
period. Plus, they have a number of subtle bugs in older versions that
sometimes prevent renewal.

What you suggest, sounds like a good approach to me. It would
basically mean that we handle have our own renewal system and do not
rely on Hadoop's token renewal system.

Improving long-running Kerberos jobs:
https://issues.apache.org/jira/browse/FLINK-3670
Kafka Kerberos support JIRA: https://issues.apache.org/jira/browse/FLINK-3239

Thanks,
Max

On Sat, Mar 26, 2016 at 12:18 AM, Eron Wright <ew...@live.com> wrote:
> (fixed bad formatting)
>
> Hi,
> Given the other thread about per-job Kerberos identity, now's a good time to discuss some problems with the current delegation-token approach, since the answer could bear on the per-job enhancement.
>
> I see two problems:
>
> 1. Delegation tokens expire. For a continuous streaming job to survive, the original keytab is needed to re-authenticate. Spark Streaming solved this problem with `--keytab` on spark-submit (see AMDelegationTokenRenewer.scala).
>
> 2. Kafka doesn't support delegation tokens yet (see KIP-48 and KAFKA-1696).
>
> Thoughts? Thanks!
> - Eron Wright
>
>
>
>

RE: Kerberos for Streaming & Kafka

Posted by Eron Wright <ew...@live.com>.

(fixed bad formatting)

Hi,
Given the other thread about per-job Kerberos identity, now's a good time to discuss some problems with the current delegation-token approach, since the answer could bear on the per-job enhancement.

I see two problems:

1. Delegation tokens expire. For a continuous streaming job to survive, the original keytab is needed to re-authenticate. Spark Streaming solved this problem with `--keytab` on spark-submit (see AMDelegationTokenRenewer.scala).

2. Kafka doesn't support delegation tokens yet (see KIP-48 and KAFKA-1696).

Thoughts? Thanks!
- Eron Wright