You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "Srivastava,Rajat" <Ra...@cerner.com> on 2019/10/03 19:51:18 UTC

Flink using Oozie in Kerberized cluster

Hi,

Has someone tried to run bounded Flink application using Oozie in a Kerberized cluster ? What else can we use to schedule bounded flink jobs.

Any thoughts or ideas will be appreciated!

Flink version: 1.8.0
Flink-runner version: 2.14.0

Best,
Rajat Srivastava
Software Engineer | HealtheIntent Dev
rajat.srivastava@cerner.com<ma...@cerner.com> | (816) 906-4958 direct
Cerner Corporation | www.cerner.com<http://www.cerner.com/>



CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.

Re: Flink using Oozie in Kerberized cluster

Posted by "Srivastava,Rajat" <Ra...@cerner.com>.
Sounds like a good idea. Thanks for your help!

Best,
Rajat Srivastava


From: sri hari kali charan Tummala <ka...@gmail.com>
Date: Monday, October 7, 2019 at 8:11 AM
To: "Srivastava,Rajat" <Ra...@cerner.com>
Subject: Re: Flink using Oozie in Kerberized cluster

please raise a ticket with cloudera its kerberos issue.

On Sun, Oct 6, 2019 at 11:26 PM Srivastava,Rajat <Ra...@cerner.com>> wrote:
Thanks for your response!

We are doing something very similar, building a fat jar, uploading it to HDFS, and using shell action to run the application.

The issue we are having however is running the application on a Kerberized cluster in yarn-cluster mode. We get the following error:
ERROR org.apache.flink.yarn.AbstractYarnClusterDescriptor - Hadoop security with Kerberos is enabled but the login user does not have Kerberos credentials

Best,
Rajat Srivastava


From: sri hari kali charan Tummala <ka...@gmail.com>>
Date: Saturday, October 5, 2019 at 1:47 PM
To: "Srivastava,Rajat" <Ra...@cerner.com>>
Subject: Re: Flink using Oozie in Kerberized cluster


https://youtu.be/Q6gEVidrGNE<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyoutu.be%2FQ6gEVidrGNE&data=02%7C01%7CRajat.Srivastava%40cerner.com%7C7cb1e244002b463abb0708d74b27d45a%7Cfbc493a80d244454a815f4ca58e8c09d%7C0%7C0%7C637060506745825325&sdata=cNE6Bv7mJMIBlDUr%2Fgqyge%2FROn2MSxIgz20DtRxqUF4%3D&reserved=0>

I usually build a fat jar and call it as shell action as explained in the video I also upload the jar to hdfs.


That’s all really , not sure about Apache beam pipeline.


Thanks
Sri



On Friday, October 4, 2019, Srivastava,Rajat <Ra...@cerner.com>> wrote:
Moving out of Cloudera is not an option for us.
By bounded flink, I actually meant by running a bounded beam pipeline on a flink runner.

Best,
Rajat Srivastava


From: sri hari kali charan Tummala <ka...@gmail.com>>
Date: Friday, October 4, 2019 at 11:41 AM
To: "Srivastava,Rajat" <Ra...@cerner.com>>
Subject: Re: Flink using Oozie in Kerberized cluster

AWS has kinesis data analytics which provisions manged Flink cluster and runs the job for you think go moving out of Cloudera, check screen shot attached.

Ozzie & Cloudera:-
what is bounded Flink Application mean in your email?



On Fri, Oct 4, 2019 at 10:23 AM Srivastava,Rajat <Ra...@cerner.com>> wrote:
It’s on a Cloudera managed cluster.

Best,
Rajat Srivastava


From: sri hari kali charan Tummala <ka...@gmail.com>>
Date: Friday, October 4, 2019 at 7:39 AM
To: "Srivastava,Rajat" <Ra...@cerner.com>>
Subject: Re: Flink using Oozie in Kerberized cluster

is this on AWS or AWS EMR or Cloudera ?

On Thu, Oct 3, 2019 at 3:51 PM Srivastava,Rajat <Ra...@cerner.com>> wrote:
Hi,

Has someone tried to run bounded Flink application using Oozie in a Kerberized cluster ? What else can we use to schedule bounded flink jobs.

Any thoughts or ideas will be appreciated!

Flink version: 1.8.0
Flink-runner version: 2.14.0

Best,
Rajat Srivastava
Software Engineer | HealtheIntent Dev
rajat.srivastava@cerner.com<ma...@cerner.com> | (816) 906-4958 direct
Cerner Corporation | www.cerner.com<http://www.cerner.com/>



CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.


--
Thanks & Regards
Sri Tummala



--
Thanks & Regards
Sri Tummala



--
Thanks & Regards
Sri Tummala




--
Thanks & Regards
Sri Tummala

Re: Flink using Oozie in Kerberized cluster

Posted by "Srivastava,Rajat" <Ra...@cerner.com>.
Hi Sri,

I logged a cloudera ticket, as you recommended, and got help from their support team, and was able to get my application running.

We had to “kinit” inside the shell action using a keytab in the following format: “kinit primary/instance@REALM -kt primary.keytab”

The keytab file had to be added to the lib folder the workflow root. The shell action localizes the keytab to the container’s current working directory and then can be used for knit in the oozie shell action.

Thanks for your help on this issue!

Best,
Rajat Srivastava


From: sri hari kali charan Tummala <ka...@gmail.com>
Date: Monday, October 7, 2019 at 10:41 AM
To: "Srivastava,Rajat" <Ra...@cerner.com>
Subject: Re: Flink using Oozie in Kerberized cluster

echo "password" | /usr/bin/kinit userid@ActiveDirectory.Domain.NET<ma...@ActiveDirectory.Domain.NET>

this is something we use to do in your first step in our Oozie workflow and then run the rest of the workflow, as I said Cloudera might help.

On Mon, Oct 7, 2019 at 9:10 AM sri hari kali charan Tummala <ka...@gmail.com>> wrote:
please raise a ticket with cloudera its kerberos issue.

On Sun, Oct 6, 2019 at 11:26 PM Srivastava,Rajat <Ra...@cerner.com>> wrote:
Thanks for your response!

We are doing something very similar, building a fat jar, uploading it to HDFS, and using shell action to run the application.

The issue we are having however is running the application on a Kerberized cluster in yarn-cluster mode. We get the following error:
ERROR org.apache.flink.yarn.AbstractYarnClusterDescriptor - Hadoop security with Kerberos is enabled but the login user does not have Kerberos credentials

Best,
Rajat Srivastava


From: sri hari kali charan Tummala <ka...@gmail.com>>
Date: Saturday, October 5, 2019 at 1:47 PM
To: "Srivastava,Rajat" <Ra...@cerner.com>>
Subject: Re: Flink using Oozie in Kerberized cluster


https://youtu.be/Q6gEVidrGNE<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyoutu.be%2FQ6gEVidrGNE&data=02%7C01%7CRajat.Srivastava%40cerner.com%7C82b5fb45850645f049ee08d74b3cdf92%7Cfbc493a80d244454a815f4ca58e8c09d%7C0%7C0%7C637060597146084177&sdata=HS55V4vQgh8%2BFoGCeVitWx%2BzyF3sbIqz%2FVyNxDPYNZw%3D&reserved=0>

I usually build a fat jar and call it as shell action as explained in the video I also upload the jar to hdfs.


That’s all really , not sure about Apache beam pipeline.


Thanks
Sri



On Friday, October 4, 2019, Srivastava,Rajat <Ra...@cerner.com>> wrote:
Moving out of Cloudera is not an option for us.
By bounded flink, I actually meant by running a bounded beam pipeline on a flink runner.

Best,
Rajat Srivastava


From: sri hari kali charan Tummala <ka...@gmail.com>>
Date: Friday, October 4, 2019 at 11:41 AM
To: "Srivastava,Rajat" <Ra...@cerner.com>>
Subject: Re: Flink using Oozie in Kerberized cluster

AWS has kinesis data analytics which provisions manged Flink cluster and runs the job for you think go moving out of Cloudera, check screen shot attached.

Ozzie & Cloudera:-
what is bounded Flink Application mean in your email?



On Fri, Oct 4, 2019 at 10:23 AM Srivastava,Rajat <Ra...@cerner.com>> wrote:
It’s on a Cloudera managed cluster.

Best,
Rajat Srivastava


From: sri hari kali charan Tummala <ka...@gmail.com>>
Date: Friday, October 4, 2019 at 7:39 AM
To: "Srivastava,Rajat" <Ra...@cerner.com>>
Subject: Re: Flink using Oozie in Kerberized cluster

is this on AWS or AWS EMR or Cloudera ?

On Thu, Oct 3, 2019 at 3:51 PM Srivastava,Rajat <Ra...@cerner.com>> wrote:
Hi,

Has someone tried to run bounded Flink application using Oozie in a Kerberized cluster ? What else can we use to schedule bounded flink jobs.

Any thoughts or ideas will be appreciated!

Flink version: 1.8.0
Flink-runner version: 2.14.0

Best,
Rajat Srivastava
Software Engineer | HealtheIntent Dev
rajat.srivastava@cerner.com<ma...@cerner.com> | (816) 906-4958 direct
Cerner Corporation | www.cerner.com<http://www.cerner.com/>



CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.


--
Thanks & Regards
Sri Tummala



--
Thanks & Regards
Sri Tummala



--
Thanks & Regards
Sri Tummala




--
Thanks & Regards
Sri Tummala



--
Thanks & Regards
Sri Tummala

Re: Flink using Oozie in Kerberized cluster

Posted by "Srivastava,Rajat" <Ra...@cerner.com>.
Thanks for your response!

We are doing something very similar, building a fat jar, uploading it to HDFS, and using shell action to run the application.

The issue we are having however is running the application on a Kerberized cluster in yarn-cluster mode. We get the following error:

ERROR org.apache.flink.yarn.AbstractYarnClusterDescriptor - Hadoop security with Kerberos is enabled but the login user does not have Kerberos credentials

Best,
Rajat Srivastava


From: sri hari kali charan Tummala <ka...@gmail.com>
Date: Saturday, October 5, 2019 at 1:47 PM
To: "Srivastava,Rajat" <Ra...@cerner.com>
Subject: Re: Flink using Oozie in Kerberized cluster


https://youtu.be/Q6gEVidrGNE<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyoutu.be%2FQ6gEVidrGNE&data=02%7C01%7CRajat.Srivastava%40cerner.com%7C77fdc163c1f748ee277a08d749c486ce%7Cfbc493a80d244454a815f4ca58e8c09d%7C0%7C0%7C637058980744781639&sdata=ynD%2FQx%2Fj4d0lsBQ9KCCy2q5MhwjmAVItcXzJkoxfxbs%3D&reserved=0>

I usually build a fat jar and call it as shell action as explained in the video I also upload the jar to hdfs.


That’s all really , not sure about Apache beam pipeline.


Thanks
Sri



On Friday, October 4, 2019, Srivastava,Rajat <Ra...@cerner.com>> wrote:
Moving out of Cloudera is not an option for us.
By bounded flink, I actually meant by running a bounded beam pipeline on a flink runner.

Best,
Rajat Srivastava


From: sri hari kali charan Tummala <ka...@gmail.com>>
Date: Friday, October 4, 2019 at 11:41 AM
To: "Srivastava,Rajat" <Ra...@cerner.com>>
Subject: Re: Flink using Oozie in Kerberized cluster

AWS has kinesis data analytics which provisions manged Flink cluster and runs the job for you think go moving out of Cloudera, check screen shot attached.

Ozzie & Cloudera:-
what is bounded Flink Application mean in your email?



On Fri, Oct 4, 2019 at 10:23 AM Srivastava,Rajat <Ra...@cerner.com>> wrote:
It’s on a Cloudera managed cluster.

Best,
Rajat Srivastava


From: sri hari kali charan Tummala <ka...@gmail.com>>
Date: Friday, October 4, 2019 at 7:39 AM
To: "Srivastava,Rajat" <Ra...@cerner.com>>
Subject: Re: Flink using Oozie in Kerberized cluster

is this on AWS or AWS EMR or Cloudera ?

On Thu, Oct 3, 2019 at 3:51 PM Srivastava,Rajat <Ra...@cerner.com>> wrote:
Hi,

Has someone tried to run bounded Flink application using Oozie in a Kerberized cluster ? What else can we use to schedule bounded flink jobs.

Any thoughts or ideas will be appreciated!

Flink version: 1.8.0
Flink-runner version: 2.14.0

Best,
Rajat Srivastava
Software Engineer | HealtheIntent Dev
rajat.srivastava@cerner.com<ma...@cerner.com> | (816) 906-4958 direct
Cerner Corporation | www.cerner.com<http://www.cerner.com/>



CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.


--
Thanks & Regards
Sri Tummala



--
Thanks & Regards
Sri Tummala



--
Thanks & Regards
Sri Tummala

Re: Flink using Oozie in Kerberized cluster

Posted by "Srivastava,Rajat" <Ra...@cerner.com>.
Moving out of Cloudera is not an option for us.
By bounded flink, I actually meant by running a bounded beam pipeline on a flink runner.

Best,
Rajat Srivastava


From: sri hari kali charan Tummala <ka...@gmail.com>
Date: Friday, October 4, 2019 at 11:41 AM
To: "Srivastava,Rajat" <Ra...@cerner.com>
Subject: Re: Flink using Oozie in Kerberized cluster

AWS has kinesis data analytics which provisions manged Flink cluster and runs the job for you think go moving out of Cloudera, check screen shot attached.

Ozzie & Cloudera:-
what is bounded Flink Application mean in your email?



On Fri, Oct 4, 2019 at 10:23 AM Srivastava,Rajat <Ra...@cerner.com>> wrote:
It’s on a Cloudera managed cluster.

Best,
Rajat Srivastava


From: sri hari kali charan Tummala <ka...@gmail.com>>
Date: Friday, October 4, 2019 at 7:39 AM
To: "Srivastava,Rajat" <Ra...@cerner.com>>
Subject: Re: Flink using Oozie in Kerberized cluster

is this on AWS or AWS EMR or Cloudera ?

On Thu, Oct 3, 2019 at 3:51 PM Srivastava,Rajat <Ra...@cerner.com>> wrote:
Hi,

Has someone tried to run bounded Flink application using Oozie in a Kerberized cluster ? What else can we use to schedule bounded flink jobs.

Any thoughts or ideas will be appreciated!

Flink version: 1.8.0
Flink-runner version: 2.14.0

Best,
Rajat Srivastava
Software Engineer | HealtheIntent Dev
rajat.srivastava@cerner.com<ma...@cerner.com> | (816) 906-4958 direct
Cerner Corporation | www.cerner.com<http://www.cerner.com/>



CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.


--
Thanks & Regards
Sri Tummala



--
Thanks & Regards
Sri Tummala

Re: Flink using Oozie in Kerberized cluster

Posted by "Srivastava,Rajat" <Ra...@cerner.com>.
It’s on a Cloudera managed cluster.

Best,
Rajat Srivastava


From: sri hari kali charan Tummala <ka...@gmail.com>
Date: Friday, October 4, 2019 at 7:39 AM
To: "Srivastava,Rajat" <Ra...@cerner.com>
Subject: Re: Flink using Oozie in Kerberized cluster

is this on AWS or AWS EMR or Cloudera ?

On Thu, Oct 3, 2019 at 3:51 PM Srivastava,Rajat <Ra...@cerner.com>> wrote:
Hi,

Has someone tried to run bounded Flink application using Oozie in a Kerberized cluster ? What else can we use to schedule bounded flink jobs.

Any thoughts or ideas will be appreciated!

Flink version: 1.8.0
Flink-runner version: 2.14.0

Best,
Rajat Srivastava
Software Engineer | HealtheIntent Dev
rajat.srivastava@cerner.com<ma...@cerner.com> | (816) 906-4958 direct
Cerner Corporation | www.cerner.com<http://www.cerner.com/>



CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.


--
Thanks & Regards
Sri Tummala