You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Gordon Tai <go...@vm5.com> on 2015/12/08 18:57:24 UTC
Passing Java system properties to YARN containers of Samza Job
Hi,
Is there any convenient way to pass custom Java system properties to Samza
jobs?
Specifically, since the system properties I wish to pass are
aws.accessKeyId and aws.secretKey for AWS credentials (
http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/credentials.html),
writing
them into the Samza job config file doesn't seem like a good idea.
Any help is great, thanks!
BR,
Gordon
--
*Gordon Tai* | Data Engineer | gordon@vm5.com | http://www.vm5.com
*VMFIVE CONFIDENTIAL*
This message, including any attachments, may contain information that is
confidential, proprietary, privileged or otherwise protected by law, and is
intended only for the parties involved in the subject discussed therein. If
you have received this e-mail by mistake, please immediately delete the
message and notify the sender of such incident. Please be noted that any
unauthorized use, dissemination, distribution or copying of this email is
strictly prohibited.
Re: Passing Java system properties to YARN containers of Samza Job
Posted by Boris Shkolnik <bo...@gmail.com>.
I am not sure about AWS credentials, but, in general, you can use task.opts
config value to pass any java level settings, which will be passed to each
container jvm.
On Tue, Dec 8, 2015 at 9:57 AM, Gordon Tai <go...@vm5.com> wrote:
> Hi,
>
> Is there any convenient way to pass custom Java system properties to Samza
> jobs?
> Specifically, since the system properties I wish to pass are
> aws.accessKeyId and aws.secretKey for AWS credentials (
>
> http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/credentials.html
> ),
> writing
> them into the Samza job config file doesn't seem like a good idea.
> Any help is great, thanks!
>
> BR,
> Gordon
>
> --
> *Gordon Tai* | Data Engineer | gordon@vm5.com | http://www.vm5.com
>
>
>
> *VMFIVE CONFIDENTIAL*
> This message, including any attachments, may contain information that is
> confidential, proprietary, privileged or otherwise protected by law, and is
> intended only for the parties involved in the subject discussed therein. If
> you have received this e-mail by mistake, please immediately delete the
> message and notify the sender of such incident. Please be noted that any
> unauthorized use, dissemination, distribution or copying of this email is
> strictly prohibited.
>
Re: Passing Java system properties to YARN containers of Samza Job
Posted by Tommy Becker <to...@tivo.com>.
You should also prefix the names of the API keys with "sensitive.". This will mask the values when the config is logged and in the AM UI.
On 12/09/2015 01:19 AM, Yi Pan wrote:
Hi, Gordon,
Try to use --config X.Y.Z=value from the command line to run run-job.sh
On Tue, Dec 8, 2015 at 5:57 PM, Gordon Tai <go...@vm5.com> wrote:
Hi Rad,
Our Samza jobs run on on-premiere clusters, on top of YARN and Kafka. So
IAM roles won't be an option either.
BR,
On 9 December 2015 at 04:09, Rad Gruchalski <ra...@gruchalski.com> wrote:
Gordon,
If your tasks run on EC2, you should use IAM roles.
https://www.youtube.com/watch?v=XuRM4Id6uDY
Kind regards,
Radek Gruchalski
radek@gruchalski.com<ma...@gruchalski.com> (mailto:radek@gruchalski.com) (mailto:
radek@gruchalski.com<ma...@gruchalski.com>)
de.linkedin.com/in/radgruchalski/ (
http://de.linkedin.com/in/radgruchalski/)
Confidentiality:
This communication is intended for the above-named person and may be
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor
must you copy or show it to anyone; please delete/destroy and inform the
sender immediately.
On Tuesday, 8 December 2015 at 18:57, Gordon Tai wrote:
Hi,
Is there any convenient way to pass custom Java system properties to
Samza
jobs?
Specifically, since the system properties I wish to pass are
aws.accessKeyId and aws.secretKey for AWS credentials (
http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/credentials.html
),
writing
them into the Samza job config file doesn't seem like a good idea.
Any help is great, thanks!
BR,
Gordon
--
*Gordon Tai* | Data Engineer | gordon@vm5.com<ma...@vm5.com> | http://www.vm5.com
*VMFIVE CONFIDENTIAL*
This message, including any attachments, may contain information that
is
confidential, proprietary, privileged or otherwise protected by law,
and
is
intended only for the parties involved in the subject discussed
therein.
If
you have received this e-mail by mistake, please immediately delete the
message and notify the sender of such incident. Please be noted that
any
unauthorized use, dissemination, distribution or copying of this email
is
strictly prohibited.
--
*Gordon Tai* | Data Engineer | gordon@vm5.com<ma...@vm5.com> | http://www.vm5.com
*VMFIVE CONFIDENTIAL*
This message, including any attachments, may contain information that is
confidential, proprietary, privileged or otherwise protected by law, and is
intended only for the parties involved in the subject discussed therein. If
you have received this e-mail by mistake, please immediately delete the
message and notify the sender of such incident. Please be noted that any
unauthorized use, dissemination, distribution or copying of this email is
strictly prohibited.
--
Tommy Becker
Senior Software Engineer
Digitalsmiths
A TiVo Company
www.digitalsmiths.com<http://www.digitalsmiths.com>
tobecker@tivo.com<ma...@tivo.com>
________________________________
This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.
Re: Passing Java system properties to YARN containers of Samza Job
Posted by Yi Pan <ni...@gmail.com>.
Hi, Gordon,
Try to use --config X.Y.Z=value from the command line to run run-job.sh
On Tue, Dec 8, 2015 at 5:57 PM, Gordon Tai <go...@vm5.com> wrote:
> Hi Rad,
>
> Our Samza jobs run on on-premiere clusters, on top of YARN and Kafka. So
> IAM roles won't be an option either.
>
> BR,
>
> On 9 December 2015 at 04:09, Rad Gruchalski <ra...@gruchalski.com> wrote:
>
> > Gordon,
> >
> > If your tasks run on EC2, you should use IAM roles.
> > https://www.youtube.com/watch?v=XuRM4Id6uDY
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Kind regards,
> > Radek Gruchalski
> > radek@gruchalski.com (mailto:radek@gruchalski.com) (mailto:
> > radek@gruchalski.com)
> > de.linkedin.com/in/radgruchalski/ (
> > http://de.linkedin.com/in/radgruchalski/)
> >
> > Confidentiality:
> > This communication is intended for the above-named person and may be
> > confidential and/or legally privileged.
> > If it has come to you in error you must take no action based on it, nor
> > must you copy or show it to anyone; please delete/destroy and inform the
> > sender immediately.
> >
> >
> >
> > On Tuesday, 8 December 2015 at 18:57, Gordon Tai wrote:
> >
> > > Hi,
> > >
> > > Is there any convenient way to pass custom Java system properties to
> > Samza
> > > jobs?
> > > Specifically, since the system properties I wish to pass are
> > > aws.accessKeyId and aws.secretKey for AWS credentials (
> > >
> >
> http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/credentials.html
> > ),
> > > writing
> > > them into the Samza job config file doesn't seem like a good idea.
> > > Any help is great, thanks!
> > >
> > > BR,
> > > Gordon
> > >
> > > --
> > > *Gordon Tai* | Data Engineer | gordon@vm5.com | http://www.vm5.com
> > >
> > >
> > >
> > > *VMFIVE CONFIDENTIAL*
> > > This message, including any attachments, may contain information that
> is
> > > confidential, proprietary, privileged or otherwise protected by law,
> and
> > is
> > > intended only for the parties involved in the subject discussed
> therein.
> > If
> > > you have received this e-mail by mistake, please immediately delete the
> > > message and notify the sender of such incident. Please be noted that
> any
> > > unauthorized use, dissemination, distribution or copying of this email
> is
> > > strictly prohibited.
> > >
> > >
> >
> >
> >
>
>
> --
> *Gordon Tai* | Data Engineer | gordon@vm5.com | http://www.vm5.com
>
>
>
> *VMFIVE CONFIDENTIAL*
> This message, including any attachments, may contain information that is
> confidential, proprietary, privileged or otherwise protected by law, and is
> intended only for the parties involved in the subject discussed therein. If
> you have received this e-mail by mistake, please immediately delete the
> message and notify the sender of such incident. Please be noted that any
> unauthorized use, dissemination, distribution or copying of this email is
> strictly prohibited.
>
Re: Passing Java system properties to YARN containers of Samza Job
Posted by Gordon Tai <go...@vm5.com>.
Hi Rad,
Our Samza jobs run on on-premiere clusters, on top of YARN and Kafka. So
IAM roles won't be an option either.
BR,
On 9 December 2015 at 04:09, Rad Gruchalski <ra...@gruchalski.com> wrote:
> Gordon,
>
> If your tasks run on EC2, you should use IAM roles.
> https://www.youtube.com/watch?v=XuRM4Id6uDY
>
>
>
>
>
>
>
>
>
>
> Kind regards,
> Radek Gruchalski
> radek@gruchalski.com (mailto:radek@gruchalski.com) (mailto:
> radek@gruchalski.com)
> de.linkedin.com/in/radgruchalski/ (
> http://de.linkedin.com/in/radgruchalski/)
>
> Confidentiality:
> This communication is intended for the above-named person and may be
> confidential and/or legally privileged.
> If it has come to you in error you must take no action based on it, nor
> must you copy or show it to anyone; please delete/destroy and inform the
> sender immediately.
>
>
>
> On Tuesday, 8 December 2015 at 18:57, Gordon Tai wrote:
>
> > Hi,
> >
> > Is there any convenient way to pass custom Java system properties to
> Samza
> > jobs?
> > Specifically, since the system properties I wish to pass are
> > aws.accessKeyId and aws.secretKey for AWS credentials (
> >
> http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/credentials.html
> ),
> > writing
> > them into the Samza job config file doesn't seem like a good idea.
> > Any help is great, thanks!
> >
> > BR,
> > Gordon
> >
> > --
> > *Gordon Tai* | Data Engineer | gordon@vm5.com | http://www.vm5.com
> >
> >
> >
> > *VMFIVE CONFIDENTIAL*
> > This message, including any attachments, may contain information that is
> > confidential, proprietary, privileged or otherwise protected by law, and
> is
> > intended only for the parties involved in the subject discussed therein.
> If
> > you have received this e-mail by mistake, please immediately delete the
> > message and notify the sender of such incident. Please be noted that any
> > unauthorized use, dissemination, distribution or copying of this email is
> > strictly prohibited.
> >
> >
>
>
>
--
*Gordon Tai* | Data Engineer | gordon@vm5.com | http://www.vm5.com
*VMFIVE CONFIDENTIAL*
This message, including any attachments, may contain information that is
confidential, proprietary, privileged or otherwise protected by law, and is
intended only for the parties involved in the subject discussed therein. If
you have received this e-mail by mistake, please immediately delete the
message and notify the sender of such incident. Please be noted that any
unauthorized use, dissemination, distribution or copying of this email is
strictly prohibited.
Re: Passing Java system properties to YARN containers of Samza
Job
Posted by Rad Gruchalski <ra...@gruchalski.com>.
Gordon,
If your tasks run on EC2, you should use IAM roles.
https://www.youtube.com/watch?v=XuRM4Id6uDY
Kind regards,
Radek Gruchalski
radek@gruchalski.com (mailto:radek@gruchalski.com)
(mailto:radek@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
Confidentiality:
This communication is intended for the above-named person and may be confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately.
On Tuesday, 8 December 2015 at 18:57, Gordon Tai wrote:
> Hi,
>
> Is there any convenient way to pass custom Java system properties to Samza
> jobs?
> Specifically, since the system properties I wish to pass are
> aws.accessKeyId and aws.secretKey for AWS credentials (
> http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/credentials.html),
> writing
> them into the Samza job config file doesn't seem like a good idea.
> Any help is great, thanks!
>
> BR,
> Gordon
>
> --
> *Gordon Tai* | Data Engineer | gordon@vm5.com | http://www.vm5.com
>
>
>
> *VMFIVE CONFIDENTIAL*
> This message, including any attachments, may contain information that is
> confidential, proprietary, privileged or otherwise protected by law, and is
> intended only for the parties involved in the subject discussed therein. If
> you have received this e-mail by mistake, please immediately delete the
> message and notify the sender of such incident. Please be noted that any
> unauthorized use, dissemination, distribution or copying of this email is
> strictly prohibited.
>
>