You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Jins George <ji...@aeris.net> on 2017/07/08 16:08:59 UTC
System properties when submitting flink job to YARN Session
Hello,
I want to set the path of a properties file as System property in my
application(something like -Dkey=value).
Is there a way to set it while submitting a flink job to running YARN
Session? I am using //bin/flink run/ to submit the job to a already
running YARN session.
Thanks,
Jins George
Re: System properties when submitting flink job to YARN Session
Posted by yunfan123 <yu...@foxmail.com>.
Can I The specific the jars that I depend on when I submit my project?
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/System-properties-when-submitting-flink-job-to-YARN-Session-tp14158p14207.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: System properties when submitting flink job to YARN Session
Posted by Aljoscha Krettek <al...@apache.org>.
Hi Jins,
Do these settings have to be in the Jar File? Since you’re using Beam, you could also use PipelineOptions to make the options accessible to functions at runtime.
Best,
Aljoscha
> On 12. Jul 2017, at 20:21, Jins George <ji...@aeris.net> wrote:
>
> Hi Aljoscha,
>
> I am still using Beam on Flink. I have one yarn session running multiple streaming jobs. The application jar contains some environment specific run time properties( like ip addresses, rest api end points etc). This adds overhead in my usecase as we have to deploy this in multiple environments. I was trying to decouple these properties files from the uber jar and provide as as either a classpath resource or pass the path of the file as a system property to the jvm.
>
> So far I noticed following options to achieve this.
> put all properties in a file and use --classpath file:// <file:///><path to file> option in flink run command . This needs the url to be accessible from all nodes, something like NFS
> use -D in yarn-session to pass each properties. This will need to restart the yarn session if a new property gets added.
> An ideal solution for me would to provide a local classpath to flink run command and that gets propagated to other workers automatically :)
>
> Thanks,
> Jins
> On 07/12/2017 02:25 AM, Aljoscha Krettek wrote:
>> Hi,
>>
>> Yes, setting the property using -D when creating the session should work to make it available on all workers. I think after that it cannot be changed since they JVMs are already running.
>>
>> If I may ask, what’s your use case for this? Are you still using Beam on Flink or are you using vanilla Flink with this?
>>
>> Best,
>> Aljoscha
>>
>>> On 11. Jul 2017, at 07:24, Jins George <ji...@aeris.net> <ma...@aeris.net> wrote:
>>>
>>> Thanks Nico. I am able to pass arguments to the main program, that works, but not exactly that I was looking for.
>>>
>>> I guess to have all worker jvms the same system property, I have to set it at yarn-session creation time using -D ( haven't tried it yet)
>>>
>>> Thanks,
>>> Jins George
>>>
>>> On 07/10/2017 06:56 AM, Nico Kruber wrote:
>>>> Hi Jins,
>>>> I'm not sure whether you can define a system property, but you can include it
>>>> in the program arguments of "flink run [OPTIONS] <jar-file> <arguments>"
>>>>
>>>> You may also be able to define system properties but these are probably only
>>>> valid in your main() function executed within the flink run script, not any
>>>> operators run on other JVM nodes. Have you tried that?
>>>>
>>>>
>>>> Nico
>>>>
>>>> On Saturday, 8 July 2017 18:08:59 CEST Jins George wrote:
>>>>> Hello,
>>>>>
>>>>> I want to set the path of a properties file as System property in my
>>>>> application(something like -Dkey=value).
>>>>> Is there a way to set it while submitting a flink job to running YARN
>>>>> Session? I am using //bin/flink run/ to submit the job to a already
>>>>> running YARN session.
>>>>>
>>>>> Thanks,
>>>>> Jins George
>>
>
Re: System properties when submitting flink job to YARN Session
Posted by Jins George <ji...@aeris.net>.
Hi Aljoscha,
I am still using Beam on Flink. I have one yarn session running multiple
streaming jobs. The application jar contains some environment specific
run time properties( like ip addresses, rest api end points etc). This
adds overhead in my usecase as we have to deploy this in multiple
environments. I was trying to decouple these properties files from the
uber jar and provide as as either a classpath resource or pass the path
of the file as a system property to the jvm.
So far I noticed following options to achieve this.
* put all properties in a file and use /--classpath/ file://<path to
file> option in /flink run /command . This needs the url to be
accessible from all nodes, something like NFS
* use -D in yarn-session to pass each properties. This will need to
restart the yarn session if a new property gets added.
An ideal solution for me would to provide a local classpath to flink run
command and that gets propagated to other workers automatically :)
Thanks,
Jins
On 07/12/2017 02:25 AM, Aljoscha Krettek wrote:
> Hi,
>
> Yes, setting the property using -D when creating the session should work to make it available on all workers. I think after that it cannot be changed since they JVMs are already running.
>
> If I may ask, what’s your use case for this? Are you still using Beam on Flink or are you using vanilla Flink with this?
>
> Best,
> Aljoscha
>
>> On 11. Jul 2017, at 07:24, Jins George <ji...@aeris.net> wrote:
>>
>> Thanks Nico. I am able to pass arguments to the main program, that works, but not exactly that I was looking for.
>>
>> I guess to have all worker jvms the same system property, I have to set it at yarn-session creation time using -D ( haven't tried it yet)
>>
>> Thanks,
>> Jins George
>>
>> On 07/10/2017 06:56 AM, Nico Kruber wrote:
>>> Hi Jins,
>>> I'm not sure whether you can define a system property, but you can include it
>>> in the program arguments of "flink run [OPTIONS] <jar-file> <arguments>"
>>>
>>> You may also be able to define system properties but these are probably only
>>> valid in your main() function executed within the flink run script, not any
>>> operators run on other JVM nodes. Have you tried that?
>>>
>>>
>>> Nico
>>>
>>> On Saturday, 8 July 2017 18:08:59 CEST Jins George wrote:
>>>> Hello,
>>>>
>>>> I want to set the path of a properties file as System property in my
>>>> application(something like -Dkey=value).
>>>> Is there a way to set it while submitting a flink job to running YARN
>>>> Session? I am using //bin/flink run/ to submit the job to a already
>>>> running YARN session.
>>>>
>>>> Thanks,
>>>> Jins George
>
Re: System properties when submitting flink job to YARN Session
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
Yes, setting the property using -D when creating the session should work to make it available on all workers. I think after that it cannot be changed since they JVMs are already running.
If I may ask, what’s your use case for this? Are you still using Beam on Flink or are you using vanilla Flink with this?
Best,
Aljoscha
> On 11. Jul 2017, at 07:24, Jins George <ji...@aeris.net> wrote:
>
> Thanks Nico. I am able to pass arguments to the main program, that works, but not exactly that I was looking for.
>
> I guess to have all worker jvms the same system property, I have to set it at yarn-session creation time using -D ( haven't tried it yet)
>
> Thanks,
> Jins George
>
> On 07/10/2017 06:56 AM, Nico Kruber wrote:
>> Hi Jins,
>> I'm not sure whether you can define a system property, but you can include it
>> in the program arguments of "flink run [OPTIONS] <jar-file> <arguments>"
>>
>> You may also be able to define system properties but these are probably only
>> valid in your main() function executed within the flink run script, not any
>> operators run on other JVM nodes. Have you tried that?
>>
>>
>> Nico
>>
>> On Saturday, 8 July 2017 18:08:59 CEST Jins George wrote:
>>> Hello,
>>>
>>> I want to set the path of a properties file as System property in my
>>> application(something like -Dkey=value).
>>> Is there a way to set it while submitting a flink job to running YARN
>>> Session? I am using //bin/flink run/ to submit the job to a already
>>> running YARN session.
>>>
>>> Thanks,
>>> Jins George
>
Re: System properties when submitting flink job to YARN Session
Posted by Jins George <ji...@aeris.net>.
Thanks Nico. I am able to pass arguments to the main program, that
works, but not exactly that I was looking for.
I guess to have all worker jvms the same system property, I have to
set it at yarn-session creation time using -D ( haven't tried it yet)
Thanks,
Jins George
On 07/10/2017 06:56 AM, Nico Kruber wrote:
> Hi Jins,
> I'm not sure whether you can define a system property, but you can include it
> in the program arguments of "flink run [OPTIONS] <jar-file> <arguments>"
>
> You may also be able to define system properties but these are probably only
> valid in your main() function executed within the flink run script, not any
> operators run on other JVM nodes. Have you tried that?
>
>
> Nico
>
> On Saturday, 8 July 2017 18:08:59 CEST Jins George wrote:
>> Hello,
>>
>> I want to set the path of a properties file as System property in my
>> application(something like -Dkey=value).
>> Is there a way to set it while submitting a flink job to running YARN
>> Session? I am using //bin/flink run/ to submit the job to a already
>> running YARN session.
>>
>> Thanks,
>> Jins George
Re: System properties when submitting flink job to YARN Session
Posted by Nico Kruber <ni...@data-artisans.com>.
Hi Jins,
I'm not sure whether you can define a system property, but you can include it
in the program arguments of "flink run [OPTIONS] <jar-file> <arguments>"
You may also be able to define system properties but these are probably only
valid in your main() function executed within the flink run script, not any
operators run on other JVM nodes. Have you tried that?
Nico
On Saturday, 8 July 2017 18:08:59 CEST Jins George wrote:
> Hello,
>
> I want to set the path of a properties file as System property in my
> application(something like -Dkey=value).
> Is there a way to set it while submitting a flink job to running YARN
> Session? I am using //bin/flink run/ to submit the job to a already
> running YARN session.
>
> Thanks,
> Jins George