externalizing config flies for flink class loader

[Vishwas Siravara <vs...@gmail.com>]

I have a standalone cluster. I have added my own library(jar file) to the
lib/ folder in flink . I submit my job from cli after I start the cluster.
Now I want to externalize a property file which has to be read by this
library. Since this library is loaded by flink's classloader and not the
application class loader I cannot supply this using flink run -C ... since
this works only for user class loader.


Thanks,
Vishwas

Re: externalizing config flies for flink class loader

[Vijay Bhaskar <bh...@gmail.com>]

Sorry there is a typo, corrected it:
val pmtool = ParameterTool.fromArgs(args)
val defaultConfig = ConfigFactory.load() //Default config in
reference.conf/application.conf/system properties/env  of typesafe
val overrideConfigFromArgs =  ConfigFactory.load(pmtool.toMap)
val finalConfig =  overrideConfigFromArgs.withFallBack(defaultConfig)
//This is going to override your default configuration  params

On Fri, Sep 13, 2019 at 11:38 AM Vijay Bhaskar <bh...@gmail.com>
wrote:

> Hi
> You can use this way:
> Use typesafe configuration, which provides excellent configuration
> methodologies.
> You supply default configuration, which is read by your application
> through reference.conf file of typesafe. If you want to override any of the
> defaults you can supply to command line arguments
> From the command line arguments, you can read them into Paramter tool map,
> which is supplied by flink. You can use this map to override the default
> configuration of the typesafe.
> Following is the code snippet:
>
> val pmtool = ParameterTool.fromArgs(args)
> val defaultConfig = ConfigFactory.load() //Default config in
> reference.conf/application.conf/system properties/env  of typesafe
> val overrideConfigFromArgs =  ConfigFactory.load(pmtool.toMap)
> val finalConfig =  overrideConfigFromArgs.withFallBack(defaultConfig)
> //This is going to override your command line params
>
> Regards
> Bhaskar
>
>
>
> On Fri, Sep 13, 2019 at 12:51 AM Vishwas Siravara <vs...@gmail.com>
> wrote:
>
>> I have a standalone cluster. I have added my own library(jar file) to the
>> lib/ folder in flink . I submit my job from cli after I start the cluster.
>> Now I want to externalize a property file which has to be read by this
>> library. Since this library is loaded by flink's classloader and not the
>> application class loader I cannot supply this using flink run -C ... since
>> this works only for user class loader.
>>
>>
>> Thanks,
>> Vishwas
>>
>

Re: externalizing config flies for flink class loader

[Vijay Bhaskar <bh...@gmail.com>]

Hi
You can use this way:
Use typesafe configuration, which provides excellent configuration
methodologies.
You supply default configuration, which is read by your application through
reference.conf file of typesafe. If you want to override any of the
defaults you can supply to command line arguments
From the command line arguments, you can read them into Paramter tool map,
which is supplied by flink. You can use this map to override the default
configuration of the typesafe.
Following is the code snippet:

val pmtool = ParameterTool.fromArgs(args)
val defaultConfig = ConfigFactory.load() //Default config in
reference.conf/application.conf/system properties/env  of typesafe
val overrideConfigFromArgs =  ConfigFactory.load(pmtool.toMap)
val finalConfig =  overrideConfigFromArgs.withFallBack(defaultConfig)
//This is going to override your command line params

Regards
Bhaskar



On Fri, Sep 13, 2019 at 12:51 AM Vishwas Siravara <vs...@gmail.com>
wrote:

> I have a standalone cluster. I have added my own library(jar file) to the
> lib/ folder in flink . I submit my job from cli after I start the cluster.
> Now I want to externalize a property file which has to be read by this
> library. Since this library is loaded by flink's classloader and not the
> application class loader I cannot supply this using flink run -C ... since
> this works only for user class loader.
>
>
> Thanks,
> Vishwas
>