You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Vishwas Siravara <vs...@gmail.com> on 2019/06/28 16:43:38 UTC
Providing external files to flink classpath
Hi ,
I am trying to add external property files to the flink classpath for
my application. These files are not a part of the fat jar. I put them
under the lib folder but flink cant find them? How can I manage
external property files that needs to be read by flink ?
Thanks,
Vishwas
Re: Providing external files to flink classpath
Posted by Maxim Parkachov <la...@gmail.com>.
Hi Vishwas,
took me some time to find out as well. If you have your properties file
under lib following will work:
val kafkaPropertiesInputStream =
getClass.getClassLoader.getResourceAsStream("lib/config/kafka.properties")
Hope this helps,
Maxim.
On Wed, Jul 17, 2019 at 7:23 PM Vishwas Siravara <vs...@gmail.com>
wrote:
> Does the -yt option work for standalone cluster without dedicated resource
> manager ? So this property file is read by one of the dependencies inside
> my application as a file, so I can't really use Parameter tool to parse the
> config file.
>
> Thanks,
> Vishwas
>
> On Fri, Jun 28, 2019 at 11:08 PM Yun Tang <my...@live.com> wrote:
>
>> Hi Vishwas
>>
>>
>> 1. You could use '-yt' to ship specified files to the class path,
>> please refer to [1] for more details.
>> 2. If the properties are only loaded on client side before executing
>> the application, you could let your application to just read from local
>> property data. Flink support to load properties within the
>> ParameterTool [2].
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/cli.html#usage
>> [2]
>> https://github.com/apache/flink/blob/f1721293b0701d584d42bd68671181e332d2ad04/flink-java/src/main/java/org/apache/flink/api/java/utils/ParameterTool.java#L120
>>
>> Best
>> Yun Tang
>>
>> ------------------------------
>> *From:* Vishwas Siravara <vs...@gmail.com>
>> *Sent:* Saturday, June 29, 2019 0:43
>> *To:* user
>> *Subject:* Providing external files to flink classpath
>>
>> Hi ,
>> I am trying to add external property files to the flink classpath for
>> my application. These files are not a part of the fat jar. I put them
>> under the lib folder but flink cant find them? How can I manage
>> external property files that needs to be read by flink ?
>>
>> Thanks,
>> Vishwas
>>
>
Re: Providing external files to flink classpath
Posted by Vishwas Siravara <vs...@gmail.com>.
Does the -yt option work for standalone cluster without dedicated resource
manager ? So this property file is read by one of the dependencies inside
my application as a file, so I can't really use Parameter tool to parse the
config file.
Thanks,
Vishwas
On Fri, Jun 28, 2019 at 11:08 PM Yun Tang <my...@live.com> wrote:
> Hi Vishwas
>
>
> 1. You could use '-yt' to ship specified files to the class path,
> please refer to [1] for more details.
> 2. If the properties are only loaded on client side before executing
> the application, you could let your application to just read from local
> property data. Flink support to load properties within the
> ParameterTool [2].
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/cli.html#usage
> [2]
> https://github.com/apache/flink/blob/f1721293b0701d584d42bd68671181e332d2ad04/flink-java/src/main/java/org/apache/flink/api/java/utils/ParameterTool.java#L120
>
> Best
> Yun Tang
>
> ------------------------------
> *From:* Vishwas Siravara <vs...@gmail.com>
> *Sent:* Saturday, June 29, 2019 0:43
> *To:* user
> *Subject:* Providing external files to flink classpath
>
> Hi ,
> I am trying to add external property files to the flink classpath for
> my application. These files are not a part of the fat jar. I put them
> under the lib folder but flink cant find them? How can I manage
> external property files that needs to be read by flink ?
>
> Thanks,
> Vishwas
>
Re: Providing external files to flink classpath
Posted by Yun Tang <my...@live.com>.
Hi Vishwas
1. You could use '-yt' to ship specified files to the class path, please refer to [1] for more details.
2. If the properties are only loaded on client side before executing the application, you could let your application to just read from local property data. Flink support to load properties within the ParameterTool [2].
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/cli.html#usage
[2] https://github.com/apache/flink/blob/f1721293b0701d584d42bd68671181e332d2ad04/flink-java/src/main/java/org/apache/flink/api/java/utils/ParameterTool.java#L120
Best
Yun Tang
________________________________
From: Vishwas Siravara <vs...@gmail.com>
Sent: Saturday, June 29, 2019 0:43
To: user
Subject: Providing external files to flink classpath
Hi ,
I am trying to add external property files to the flink classpath for
my application. These files are not a part of the fat jar. I put them
under the lib folder but flink cant find them? How can I manage
external property files that needs to be read by flink ?
Thanks,
Vishwas