You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Manas Kale <ma...@gmail.com> on 2020/10/21 12:24:34 UTC
Correct way to package application.properties file with Flink JAR
Hi,
I have a Flink job that I am packaging as a JAR that is submitted to the
Flink cluster runtime. However, this JAR reads a few configuration values
from a .properties file.
What is the recommended way to package this properties file when submitting
to a cluster? Do I have to copy it to a folder in my flink cluster
installation?
My own attempt is a somewhat convoluted method that is not working.
Basically I set an environment variable that points to the properties file,
and I use that at runtime to read configuration values. This works when I
run it in my IDE as a minicluster but fails when I submit it to the
cluster. I'm kind of stuck debugging this as for some reason I am not able
to see the logs from the configuration reader class (asked a question about
that in a separate thread).
Re: Correct way to package application.properties file with Flink JAR
Posted by Manas Kale <ma...@gmail.com>.
Okay, I solved the other issue with viewing logs which proved that correct,
non-null values are being loaded. I believe I have a different issue
altogether so will create a separate thread for that. Thanks for the help
Chesnay!
On Thu, Oct 22, 2020 at 11:30 AM Manas Kale <ma...@gmail.com> wrote:
> Hi Chesnay,
> The Config reader has everything static, so I tried using
>
> Config.class.getClassLoader().getResourceAsStream("pipeline.properties");
>
> Here's the .pom file for that file:
>
> <build>
>
> <resources>
> <resource>
> <directory>src/main/resources</directory>
> <includes>
> <include>pipeline.properties</include>
> <include>log4j.properties</include>
> </includes>
> </resource>
> </resources>
>
> ...
>
> </build>
>
>
> I can see the pipeline.properties packaged in my JAR at the root level
> (using maven package command).
> However, this works on my IDE minicluster but loads null values when
> submitted to the cluster.
> The Config class is not at the package root, rather it is a few levels
> deep. Does that make a difference and cause the getClassLoader() to treat
> an inner package as root?
>
>
>
> On Wed, Oct 21, 2020 at 6:06 PM Chesnay Schepler <ch...@apache.org>
> wrote:
>
>> You could bundle said file in the jar and retrieve it via
>> getClass().getClassLoader().getResource("<file name>").
>>
>> On 10/21/2020 2:24 PM, Manas Kale wrote:
>> > Hi,
>> > I have a Flink job that I am packaging as a JAR that is submitted to
>> > the Flink cluster runtime. However, this JAR reads a few configuration
>> > values from a .properties file.
>> > What is the recommended way to package this properties file when
>> > submitting to a cluster? Do I have to copy it to a folder in my flink
>> > cluster installation?
>> >
>> > My own attempt is a somewhat convoluted method that is not working.
>> > Basically I set an environment variable that points to the properties
>> > file, and I use that at runtime to read configuration values. This
>> > works when I run it in my IDE as a minicluster but fails when I submit
>> > it to the cluster. I'm kind of stuck debugging this as for some reason
>> > I am not able to see the logs from the configuration reader class
>> > (asked a question about that in a separate thread).
>>
>>
>>
Re: Correct way to package application.properties file with Flink JAR
Posted by Manas Kale <ma...@gmail.com>.
Hi Chesnay,
The Config reader has everything static, so I tried using
Config.class.getClassLoader().getResourceAsStream("pipeline.properties");
Here's the .pom file for that file:
<build>
<resources>
<resource>
<directory>src/main/resources</directory>
<includes>
<include>pipeline.properties</include>
<include>log4j.properties</include>
</includes>
</resource>
</resources>
...
</build>
I can see the pipeline.properties packaged in my JAR at the root level
(using maven package command).
However, this works on my IDE minicluster but loads null values when
submitted to the cluster.
The Config class is not at the package root, rather it is a few levels
deep. Does that make a difference and cause the getClassLoader() to treat
an inner package as root?
On Wed, Oct 21, 2020 at 6:06 PM Chesnay Schepler <ch...@apache.org> wrote:
> You could bundle said file in the jar and retrieve it via
> getClass().getClassLoader().getResource("<file name>").
>
> On 10/21/2020 2:24 PM, Manas Kale wrote:
> > Hi,
> > I have a Flink job that I am packaging as a JAR that is submitted to
> > the Flink cluster runtime. However, this JAR reads a few configuration
> > values from a .properties file.
> > What is the recommended way to package this properties file when
> > submitting to a cluster? Do I have to copy it to a folder in my flink
> > cluster installation?
> >
> > My own attempt is a somewhat convoluted method that is not working.
> > Basically I set an environment variable that points to the properties
> > file, and I use that at runtime to read configuration values. This
> > works when I run it in my IDE as a minicluster but fails when I submit
> > it to the cluster. I'm kind of stuck debugging this as for some reason
> > I am not able to see the logs from the configuration reader class
> > (asked a question about that in a separate thread).
>
>
>
Re: Correct way to package application.properties file with Flink JAR
Posted by Chesnay Schepler <ch...@apache.org>.
You could bundle said file in the jar and retrieve it via
getClass().getClassLoader().getResource("<file name>").
On 10/21/2020 2:24 PM, Manas Kale wrote:
> Hi,
> I have a Flink job that I am packaging as a JAR that is submitted to
> the Flink cluster runtime. However, this JAR reads a few configuration
> values from a .properties file.
> What is the recommended way to package this properties file when
> submitting to a cluster? Do I have to copy it to a folder in my flink
> cluster installation?
>
> My own attempt is a somewhat convoluted method that is not working.
> Basically I set an environment variable that points to the properties
> file, and I use that at runtime to read configuration values. This
> works when I run it in my IDE as a minicluster but fails when I submit
> it to the cluster. I'm kind of stuck debugging this as for some reason
> I am not able to see the logs from the configuration reader class
> (asked a question about that in a separate thread).