You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Tony Wei <to...@gmail.com> on 2017/09/08 14:32:34 UTC
How to user ParameterTool.fromPropertiesFile() to get resource file
inside my jar
Hi,
I put the my configuration file in `./src/main/resources/` and packed it
inside my jar.
I want to run it on standalone cluster by using web UI to submit my job.
No matter which way I tried, the ParameterTool.fromPropertiesFile()
couldn't find the file path, but threw `FileNotFoundException` instead.
Is there any best practice to deal with such problem? Thanks for your help.
Best Regards,
Tony Wei
Re: How to user ParameterTool.fromPropertiesFile() to get resource
file inside my jar
Posted by Aljoscha Krettek <al...@apache.org>.
Hi Ron,
This is the issue and it was already implemented for upcoming 1.3.3 and 1.4.0.
https://issues.apache.org/jira/browse/FLINK-7630 <https://issues.apache.org/jira/browse/FLINK-7630>
Best,
Aljoscha
> On 26. Sep 2017, at 21:10, Ron Crocker <rc...@newrelic.com> wrote:
>
> What’s crazy is that I just stumbled on the same issue. Thanks for sharing!
>
> Ron
> —
> Ron Crocker
> Principal Engineer & Architect
> ( ( •)) New Relic
> rcrocker@newrelic.com <ma...@newrelic.com>
> M: +1 630 363 8835
>
>> On Sep 15, 2017, at 7:30 AM, Tony Wei <tony19920430@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hi Aljoscha,
>>
>> Thanks for your reply. It looks great to have hat feature. I will create a Jira issue for that and try to solve it.
>>
>> Best Regards,
>> Tony Wei
>>
>> 2017-09-15 20:51 GMT+08:00 Aljoscha Krettek <aljoscha@apache.org <ma...@apache.org>>:
>> Hi,
>>
>> I think calling getPath() on the URL returned from getResource() loses some of the information that is required to resolve the file in the jar. The solution should be to allow passing a "File" to ParameterTool.fromPropertiesFile() or to allow passing an InputStream to ParameterTool.fromPropertiesFile(). Passing a File should work because a File can be constructed from an URI and a URL can be turned into a URI.
>>
>> Would you be interested in opening a Jira issue for that and working on it?
>>
>> Best,
>> Aljoscha
>>
>>> On 15. Sep 2017, at 03:32, Tony Wei <tony19920430@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> Hi Aljoscha,
>>>
>>> I used Maven with this command "mvn clean package -Pbuild-jar" to create the jar.
>>>
>>> Best Regards,
>>> Tony Wei
>>>
>>> Aljoscha Krettek <aljoscha@apache.org <ma...@apache.org>>於 2017年9月14日 週四，下午6:24寫道：
>>> Hi,
>>>
>>> Are you using Maven to create the Jar or your IDE? I think this might be a problem only when creating the Jar via the IDE.
>>>
>>> Best,
>>> Aljoscha
>>>
>>>> On 11. Sep 2017, at 04:46, Tony Wei <tony19920430@gmail.com <ma...@gmail.com>> wrote:
>>>>
>>>> Hi Aljoscha,
>>>>
>>>> I found the root cause of my problem from this reference https://stackoverflow.com/questions/18151072/cant-find-resource-file-after-exporting-to-a-runnable-jar <https://stackoverflow.com/questions/18151072/cant-find-resource-file-after-exporting-to-a-runnable-jar>.
>>>> So I changed the way to use ParameterTool. I read the configurations from InputStream, construct them as argument format and used ParameterTool.fromArgs() to parse them with other arguments.
>>>> I'm not sure if this is a good solution. If you have any better one, please let me know. Thanks for your help.
>>>>
>>>> Best Regards,
>>>> Tony Wei
>>>>
>>>> 2017-09-08 23:40 GMT+08:00 Tony Wei <tony19920430@gmail.com <ma...@gmail.com>>:
>>>> Hi Aljoscha,
>>>>
>>>> I have tried `StreamJob.class.getClassLoader().getResource("application.conf").getPath()`, but I got this exception.
>>>>
>>>> Caused by: java.io.FileNotFoundException: Properties file /home/tonywei/flink/file:/tmp/flink-web-24351e69-a261-45be-9503-087db8155a8f/d69a3ca9-bfa0-43ef-83e8-e15f38162a87_quickstart-0.1.jar!/application.conf
>>>>
>>>> Best Regards,
>>>> Tony Wei
>>>>
>>>> 2017-09-08 23:24 GMT+08:00 Aljoscha Krettek <aljoscha@apache.org <ma...@apache.org>>:
>>>> Hi,
>>>>
>>>> How are you specifying the path for the properties file? Have you tried reading the properties by using this.getClass().getClassLoader().getResource()?
>>>>
>>>> Best,
>>>> Aljoscha
>>>>
>>>> > On 8. Sep 2017, at 16:32, Tony Wei <tony19920430@gmail.com <ma...@gmail.com>> wrote:
>>>> >
>>>> > Hi,
>>>> >
>>>> > I put the my configuration file in `./src/main/resources/` and packed it inside my jar.
>>>> > I want to run it on standalone cluster by using web UI to submit my job.
>>>> > No matter which way I tried, the ParameterTool.fromPropertiesFile() couldn't find the file path, but threw `FileNotFoundException` instead.
>>>> > Is there any best practice to deal with such problem? Thanks for your help.
>>>> >
>>>> > Best Regards,
>>>> > Tony Wei
>>>>
>>>>
>>>>
>>>
>>
>>
>
Re: How to user ParameterTool.fromPropertiesFile() to get resource
file inside my jar
Posted by Ron Crocker <rc...@newrelic.com>.
What’s crazy is that I just stumbled on the same issue. Thanks for sharing!
Ron
—
Ron Crocker
Principal Engineer & Architect
( ( •)) New Relic
rcrocker@newrelic.com
M: +1 630 363 8835
> On Sep 15, 2017, at 7:30 AM, Tony Wei <to...@gmail.com> wrote:
>
> Hi Aljoscha,
>
> Thanks for your reply. It looks great to have hat feature. I will create a Jira issue for that and try to solve it.
>
> Best Regards,
> Tony Wei
>
> 2017-09-15 20:51 GMT+08:00 Aljoscha Krettek <aljoscha@apache.org <ma...@apache.org>>:
> Hi,
>
> I think calling getPath() on the URL returned from getResource() loses some of the information that is required to resolve the file in the jar. The solution should be to allow passing a "File" to ParameterTool.fromPropertiesFile() or to allow passing an InputStream to ParameterTool.fromPropertiesFile(). Passing a File should work because a File can be constructed from an URI and a URL can be turned into a URI.
>
> Would you be interested in opening a Jira issue for that and working on it?
>
> Best,
> Aljoscha
>
>> On 15. Sep 2017, at 03:32, Tony Wei <tony19920430@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hi Aljoscha,
>>
>> I used Maven with this command "mvn clean package -Pbuild-jar" to create the jar.
>>
>> Best Regards,
>> Tony Wei
>>
>> Aljoscha Krettek <aljoscha@apache.org <ma...@apache.org>>於 2017年9月14日 週四，下午6:24寫道：
>> Hi,
>>
>> Are you using Maven to create the Jar or your IDE? I think this might be a problem only when creating the Jar via the IDE.
>>
>> Best,
>> Aljoscha
>>
>>> On 11. Sep 2017, at 04:46, Tony Wei <tony19920430@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> Hi Aljoscha,
>>>
>>> I found the root cause of my problem from this reference https://stackoverflow.com/questions/18151072/cant-find-resource-file-after-exporting-to-a-runnable-jar <https://stackoverflow.com/questions/18151072/cant-find-resource-file-after-exporting-to-a-runnable-jar>.
>>> So I changed the way to use ParameterTool. I read the configurations from InputStream, construct them as argument format and used ParameterTool.fromArgs() to parse them with other arguments.
>>> I'm not sure if this is a good solution. If you have any better one, please let me know. Thanks for your help.
>>>
>>> Best Regards,
>>> Tony Wei
>>>
>>> 2017-09-08 23:40 GMT+08:00 Tony Wei <tony19920430@gmail.com <ma...@gmail.com>>:
>>> Hi Aljoscha,
>>>
>>> I have tried `StreamJob.class.getClassLoader().getResource("application.conf").getPath()`, but I got this exception.
>>>
>>> Caused by: java.io.FileNotFoundException: Properties file /home/tonywei/flink/file:/tmp/flink-web-24351e69-a261-45be-9503-087db8155a8f/d69a3ca9-bfa0-43ef-83e8-e15f38162a87_quickstart-0.1.jar!/application.conf
>>>
>>> Best Regards,
>>> Tony Wei
>>>
>>> 2017-09-08 23:24 GMT+08:00 Aljoscha Krettek <aljoscha@apache.org <ma...@apache.org>>:
>>> Hi,
>>>
>>> How are you specifying the path for the properties file? Have you tried reading the properties by using this.getClass().getClassLoader().getResource()?
>>>
>>> Best,
>>> Aljoscha
>>>
>>> > On 8. Sep 2017, at 16:32, Tony Wei <tony19920430@gmail.com <ma...@gmail.com>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I put the my configuration file in `./src/main/resources/` and packed it inside my jar.
>>> > I want to run it on standalone cluster by using web UI to submit my job.
>>> > No matter which way I tried, the ParameterTool.fromPropertiesFile() couldn't find the file path, but threw `FileNotFoundException` instead.
>>> > Is there any best practice to deal with such problem? Thanks for your help.
>>> >
>>> > Best Regards,
>>> > Tony Wei
>>>
>>>
>>>
>>
>
>
Re: How to user ParameterTool.fromPropertiesFile() to get resource
file inside my jar
Posted by Tony Wei <to...@gmail.com>.
Hi Aljoscha,
Thanks for your reply. It looks great to have hat feature. I will create a
Jira issue for that and try to solve it.
Best Regards,
Tony Wei
2017-09-15 20:51 GMT+08:00 Aljoscha Krettek <al...@apache.org>:
> Hi,
>
> I think calling getPath() on the URL returned from getResource() loses
> some of the information that is required to resolve the file in the jar.
> The solution should be to allow passing a "File" to ParameterTool.fromPropertiesFile()
> or to allow passing an InputStream to ParameterTool.fromPropertiesFile().
> Passing a File should work because a File can be constructed from an URI
> and a URL can be turned into a URI.
>
> Would you be interested in opening a Jira issue for that and working on it?
>
> Best,
> Aljoscha
>
> On 15. Sep 2017, at 03:32, Tony Wei <to...@gmail.com> wrote:
>
> Hi Aljoscha,
>
> I used Maven with this command "mvn clean package -Pbuild-jar" to create
> the jar.
>
> Best Regards,
> Tony Wei
>
> Aljoscha Krettek <al...@apache.org>於 2017年9月14日 週四，下午6:24寫道：
>
>> Hi,
>>
>> Are you using Maven to create the Jar or your IDE? I think this might be
>> a problem only when creating the Jar via the IDE.
>>
>> Best,
>> Aljoscha
>>
>> On 11. Sep 2017, at 04:46, Tony Wei <to...@gmail.com> wrote:
>>
>> Hi Aljoscha,
>>
>> I found the root cause of my problem from this reference
>> https://stackoverflow.com/questions/18151072/cant-
>> find-resource-file-after-exporting-to-a-runnable-jar.
>> So I changed the way to use ParameterTool. I read the configurations from
>> InputStream, construct them as argument format and used
>> ParameterTool.fromArgs() to parse them with other arguments.
>> I'm not sure if this is a good solution. If you have any better one,
>> please let me know. Thanks for your help.
>>
>> Best Regards,
>> Tony Wei
>>
>> 2017-09-08 23:40 GMT+08:00 Tony Wei <to...@gmail.com>:
>>
>>> Hi Aljoscha,
>>>
>>> I have tried `StreamJob.class.getClassLoader().getResource("application.conf").getPath()`,
>>> but I got this exception.
>>>
>>> Caused by: java.io.FileNotFoundException: Properties file
>>> /home/tonywei/flink/file:/tmp/flink-web-24351e69-a261-45be-9
>>> 503-087db8155a8f/d69a3ca9-bfa0-43ef-83e8-e15f38162a87_quicks
>>> tart-0.1.jar!/application.conf
>>>
>>> Best Regards,
>>> Tony Wei
>>>
>>> 2017-09-08 23:24 GMT+08:00 Aljoscha Krettek <al...@apache.org>:
>>>
>>>> Hi,
>>>>
>>>> How are you specifying the path for the properties file? Have you tried
>>>> reading the properties by using this.getClass().getClassLoader
>>>> ().getResource()?
>>>>
>>>> Best,
>>>> Aljoscha
>>>>
>>>> > On 8. Sep 2017, at 16:32, Tony Wei <to...@gmail.com> wrote:
>>>> >
>>>> > Hi,
>>>> >
>>>> > I put the my configuration file in `./src/main/resources/` and packed
>>>> it inside my jar.
>>>> > I want to run it on standalone cluster by using web UI to submit my
>>>> job.
>>>> > No matter which way I tried, the ParameterTool.fromPropertiesFile()
>>>> couldn't find the file path, but threw `FileNotFoundException` instead.
>>>> > Is there any best practice to deal with such problem? Thanks for your
>>>> help.
>>>> >
>>>> > Best Regards,
>>>> > Tony Wei
>>>>
>>>>
>>>
>>
>>
>
Re: How to user ParameterTool.fromPropertiesFile() to get resource
file inside my jar
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
I think calling getPath() on the URL returned from getResource() loses some of the information that is required to resolve the file in the jar. The solution should be to allow passing a "File" to ParameterTool.fromPropertiesFile() or to allow passing an InputStream to ParameterTool.fromPropertiesFile(). Passing a File should work because a File can be constructed from an URI and a URL can be turned into a URI.
Would you be interested in opening a Jira issue for that and working on it?
Best,
Aljoscha
> On 15. Sep 2017, at 03:32, Tony Wei <to...@gmail.com> wrote:
>
> Hi Aljoscha,
>
> I used Maven with this command "mvn clean package -Pbuild-jar" to create the jar.
>
> Best Regards,
> Tony Wei
>
> Aljoscha Krettek <aljoscha@apache.org <ma...@apache.org>>於 2017年9月14日 週四，下午6:24寫道：
> Hi,
>
> Are you using Maven to create the Jar or your IDE? I think this might be a problem only when creating the Jar via the IDE.
>
> Best,
> Aljoscha
>
>> On 11. Sep 2017, at 04:46, Tony Wei <tony19920430@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hi Aljoscha,
>>
>> I found the root cause of my problem from this reference https://stackoverflow.com/questions/18151072/cant-find-resource-file-after-exporting-to-a-runnable-jar <https://stackoverflow.com/questions/18151072/cant-find-resource-file-after-exporting-to-a-runnable-jar>.
>> So I changed the way to use ParameterTool. I read the configurations from InputStream, construct them as argument format and used ParameterTool.fromArgs() to parse them with other arguments.
>> I'm not sure if this is a good solution. If you have any better one, please let me know. Thanks for your help.
>>
>> Best Regards,
>> Tony Wei
>>
>> 2017-09-08 23:40 GMT+08:00 Tony Wei <tony19920430@gmail.com <ma...@gmail.com>>:
>> Hi Aljoscha,
>>
>> I have tried `StreamJob.class.getClassLoader().getResource("application.conf").getPath()`, but I got this exception.
>>
>> Caused by: java.io.FileNotFoundException: Properties file /home/tonywei/flink/file:/tmp/flink-web-24351e69-a261-45be-9503-087db8155a8f/d69a3ca9-bfa0-43ef-83e8-e15f38162a87_quickstart-0.1.jar!/application.conf
>>
>> Best Regards,
>> Tony Wei
>>
>> 2017-09-08 23:24 GMT+08:00 Aljoscha Krettek <aljoscha@apache.org <ma...@apache.org>>:
>> Hi,
>>
>> How are you specifying the path for the properties file? Have you tried reading the properties by using this.getClass().getClassLoader().getResource()?
>>
>> Best,
>> Aljoscha
>>
>> > On 8. Sep 2017, at 16:32, Tony Wei <tony19920430@gmail.com <ma...@gmail.com>> wrote:
>> >
>> > Hi,
>> >
>> > I put the my configuration file in `./src/main/resources/` and packed it inside my jar.
>> > I want to run it on standalone cluster by using web UI to submit my job.
>> > No matter which way I tried, the ParameterTool.fromPropertiesFile() couldn't find the file path, but threw `FileNotFoundException` instead.
>> > Is there any best practice to deal with such problem? Thanks for your help.
>> >
>> > Best Regards,
>> > Tony Wei
>>
>>
>>
>
Re: How to user ParameterTool.fromPropertiesFile() to get resource
file inside my jar
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
Are you using Maven to create the Jar or your IDE? I think this might be a problem only when creating the Jar via the IDE.
Best,
Aljoscha
> On 11. Sep 2017, at 04:46, Tony Wei <to...@gmail.com> wrote:
>
> Hi Aljoscha,
>
> I found the root cause of my problem from this reference https://stackoverflow.com/questions/18151072/cant-find-resource-file-after-exporting-to-a-runnable-jar <https://stackoverflow.com/questions/18151072/cant-find-resource-file-after-exporting-to-a-runnable-jar>.
> So I changed the way to use ParameterTool. I read the configurations from InputStream, construct them as argument format and used ParameterTool.fromArgs() to parse them with other arguments.
> I'm not sure if this is a good solution. If you have any better one, please let me know. Thanks for your help.
>
> Best Regards,
> Tony Wei
>
> 2017-09-08 23:40 GMT+08:00 Tony Wei <tony19920430@gmail.com <ma...@gmail.com>>:
> Hi Aljoscha,
>
> I have tried `StreamJob.class.getClassLoader().getResource("application.conf").getPath()`, but I got this exception.
>
> Caused by: java.io.FileNotFoundException: Properties file /home/tonywei/flink/file:/tmp/flink-web-24351e69-a261-45be-9503-087db8155a8f/d69a3ca9-bfa0-43ef-83e8-e15f38162a87_quickstart-0.1.jar!/application.conf
>
> Best Regards,
> Tony Wei
>
> 2017-09-08 23:24 GMT+08:00 Aljoscha Krettek <aljoscha@apache.org <ma...@apache.org>>:
> Hi,
>
> How are you specifying the path for the properties file? Have you tried reading the properties by using this.getClass().getClassLoader().getResource()?
>
> Best,
> Aljoscha
>
> > On 8. Sep 2017, at 16:32, Tony Wei <tony19920430@gmail.com <ma...@gmail.com>> wrote:
> >
> > Hi,
> >
> > I put the my configuration file in `./src/main/resources/` and packed it inside my jar.
> > I want to run it on standalone cluster by using web UI to submit my job.
> > No matter which way I tried, the ParameterTool.fromPropertiesFile() couldn't find the file path, but threw `FileNotFoundException` instead.
> > Is there any best practice to deal with such problem? Thanks for your help.
> >
> > Best Regards,
> > Tony Wei
>
>
>
Re: How to user ParameterTool.fromPropertiesFile() to get resource
file inside my jar
Posted by Tony Wei <to...@gmail.com>.
Hi Aljoscha,
I found the root cause of my problem from this reference
https://stackoverflow.com/questions/18151072/cant-find-resource-file-after-exporting-to-a-runnable-jar
.
So I changed the way to use ParameterTool. I read the configurations from
InputStream, construct them as argument format and used
ParameterTool.fromArgs() to parse them with other arguments.
I'm not sure if this is a good solution. If you have any better one, please
let me know. Thanks for your help.
Best Regards,
Tony Wei
2017-09-08 23:40 GMT+08:00 Tony Wei <to...@gmail.com>:
> Hi Aljoscha,
>
> I have tried `StreamJob.class.getClassLoader().getResource("application.conf").getPath()`,
> but I got this exception.
>
> Caused by: java.io.FileNotFoundException: Properties file
> /home/tonywei/flink/file:/tmp/flink-web-24351e69-a261-45be-
> 9503-087db8155a8f/d69a3ca9-bfa0-43ef-83e8-e15f38162a87_
> quickstart-0.1.jar!/application.conf
>
> Best Regards,
> Tony Wei
>
> 2017-09-08 23:24 GMT+08:00 Aljoscha Krettek <al...@apache.org>:
>
>> Hi,
>>
>> How are you specifying the path for the properties file? Have you tried
>> reading the properties by using this.getClass().getClassLoader
>> ().getResource()?
>>
>> Best,
>> Aljoscha
>>
>> > On 8. Sep 2017, at 16:32, Tony Wei <to...@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > I put the my configuration file in `./src/main/resources/` and packed
>> it inside my jar.
>> > I want to run it on standalone cluster by using web UI to submit my job.
>> > No matter which way I tried, the ParameterTool.fromPropertiesFile()
>> couldn't find the file path, but threw `FileNotFoundException` instead.
>> > Is there any best practice to deal with such problem? Thanks for your
>> help.
>> >
>> > Best Regards,
>> > Tony Wei
>>
>>
>
Re: How to user ParameterTool.fromPropertiesFile() to get resource
file inside my jar
Posted by Tony Wei <to...@gmail.com>.
Hi Aljoscha,
I have tried
`StreamJob.class.getClassLoader().getResource("application.conf").getPath()`,
but I got this exception.
Caused by: java.io.FileNotFoundException: Properties file
/home/tonywei/flink/file:/tmp/flink-web-24351e69-a261-45be-9503-087db8155a8f/d69a3ca9-bfa0-43ef-83e8-e15f38162a87_quickstart-0.1.jar!/application.conf
Best Regards,
Tony Wei
2017-09-08 23:24 GMT+08:00 Aljoscha Krettek <al...@apache.org>:
> Hi,
>
> How are you specifying the path for the properties file? Have you tried
> reading the properties by using this.getClass().
> getClassLoader().getResource()?
>
> Best,
> Aljoscha
>
> > On 8. Sep 2017, at 16:32, Tony Wei <to...@gmail.com> wrote:
> >
> > Hi,
> >
> > I put the my configuration file in `./src/main/resources/` and packed it
> inside my jar.
> > I want to run it on standalone cluster by using web UI to submit my job.
> > No matter which way I tried, the ParameterTool.fromPropertiesFile()
> couldn't find the file path, but threw `FileNotFoundException` instead.
> > Is there any best practice to deal with such problem? Thanks for your
> help.
> >
> > Best Regards,
> > Tony Wei
>
>
Re: How to user ParameterTool.fromPropertiesFile() to get resource
file inside my jar
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
How are you specifying the path for the properties file? Have you tried reading the properties by using this.getClass().getClassLoader().getResource()?
Best,
Aljoscha
> On 8. Sep 2017, at 16:32, Tony Wei <to...@gmail.com> wrote:
>
> Hi,
>
> I put the my configuration file in `./src/main/resources/` and packed it inside my jar.
> I want to run it on standalone cluster by using web UI to submit my job.
> No matter which way I tried, the ParameterTool.fromPropertiesFile() couldn't find the file path, but threw `FileNotFoundException` instead.
> Is there any best practice to deal with such problem? Thanks for your help.
>
> Best Regards,
> Tony Wei