You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by simon peyer <si...@soom-it.ch> on 2016/05/23 13:56:59 UTC
Import Configuration File in Flink Cluster
Hi together
Currently I'm using flink on a docker cluster on AWS.
I would like to use property files with the integrated ParameterTool.fromPropertiesFile function of Flink.
Locally this version works absolutely fine:
val configuration = ParameterTool.fromPropertiesFile("src/main/resources/config.properties")
But on the cluster this didn't work, so we introduced this version, which also doesn't work:
val baseParameters = ParameterTool.fromPropertiesFile(getClass.getClassLoader.getResource("config.properties").getFile)
gives
java.io.FileNotFoundException: Properties file file:/tmp/flink-web-upload-57bcc912-bc98-4c89-b5ee-c5176155abd5/992186c1-b3c3-4342-a5c8-67af133155e4pipeline-0.1.0-all.jar!/config.properties does not exist
The property file is located in src/main/resources.
Do you have any idea of how to integrate such property files into the jar package?
-Thanks
Simon
Re: Import Configuration File in Flink Cluster
Posted by Maximilian Michels <mx...@apache.org>.
Hi Simon,
You'll have to write the property file to disk first to load it using
the ParameterTool.fromPropertiesFile method.
For example:
// copy config from Java resource to a file
File configOnDisk = new File("/path/to/config.properties");
Files.copy(getClass.getClassLoader.getResourceAsStream("config.properties"),
configOnDisk.toPath());
// load the new file
ParameterTool.fromPropertiesFile(configOnDisk);
Cheers,
Max
On Mon, May 23, 2016 at 3:56 PM, simon peyer <si...@soom-it.ch> wrote:
> Hi together
>
> Currently I'm using flink on a docker cluster on AWS.
> I would like to use property files with the integrated
> ParameterTool.fromPropertiesFile function of Flink.
>
> Locally this version works absolutely fine:
> val configuration =
> ParameterTool.fromPropertiesFile("src/main/resources/config.properties")
>
> But on the cluster this didn't work, so we introduced this version, which
> also doesn't work:
>
> val baseParameters =
> ParameterTool.fromPropertiesFile(getClass.getClassLoader.getResource("config.properties").getFile)
>
> gives
>
> java.io.FileNotFoundException: Properties file
> file:/tmp/flink-web-upload-57bcc912-bc98-4c89-b5ee-c5176155abd5/992186c1-b3c3-4342-a5c8-67af133155e4pipeline-0.1.0-all.jar!/config.properties
> does not exist
>
> The property file is located in src/main/resources.
> Do you have any idea of how to integrate such property files into the jar
> package?
>
> -Thanks
> Simon
Re: Import Configuration File in Flink Cluster
Posted by simon peyer <si...@soom-it.ch>.
Hi
I just want to let you know, that I did implement a version for wrapping properties in the cluster:
object PropertiesWrapper {
def getFromConfig(configFile: String): ParameterTool = {
val prop: Properties = new Properties()
var input: InputStream = null
try {
input = PropertiesWrapper.getClass.getResourceAsStream("/" + configFile);
// load a properties file
prop.load(input)
} finally {
if (input != null) {
input.close();
}
}
ParameterTool.fromMap(prop.asInstanceOf[Map[String,String]])
}
def mergeArgsWithProperties(configuration:Properties, args: Array[String]):Properties = {
configuration
}
}
Have a nice day
--Simon
> On 24 May 2016, at 01:51, Bajaj, Abhinav <ab...@here.com> wrote:
>
> I was gonna post the exact question and noticed this thread.
>
> It will be great if we can have a method in parameter tool to load from resources.
>
> Thanks Simon :)
>
> Abhinav Bajaj
> Senior Engineer
> HERE Predictive Analytics
> Office: +12062092767
> Mobile: +17083299516
> HERE Seattle
> 701 Pike Street, #2000, Seattle, WA 98101, USA
> 47° 36' 41" N. 122° 19' 57" W
> HERE Maps
>
>
>
>
>
>
>
>
>
>
>
> On 5/23/16, 8:54 AM, "simon peyer" <si...@soom-it.ch> wrote:
>
>> Hi Max
>>
>> Thanks a lot for your helpful answer.
>> It now works on the cluster.
>> It would be great to have a method for loading from resources.
>>
>> -Cheers
>> Simon
>>
>>
>>> On 23 May 2016, at 17:52, Maximilian Michels <mx...@apache.org> wrote:
>>>
>>> Hi Simon,
>>>
>>> AFAIK this is the way to go. We could add a method to the
>>> ParameterTool which loads from a resource to make it more convenient.
>>>
>>> Cheers,
>>> Max
>>>
>>> On Mon, May 23, 2016 at 4:42 PM, simon peyer <si...@soom-it.ch> wrote:
>>>> Hi
>>>>
>>>> @Max
>>>> So for each file in the src/main/resources folder, I first have to create a
>>>> new file, copy the file from the resources folder to this new file and then
>>>> I'm able to parse it?
>>>>
>>>> @Stefano
>>>> I think the files in src/main/resources are integrated automatically right?
>>>> Or am I missing something?
>>>>
>>>> Cheers
>>>> Simon
>>>>
>>>>
>>>> On 23 May 2016, at 16:30, Stefano Baghino <st...@radicalbit.io>
>>>> wrote:
>>>>
>>>> Are you using Maven to package your project? I believe the resources
>>>> plugin[1] can suit your needs.
>>>>
>>>> [1]:
>>>> http://maven.apache.org/plugins/maven-resources-plugin/examples/include-exclude.html
>>>>
>>>>
>>>> On 23 May 2016, at 16:28, Maximilian Michels <mx...@apache.org> wrote:
>>>>
>>>> Hi Simon,
>>>>
>>>> You'll have to write the property file to disk first to load it using
>>>> the ParameterTool.fromPropertiesFile method.
>>>>
>>>> For example:
>>>>
>>>> // copy config from Java resource to a file
>>>> File configOnDisk = new File("/path/to/config.properties");
>>>> Files.copy(getClass.getClassLoader.getResourceAsStream("config.properties"),
>>>> configOnDisk.toPath());
>>>> // load the new file
>>>> ParameterTool.fromPropertiesFile(configOnDisk);
>>>>
>>>>
>>>> Cheers,
>>>> Max
>>>>
>>>>
>>>>
>>>> On Mon, May 23, 2016 at 3:56 PM, simon peyer <si...@soom-it.ch> wrote:
>>>>>
>>>>> Hi together
>>>>>
>>>>> Currently I'm using flink on a docker cluster on AWS.
>>>>> I would like to use property files with the integrated
>>>>> ParameterTool.fromPropertiesFile function of Flink.
>>>>>
>>>>> Locally this version works absolutely fine:
>>>>> val configuration =
>>>>> ParameterTool.fromPropertiesFile("src/main/resources/config.properties")
>>>>>
>>>>> But on the cluster this didn't work, so we introduced this version, which
>>>>> also doesn't work:
>>>>>
>>>>> val baseParameters =
>>>>> ParameterTool.fromPropertiesFile(getClass.getClassLoader.getResource("config.properties").getFile)
>>>>>
>>>>> gives
>>>>>
>>>>> java.io.FileNotFoundException: Properties file
>>>>> file:/tmp/flink-web-upload-57bcc912-bc98-4c89-b5ee-c5176155abd5/992186c1-b3c3-4342-a5c8-67af133155e4pipeline-0.1.0-all.jar!/config.properties
>>>>> does not exist
>>>>>
>>>>> The property file is located in src/main/resources.
>>>>> Do you have any idea of how to integrate such property files into the jar
>>>>> package?
>>>>>
>>>>> -Thanks
>>>>> Simon
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> BR,
>>>> Stefano Baghino
>>>>
>>>> Software Engineer @ Radicalbit
>>>>
>>>>
>>
Re: Import Configuration File in Flink Cluster
Posted by "Bajaj, Abhinav" <ab...@here.com>.
I was gonna post the exact question and noticed this thread.
It will be great if we can have a method in parameter tool to load from resources.
Thanks Simon :)
Abhinav Bajaj
Senior Engineer
HERE Predictive Analytics
Office: +12062092767
Mobile: +17083299516
HERE Seattle
701 Pike Street, #2000, Seattle, WA 98101, USA
47° 36' 41" N. 122° 19' 57" W
HERE Maps
On 5/23/16, 8:54 AM, "simon peyer" <si...@soom-it.ch> wrote:
>Hi Max
>
>Thanks a lot for your helpful answer.
>It now works on the cluster.
>It would be great to have a method for loading from resources.
>
>-Cheers
>Simon
>
>
>> On 23 May 2016, at 17:52, Maximilian Michels <mx...@apache.org> wrote:
>>
>> Hi Simon,
>>
>> AFAIK this is the way to go. We could add a method to the
>> ParameterTool which loads from a resource to make it more convenient.
>>
>> Cheers,
>> Max
>>
>> On Mon, May 23, 2016 at 4:42 PM, simon peyer <si...@soom-it.ch> wrote:
>>> Hi
>>>
>>> @Max
>>> So for each file in the src/main/resources folder, I first have to create a
>>> new file, copy the file from the resources folder to this new file and then
>>> I'm able to parse it?
>>>
>>> @Stefano
>>> I think the files in src/main/resources are integrated automatically right?
>>> Or am I missing something?
>>>
>>> Cheers
>>> Simon
>>>
>>>
>>> On 23 May 2016, at 16:30, Stefano Baghino <st...@radicalbit.io>
>>> wrote:
>>>
>>> Are you using Maven to package your project? I believe the resources
>>> plugin[1] can suit your needs.
>>>
>>> [1]:
>>> http://maven.apache.org/plugins/maven-resources-plugin/examples/include-exclude.html
>>>
>>>
>>> On 23 May 2016, at 16:28, Maximilian Michels <mx...@apache.org> wrote:
>>>
>>> Hi Simon,
>>>
>>> You'll have to write the property file to disk first to load it using
>>> the ParameterTool.fromPropertiesFile method.
>>>
>>> For example:
>>>
>>> // copy config from Java resource to a file
>>> File configOnDisk = new File("/path/to/config.properties");
>>> Files.copy(getClass.getClassLoader.getResourceAsStream("config.properties"),
>>> configOnDisk.toPath());
>>> // load the new file
>>> ParameterTool.fromPropertiesFile(configOnDisk);
>>>
>>>
>>> Cheers,
>>> Max
>>>
>>>
>>>
>>> On Mon, May 23, 2016 at 3:56 PM, simon peyer <si...@soom-it.ch> wrote:
>>>>
>>>> Hi together
>>>>
>>>> Currently I'm using flink on a docker cluster on AWS.
>>>> I would like to use property files with the integrated
>>>> ParameterTool.fromPropertiesFile function of Flink.
>>>>
>>>> Locally this version works absolutely fine:
>>>> val configuration =
>>>> ParameterTool.fromPropertiesFile("src/main/resources/config.properties")
>>>>
>>>> But on the cluster this didn't work, so we introduced this version, which
>>>> also doesn't work:
>>>>
>>>> val baseParameters =
>>>> ParameterTool.fromPropertiesFile(getClass.getClassLoader.getResource("config.properties").getFile)
>>>>
>>>> gives
>>>>
>>>> java.io.FileNotFoundException: Properties file
>>>> file:/tmp/flink-web-upload-57bcc912-bc98-4c89-b5ee-c5176155abd5/992186c1-b3c3-4342-a5c8-67af133155e4pipeline-0.1.0-all.jar!/config.properties
>>>> does not exist
>>>>
>>>> The property file is located in src/main/resources.
>>>> Do you have any idea of how to integrate such property files into the jar
>>>> package?
>>>>
>>>> -Thanks
>>>> Simon
>>>
>>>
>>>
>>>
>>> --
>>> BR,
>>> Stefano Baghino
>>>
>>> Software Engineer @ Radicalbit
>>>
>>>
>
Re: Import Configuration File in Flink Cluster
Posted by simon peyer <si...@soom-it.ch>.
Hi Max
Thanks a lot for your helpful answer.
It now works on the cluster.
It would be great to have a method for loading from resources.
-Cheers
Simon
> On 23 May 2016, at 17:52, Maximilian Michels <mx...@apache.org> wrote:
>
> Hi Simon,
>
> AFAIK this is the way to go. We could add a method to the
> ParameterTool which loads from a resource to make it more convenient.
>
> Cheers,
> Max
>
> On Mon, May 23, 2016 at 4:42 PM, simon peyer <si...@soom-it.ch> wrote:
>> Hi
>>
>> @Max
>> So for each file in the src/main/resources folder, I first have to create a
>> new file, copy the file from the resources folder to this new file and then
>> I'm able to parse it?
>>
>> @Stefano
>> I think the files in src/main/resources are integrated automatically right?
>> Or am I missing something?
>>
>> Cheers
>> Simon
>>
>>
>> On 23 May 2016, at 16:30, Stefano Baghino <st...@radicalbit.io>
>> wrote:
>>
>> Are you using Maven to package your project? I believe the resources
>> plugin[1] can suit your needs.
>>
>> [1]:
>> http://maven.apache.org/plugins/maven-resources-plugin/examples/include-exclude.html
>>
>>
>> On 23 May 2016, at 16:28, Maximilian Michels <mx...@apache.org> wrote:
>>
>> Hi Simon,
>>
>> You'll have to write the property file to disk first to load it using
>> the ParameterTool.fromPropertiesFile method.
>>
>> For example:
>>
>> // copy config from Java resource to a file
>> File configOnDisk = new File("/path/to/config.properties");
>> Files.copy(getClass.getClassLoader.getResourceAsStream("config.properties"),
>> configOnDisk.toPath());
>> // load the new file
>> ParameterTool.fromPropertiesFile(configOnDisk);
>>
>>
>> Cheers,
>> Max
>>
>>
>>
>> On Mon, May 23, 2016 at 3:56 PM, simon peyer <si...@soom-it.ch> wrote:
>>>
>>> Hi together
>>>
>>> Currently I'm using flink on a docker cluster on AWS.
>>> I would like to use property files with the integrated
>>> ParameterTool.fromPropertiesFile function of Flink.
>>>
>>> Locally this version works absolutely fine:
>>> val configuration =
>>> ParameterTool.fromPropertiesFile("src/main/resources/config.properties")
>>>
>>> But on the cluster this didn't work, so we introduced this version, which
>>> also doesn't work:
>>>
>>> val baseParameters =
>>> ParameterTool.fromPropertiesFile(getClass.getClassLoader.getResource("config.properties").getFile)
>>>
>>> gives
>>>
>>> java.io.FileNotFoundException: Properties file
>>> file:/tmp/flink-web-upload-57bcc912-bc98-4c89-b5ee-c5176155abd5/992186c1-b3c3-4342-a5c8-67af133155e4pipeline-0.1.0-all.jar!/config.properties
>>> does not exist
>>>
>>> The property file is located in src/main/resources.
>>> Do you have any idea of how to integrate such property files into the jar
>>> package?
>>>
>>> -Thanks
>>> Simon
>>
>>
>>
>>
>> --
>> BR,
>> Stefano Baghino
>>
>> Software Engineer @ Radicalbit
>>
>>
Re: Import Configuration File in Flink Cluster
Posted by Maximilian Michels <mx...@apache.org>.
Hi Simon,
AFAIK this is the way to go. We could add a method to the
ParameterTool which loads from a resource to make it more convenient.
Cheers,
Max
On Mon, May 23, 2016 at 4:42 PM, simon peyer <si...@soom-it.ch> wrote:
> Hi
>
> @Max
> So for each file in the src/main/resources folder, I first have to create a
> new file, copy the file from the resources folder to this new file and then
> I'm able to parse it?
>
> @Stefano
> I think the files in src/main/resources are integrated automatically right?
> Or am I missing something?
>
> Cheers
> Simon
>
>
> On 23 May 2016, at 16:30, Stefano Baghino <st...@radicalbit.io>
> wrote:
>
> Are you using Maven to package your project? I believe the resources
> plugin[1] can suit your needs.
>
> [1]:
> http://maven.apache.org/plugins/maven-resources-plugin/examples/include-exclude.html
>
>
> On 23 May 2016, at 16:28, Maximilian Michels <mx...@apache.org> wrote:
>
> Hi Simon,
>
> You'll have to write the property file to disk first to load it using
> the ParameterTool.fromPropertiesFile method.
>
> For example:
>
> // copy config from Java resource to a file
> File configOnDisk = new File("/path/to/config.properties");
> Files.copy(getClass.getClassLoader.getResourceAsStream("config.properties"),
> configOnDisk.toPath());
> // load the new file
> ParameterTool.fromPropertiesFile(configOnDisk);
>
>
> Cheers,
> Max
>
>
>
> On Mon, May 23, 2016 at 3:56 PM, simon peyer <si...@soom-it.ch> wrote:
>>
>> Hi together
>>
>> Currently I'm using flink on a docker cluster on AWS.
>> I would like to use property files with the integrated
>> ParameterTool.fromPropertiesFile function of Flink.
>>
>> Locally this version works absolutely fine:
>> val configuration =
>> ParameterTool.fromPropertiesFile("src/main/resources/config.properties")
>>
>> But on the cluster this didn't work, so we introduced this version, which
>> also doesn't work:
>>
>> val baseParameters =
>> ParameterTool.fromPropertiesFile(getClass.getClassLoader.getResource("config.properties").getFile)
>>
>> gives
>>
>> java.io.FileNotFoundException: Properties file
>> file:/tmp/flink-web-upload-57bcc912-bc98-4c89-b5ee-c5176155abd5/992186c1-b3c3-4342-a5c8-67af133155e4pipeline-0.1.0-all.jar!/config.properties
>> does not exist
>>
>> The property file is located in src/main/resources.
>> Do you have any idea of how to integrate such property files into the jar
>> package?
>>
>> -Thanks
>> Simon
>
>
>
>
> --
> BR,
> Stefano Baghino
>
> Software Engineer @ Radicalbit
>
>
Re: Import Configuration File in Flink Cluster
Posted by simon peyer <si...@soom-it.ch>.
Hi
@Max
So for each file in the src/main/resources folder, I first have to create a new file, copy the file from the resources folder to this new file and then I'm able to parse it?
@Stefano
I think the files in src/main/resources are integrated automatically right? Or am I missing something?
Cheers
Simon
> On 23 May 2016, at 16:30, Stefano Baghino <st...@radicalbit.io> wrote:
>
> Are you using Maven to package your project? I believe the resources plugin[1] can suit your needs.
>
> [1]: http://maven.apache.org/plugins/maven-resources-plugin/examples/include-exclude.html <http://maven.apache.org/plugins/maven-resources-plugin/examples/include-exclude.html>
On 23 May 2016, at 16:28, Maximilian Michels <mx...@apache.org> wrote:
Hi Simon,
You'll have to write the property file to disk first to load it using
the ParameterTool.fromPropertiesFile method.
For example:
// copy config from Java resource to a file
File configOnDisk = new File("/path/to/config.properties");
Files.copy(getClass.getClassLoader.getResourceAsStream("config.properties"),
configOnDisk.toPath());
// load the new file
ParameterTool.fromPropertiesFile(configOnDisk);
Cheers,
Max
>
> On Mon, May 23, 2016 at 3:56 PM, simon peyer <simon.peyer@soom-it.ch <ma...@soom-it.ch>> wrote:
> Hi together
>
> Currently I'm using flink on a docker cluster on AWS.
> I would like to use property files with the integrated ParameterTool.fromPropertiesFile function of Flink.
>
> Locally this version works absolutely fine:
> val configuration = ParameterTool.fromPropertiesFile("src/main/resources/config.properties")
>
> But on the cluster this didn't work, so we introduced this version, which also doesn't work:
>
> val baseParameters = ParameterTool.fromPropertiesFile(getClass.getClassLoader.getResource("config.properties").getFile)
>
> gives
> java.io.FileNotFoundException: Properties file file:/tmp/flink-web-upload-57bcc912-bc98-4c89-b5ee-c5176155abd5/992186c1-b3c3-4342-a5c8-67af133155e4pipeline-0.1.0-all.jar!/config.properties does not exist
> The property file is located in src/main/resources.
> Do you have any idea of how to integrate such property files into the jar package?
>
> -Thanks
> Simon
>
>
>
> --
> BR,
> Stefano Baghino
>
> Software Engineer @ Radicalbit
Re: Import Configuration File in Flink Cluster
Posted by Stefano Baghino <st...@radicalbit.io>.
Are you using Maven to package your project? I believe the resources
plugin[1] can suit your needs.
[1]:
http://maven.apache.org/plugins/maven-resources-plugin/examples/include-exclude.html
On Mon, May 23, 2016 at 3:56 PM, simon peyer <si...@soom-it.ch> wrote:
> Hi together
>
> Currently I'm using flink on a docker cluster on AWS.
> I would like to use property files with the integrated
> ParameterTool.fromPropertiesFile function of Flink.
>
> Locally this version works absolutely fine:
> val configuration = ParameterTool.fromPropertiesFile(
> "src/main/resources/config.properties")
>
> But on the cluster this didn't work, so we introduced this version, which
> also doesn't work:
>
> val baseParameters =
> ParameterTool.fromPropertiesFile(getClass.getClassLoader.getResource(
> "config.properties").getFile)
>
> gives
>
> java.io.FileNotFoundException: Properties file file:/tmp/flink-web-upload-57bcc912-bc98-4c89-b5ee-c5176155abd5/992186c1-b3c3-4342-a5c8-67af133155e4pipeline-0.1.0-all.jar!/config.properties does not exist
>
> The property file is located in src/main/resources.
> Do you have any idea of how to integrate such property files into the jar
> package?
>
> -Thanks
> Simon
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit