You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Flavio Pompermaier <po...@okkam.it> on 2015/10/06 10:36:42 UTC
ExecutionEnvironment setConfiguration API
Hi to all,
today my code doesn't compile anymore because ExecutionEnvironment doesn't
have setConfiguration() anymore..how can I set the following parameters in
my unit tests?
- ConfigConstants.TASK_MANAGER_TMP_DIR_KEY
- ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY
- ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY
Best,
Flavio
Re: ExecutionEnvironment setConfiguration API
Posted by Fabian Hueske <fh...@gmail.com>.
I think it's not a nice solution to check for the type of the returned
execution environment to determine whether it is a local or a remote
execution environment.
Wouldn't it be better to add a method isLocal() to ExecutionEnvironment?
Cheers, Fabian
2015-10-14 19:14 GMT+02:00 Flavio Pompermaier <po...@okkam.it>:
> of course,I tried to configure the task slot during a debug test and I
> forgot to remove it..
> Just for curiosity, is there any good reason why you've changed the
> default parallellelism that way?and moreover, is it the only unexpected
> changed behaviour wrt the previous API version?
> On 14 Oct 2015 18:35, "Stephan Ewen" <se...@apache.org> wrote:
>
>> Hi Flavio!
>>
>> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment()
>> by default picks up the number of cores as the parallelism, while the
>> manual environments do not do that.
>> You can still set it manually set the parallelism
>> "env.setParallelism(Runtime.getRuntime().availableProcessors());"
>>
>> I would not configure the slots for the local execution, they should be
>> automatically configured based on the max parallelism.
>>
>> Greetings,
>> Stephan
>>
>>
>> On Wed, Oct 14, 2015 at 3:36 PM, Flavio Pompermaier <pompermaier@okkam.it
>> > wrote:
>>
>>> Hi Fabian and Stephan, back to work :)
>>>
>>> I finally managed to find the problem of the parallelism encountered by
>>> my colleague!
>>> Basically that was introduced by this API change. Before I was using
>>> env.setConfiguration() to merge the default params with some custom ones.
>>> Now, after the API change I was using the following code:
>>>
>>> ExecutionEnvironment env =
>>> ExecutionEnvironment.getExecutionEnvironment();
>>> if (env instanceof LocalEnvironment) {
>>> Configuration c = new Configuration();
>>> c.setString(ConfigConstants.TASK_MANAGER_TMP_DIR_KEY,
>>> FLINK_TEST_TMP_DIR);
>>>
>>> c.setString(ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY,FLINK_TEST_TMP_DIR);
>>> c.setLong(ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY, 2048 *
>>> 2);
>>> c.setLong(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, 4);
>>> env = ExecutionEnvironment.createLocalEnvironment(c);
>>> }
>>>
>>> However, the first env and the reassigned one doesn't behave in the same
>>> manner.
>>> If I don't reassign env I have parallelism=8, otherwise it's 1 :(
>>> Am I using the wrong APIs or the execution environment doesn't allow now
>>> to configure such parameters anymore?
>>>
>>> Thanks in advance,
>>> Flavio
>>>
>>>
>>> On Tue, Oct 6, 2015 at 11:31 AM, Flavio Pompermaier <
>>> pompermaier@okkam.it> wrote:
>>>
>>>> That makes sense: what can be configured should be differentiated
>>>> between local and remote envs (obviously this is a minor issue/improvement)
>>>>
>>>> Thanks again,
>>>> Flavio
>>>>
>>>> On Tue, Oct 6, 2015 at 11:25 AM, Stephan Ewen <se...@apache.org> wrote:
>>>>
>>>>> We can think about that, but I think it may be quite confusing. The
>>>>> configurations actually mean something different for local and remote
>>>>> environments:
>>>>>
>>>>> - For the local environment, the config basically describes the
>>>>> entire Flink cluster setup (for the local execution cluster in the
>>>>> background)
>>>>> - For the remote environment, the config describes the parameters
>>>>> for the client that connects to the cluster (akka paramters, optimizer
>>>>> parameters, ...), but not parameters of the cluster itself (like
>>>>> taskmanager slots and memory).
>>>>>
>>>>> Greetings,
>>>>> Stephan
>>>>>
>>>>>
>>>>> On Tue, Oct 6, 2015 at 10:56 AM, Flavio Pompermaier <
>>>>> pompermaier@okkam.it> wrote:
>>>>>
>>>>>> However it could be a good idea to overload also
>>>>>> the getExecutionEnvironment() to be able to pass a custom
>>>>>> configuration..what do you think?
>>>>>> Otherwise I have to know a priori if I'm working in a local
>>>>>> deployment or in a remote one, or check if getExecutionEnvironment()
>>>>>> returned an instance of LocalEnvironment/RemoteEnvironment..
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 6, 2015 at 10:53 AM, Flavio Pompermaier <
>>>>>> pompermaier@okkam.it> wrote:
>>>>>>
>>>>>>> Yes Stephan!
>>>>>>> I usually work with the master version, at least in development ;)
>>>>>>> Thanks for the quick support!
>>>>>>>
>>>>>>> Best,
>>>>>>> Flavio
>>>>>>>
>>>>>>> On Tue, Oct 6, 2015 at 10:48 AM, Stephan Ewen <se...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> Are you on the SNAPSHOT master version?
>>>>>>>>
>>>>>>>> You can pass the configuration to the constructor of the execution
>>>>>>>> environment, or create one via
>>>>>>>> ExecutionEnvironment.createLocalEnvironment(config) or via
>>>>>>>> createRemoteEnvironment(host, port, configuration, jarFiles);
>>>>>>>>
>>>>>>>> The change of the signature was part of an API cleanup for the next
>>>>>>>> release. Sorry for the inconvenience...
>>>>>>>>
>>>>>>>> Stephan
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Oct 6, 2015 at 10:36 AM, Flavio Pompermaier <
>>>>>>>> pompermaier@okkam.it> wrote:
>>>>>>>>
>>>>>>>>> Hi to all,
>>>>>>>>> today my code doesn't compile anymore because ExecutionEnvironment
>>>>>>>>> doesn't have setConfiguration() anymore..how can I set the following
>>>>>>>>> parameters in my unit tests?
>>>>>>>>>
>>>>>>>>> - ConfigConstants.TASK_MANAGER_TMP_DIR_KEY
>>>>>>>>> - ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY
>>>>>>>>> - ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Flavio
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Re: ExecutionEnvironment setConfiguration API
Posted by Flavio Pompermaier <po...@okkam.it>.
of course,I tried to configure the task slot during a debug test and I
forgot to remove it..
Just for curiosity, is there any good reason why you've changed the default
parallellelism that way?and moreover, is it the only unexpected changed
behaviour wrt the previous API version?
On 14 Oct 2015 18:35, "Stephan Ewen" <se...@apache.org> wrote:
> Hi Flavio!
>
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment()
> by default picks up the number of cores as the parallelism, while the
> manual environments do not do that.
> You can still set it manually set the parallelism
> "env.setParallelism(Runtime.getRuntime().availableProcessors());"
>
> I would not configure the slots for the local execution, they should be
> automatically configured based on the max parallelism.
>
> Greetings,
> Stephan
>
>
> On Wed, Oct 14, 2015 at 3:36 PM, Flavio Pompermaier <po...@okkam.it>
> wrote:
>
>> Hi Fabian and Stephan, back to work :)
>>
>> I finally managed to find the problem of the parallelism encountered by
>> my colleague!
>> Basically that was introduced by this API change. Before I was using
>> env.setConfiguration() to merge the default params with some custom ones.
>> Now, after the API change I was using the following code:
>>
>> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
>> if (env instanceof LocalEnvironment) {
>> Configuration c = new Configuration();
>> c.setString(ConfigConstants.TASK_MANAGER_TMP_DIR_KEY, FLINK_TEST_TMP_DIR);
>>
>> c.setString(ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY,FLINK_TEST_TMP_DIR);
>> c.setLong(ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY, 2048 * 2);
>> c.setLong(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, 4);
>> env = ExecutionEnvironment.createLocalEnvironment(c);
>> }
>>
>> However, the first env and the reassigned one doesn't behave in the same
>> manner.
>> If I don't reassign env I have parallelism=8, otherwise it's 1 :(
>> Am I using the wrong APIs or the execution environment doesn't allow now
>> to configure such parameters anymore?
>>
>> Thanks in advance,
>> Flavio
>>
>>
>> On Tue, Oct 6, 2015 at 11:31 AM, Flavio Pompermaier <pompermaier@okkam.it
>> > wrote:
>>
>>> That makes sense: what can be configured should be differentiated
>>> between local and remote envs (obviously this is a minor issue/improvement)
>>>
>>> Thanks again,
>>> Flavio
>>>
>>> On Tue, Oct 6, 2015 at 11:25 AM, Stephan Ewen <se...@apache.org> wrote:
>>>
>>>> We can think about that, but I think it may be quite confusing. The
>>>> configurations actually mean something different for local and remote
>>>> environments:
>>>>
>>>> - For the local environment, the config basically describes the
>>>> entire Flink cluster setup (for the local execution cluster in the
>>>> background)
>>>> - For the remote environment, the config describes the parameters for
>>>> the client that connects to the cluster (akka paramters, optimizer
>>>> parameters, ...), but not parameters of the cluster itself (like
>>>> taskmanager slots and memory).
>>>>
>>>> Greetings,
>>>> Stephan
>>>>
>>>>
>>>> On Tue, Oct 6, 2015 at 10:56 AM, Flavio Pompermaier <
>>>> pompermaier@okkam.it> wrote:
>>>>
>>>>> However it could be a good idea to overload also
>>>>> the getExecutionEnvironment() to be able to pass a custom
>>>>> configuration..what do you think?
>>>>> Otherwise I have to know a priori if I'm working in a local deployment
>>>>> or in a remote one, or check if getExecutionEnvironment() returned an
>>>>> instance of LocalEnvironment/RemoteEnvironment..
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Oct 6, 2015 at 10:53 AM, Flavio Pompermaier <
>>>>> pompermaier@okkam.it> wrote:
>>>>>
>>>>>> Yes Stephan!
>>>>>> I usually work with the master version, at least in development ;)
>>>>>> Thanks for the quick support!
>>>>>>
>>>>>> Best,
>>>>>> Flavio
>>>>>>
>>>>>> On Tue, Oct 6, 2015 at 10:48 AM, Stephan Ewen <se...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi!
>>>>>>>
>>>>>>> Are you on the SNAPSHOT master version?
>>>>>>>
>>>>>>> You can pass the configuration to the constructor of the execution
>>>>>>> environment, or create one via
>>>>>>> ExecutionEnvironment.createLocalEnvironment(config) or via
>>>>>>> createRemoteEnvironment(host, port, configuration, jarFiles);
>>>>>>>
>>>>>>> The change of the signature was part of an API cleanup for the next
>>>>>>> release. Sorry for the inconvenience...
>>>>>>>
>>>>>>> Stephan
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Oct 6, 2015 at 10:36 AM, Flavio Pompermaier <
>>>>>>> pompermaier@okkam.it> wrote:
>>>>>>>
>>>>>>>> Hi to all,
>>>>>>>>
>>>>>>>> today my code doesn't compile anymore because ExecutionEnvironment
>>>>>>>> doesn't have setConfiguration() anymore..how can I set the following
>>>>>>>> parameters in my unit tests?
>>>>>>>>
>>>>>>>> - ConfigConstants.TASK_MANAGER_TMP_DIR_KEY
>>>>>>>> - ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY
>>>>>>>> - ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Flavio
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: ExecutionEnvironment setConfiguration API
Posted by Stephan Ewen <se...@apache.org>.
Hi Flavio!
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment()
by default picks up the number of cores as the parallelism, while the
manual environments do not do that.
You can still set it manually set the parallelism
"env.setParallelism(Runtime.getRuntime().availableProcessors());"
I would not configure the slots for the local execution, they should be
automatically configured based on the max parallelism.
Greetings,
Stephan
On Wed, Oct 14, 2015 at 3:36 PM, Flavio Pompermaier <po...@okkam.it>
wrote:
> Hi Fabian and Stephan, back to work :)
>
> I finally managed to find the problem of the parallelism encountered by my
> colleague!
> Basically that was introduced by this API change. Before I was using
> env.setConfiguration() to merge the default params with some custom ones.
> Now, after the API change I was using the following code:
>
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
> if (env instanceof LocalEnvironment) {
> Configuration c = new Configuration();
> c.setString(ConfigConstants.TASK_MANAGER_TMP_DIR_KEY, FLINK_TEST_TMP_DIR);
> c.setString(ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY,FLINK_TEST_TMP_DIR);
> c.setLong(ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY, 2048 * 2);
> c.setLong(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, 4);
> env = ExecutionEnvironment.createLocalEnvironment(c);
> }
>
> However, the first env and the reassigned one doesn't behave in the same
> manner.
> If I don't reassign env I have parallelism=8, otherwise it's 1 :(
> Am I using the wrong APIs or the execution environment doesn't allow now
> to configure such parameters anymore?
>
> Thanks in advance,
> Flavio
>
>
> On Tue, Oct 6, 2015 at 11:31 AM, Flavio Pompermaier <po...@okkam.it>
> wrote:
>
>> That makes sense: what can be configured should be differentiated between
>> local and remote envs (obviously this is a minor issue/improvement)
>>
>> Thanks again,
>> Flavio
>>
>> On Tue, Oct 6, 2015 at 11:25 AM, Stephan Ewen <se...@apache.org> wrote:
>>
>>> We can think about that, but I think it may be quite confusing. The
>>> configurations actually mean something different for local and remote
>>> environments:
>>>
>>> - For the local environment, the config basically describes the entire
>>> Flink cluster setup (for the local execution cluster in the background)
>>> - For the remote environment, the config describes the parameters for
>>> the client that connects to the cluster (akka paramters, optimizer
>>> parameters, ...), but not parameters of the cluster itself (like
>>> taskmanager slots and memory).
>>>
>>> Greetings,
>>> Stephan
>>>
>>>
>>> On Tue, Oct 6, 2015 at 10:56 AM, Flavio Pompermaier <
>>> pompermaier@okkam.it> wrote:
>>>
>>>> However it could be a good idea to overload also
>>>> the getExecutionEnvironment() to be able to pass a custom
>>>> configuration..what do you think?
>>>> Otherwise I have to know a priori if I'm working in a local deployment
>>>> or in a remote one, or check if getExecutionEnvironment() returned an
>>>> instance of LocalEnvironment/RemoteEnvironment..
>>>>
>>>>
>>>>
>>>> On Tue, Oct 6, 2015 at 10:53 AM, Flavio Pompermaier <
>>>> pompermaier@okkam.it> wrote:
>>>>
>>>>> Yes Stephan!
>>>>> I usually work with the master version, at least in development ;)
>>>>> Thanks for the quick support!
>>>>>
>>>>> Best,
>>>>> Flavio
>>>>>
>>>>> On Tue, Oct 6, 2015 at 10:48 AM, Stephan Ewen <se...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> Are you on the SNAPSHOT master version?
>>>>>>
>>>>>> You can pass the configuration to the constructor of the execution
>>>>>> environment, or create one via
>>>>>> ExecutionEnvironment.createLocalEnvironment(config) or via
>>>>>> createRemoteEnvironment(host, port, configuration, jarFiles);
>>>>>>
>>>>>> The change of the signature was part of an API cleanup for the next
>>>>>> release. Sorry for the inconvenience...
>>>>>>
>>>>>> Stephan
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 6, 2015 at 10:36 AM, Flavio Pompermaier <
>>>>>> pompermaier@okkam.it> wrote:
>>>>>>
>>>>>>> Hi to all,
>>>>>>>
>>>>>>> today my code doesn't compile anymore because ExecutionEnvironment
>>>>>>> doesn't have setConfiguration() anymore..how can I set the following
>>>>>>> parameters in my unit tests?
>>>>>>>
>>>>>>> - ConfigConstants.TASK_MANAGER_TMP_DIR_KEY
>>>>>>> - ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY
>>>>>>> - ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY
>>>>>>>
>>>>>>> Best,
>>>>>>> Flavio
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: ExecutionEnvironment setConfiguration API
Posted by Flavio Pompermaier <po...@okkam.it>.
Hi Fabian and Stephan, back to work :)
I finally managed to find the problem of the parallelism encountered by my
colleague!
Basically that was introduced by this API change. Before I was using
env.setConfiguration() to merge the default params with some custom ones.
Now, after the API change I was using the following code:
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
if (env instanceof LocalEnvironment) {
Configuration c = new Configuration();
c.setString(ConfigConstants.TASK_MANAGER_TMP_DIR_KEY, FLINK_TEST_TMP_DIR);
c.setString(ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY,FLINK_TEST_TMP_DIR);
c.setLong(ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY, 2048 * 2);
c.setLong(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, 4);
env = ExecutionEnvironment.createLocalEnvironment(c);
}
However, the first env and the reassigned one doesn't behave in the same
manner.
If I don't reassign env I have parallelism=8, otherwise it's 1 :(
Am I using the wrong APIs or the execution environment doesn't allow now to
configure such parameters anymore?
Thanks in advance,
Flavio
On Tue, Oct 6, 2015 at 11:31 AM, Flavio Pompermaier <po...@okkam.it>
wrote:
> That makes sense: what can be configured should be differentiated between
> local and remote envs (obviously this is a minor issue/improvement)
>
> Thanks again,
> Flavio
>
> On Tue, Oct 6, 2015 at 11:25 AM, Stephan Ewen <se...@apache.org> wrote:
>
>> We can think about that, but I think it may be quite confusing. The
>> configurations actually mean something different for local and remote
>> environments:
>>
>> - For the local environment, the config basically describes the entire
>> Flink cluster setup (for the local execution cluster in the background)
>> - For the remote environment, the config describes the parameters for
>> the client that connects to the cluster (akka paramters, optimizer
>> parameters, ...), but not parameters of the cluster itself (like
>> taskmanager slots and memory).
>>
>> Greetings,
>> Stephan
>>
>>
>> On Tue, Oct 6, 2015 at 10:56 AM, Flavio Pompermaier <pompermaier@okkam.it
>> > wrote:
>>
>>> However it could be a good idea to overload also
>>> the getExecutionEnvironment() to be able to pass a custom
>>> configuration..what do you think?
>>> Otherwise I have to know a priori if I'm working in a local deployment
>>> or in a remote one, or check if getExecutionEnvironment() returned an
>>> instance of LocalEnvironment/RemoteEnvironment..
>>>
>>>
>>>
>>> On Tue, Oct 6, 2015 at 10:53 AM, Flavio Pompermaier <
>>> pompermaier@okkam.it> wrote:
>>>
>>>> Yes Stephan!
>>>> I usually work with the master version, at least in development ;)
>>>> Thanks for the quick support!
>>>>
>>>> Best,
>>>> Flavio
>>>>
>>>> On Tue, Oct 6, 2015 at 10:48 AM, Stephan Ewen <se...@apache.org> wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>> Are you on the SNAPSHOT master version?
>>>>>
>>>>> You can pass the configuration to the constructor of the execution
>>>>> environment, or create one via
>>>>> ExecutionEnvironment.createLocalEnvironment(config) or via
>>>>> createRemoteEnvironment(host, port, configuration, jarFiles);
>>>>>
>>>>> The change of the signature was part of an API cleanup for the next
>>>>> release. Sorry for the inconvenience...
>>>>>
>>>>> Stephan
>>>>>
>>>>>
>>>>> On Tue, Oct 6, 2015 at 10:36 AM, Flavio Pompermaier <
>>>>> pompermaier@okkam.it> wrote:
>>>>>
>>>>>> Hi to all,
>>>>>>
>>>>>> today my code doesn't compile anymore because ExecutionEnvironment
>>>>>> doesn't have setConfiguration() anymore..how can I set the following
>>>>>> parameters in my unit tests?
>>>>>>
>>>>>> - ConfigConstants.TASK_MANAGER_TMP_DIR_KEY
>>>>>> - ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY
>>>>>> - ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY
>>>>>>
>>>>>> Best,
>>>>>> Flavio
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: ExecutionEnvironment setConfiguration API
Posted by Flavio Pompermaier <po...@okkam.it>.
That makes sense: what can be configured should be differentiated between
local and remote envs (obviously this is a minor issue/improvement)
Thanks again,
Flavio
On Tue, Oct 6, 2015 at 11:25 AM, Stephan Ewen <se...@apache.org> wrote:
> We can think about that, but I think it may be quite confusing. The
> configurations actually mean something different for local and remote
> environments:
>
> - For the local environment, the config basically describes the entire
> Flink cluster setup (for the local execution cluster in the background)
> - For the remote environment, the config describes the parameters for
> the client that connects to the cluster (akka paramters, optimizer
> parameters, ...), but not parameters of the cluster itself (like
> taskmanager slots and memory).
>
> Greetings,
> Stephan
>
>
> On Tue, Oct 6, 2015 at 10:56 AM, Flavio Pompermaier <po...@okkam.it>
> wrote:
>
>> However it could be a good idea to overload also
>> the getExecutionEnvironment() to be able to pass a custom
>> configuration..what do you think?
>> Otherwise I have to know a priori if I'm working in a local deployment or
>> in a remote one, or check if getExecutionEnvironment() returned an instance
>> of LocalEnvironment/RemoteEnvironment..
>>
>>
>>
>> On Tue, Oct 6, 2015 at 10:53 AM, Flavio Pompermaier <pompermaier@okkam.it
>> > wrote:
>>
>>> Yes Stephan!
>>> I usually work with the master version, at least in development ;)
>>> Thanks for the quick support!
>>>
>>> Best,
>>> Flavio
>>>
>>> On Tue, Oct 6, 2015 at 10:48 AM, Stephan Ewen <se...@apache.org> wrote:
>>>
>>>> Hi!
>>>>
>>>> Are you on the SNAPSHOT master version?
>>>>
>>>> You can pass the configuration to the constructor of the execution
>>>> environment, or create one via
>>>> ExecutionEnvironment.createLocalEnvironment(config) or via
>>>> createRemoteEnvironment(host, port, configuration, jarFiles);
>>>>
>>>> The change of the signature was part of an API cleanup for the next
>>>> release. Sorry for the inconvenience...
>>>>
>>>> Stephan
>>>>
>>>>
>>>> On Tue, Oct 6, 2015 at 10:36 AM, Flavio Pompermaier <
>>>> pompermaier@okkam.it> wrote:
>>>>
>>>>> Hi to all,
>>>>>
>>>>> today my code doesn't compile anymore because ExecutionEnvironment
>>>>> doesn't have setConfiguration() anymore..how can I set the following
>>>>> parameters in my unit tests?
>>>>>
>>>>> - ConfigConstants.TASK_MANAGER_TMP_DIR_KEY
>>>>> - ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY
>>>>> - ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY
>>>>>
>>>>> Best,
>>>>> Flavio
>>>>>
>>>>
>>>>
>>>
>>
>
Re: ExecutionEnvironment setConfiguration API
Posted by Stephan Ewen <se...@apache.org>.
We can think about that, but I think it may be quite confusing. The
configurations actually mean something different for local and remote
environments:
- For the local environment, the config basically describes the entire
Flink cluster setup (for the local execution cluster in the background)
- For the remote environment, the config describes the parameters for the
client that connects to the cluster (akka paramters, optimizer parameters,
...), but not parameters of the cluster itself (like taskmanager slots and
memory).
Greetings,
Stephan
On Tue, Oct 6, 2015 at 10:56 AM, Flavio Pompermaier <po...@okkam.it>
wrote:
> However it could be a good idea to overload also
> the getExecutionEnvironment() to be able to pass a custom
> configuration..what do you think?
> Otherwise I have to know a priori if I'm working in a local deployment or
> in a remote one, or check if getExecutionEnvironment() returned an instance
> of LocalEnvironment/RemoteEnvironment..
>
>
>
> On Tue, Oct 6, 2015 at 10:53 AM, Flavio Pompermaier <po...@okkam.it>
> wrote:
>
>> Yes Stephan!
>> I usually work with the master version, at least in development ;)
>> Thanks for the quick support!
>>
>> Best,
>> Flavio
>>
>> On Tue, Oct 6, 2015 at 10:48 AM, Stephan Ewen <se...@apache.org> wrote:
>>
>>> Hi!
>>>
>>> Are you on the SNAPSHOT master version?
>>>
>>> You can pass the configuration to the constructor of the execution
>>> environment, or create one via
>>> ExecutionEnvironment.createLocalEnvironment(config) or via
>>> createRemoteEnvironment(host, port, configuration, jarFiles);
>>>
>>> The change of the signature was part of an API cleanup for the next
>>> release. Sorry for the inconvenience...
>>>
>>> Stephan
>>>
>>>
>>> On Tue, Oct 6, 2015 at 10:36 AM, Flavio Pompermaier <
>>> pompermaier@okkam.it> wrote:
>>>
>>>> Hi to all,
>>>>
>>>> today my code doesn't compile anymore because ExecutionEnvironment
>>>> doesn't have setConfiguration() anymore..how can I set the following
>>>> parameters in my unit tests?
>>>>
>>>> - ConfigConstants.TASK_MANAGER_TMP_DIR_KEY
>>>> - ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY
>>>> - ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY
>>>>
>>>> Best,
>>>> Flavio
>>>>
>>>
>>>
>>
>
Re: ExecutionEnvironment setConfiguration API
Posted by Flavio Pompermaier <po...@okkam.it>.
However it could be a good idea to overload also
the getExecutionEnvironment() to be able to pass a custom
configuration..what do you think?
Otherwise I have to know a priori if I'm working in a local deployment or
in a remote one, or check if getExecutionEnvironment() returned an instance
of LocalEnvironment/RemoteEnvironment..
On Tue, Oct 6, 2015 at 10:53 AM, Flavio Pompermaier <po...@okkam.it>
wrote:
> Yes Stephan!
> I usually work with the master version, at least in development ;)
> Thanks for the quick support!
>
> Best,
> Flavio
>
> On Tue, Oct 6, 2015 at 10:48 AM, Stephan Ewen <se...@apache.org> wrote:
>
>> Hi!
>>
>> Are you on the SNAPSHOT master version?
>>
>> You can pass the configuration to the constructor of the execution
>> environment, or create one via
>> ExecutionEnvironment.createLocalEnvironment(config) or via
>> createRemoteEnvironment(host, port, configuration, jarFiles);
>>
>> The change of the signature was part of an API cleanup for the next
>> release. Sorry for the inconvenience...
>>
>> Stephan
>>
>>
>> On Tue, Oct 6, 2015 at 10:36 AM, Flavio Pompermaier <pompermaier@okkam.it
>> > wrote:
>>
>>> Hi to all,
>>>
>>> today my code doesn't compile anymore because ExecutionEnvironment
>>> doesn't have setConfiguration() anymore..how can I set the following
>>> parameters in my unit tests?
>>>
>>> - ConfigConstants.TASK_MANAGER_TMP_DIR_KEY
>>> - ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY
>>> - ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY
>>>
>>> Best,
>>> Flavio
>>>
>>
>>
>
Re: ExecutionEnvironment setConfiguration API
Posted by Flavio Pompermaier <po...@okkam.it>.
Yes Stephan!
I usually work with the master version, at least in development ;)
Thanks for the quick support!
Best,
Flavio
On Tue, Oct 6, 2015 at 10:48 AM, Stephan Ewen <se...@apache.org> wrote:
> Hi!
>
> Are you on the SNAPSHOT master version?
>
> You can pass the configuration to the constructor of the execution
> environment, or create one via
> ExecutionEnvironment.createLocalEnvironment(config) or via
> createRemoteEnvironment(host, port, configuration, jarFiles);
>
> The change of the signature was part of an API cleanup for the next
> release. Sorry for the inconvenience...
>
> Stephan
>
>
> On Tue, Oct 6, 2015 at 10:36 AM, Flavio Pompermaier <po...@okkam.it>
> wrote:
>
>> Hi to all,
>>
>> today my code doesn't compile anymore because ExecutionEnvironment
>> doesn't have setConfiguration() anymore..how can I set the following
>> parameters in my unit tests?
>>
>> - ConfigConstants.TASK_MANAGER_TMP_DIR_KEY
>> - ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY
>> - ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY
>>
>> Best,
>> Flavio
>>
>
>
Re: ExecutionEnvironment setConfiguration API
Posted by Stephan Ewen <se...@apache.org>.
Hi!
Are you on the SNAPSHOT master version?
You can pass the configuration to the constructor of the execution
environment, or create one via
ExecutionEnvironment.createLocalEnvironment(config) or via
createRemoteEnvironment(host, port, configuration, jarFiles);
The change of the signature was part of an API cleanup for the next
release. Sorry for the inconvenience...
Stephan
On Tue, Oct 6, 2015 at 10:36 AM, Flavio Pompermaier <po...@okkam.it>
wrote:
> Hi to all,
>
> today my code doesn't compile anymore because ExecutionEnvironment doesn't
> have setConfiguration() anymore..how can I set the following parameters in
> my unit tests?
>
> - ConfigConstants.TASK_MANAGER_TMP_DIR_KEY
> - ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY
> - ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY
>
> Best,
> Flavio
>