You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Taras Ledkov <tl...@gridgain.com> on 2017/04/18 08:23:53 UTC
Introduce custom executors for compute grid
Igniters,
Custom executor (user's thread pool) is added fro compute grid with
following semantics:
1. Configuration:
IgniteConfiguration cfg;
...
cfg.setExecutorConfiguration(
new ExecutorConfiguration().setName("executor0").setSize(2),
new ExecutorConfiguration().setName("executor1").setSize(4));
Where
name - name of executor and thread pool;
size - thread pool size.
2. Usage:
Ignite ignite;
...
IgniteCompute comp = ignite.compute().withExecutor("executor0");
comp.broadcast(new IgniteRunnable() {
@Override public void run() {
...
}
});
So, 'withExecutor(String)' returns the compute associated with custom
named executor.
All jobs submitted by the components will be processed by thread pool
corresponds to named executor.
If the executor isn't configured on the target host the warning will be
printed in the log and a job will be processed in the public pool.
e.g.:
[11:20:01,023][WARN
][grid-nio-worker-tcp-comm-0-#27%compute.IgniteComputeCustomExecutorSelfTest1%][GridIoManager]
Custom executor 'invalid' doesn't exist. The job will be submit to
public pool: b2e85208b51-4fbcb569-07a2-480e-9be1-512bc3200000
Issue: https://issues.apache.org/jira/browse/IGNITE-4699
Please share your thoughts or ask questions.
--
Taras Ledkov
Mail-To: tledkov@gridgain.com
Re: Introduce custom executors for compute grid
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Yes.
On Wed, Apr 19, 2017 at 12:06 AM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> On Tue, Apr 18, 2017 at 1:49 PM, Vladimir Ozerov <vo...@gridgain.com>
> wrote:
>
> > Yes, task will be routed to public pool with a warning to the log.
> >
>
> Let's make sure we throttle these warnings for every computation so we do
> not flood the logs.
>
>
> >
> > On Tue, Apr 18, 2017 at 9:21 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> > wrote:
> >
> > > Christos,
> > >
> > > I think you are right in general, however, is the executor service is
> not
> > > found, there should not be an exception, we should default to the
> public
> > > pool.
> > >
> > > D.
> > >
> > > On Tue, Apr 18, 2017 at 8:53 AM, Christos Erotocritou <
> > > christos@gridgain.com
> > > > wrote:
> > >
> > > > Taras,
> > > >
> > > > If I’m understanding right:
> > > >
> > > > Executor is configured local to a node, if same configuration is used
> > to
> > > > start multiple nodes then they will all have the same executor hence
> a
> > > > broadcast task would execute on both all nodes. If one of those nodes
> > > > didn’t have this executor configured then task on that node will
> throw
> > an
> > > > exception. Now in a scenario where 2 nodes do not have same executor
> > and
> > > we
> > > > just use compute apply api call to execute task on next available
> node
> > > > chances are it can end up on a node without that executor, correct?
> So
> > we
> > > > don’t have a way of routing tasks automatically to nodes with
> specific
> > > > executors, we would have to use cluster groups to ensure that,
> correct?
> > > >
> > > > I think its important that we document all these points.
> > > >
> > > > C.
> > > >
> > > > > On 18 Apr 2017, at 16:12, Taras Ledkov <tl...@gridgain.com>
> wrote:
> > > > >
> > > > > Hi, Christos.
> > > > >
> > > > > 1. The custom executor executor is configured for specified node.
> > There
> > > > is not restriction that guaranties the similar set of executors on
> the
> > > > whole cluster.
> > > > >
> > > > > 2. I guess You can use ClusterGroup.forPredicate() and custom node
> > > > attribute when you want to submit the job only to nodes where the
> > > specific
> > > > executor is configured.
> > > > >
> > > > >
> > > > > On 18.04.2017 18:00, Christos Erotocritou wrote:
> > > > >> I assume its not currently possible to constrain an executor pool
> to
> > > be
> > > > started only on a specific node?
> > > > >
> > > > > --
> > > > > Taras Ledkov
> > > > > Mail-To: tledkov@gridgain.com
> > > > >
> > > >
> > > >
> > >
> >
>
Re: Introduce custom executors for compute grid
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Tue, Apr 18, 2017 at 1:49 PM, Vladimir Ozerov <vo...@gridgain.com>
wrote:
> Yes, task will be routed to public pool with a warning to the log.
>
Let's make sure we throttle these warnings for every computation so we do
not flood the logs.
>
> On Tue, Apr 18, 2017 at 9:21 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
> > Christos,
> >
> > I think you are right in general, however, is the executor service is not
> > found, there should not be an exception, we should default to the public
> > pool.
> >
> > D.
> >
> > On Tue, Apr 18, 2017 at 8:53 AM, Christos Erotocritou <
> > christos@gridgain.com
> > > wrote:
> >
> > > Taras,
> > >
> > > If I’m understanding right:
> > >
> > > Executor is configured local to a node, if same configuration is used
> to
> > > start multiple nodes then they will all have the same executor hence a
> > > broadcast task would execute on both all nodes. If one of those nodes
> > > didn’t have this executor configured then task on that node will throw
> an
> > > exception. Now in a scenario where 2 nodes do not have same executor
> and
> > we
> > > just use compute apply api call to execute task on next available node
> > > chances are it can end up on a node without that executor, correct? So
> we
> > > don’t have a way of routing tasks automatically to nodes with specific
> > > executors, we would have to use cluster groups to ensure that, correct?
> > >
> > > I think its important that we document all these points.
> > >
> > > C.
> > >
> > > > On 18 Apr 2017, at 16:12, Taras Ledkov <tl...@gridgain.com> wrote:
> > > >
> > > > Hi, Christos.
> > > >
> > > > 1. The custom executor executor is configured for specified node.
> There
> > > is not restriction that guaranties the similar set of executors on the
> > > whole cluster.
> > > >
> > > > 2. I guess You can use ClusterGroup.forPredicate() and custom node
> > > attribute when you want to submit the job only to nodes where the
> > specific
> > > executor is configured.
> > > >
> > > >
> > > > On 18.04.2017 18:00, Christos Erotocritou wrote:
> > > >> I assume its not currently possible to constrain an executor pool to
> > be
> > > started only on a specific node?
> > > >
> > > > --
> > > > Taras Ledkov
> > > > Mail-To: tledkov@gridgain.com
> > > >
> > >
> > >
> >
>
Re: Introduce custom executors for compute grid
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Yes, task will be routed to public pool with a warning to the log.
On Tue, Apr 18, 2017 at 9:21 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> Christos,
>
> I think you are right in general, however, is the executor service is not
> found, there should not be an exception, we should default to the public
> pool.
>
> D.
>
> On Tue, Apr 18, 2017 at 8:53 AM, Christos Erotocritou <
> christos@gridgain.com
> > wrote:
>
> > Taras,
> >
> > If I’m understanding right:
> >
> > Executor is configured local to a node, if same configuration is used to
> > start multiple nodes then they will all have the same executor hence a
> > broadcast task would execute on both all nodes. If one of those nodes
> > didn’t have this executor configured then task on that node will throw an
> > exception. Now in a scenario where 2 nodes do not have same executor and
> we
> > just use compute apply api call to execute task on next available node
> > chances are it can end up on a node without that executor, correct? So we
> > don’t have a way of routing tasks automatically to nodes with specific
> > executors, we would have to use cluster groups to ensure that, correct?
> >
> > I think its important that we document all these points.
> >
> > C.
> >
> > > On 18 Apr 2017, at 16:12, Taras Ledkov <tl...@gridgain.com> wrote:
> > >
> > > Hi, Christos.
> > >
> > > 1. The custom executor executor is configured for specified node. There
> > is not restriction that guaranties the similar set of executors on the
> > whole cluster.
> > >
> > > 2. I guess You can use ClusterGroup.forPredicate() and custom node
> > attribute when you want to submit the job only to nodes where the
> specific
> > executor is configured.
> > >
> > >
> > > On 18.04.2017 18:00, Christos Erotocritou wrote:
> > >> I assume its not currently possible to constrain an executor pool to
> be
> > started only on a specific node?
> > >
> > > --
> > > Taras Ledkov
> > > Mail-To: tledkov@gridgain.com
> > >
> >
> >
>
Re: Introduce custom executors for compute grid
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Christos,
I think you are right in general, however, is the executor service is not
found, there should not be an exception, we should default to the public
pool.
D.
On Tue, Apr 18, 2017 at 8:53 AM, Christos Erotocritou <christos@gridgain.com
> wrote:
> Taras,
>
> If I’m understanding right:
>
> Executor is configured local to a node, if same configuration is used to
> start multiple nodes then they will all have the same executor hence a
> broadcast task would execute on both all nodes. If one of those nodes
> didn’t have this executor configured then task on that node will throw an
> exception. Now in a scenario where 2 nodes do not have same executor and we
> just use compute apply api call to execute task on next available node
> chances are it can end up on a node without that executor, correct? So we
> don’t have a way of routing tasks automatically to nodes with specific
> executors, we would have to use cluster groups to ensure that, correct?
>
> I think its important that we document all these points.
>
> C.
>
> > On 18 Apr 2017, at 16:12, Taras Ledkov <tl...@gridgain.com> wrote:
> >
> > Hi, Christos.
> >
> > 1. The custom executor executor is configured for specified node. There
> is not restriction that guaranties the similar set of executors on the
> whole cluster.
> >
> > 2. I guess You can use ClusterGroup.forPredicate() and custom node
> attribute when you want to submit the job only to nodes where the specific
> executor is configured.
> >
> >
> > On 18.04.2017 18:00, Christos Erotocritou wrote:
> >> I assume its not currently possible to constrain an executor pool to be
> started only on a specific node?
> >
> > --
> > Taras Ledkov
> > Mail-To: tledkov@gridgain.com
> >
>
>
Re: Introduce custom executors for compute grid
Posted by Christos Erotocritou <ch...@gridgain.com>.
Taras,
If I’m understanding right:
Executor is configured local to a node, if same configuration is used to start multiple nodes then they will all have the same executor hence a broadcast task would execute on both all nodes. If one of those nodes didn’t have this executor configured then task on that node will throw an exception. Now in a scenario where 2 nodes do not have same executor and we just use compute apply api call to execute task on next available node chances are it can end up on a node without that executor, correct? So we don’t have a way of routing tasks automatically to nodes with specific executors, we would have to use cluster groups to ensure that, correct?
I think its important that we document all these points.
C.
> On 18 Apr 2017, at 16:12, Taras Ledkov <tl...@gridgain.com> wrote:
>
> Hi, Christos.
>
> 1. The custom executor executor is configured for specified node. There is not restriction that guaranties the similar set of executors on the whole cluster.
>
> 2. I guess You can use ClusterGroup.forPredicate() and custom node attribute when you want to submit the job only to nodes where the specific executor is configured.
>
>
> On 18.04.2017 18:00, Christos Erotocritou wrote:
>> I assume its not currently possible to constrain an executor pool to be started only on a specific node?
>
> --
> Taras Ledkov
> Mail-To: tledkov@gridgain.com
>
Re: Introduce custom executors for compute grid
Posted by Taras Ledkov <tl...@gridgain.com>.
Hi, Christos.
1. The custom executor executor is configured for specified node. There
is not restriction that guaranties the similar set of executors on the
whole cluster.
2. I guess You can use ClusterGroup.forPredicate() and custom node
attribute when you want to submit the job only to nodes where the
specific executor is configured.
On 18.04.2017 18:00, Christos Erotocritou wrote:
> I assume its not currently possible to constrain an executor pool to be started only on a specific node?
--
Taras Ledkov
Mail-To: tledkov@gridgain.com
Re: Introduce custom executors for compute grid
Posted by Christos Erotocritou <ch...@gridgain.com>.
Thanks Taras, this will be very handy.
I assume its not currently possible to constrain an executor pool to be started only on a specific node?
C.
> On 18 Apr 2017, at 15:55, Denis Magda <dm...@apache.org> wrote:
>
> Taras, that’s an excellent addition to the project!
>
> Please don’t forget to document it: https://issues.apache.org/jira/browse/IGNITE-4969 <https://issues.apache.org/jira/browse/IGNITE-4969>
>
> —
> Denis
>
>> On Apr 18, 2017, at 1:23 AM, Taras Ledkov <tl...@gridgain.com> wrote:
>>
>> Igniters,
>>
>> Custom executor (user's thread pool) is added fro compute grid with following semantics:
>>
>> 1. Configuration:
>>
>> IgniteConfiguration cfg;
>> ...
>> cfg.setExecutorConfiguration(
>> new ExecutorConfiguration().setName("executor0").setSize(2),
>> new ExecutorConfiguration().setName("executor1").setSize(4));
>>
>> Where
>> name - name of executor and thread pool;
>> size - thread pool size.
>>
>> 2. Usage:
>>
>> Ignite ignite;
>> ...
>> IgniteCompute comp = ignite.compute().withExecutor("executor0");
>> comp.broadcast(new IgniteRunnable() {
>> @Override public void run() {
>> ...
>> }
>> });
>>
>> So, 'withExecutor(String)' returns the compute associated with custom named executor.
>> All jobs submitted by the components will be processed by thread pool corresponds to named executor.
>> If the executor isn't configured on the target host the warning will be printed in the log and a job will be processed in the public pool.
>> e.g.:
>> [11:20:01,023][WARN ][grid-nio-worker-tcp-comm-0-#27%compute.IgniteComputeCustomExecutorSelfTest1%][GridIoManager] Custom executor 'invalid' doesn't exist. The job will be submit to public pool: b2e85208b51-4fbcb569-07a2-480e-9be1-512bc3200000
>>
>> Issue: https://issues.apache.org/jira/browse/IGNITE-4699
>>
>> Please share your thoughts or ask questions.
>>
>> --
>> Taras Ledkov
>> Mail-To: tledkov@gridgain.com
>>
>
Re: Introduce custom executors for compute grid
Posted by Christos Erotocritou <ch...@gridgain.com>.
D, I think this fixed with this ticket: https://issues.apache.org/jira/browse/IGNITE-4802 <https://issues.apache.org/jira/browse/IGNITE-4802>
> On 18 Apr 2017, at 16:30, Dmitriy Setrakyan <ds...@apache.org> wrote:
>
> Agree, very useful.
>
> Does the service grid run in its own thread pool? Just wondering if this
> feature can be useful there as well.
>
> D.
>
> On Tue, Apr 18, 2017 at 7:55 AM, Denis Magda <dm...@apache.org> wrote:
>
>> Taras, that’s an excellent addition to the project!
>>
>> Please don’t forget to document it: https://issues.apache.org/
>> jira/browse/IGNITE-4969 <https://issues.apache.org/jira/browse/IGNITE-4969
>>>
>>
>> —
>> Denis
>>
>>> On Apr 18, 2017, at 1:23 AM, Taras Ledkov <tl...@gridgain.com> wrote:
>>>
>>> Igniters,
>>>
>>> Custom executor (user's thread pool) is added fro compute grid with
>> following semantics:
>>>
>>> 1. Configuration:
>>>
>>> IgniteConfiguration cfg;
>>> ...
>>> cfg.setExecutorConfiguration(
>>> new ExecutorConfiguration().setName("executor0").setSize(2),
>>> new ExecutorConfiguration().setName("executor1").setSize(4));
>>>
>>> Where
>>> name - name of executor and thread pool;
>>> size - thread pool size.
>>>
>>> 2. Usage:
>>>
>>> Ignite ignite;
>>> ...
>>> IgniteCompute comp = ignite.compute().withExecutor("executor0");
>>> comp.broadcast(new IgniteRunnable() {
>>> @Override public void run() {
>>> ...
>>> }
>>> });
>>>
>>> So, 'withExecutor(String)' returns the compute associated with custom
>> named executor.
>>> All jobs submitted by the components will be processed by thread pool
>> corresponds to named executor.
>>> If the executor isn't configured on the target host the warning will be
>> printed in the log and a job will be processed in the public pool.
>>> e.g.:
>>> [11:20:01,023][WARN ][grid-nio-worker-tcp-comm-0-#27%compute.
>> IgniteComputeCustomExecutorSelfTest1%][GridIoManager] Custom executor
>> 'invalid' doesn't exist. The job will be submit to public pool:
>> b2e85208b51-4fbcb569-07a2-480e-9be1-512bc3200000
>>>
>>> Issue: https://issues.apache.org/jira/browse/IGNITE-4699
>>>
>>> Please share your thoughts or ask questions.
>>>
>>> --
>>> Taras Ledkov
>>> Mail-To: tledkov@gridgain.com
>>>
>>
>>
Re: Introduce custom executors for compute grid
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Agree, very useful.
Does the service grid run in its own thread pool? Just wondering if this
feature can be useful there as well.
D.
On Tue, Apr 18, 2017 at 7:55 AM, Denis Magda <dm...@apache.org> wrote:
> Taras, that’s an excellent addition to the project!
>
> Please don’t forget to document it: https://issues.apache.org/
> jira/browse/IGNITE-4969 <https://issues.apache.org/jira/browse/IGNITE-4969
> >
>
> —
> Denis
>
> > On Apr 18, 2017, at 1:23 AM, Taras Ledkov <tl...@gridgain.com> wrote:
> >
> > Igniters,
> >
> > Custom executor (user's thread pool) is added fro compute grid with
> following semantics:
> >
> > 1. Configuration:
> >
> > IgniteConfiguration cfg;
> > ...
> > cfg.setExecutorConfiguration(
> > new ExecutorConfiguration().setName("executor0").setSize(2),
> > new ExecutorConfiguration().setName("executor1").setSize(4));
> >
> > Where
> > name - name of executor and thread pool;
> > size - thread pool size.
> >
> > 2. Usage:
> >
> > Ignite ignite;
> > ...
> > IgniteCompute comp = ignite.compute().withExecutor("executor0");
> > comp.broadcast(new IgniteRunnable() {
> > @Override public void run() {
> > ...
> > }
> > });
> >
> > So, 'withExecutor(String)' returns the compute associated with custom
> named executor.
> > All jobs submitted by the components will be processed by thread pool
> corresponds to named executor.
> > If the executor isn't configured on the target host the warning will be
> printed in the log and a job will be processed in the public pool.
> > e.g.:
> > [11:20:01,023][WARN ][grid-nio-worker-tcp-comm-0-#27%compute.
> IgniteComputeCustomExecutorSelfTest1%][GridIoManager] Custom executor
> 'invalid' doesn't exist. The job will be submit to public pool:
> b2e85208b51-4fbcb569-07a2-480e-9be1-512bc3200000
> >
> > Issue: https://issues.apache.org/jira/browse/IGNITE-4699
> >
> > Please share your thoughts or ask questions.
> >
> > --
> > Taras Ledkov
> > Mail-To: tledkov@gridgain.com
> >
>
>
Re: Introduce custom executors for compute grid
Posted by Denis Magda <dm...@apache.org>.
Taras, that’s an excellent addition to the project!
Please don’t forget to document it: https://issues.apache.org/jira/browse/IGNITE-4969 <https://issues.apache.org/jira/browse/IGNITE-4969>
—
Denis
> On Apr 18, 2017, at 1:23 AM, Taras Ledkov <tl...@gridgain.com> wrote:
>
> Igniters,
>
> Custom executor (user's thread pool) is added fro compute grid with following semantics:
>
> 1. Configuration:
>
> IgniteConfiguration cfg;
> ...
> cfg.setExecutorConfiguration(
> new ExecutorConfiguration().setName("executor0").setSize(2),
> new ExecutorConfiguration().setName("executor1").setSize(4));
>
> Where
> name - name of executor and thread pool;
> size - thread pool size.
>
> 2. Usage:
>
> Ignite ignite;
> ...
> IgniteCompute comp = ignite.compute().withExecutor("executor0");
> comp.broadcast(new IgniteRunnable() {
> @Override public void run() {
> ...
> }
> });
>
> So, 'withExecutor(String)' returns the compute associated with custom named executor.
> All jobs submitted by the components will be processed by thread pool corresponds to named executor.
> If the executor isn't configured on the target host the warning will be printed in the log and a job will be processed in the public pool.
> e.g.:
> [11:20:01,023][WARN ][grid-nio-worker-tcp-comm-0-#27%compute.IgniteComputeCustomExecutorSelfTest1%][GridIoManager] Custom executor 'invalid' doesn't exist. The job will be submit to public pool: b2e85208b51-4fbcb569-07a2-480e-9be1-512bc3200000
>
> Issue: https://issues.apache.org/jira/browse/IGNITE-4699
>
> Please share your thoughts or ask questions.
>
> --
> Taras Ledkov
> Mail-To: tledkov@gridgain.com
>