New thread pool for queries

[Sergi Vladykin <se...@gmail.com>]

Guys,

We had a problem with thread starvation in PUBLIC pool because of queries
as described in https://issues.apache.org/jira/browse/IGNITE-1174

I switched it to SYSTEM pool but this can cause delays for other cache
operations like put/get if we have many long running queries.

I propose adding one more thread pool for queries to make them independent.

Thoughts?

Sergi

Re: New thread pool for queries

[Vladimir Ozerov <vo...@gridgain.com>]

We already worked on the problem a bit. Now IO pool is not an enum, but a
byte. We have 0..31 bytes reserved for Ignite pools (and some of them are
already used for existing pools), and the rest values are for plugins which
can register their own pools and send messages to them.

It seems to me that only minor changes to support backward compatibility
are required here.

On Tue, Aug 4, 2015 at 11:25 AM, Sergi Vladykin <se...@gmail.com>
wrote:

> I created one here:
> https://issues.apache.org/jira/browse/IGNITE-1193
>
> Sergi
>
> 2015-08-04 3:21 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>
> > Is there a ticket about supporting multiple thread pools? I wanted to
> leave
> > some design thoughts on this subject, but can't see to find the
> appropriate
> > ticket for it.
> >
> > On Fri, Jul 31, 2015 at 6:49 AM, Sergi Vladykin <
> sergi.vladykin@gmail.com>
> > wrote:
> >
> > > Agree with Val, we already have multiple pools support, the only thing
> we
> > > should take care about is backward compatibility, when new node sends
> > > message to an old one which does not have a needed pool yet. We can
> > handle
> > > it manually each time, but I prefer just to fallback to some default
> pool
> > > in old nodes to avoid complexity.
> > >
> > > Sergi
> > > On Jul 31, 2015 10:50 AM, "Valentin Kulichenko" <
> > > valentin.kulichenko@gmail.com> wrote:
> > >
> > > > On Fri, Jul 31, 2015 at 12:30 AM, Dmitriy Setrakyan <
> > > dsetrakyan@apache.org
> > > > >
> > > > wrote:
> > > >
> > > > > On Fri, Jul 31, 2015 at 12:06 AM, Sergi Vladykin <
> > > > sergi.vladykin@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Guys,
> > > > > >
> > > > > > We had a problem with thread starvation in PUBLIC pool because of
> > > > queries
> > > > > > as described in
> https://issues.apache.org/jira/browse/IGNITE-1174
> > > > > >
> > > > > > I switched it to SYSTEM pool but this can cause delays for other
> > > cache
> > > > > > operations like put/get if we have many long running queries.
> > > > > >
> > > > > > I propose adding one more thread pool for queries to make them
> > > > > independent.
> > > > > >
> > > > >
> > > > > I like the idea, however, I think the design should be more
> generic.
> > I
> > > > > think Ignite should be able to have N thread pools. Then for each
> set
> > > of
> > > > > operations, e.g. task-execution, message-replies, query-processing,
> > > etc.
> > > > > user should be able to assign any of the available N thread-pools.
> > This
> > > > can
> > > > > be done in the configuration.
> > > > >
> > > >
> > > > We actually already have similar design for plugins (see IoPool
> > > extension).
> > > > I think we can reuse it for all pools in the system.
> > > >
> > > >
> > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > Sergi
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: New thread pool for queries

[Sergi Vladykin <se...@gmail.com>]

I created one here:
https://issues.apache.org/jira/browse/IGNITE-1193

Sergi

2015-08-04 3:21 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:

> Is there a ticket about supporting multiple thread pools? I wanted to leave
> some design thoughts on this subject, but can't see to find the appropriate
> ticket for it.
>
> On Fri, Jul 31, 2015 at 6:49 AM, Sergi Vladykin <se...@gmail.com>
> wrote:
>
> > Agree with Val, we already have multiple pools support, the only thing we
> > should take care about is backward compatibility, when new node sends
> > message to an old one which does not have a needed pool yet. We can
> handle
> > it manually each time, but I prefer just to fallback to some default pool
> > in old nodes to avoid complexity.
> >
> > Sergi
> > On Jul 31, 2015 10:50 AM, "Valentin Kulichenko" <
> > valentin.kulichenko@gmail.com> wrote:
> >
> > > On Fri, Jul 31, 2015 at 12:30 AM, Dmitriy Setrakyan <
> > dsetrakyan@apache.org
> > > >
> > > wrote:
> > >
> > > > On Fri, Jul 31, 2015 at 12:06 AM, Sergi Vladykin <
> > > sergi.vladykin@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Guys,
> > > > >
> > > > > We had a problem with thread starvation in PUBLIC pool because of
> > > queries
> > > > > as described in https://issues.apache.org/jira/browse/IGNITE-1174
> > > > >
> > > > > I switched it to SYSTEM pool but this can cause delays for other
> > cache
> > > > > operations like put/get if we have many long running queries.
> > > > >
> > > > > I propose adding one more thread pool for queries to make them
> > > > independent.
> > > > >
> > > >
> > > > I like the idea, however, I think the design should be more generic.
> I
> > > > think Ignite should be able to have N thread pools. Then for each set
> > of
> > > > operations, e.g. task-execution, message-replies, query-processing,
> > etc.
> > > > user should be able to assign any of the available N thread-pools.
> This
> > > can
> > > > be done in the configuration.
> > > >
> > >
> > > We actually already have similar design for plugins (see IoPool
> > extension).
> > > I think we can reuse it for all pools in the system.
> > >
> > >
> > > >
> > > > > Thoughts?
> > > > >
> > > > > Sergi
> > > > >
> > > >
> > >
> >
>

Re: New thread pool for queries

[Dmitriy Setrakyan <ds...@apache.org>]

Is there a ticket about supporting multiple thread pools? I wanted to leave
some design thoughts on this subject, but can't see to find the appropriate
ticket for it.

On Fri, Jul 31, 2015 at 6:49 AM, Sergi Vladykin <se...@gmail.com>
wrote:

> Agree with Val, we already have multiple pools support, the only thing we
> should take care about is backward compatibility, when new node sends
> message to an old one which does not have a needed pool yet. We can handle
> it manually each time, but I prefer just to fallback to some default pool
> in old nodes to avoid complexity.
>
> Sergi
> On Jul 31, 2015 10:50 AM, "Valentin Kulichenko" <
> valentin.kulichenko@gmail.com> wrote:
>
> > On Fri, Jul 31, 2015 at 12:30 AM, Dmitriy Setrakyan <
> dsetrakyan@apache.org
> > >
> > wrote:
> >
> > > On Fri, Jul 31, 2015 at 12:06 AM, Sergi Vladykin <
> > sergi.vladykin@gmail.com
> > > >
> > > wrote:
> > >
> > > > Guys,
> > > >
> > > > We had a problem with thread starvation in PUBLIC pool because of
> > queries
> > > > as described in https://issues.apache.org/jira/browse/IGNITE-1174
> > > >
> > > > I switched it to SYSTEM pool but this can cause delays for other
> cache
> > > > operations like put/get if we have many long running queries.
> > > >
> > > > I propose adding one more thread pool for queries to make them
> > > independent.
> > > >
> > >
> > > I like the idea, however, I think the design should be more generic. I
> > > think Ignite should be able to have N thread pools. Then for each set
> of
> > > operations, e.g. task-execution, message-replies, query-processing,
> etc.
> > > user should be able to assign any of the available N thread-pools. This
> > can
> > > be done in the configuration.
> > >
> >
> > We actually already have similar design for plugins (see IoPool
> extension).
> > I think we can reuse it for all pools in the system.
> >
> >
> > >
> > > > Thoughts?
> > > >
> > > > Sergi
> > > >
> > >
> >
>

Re: New thread pool for queries

[Sergi Vladykin <se...@gmail.com>]

Agree with Val, we already have multiple pools support, the only thing we
should take care about is backward compatibility, when new node sends
message to an old one which does not have a needed pool yet. We can handle
it manually each time, but I prefer just to fallback to some default pool
in old nodes to avoid complexity.

Sergi
On Jul 31, 2015 10:50 AM, "Valentin Kulichenko" <
valentin.kulichenko@gmail.com> wrote:

> On Fri, Jul 31, 2015 at 12:30 AM, Dmitriy Setrakyan <dsetrakyan@apache.org
> >
> wrote:
>
> > On Fri, Jul 31, 2015 at 12:06 AM, Sergi Vladykin <
> sergi.vladykin@gmail.com
> > >
> > wrote:
> >
> > > Guys,
> > >
> > > We had a problem with thread starvation in PUBLIC pool because of
> queries
> > > as described in https://issues.apache.org/jira/browse/IGNITE-1174
> > >
> > > I switched it to SYSTEM pool but this can cause delays for other cache
> > > operations like put/get if we have many long running queries.
> > >
> > > I propose adding one more thread pool for queries to make them
> > independent.
> > >
> >
> > I like the idea, however, I think the design should be more generic. I
> > think Ignite should be able to have N thread pools. Then for each set of
> > operations, e.g. task-execution, message-replies, query-processing, etc.
> > user should be able to assign any of the available N thread-pools. This
> can
> > be done in the configuration.
> >
>
> We actually already have similar design for plugins (see IoPool extension).
> I think we can reuse it for all pools in the system.
>
>
> >
> > > Thoughts?
> > >
> > > Sergi
> > >
> >
>

Re: New thread pool for queries

[Valentin Kulichenko <va...@gmail.com>]

On Fri, Jul 31, 2015 at 12:30 AM, Dmitriy Setrakyan <ds...@apache.org>
wrote:

> On Fri, Jul 31, 2015 at 12:06 AM, Sergi Vladykin <sergi.vladykin@gmail.com
> >
> wrote:
>
> > Guys,
> >
> > We had a problem with thread starvation in PUBLIC pool because of queries
> > as described in https://issues.apache.org/jira/browse/IGNITE-1174
> >
> > I switched it to SYSTEM pool but this can cause delays for other cache
> > operations like put/get if we have many long running queries.
> >
> > I propose adding one more thread pool for queries to make them
> independent.
> >
>
> I like the idea, however, I think the design should be more generic. I
> think Ignite should be able to have N thread pools. Then for each set of
> operations, e.g. task-execution, message-replies, query-processing, etc.
> user should be able to assign any of the available N thread-pools. This can
> be done in the configuration.
>

We actually already have similar design for plugins (see IoPool extension).
I think we can reuse it for all pools in the system.


>
> > Thoughts?
> >
> > Sergi
> >
>

Re: New thread pool for queries

[Dmitriy Setrakyan <ds...@apache.org>]

On Fri, Jul 31, 2015 at 12:06 AM, Sergi Vladykin <se...@gmail.com>
wrote:

> Guys,
>
> We had a problem with thread starvation in PUBLIC pool because of queries
> as described in https://issues.apache.org/jira/browse/IGNITE-1174
>
> I switched it to SYSTEM pool but this can cause delays for other cache
> operations like put/get if we have many long running queries.
>
> I propose adding one more thread pool for queries to make them independent.
>

I like the idea, however, I think the design should be more generic. I
think Ignite should be able to have N thread pools. Then for each set of
operations, e.g. task-execution, message-replies, query-processing, etc.
user should be able to assign any of the available N thread-pools. This can
be done in the configuration.

I believe this approach leaves the door open to have more thread-pools down
the road.


> Thoughts?
>
> Sergi
>

Re: New thread pool for queries

[Sergi Vladykin <se...@gmail.com>]

Atri,

I don't think that we'll see any meaningful changes in benchmark results.
There can be little throughput drop but I always can fix this by changing
number of threads in system pool so that it will work the same way as it
did with public pool.

Sergi
On Jul 31, 2015 10:12 AM, "Atri Sharma" <at...@gmail.com> wrote:

> Were the benchmarks run on the patch attached in the JIRA?
>
> On Fri, Jul 31, 2015 at 12:36 PM, Sergi Vladykin <sergi.vladykin@gmail.com
> >
> wrote:
>
> > Guys,
> >
> > We had a problem with thread starvation in PUBLIC pool because of queries
> > as described in https://issues.apache.org/jira/browse/IGNITE-1174
> >
> > I switched it to SYSTEM pool but this can cause delays for other cache
> > operations like put/get if we have many long running queries.
> >
> > I propose adding one more thread pool for queries to make them
> independent.
> >
> > Thoughts?
> >
> > Sergi
> >
>
>
>
> --
> Regards,
>
> Atri
> *l'apprenant*
>

Re: New thread pool for queries

[Alexey Kuznetsov <ak...@gridgain.com>]

Sergi,

What is the difference between system and public pool?
It is possible to get thread starvation in this case also?

In general, separate thread pool looks good for me, or we need to somehow
to balance queries and cache operations.




-- 
Alexey Kuznetsov
GridGain Systems
www.gridgain.com

Re: New thread pool for queries

[Atri Sharma <at...@gmail.com>]

Were the benchmarks run on the patch attached in the JIRA?

On Fri, Jul 31, 2015 at 12:36 PM, Sergi Vladykin <se...@gmail.com>
wrote:

> Guys,
>
> We had a problem with thread starvation in PUBLIC pool because of queries
> as described in https://issues.apache.org/jira/browse/IGNITE-1174
>
> I switched it to SYSTEM pool but this can cause delays for other cache
> operations like put/get if we have many long running queries.
>
> I propose adding one more thread pool for queries to make them independent.
>
> Thoughts?
>
> Sergi
>



-- 
Regards,

Atri
*l'apprenant*