You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Andrey Kornev <an...@hotmail.com> on 2015/11/12 19:02:52 UTC
Custom ThreadFactory
Hello,
If my memory doesn't fail me, in the pre-Ignite versions of GridGain, it was possible to configure custom executor services which would then be used to create the public, system, utility, etc. thread pools. In Ignite however it's only possible to configure the size of the thread pools and not their implementations.
This is unfortunate as I'd like to be able to configure my own ThreadFactory. My implementation would for example ensure that newly created threads have their thread locals properly initialized (for example, by storing the local instance of Ignite in it). Specific use case is being able to get hold of the local Ignite instance during deserialization when the JVM instance has multiple Ignite nodes started. Some of my classes must be able to access resources that are local to the node on which they are being deserialized. At the moment there is absolutely no way of achieving something like that.
I'm wondering if it would be possible to add this feature back to Ignite? It seems to be indispensable for unit testing.
Alternatively, to reduce the impact on the public API, an environment variable that takes an FQN of the ThreadFactory to use would also work. It would be injectable with the Ignite resources in the manner similar to how it's done for the closures and factories...
Regards
Andrey
PS. While we're at it, I also remember that in the pre-Ignite versions it was possible to inject an instance of the public executor service into the closures. Not anymore. It causes the inconvenience of starting another thread pool while there is already a public pool managed by Ignite with plenty of threads idling most of the time... It feels wasteful.
Re: Custom ThreadFactory
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Fri, Nov 13, 2015 at 4:16 AM, Yakov Zhdanov <yz...@apache.org> wrote:
> Andrey, I closed (with proper relation)
> https://issues.apache.org/jira/browse/IGNITE-1899 since we already had
> https://issues.apache.org/jira/browse/IGNITE-10
Andrey, would you be interested in contributing this feature?
>
>
> --Yakov
>
> 2015-11-13 2:40 GMT+03:00 Andrey Kornev <an...@hotmail.com>:
>
> > There you go:
> > https://issues.apache.org/jira/browse/IGNITE-1898
> > and
> > https://issues.apache.org/jira/browse/IGNITE-1899
> >
> > Cheers
> > Andrey
> >
> > > From: dsetrakyan@apache.org
> > > Date: Thu, 12 Nov 2015 12:53:52 -0800
> > > Subject: Re: Custom ThreadFactory
> > > To: dev@ignite.apache.org
> > >
> > > On Thu, Nov 12, 2015 at 10:26 AM, Andrey Kornev <
> > andrewkornev@hotmail.com>
> > > wrote:
> > >
> > > > Even better, Ignite might provide out-of-the-box access to the local
> > > > instance via a
> > > > thread local. It could be in a form of a static public
> > > > method on the Ignition class.
> > > >
> > > > Ignite itself could benefit from
> > > > this feature as it does get it wrong occasionally. A good example of
> > > > this is the ClusterGroupAdapter class or any other class that
> > serializes
> > > > the instance of IgniteKernal. Imagine the situation where you have a
> > > > single JVM with multiple Ignite nodes started -- Ignite requires the
> > > > grid names to be different. But since only the name of the grid is
> > > > serialized, during deserialization an invalid (unexpected, to put it
> > > > mildly) instance of IgniteKernal is looked up. I don't know how
> serious
> > > > it is, but it is probably a bug.
> > > >
> > >
> > > Andrey, I think its best to summarize the design thoughts in a ticket.
> > Then
> > > someone from the community will pick it up.
> > >
> > >
> > > >
> > > > Regards
> > > > Andrey
> > > >
> > > > > Date: Thu, 12 Nov 2015 21:11:38 +0300
> > > > > Subject: Re: Custom ThreadFactory
> > > > > From: vozerov@gridgain.com
> > > > > To: dev@ignite.apache.org
> > > > >
> > > > > I would avoid running any external payloads in public pool because
> it
> > > > could
> > > > > unpredictably affect Ignite internals. "Public" doesn't mean
> "opened
> > for
> > > > > everyone" here.
> > > > >
> > > > > On the other hand, I abosuletly agree that removing possibility to
> > > > > configure custom pools was not very good idea. I do not see any
> > problems
> > > > > with returning it back while still keeping "thread count" property
> > for
> > > > the
> > > > > most common use case when custom pool is not needed/
> > > > >
> > > > > On Thu, Nov 12, 2015 at 9:02 PM, Andrey Kornev <
> > andrewkornev@hotmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > If my memory doesn't fail me, in the pre-Ignite versions of
> > GridGain,
> > > > it
> > > > > > was possible to configure custom executor services which would
> > then be
> > > > used
> > > > > > to create the public, system, utility, etc. thread pools. In
> Ignite
> > > > however
> > > > > > it's only possible to configure the size of the thread pools and
> > not
> > > > their
> > > > > > implementations.
> > > > > >
> > > > > > This is unfortunate as I'd like to be able to configure my own
> > > > > > ThreadFactory. My implementation would for example ensure that
> > newly
> > > > > > created threads have their thread locals properly initialized
> (for
> > > > example,
> > > > > > by storing the local instance of Ignite in it). Specific use case
> > is
> > > > being
> > > > > > able to get hold of the local Ignite instance during
> > deserialization
> > > > when
> > > > > > the JVM instance has multiple Ignite nodes started. Some of my
> > classes
> > > > must
> > > > > > be able to access resources that are local to the node on which
> > they
> > > > are
> > > > > > being deserialized. At the moment there is absolutely no way of
> > > > achieving
> > > > > > something like that.
> > > > > >
> > > > > > I'm wondering if it would be possible to add this feature back to
> > > > Ignite?
> > > > > > It seems to be indispensable for unit testing.
> > > > > >
> > > > > > Alternatively, to reduce the impact on the public API, an
> > environment
> > > > > > variable that takes an FQN of the ThreadFactory to use would also
> > > > work. It
> > > > > > would be injectable with the Ignite resources in the manner
> > similar to
> > > > how
> > > > > > it's done for the closures and factories...
> > > > > >
> > > > > > Regards
> > > > > > Andrey
> > > > > >
> > > > > > PS. While we're at it, I also remember that in the pre-Ignite
> > versions
> > > > it
> > > > > > was possible to inject an instance of the public executor service
> > into
> > > > the
> > > > > > closures. Not anymore. It causes the inconvenience of starting
> > another
> > > > > > thread pool while there is already a public pool managed by
> Ignite
> > with
> > > > > > plenty of threads idling most of the time... It feels wasteful.
> > > > > >
> > > >
> > > >
> >
> >
>
Re: Custom ThreadFactory
Posted by Yakov Zhdanov <yz...@apache.org>.
Andrey, I closed (with proper relation)
https://issues.apache.org/jira/browse/IGNITE-1899 since we already had
https://issues.apache.org/jira/browse/IGNITE-10
--Yakov
2015-11-13 2:40 GMT+03:00 Andrey Kornev <an...@hotmail.com>:
> There you go:
> https://issues.apache.org/jira/browse/IGNITE-1898
> and
> https://issues.apache.org/jira/browse/IGNITE-1899
>
> Cheers
> Andrey
>
> > From: dsetrakyan@apache.org
> > Date: Thu, 12 Nov 2015 12:53:52 -0800
> > Subject: Re: Custom ThreadFactory
> > To: dev@ignite.apache.org
> >
> > On Thu, Nov 12, 2015 at 10:26 AM, Andrey Kornev <
> andrewkornev@hotmail.com>
> > wrote:
> >
> > > Even better, Ignite might provide out-of-the-box access to the local
> > > instance via a
> > > thread local. It could be in a form of a static public
> > > method on the Ignition class.
> > >
> > > Ignite itself could benefit from
> > > this feature as it does get it wrong occasionally. A good example of
> > > this is the ClusterGroupAdapter class or any other class that
> serializes
> > > the instance of IgniteKernal. Imagine the situation where you have a
> > > single JVM with multiple Ignite nodes started -- Ignite requires the
> > > grid names to be different. But since only the name of the grid is
> > > serialized, during deserialization an invalid (unexpected, to put it
> > > mildly) instance of IgniteKernal is looked up. I don't know how serious
> > > it is, but it is probably a bug.
> > >
> >
> > Andrey, I think its best to summarize the design thoughts in a ticket.
> Then
> > someone from the community will pick it up.
> >
> >
> > >
> > > Regards
> > > Andrey
> > >
> > > > Date: Thu, 12 Nov 2015 21:11:38 +0300
> > > > Subject: Re: Custom ThreadFactory
> > > > From: vozerov@gridgain.com
> > > > To: dev@ignite.apache.org
> > > >
> > > > I would avoid running any external payloads in public pool because it
> > > could
> > > > unpredictably affect Ignite internals. "Public" doesn't mean "opened
> for
> > > > everyone" here.
> > > >
> > > > On the other hand, I abosuletly agree that removing possibility to
> > > > configure custom pools was not very good idea. I do not see any
> problems
> > > > with returning it back while still keeping "thread count" property
> for
> > > the
> > > > most common use case when custom pool is not needed/
> > > >
> > > > On Thu, Nov 12, 2015 at 9:02 PM, Andrey Kornev <
> andrewkornev@hotmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > If my memory doesn't fail me, in the pre-Ignite versions of
> GridGain,
> > > it
> > > > > was possible to configure custom executor services which would
> then be
> > > used
> > > > > to create the public, system, utility, etc. thread pools. In Ignite
> > > however
> > > > > it's only possible to configure the size of the thread pools and
> not
> > > their
> > > > > implementations.
> > > > >
> > > > > This is unfortunate as I'd like to be able to configure my own
> > > > > ThreadFactory. My implementation would for example ensure that
> newly
> > > > > created threads have their thread locals properly initialized (for
> > > example,
> > > > > by storing the local instance of Ignite in it). Specific use case
> is
> > > being
> > > > > able to get hold of the local Ignite instance during
> deserialization
> > > when
> > > > > the JVM instance has multiple Ignite nodes started. Some of my
> classes
> > > must
> > > > > be able to access resources that are local to the node on which
> they
> > > are
> > > > > being deserialized. At the moment there is absolutely no way of
> > > achieving
> > > > > something like that.
> > > > >
> > > > > I'm wondering if it would be possible to add this feature back to
> > > Ignite?
> > > > > It seems to be indispensable for unit testing.
> > > > >
> > > > > Alternatively, to reduce the impact on the public API, an
> environment
> > > > > variable that takes an FQN of the ThreadFactory to use would also
> > > work. It
> > > > > would be injectable with the Ignite resources in the manner
> similar to
> > > how
> > > > > it's done for the closures and factories...
> > > > >
> > > > > Regards
> > > > > Andrey
> > > > >
> > > > > PS. While we're at it, I also remember that in the pre-Ignite
> versions
> > > it
> > > > > was possible to inject an instance of the public executor service
> into
> > > the
> > > > > closures. Not anymore. It causes the inconvenience of starting
> another
> > > > > thread pool while there is already a public pool managed by Ignite
> with
> > > > > plenty of threads idling most of the time... It feels wasteful.
> > > > >
> > >
> > >
>
>
RE: Custom ThreadFactory
Posted by Andrey Kornev <an...@hotmail.com>.
There you go:
https://issues.apache.org/jira/browse/IGNITE-1898
and
https://issues.apache.org/jira/browse/IGNITE-1899
Cheers
Andrey
> From: dsetrakyan@apache.org
> Date: Thu, 12 Nov 2015 12:53:52 -0800
> Subject: Re: Custom ThreadFactory
> To: dev@ignite.apache.org
>
> On Thu, Nov 12, 2015 at 10:26 AM, Andrey Kornev <an...@hotmail.com>
> wrote:
>
> > Even better, Ignite might provide out-of-the-box access to the local
> > instance via a
> > thread local. It could be in a form of a static public
> > method on the Ignition class.
> >
> > Ignite itself could benefit from
> > this feature as it does get it wrong occasionally. A good example of
> > this is the ClusterGroupAdapter class or any other class that serializes
> > the instance of IgniteKernal. Imagine the situation where you have a
> > single JVM with multiple Ignite nodes started -- Ignite requires the
> > grid names to be different. But since only the name of the grid is
> > serialized, during deserialization an invalid (unexpected, to put it
> > mildly) instance of IgniteKernal is looked up. I don't know how serious
> > it is, but it is probably a bug.
> >
>
> Andrey, I think its best to summarize the design thoughts in a ticket. Then
> someone from the community will pick it up.
>
>
> >
> > Regards
> > Andrey
> >
> > > Date: Thu, 12 Nov 2015 21:11:38 +0300
> > > Subject: Re: Custom ThreadFactory
> > > From: vozerov@gridgain.com
> > > To: dev@ignite.apache.org
> > >
> > > I would avoid running any external payloads in public pool because it
> > could
> > > unpredictably affect Ignite internals. "Public" doesn't mean "opened for
> > > everyone" here.
> > >
> > > On the other hand, I abosuletly agree that removing possibility to
> > > configure custom pools was not very good idea. I do not see any problems
> > > with returning it back while still keeping "thread count" property for
> > the
> > > most common use case when custom pool is not needed/
> > >
> > > On Thu, Nov 12, 2015 at 9:02 PM, Andrey Kornev <andrewkornev@hotmail.com
> > >
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > If my memory doesn't fail me, in the pre-Ignite versions of GridGain,
> > it
> > > > was possible to configure custom executor services which would then be
> > used
> > > > to create the public, system, utility, etc. thread pools. In Ignite
> > however
> > > > it's only possible to configure the size of the thread pools and not
> > their
> > > > implementations.
> > > >
> > > > This is unfortunate as I'd like to be able to configure my own
> > > > ThreadFactory. My implementation would for example ensure that newly
> > > > created threads have their thread locals properly initialized (for
> > example,
> > > > by storing the local instance of Ignite in it). Specific use case is
> > being
> > > > able to get hold of the local Ignite instance during deserialization
> > when
> > > > the JVM instance has multiple Ignite nodes started. Some of my classes
> > must
> > > > be able to access resources that are local to the node on which they
> > are
> > > > being deserialized. At the moment there is absolutely no way of
> > achieving
> > > > something like that.
> > > >
> > > > I'm wondering if it would be possible to add this feature back to
> > Ignite?
> > > > It seems to be indispensable for unit testing.
> > > >
> > > > Alternatively, to reduce the impact on the public API, an environment
> > > > variable that takes an FQN of the ThreadFactory to use would also
> > work. It
> > > > would be injectable with the Ignite resources in the manner similar to
> > how
> > > > it's done for the closures and factories...
> > > >
> > > > Regards
> > > > Andrey
> > > >
> > > > PS. While we're at it, I also remember that in the pre-Ignite versions
> > it
> > > > was possible to inject an instance of the public executor service into
> > the
> > > > closures. Not anymore. It causes the inconvenience of starting another
> > > > thread pool while there is already a public pool managed by Ignite with
> > > > plenty of threads idling most of the time... It feels wasteful.
> > > >
> >
> >
Re: Custom ThreadFactory
Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Thu, Nov 12, 2015 at 10:26 AM, Andrey Kornev <an...@hotmail.com>
wrote:
> Even better, Ignite might provide out-of-the-box access to the local
> instance via a
> thread local. It could be in a form of a static public
> method on the Ignition class.
>
> Ignite itself could benefit from
> this feature as it does get it wrong occasionally. A good example of
> this is the ClusterGroupAdapter class or any other class that serializes
> the instance of IgniteKernal. Imagine the situation where you have a
> single JVM with multiple Ignite nodes started -- Ignite requires the
> grid names to be different. But since only the name of the grid is
> serialized, during deserialization an invalid (unexpected, to put it
> mildly) instance of IgniteKernal is looked up. I don't know how serious
> it is, but it is probably a bug.
>
Andrey, I think its best to summarize the design thoughts in a ticket. Then
someone from the community will pick it up.
>
> Regards
> Andrey
>
> > Date: Thu, 12 Nov 2015 21:11:38 +0300
> > Subject: Re: Custom ThreadFactory
> > From: vozerov@gridgain.com
> > To: dev@ignite.apache.org
> >
> > I would avoid running any external payloads in public pool because it
> could
> > unpredictably affect Ignite internals. "Public" doesn't mean "opened for
> > everyone" here.
> >
> > On the other hand, I abosuletly agree that removing possibility to
> > configure custom pools was not very good idea. I do not see any problems
> > with returning it back while still keeping "thread count" property for
> the
> > most common use case when custom pool is not needed/
> >
> > On Thu, Nov 12, 2015 at 9:02 PM, Andrey Kornev <andrewkornev@hotmail.com
> >
> > wrote:
> >
> > > Hello,
> > >
> > > If my memory doesn't fail me, in the pre-Ignite versions of GridGain,
> it
> > > was possible to configure custom executor services which would then be
> used
> > > to create the public, system, utility, etc. thread pools. In Ignite
> however
> > > it's only possible to configure the size of the thread pools and not
> their
> > > implementations.
> > >
> > > This is unfortunate as I'd like to be able to configure my own
> > > ThreadFactory. My implementation would for example ensure that newly
> > > created threads have their thread locals properly initialized (for
> example,
> > > by storing the local instance of Ignite in it). Specific use case is
> being
> > > able to get hold of the local Ignite instance during deserialization
> when
> > > the JVM instance has multiple Ignite nodes started. Some of my classes
> must
> > > be able to access resources that are local to the node on which they
> are
> > > being deserialized. At the moment there is absolutely no way of
> achieving
> > > something like that.
> > >
> > > I'm wondering if it would be possible to add this feature back to
> Ignite?
> > > It seems to be indispensable for unit testing.
> > >
> > > Alternatively, to reduce the impact on the public API, an environment
> > > variable that takes an FQN of the ThreadFactory to use would also
> work. It
> > > would be injectable with the Ignite resources in the manner similar to
> how
> > > it's done for the closures and factories...
> > >
> > > Regards
> > > Andrey
> > >
> > > PS. While we're at it, I also remember that in the pre-Ignite versions
> it
> > > was possible to inject an instance of the public executor service into
> the
> > > closures. Not anymore. It causes the inconvenience of starting another
> > > thread pool while there is already a public pool managed by Ignite with
> > > plenty of threads idling most of the time... It feels wasteful.
> > >
>
>
RE: Custom ThreadFactory
Posted by Andrey Kornev <an...@hotmail.com>.
Even better, Ignite might provide out-of-the-box access to the local instance via a
thread local. It could be in a form of a static public
method on the Ignition class.
Ignite itself could benefit from
this feature as it does get it wrong occasionally. A good example of
this is the ClusterGroupAdapter class or any other class that serializes
the instance of IgniteKernal. Imagine the situation where you have a
single JVM with multiple Ignite nodes started -- Ignite requires the
grid names to be different. But since only the name of the grid is
serialized, during deserialization an invalid (unexpected, to put it
mildly) instance of IgniteKernal is looked up. I don't know how serious
it is, but it is probably a bug.
Regards
Andrey
> Date: Thu, 12 Nov 2015 21:11:38 +0300
> Subject: Re: Custom ThreadFactory
> From: vozerov@gridgain.com
> To: dev@ignite.apache.org
>
> I would avoid running any external payloads in public pool because it could
> unpredictably affect Ignite internals. "Public" doesn't mean "opened for
> everyone" here.
>
> On the other hand, I abosuletly agree that removing possibility to
> configure custom pools was not very good idea. I do not see any problems
> with returning it back while still keeping "thread count" property for the
> most common use case when custom pool is not needed/
>
> On Thu, Nov 12, 2015 at 9:02 PM, Andrey Kornev <an...@hotmail.com>
> wrote:
>
> > Hello,
> >
> > If my memory doesn't fail me, in the pre-Ignite versions of GridGain, it
> > was possible to configure custom executor services which would then be used
> > to create the public, system, utility, etc. thread pools. In Ignite however
> > it's only possible to configure the size of the thread pools and not their
> > implementations.
> >
> > This is unfortunate as I'd like to be able to configure my own
> > ThreadFactory. My implementation would for example ensure that newly
> > created threads have their thread locals properly initialized (for example,
> > by storing the local instance of Ignite in it). Specific use case is being
> > able to get hold of the local Ignite instance during deserialization when
> > the JVM instance has multiple Ignite nodes started. Some of my classes must
> > be able to access resources that are local to the node on which they are
> > being deserialized. At the moment there is absolutely no way of achieving
> > something like that.
> >
> > I'm wondering if it would be possible to add this feature back to Ignite?
> > It seems to be indispensable for unit testing.
> >
> > Alternatively, to reduce the impact on the public API, an environment
> > variable that takes an FQN of the ThreadFactory to use would also work. It
> > would be injectable with the Ignite resources in the manner similar to how
> > it's done for the closures and factories...
> >
> > Regards
> > Andrey
> >
> > PS. While we're at it, I also remember that in the pre-Ignite versions it
> > was possible to inject an instance of the public executor service into the
> > closures. Not anymore. It causes the inconvenience of starting another
> > thread pool while there is already a public pool managed by Ignite with
> > plenty of threads idling most of the time... It feels wasteful.
> >
Re: Custom ThreadFactory
Posted by Vladimir Ozerov <vo...@gridgain.com>.
I would avoid running any external payloads in public pool because it could
unpredictably affect Ignite internals. "Public" doesn't mean "opened for
everyone" here.
On the other hand, I abosuletly agree that removing possibility to
configure custom pools was not very good idea. I do not see any problems
with returning it back while still keeping "thread count" property for the
most common use case when custom pool is not needed/
On Thu, Nov 12, 2015 at 9:02 PM, Andrey Kornev <an...@hotmail.com>
wrote:
> Hello,
>
> If my memory doesn't fail me, in the pre-Ignite versions of GridGain, it
> was possible to configure custom executor services which would then be used
> to create the public, system, utility, etc. thread pools. In Ignite however
> it's only possible to configure the size of the thread pools and not their
> implementations.
>
> This is unfortunate as I'd like to be able to configure my own
> ThreadFactory. My implementation would for example ensure that newly
> created threads have their thread locals properly initialized (for example,
> by storing the local instance of Ignite in it). Specific use case is being
> able to get hold of the local Ignite instance during deserialization when
> the JVM instance has multiple Ignite nodes started. Some of my classes must
> be able to access resources that are local to the node on which they are
> being deserialized. At the moment there is absolutely no way of achieving
> something like that.
>
> I'm wondering if it would be possible to add this feature back to Ignite?
> It seems to be indispensable for unit testing.
>
> Alternatively, to reduce the impact on the public API, an environment
> variable that takes an FQN of the ThreadFactory to use would also work. It
> would be injectable with the Ignite resources in the manner similar to how
> it's done for the closures and factories...
>
> Regards
> Andrey
>
> PS. While we're at it, I also remember that in the pre-Ignite versions it
> was possible to inject an instance of the public executor service into the
> closures. Not anymore. It causes the inconvenience of starting another
> thread pool while there is already a public pool managed by Ignite with
> plenty of threads idling most of the time... It feels wasteful.
>