Re: Threadpools and .WithExecute() for C# clients

[Pavel Tupitsyn <pt...@apache.org>]

Most probably - yes

On Fri, Jul 26, 2019 at 1:36 AM Raymond Wilson <ra...@trimble.com>
wrote:

> Thanks Pavel!
>
> Does the priority on the Jira ticket suggest this will target IA 2.8?
>
> Thanks,
> Raymond.
>
> On Wed, Jul 24, 2019 at 8:21 PM Pavel Tupitsyn <pt...@apache.org>
> wrote:
>
>> Denis, yes, looks like a simple thing to add.
>>
>> On Tue, Jul 23, 2019 at 10:38 PM Denis Magda <dm...@apache.org> wrote:
>>
>>> Looping in the dev list.
>>>
>>> Pavel, Igor and other C# maintainers, this looks like a valuable
>>> extension
>>> of our C# APIs. Shouldn't this be a quick addition to Ignite?
>>>
>>> -
>>> Denis
>>>
>>>
>>> On Mon, Jul 22, 2019 at 3:22 PM Raymond Wilson <
>>> raymond_wilson@trimble.com>
>>> wrote:
>>>
>>> > Alexandr,
>>> >
>>> > If .WithExecute is not planned to be made available in the C# client,
>>> what
>>> > is the plan to support custom thread pools from the C# side of things?
>>> >
>>> > Thanks,
>>> > Raymond.
>>> >
>>> >
>>> > On Thu, Jul 18, 2019 at 9:28 AM Raymond Wilson <
>>> raymond_wilson@trimble.com>
>>> > wrote:
>>> >
>>> >> The source of inbound requests into Server A is from client
>>> applications.
>>> >>
>>> >> Server B is really a cluster of servers that are performing clustered
>>> >> transformations and computations across a data set.
>>> >>
>>> >> I originally used IComputeJob and similar functions which work very
>>> well
>>> >> but have the restriction that they return the entire result set from a
>>> >> Server B node in a single response. These result sets can be large
>>> (100's
>>> >> of megabytes and larger), which makes life pretty hard for Server A
>>> if it
>>> >> has to field multiple incoming responses of this size. So, these
>>> types of
>>> >> requests progressively send responses back (using Ignite messaging) to
>>> >> Server A using the Ignite messaging fabric. As Server A receives each
>>> part
>>> >> of the overall response it processes it according the business rules
>>> >> relevant to the request.
>>> >>
>>> >> The cluster config and numbers of nodes are not really material to
>>> this.
>>> >>
>>> >> Raymond.
>>> >>
>>> >> On Thu, Jul 18, 2019 at 12:26 AM Alexandr Shapkin <le...@gmail.com>
>>> >> wrote:
>>> >>
>>> >>> Hi,
>>> >>>
>>> >>>
>>> >>>
>>> >>> Can you share a more detailed use case, please?
>>> >>>
>>> >>>
>>> >>>
>>> >>> Right now it’s not clear why do you need a messaging fabric.
>>> >>>
>>> >>> If you are interesting in a progress tracking, then you could try a
>>> >>> CacheAPI or QueryContinious, for example.
>>> >>>
>>> >>>
>>> >>>
>>> >>> What are the sources of inbound requests? Is it a client requests?
>>> >>>
>>> >>>
>>> >>>
>>> >>> What is your cluster config? How many nodes do you have for your
>>> >>> distributed computations?
>>> >>>
>>> >>>
>>> >>>
>>> >>> *From: *Raymond Wilson <ra...@trimble.com>
>>> >>> *Sent: *Wednesday, July 17, 2019 1:49 PM
>>> >>> *To: *user <us...@ignite.apache.org>
>>> >>> *Subject: *Re: Threadpools and .WithExecute() for C# clients
>>> >>>
>>> >>>
>>> >>>
>>> >>> Hi Alexandr,
>>> >>>
>>> >>>
>>> >>>
>>> >>> To summarise from the original thread, say I have server A that
>>> accepts
>>> >>> requests. It contacts server B in order to help processing those
>>> requests.
>>> >>> Server B sends in-progress results to server A using the Ignite
>>> messaging
>>> >>> fabric. If the thread pool in server A is saturated with inbound
>>> requests,
>>> >>> then there are no available threads to service the messaging fabric
>>> traffic
>>> >>> from server B to server A resulting in a deadlock condition.
>>> >>>
>>> >>>
>>> >>>
>>> >>> In the original discussion it was suggested creating a custom thread
>>> >>> pool to handle the Server B to Server A traffic would resolve it.
>>> >>>
>>> >>>
>>> >>>
>>> >>> Thanks,
>>> >>>
>>> >>> Raymond.
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Wed, Jul 17, 2019 at 9:48 PM Alexandr Shapkin <le...@gmail.com>
>>> >>> wrote:
>>> >>>
>>> >>> Hi, Raymond!
>>> >>>
>>> >>>
>>> >>>
>>> >>> As far as I can see, there are no plans for porting custom executors
>>> >>> configuration in .NET client right now [1].
>>> >>>
>>> >>>
>>> >>>
>>> >>> Please, remind, why do you need a separate pool instead of a default
>>> >>> PublicPool?
>>> >>>
>>> >>>
>>> >>>
>>> >>> [1] - https://issues.apache.org/jira/browse/IGNITE-6566
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> *From: *Raymond Wilson <ra...@trimble.com>
>>> >>> *Sent: *Wednesday, July 17, 2019 10:58 AM
>>> >>> *To: *user <us...@ignite.apache.org>
>>> >>> *Subject: *Threadpools and .WithExecute() for C# clients
>>> >>>
>>> >>>
>>> >>>
>>> >>> Some time ago I ran into and issue with thread pool exhaustion and
>>> >>> deadlocking in AI 2.2.
>>> >>>
>>> >>>
>>> >>>
>>> >>> This is the original thread:
>>> >>>
>>> http://apache-ignite-users.70518.x6.nabble.com/Possible-dead-lock-when-number-of-jobs-exceeds-thread-pool-td17262.html
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> At the time .WithExecutor() was not implemented in the C# client so
>>> >>> there was little option but to expand the size of the public thread
>>> pool
>>> >>> sufficiently to prevent the deadlocking.
>>> >>>
>>> >>>
>>> >>>
>>> >>> We have been revisiting this issue and see that .WithExecutor() is
>>> not
>>> >>> supported in the AI 2.7.5 client.
>>> >>>
>>> >>>
>>> >>>
>>> >>> Can this be supported in the C# client, or is there a workaround in
>>> the
>>> >>> .Net environment? that does not require this capability?
>>> >>>
>>> >>>
>>> >>>
>>> >>> Thanks,
>>> >>>
>>> >>> Raymond.
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>
>>>
>>