You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Denis Magda <dm...@gridgain.com> on 2016/10/27 18:53:04 UTC
Making Ignite shmem library optional
Igniters,
Presently, if someone starts up a cluster and has at least two nodes running on a single Unix machine then those nodes will be communicating over the shared memory (shmem) by default.
This approach sounds absolutely reasonable for me but the shmem library is not ideal at the moment. I observed many situations when a cluster got stuck in the production or during long running tests due to some unclear issues in shmem internals. Even from Ignite community side we have the following shmem related issues
https://issues.apache.org/jira/browse/IGNITE-1578 <https://issues.apache.org/jira/browse/IGNITE-1578>
https://issues.apache.org/jira/browse/IGNITE-1294 <https://issues.apache.org/jira/browse/IGNITE-1294>
My proposal would be to make this library optional starting from the nearest Apache Ignite release. As far as I recall, it’s still mandatory because it's supposed that it speeds up our Spark and Hadoop integrations. But I would avoid using something that can accelerate a solution and fail at some unpredictable time.
Any thoughts or controversial arguments?
—
Denis
Re: Making Ignite shmem library optional
Posted by Valentin Kulichenko <va...@gmail.com>.
Absolutely agree. This functionality is unreliable right now and at the
very least it should be switched off by default as soon as possible. Going
forward, it should be either fixed or removed, because it's just a broken
piece of code.
-Val
On Thu, Oct 27, 2016 at 11:53 AM, Denis Magda <dm...@gridgain.com> wrote:
> Igniters,
>
> Presently, if someone starts up a cluster and has at least two nodes
> running on a single Unix machine then those nodes will be communicating
> over the shared memory (shmem) by default.
>
> This approach sounds absolutely reasonable for me but the shmem library is
> not ideal at the moment. I observed many situations when a cluster got
> stuck in the production or during long running tests due to some unclear
> issues in shmem internals. Even from Ignite community side we have the
> following shmem related issues
> https://issues.apache.org/jira/browse/IGNITE-1578 <
> https://issues.apache.org/jira/browse/IGNITE-1578>
> https://issues.apache.org/jira/browse/IGNITE-1294 <
> https://issues.apache.org/jira/browse/IGNITE-1294>
>
> My proposal would be to make this library optional starting from the
> nearest Apache Ignite release. As far as I recall, it’s still mandatory
> because it's supposed that it speeds up our Spark and Hadoop integrations.
> But I would avoid using something that can accelerate a solution and fail
> at some unpredictable time.
>
> Any thoughts or controversial arguments?
>
> —
> Denis
>
>
Re: Making Ignite shmem library optional
Posted by Denis Magda <dm...@gridgain.com>.
Alright, the ticket is ready to be picked up by someone
https://issues.apache.org/jira/browse/IGNITE-4152 <https://issues.apache.org/jira/browse/IGNITE-4152>
If you have something to add go to the ticket and comment there.
—
Denis
> On Oct 27, 2016, at 12:08 PM, Denis Magda <dm...@gridgain.com> wrote:
>
> Simply because it makes Ignite Spark integration unreliable as well.
>
> —
> Denis
>
>> On Oct 27, 2016, at 12:02 PM, Dmitriy Setrakyan <ds...@apache.org> wrote:
>>
>> Why not make it mandatory only with "ignite-spark" dependency?
>>
>> On Thu, Oct 27, 2016 at 11:53 AM, Denis Magda <dm...@gridgain.com> wrote:
>>
>>> Igniters,
>>>
>>> Presently, if someone starts up a cluster and has at least two nodes
>>> running on a single Unix machine then those nodes will be communicating
>>> over the shared memory (shmem) by default.
>>>
>>> This approach sounds absolutely reasonable for me but the shmem library is
>>> not ideal at the moment. I observed many situations when a cluster got
>>> stuck in the production or during long running tests due to some unclear
>>> issues in shmem internals. Even from Ignite community side we have the
>>> following shmem related issues
>>> https://issues.apache.org/jira/browse/IGNITE-1578 <
>>> https://issues.apache.org/jira/browse/IGNITE-1578>
>>> https://issues.apache.org/jira/browse/IGNITE-1294 <
>>> https://issues.apache.org/jira/browse/IGNITE-1294>
>>>
>>> My proposal would be to make this library optional starting from the
>>> nearest Apache Ignite release. As far as I recall, it’s still mandatory
>>> because it's supposed that it speeds up our Spark and Hadoop integrations.
>>> But I would avoid using something that can accelerate a solution and fail
>>> at some unpredictable time.
>>>
>>> Any thoughts or controversial arguments?
>>>
>>> —
>>> Denis
>>>
>>>
>
Re: Making Ignite shmem library optional
Posted by Denis Magda <dm...@gridgain.com>.
Simply because it makes Ignite Spark integration unreliable as well.
—
Denis
> On Oct 27, 2016, at 12:02 PM, Dmitriy Setrakyan <ds...@apache.org> wrote:
>
> Why not make it mandatory only with "ignite-spark" dependency?
>
> On Thu, Oct 27, 2016 at 11:53 AM, Denis Magda <dm...@gridgain.com> wrote:
>
>> Igniters,
>>
>> Presently, if someone starts up a cluster and has at least two nodes
>> running on a single Unix machine then those nodes will be communicating
>> over the shared memory (shmem) by default.
>>
>> This approach sounds absolutely reasonable for me but the shmem library is
>> not ideal at the moment. I observed many situations when a cluster got
>> stuck in the production or during long running tests due to some unclear
>> issues in shmem internals. Even from Ignite community side we have the
>> following shmem related issues
>> https://issues.apache.org/jira/browse/IGNITE-1578 <
>> https://issues.apache.org/jira/browse/IGNITE-1578>
>> https://issues.apache.org/jira/browse/IGNITE-1294 <
>> https://issues.apache.org/jira/browse/IGNITE-1294>
>>
>> My proposal would be to make this library optional starting from the
>> nearest Apache Ignite release. As far as I recall, it’s still mandatory
>> because it's supposed that it speeds up our Spark and Hadoop integrations.
>> But I would avoid using something that can accelerate a solution and fail
>> at some unpredictable time.
>>
>> Any thoughts or controversial arguments?
>>
>> —
>> Denis
>>
>>
Re: Making Ignite shmem library optional
Posted by Valentin Kulichenko <va...@gmail.com>.
Dmitry,
I don't think it's possible. Shared mode is supposed to help in the
standalone mode, but in this mode server nodes are not required to have
ignite-spark dependency. They just store data in caches.
-Val
On Thu, Oct 27, 2016 at 12:02 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> Why not make it mandatory only with "ignite-spark" dependency?
>
> On Thu, Oct 27, 2016 at 11:53 AM, Denis Magda <dm...@gridgain.com> wrote:
>
> > Igniters,
> >
> > Presently, if someone starts up a cluster and has at least two nodes
> > running on a single Unix machine then those nodes will be communicating
> > over the shared memory (shmem) by default.
> >
> > This approach sounds absolutely reasonable for me but the shmem library
> is
> > not ideal at the moment. I observed many situations when a cluster got
> > stuck in the production or during long running tests due to some unclear
> > issues in shmem internals. Even from Ignite community side we have the
> > following shmem related issues
> > https://issues.apache.org/jira/browse/IGNITE-1578 <
> > https://issues.apache.org/jira/browse/IGNITE-1578>
> > https://issues.apache.org/jira/browse/IGNITE-1294 <
> > https://issues.apache.org/jira/browse/IGNITE-1294>
> >
> > My proposal would be to make this library optional starting from the
> > nearest Apache Ignite release. As far as I recall, it’s still mandatory
> > because it's supposed that it speeds up our Spark and Hadoop
> integrations.
> > But I would avoid using something that can accelerate a solution and fail
> > at some unpredictable time.
> >
> > Any thoughts or controversial arguments?
> >
> > —
> > Denis
> >
> >
>
Re: Making Ignite shmem library optional
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Why not make it mandatory only with "ignite-spark" dependency?
On Thu, Oct 27, 2016 at 11:53 AM, Denis Magda <dm...@gridgain.com> wrote:
> Igniters,
>
> Presently, if someone starts up a cluster and has at least two nodes
> running on a single Unix machine then those nodes will be communicating
> over the shared memory (shmem) by default.
>
> This approach sounds absolutely reasonable for me but the shmem library is
> not ideal at the moment. I observed many situations when a cluster got
> stuck in the production or during long running tests due to some unclear
> issues in shmem internals. Even from Ignite community side we have the
> following shmem related issues
> https://issues.apache.org/jira/browse/IGNITE-1578 <
> https://issues.apache.org/jira/browse/IGNITE-1578>
> https://issues.apache.org/jira/browse/IGNITE-1294 <
> https://issues.apache.org/jira/browse/IGNITE-1294>
>
> My proposal would be to make this library optional starting from the
> nearest Apache Ignite release. As far as I recall, it’s still mandatory
> because it's supposed that it speeds up our Spark and Hadoop integrations.
> But I would avoid using something that can accelerate a solution and fail
> at some unpredictable time.
>
> Any thoughts or controversial arguments?
>
> —
> Denis
>
>