You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Valentin Kulichenko <va...@gmail.com> on 2017/11/30 23:07:43 UTC
Deprecate IgniteRDD in embedded mode
Igniters,
Currently we claim to support IgniteRDD in two modes: standalone and
embedded. Standalone means there is a separately running Ignite cluster,
and Spark start client node(s) to interact with it. In embedded node
everything runs within Spark, including Ignite server nodes that are
started embedded into Spark executors.
The latter case doesn't really work, mainly because the lifecycle of Spark
executors is not very predictable - Spark can start and stop them while
application is running. In case Ignite cluster is used to store data (which
is usually the case), this causes unnecessary rebalancing or even
unexpected data loss.
I propose to deprecate and eventually discontinue the embedded mode.
Luckily, standalone mode is the default one, so we can simply print out a
clear warning if one switches to embedded mode, and also mention this in
the docs.
Thoughts? If there are no objections, I will create a ticket and make the
change.
-Val
Re: Deprecate IgniteRDD in embedded mode
Posted by Holden Karau <ho...@pigscanfly.ca>.
So for what it's worth more and more of Spark's own services have also
moved to be in separate processes, and with the increased work around
scaling the executors are going to continue this trend.
On Thu, Nov 30, 2017 at 3:07 PM, Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> Igniters,
>
> Currently we claim to support IgniteRDD in two modes: standalone and
> embedded. Standalone means there is a separately running Ignite cluster,
> and Spark start client node(s) to interact with it. In embedded node
> everything runs within Spark, including Ignite server nodes that are
> started embedded into Spark executors.
>
> The latter case doesn't really work, mainly because the lifecycle of Spark
> executors is not very predictable - Spark can start and stop them while
> application is running. In case Ignite cluster is used to store data (which
> is usually the case), this causes unnecessary rebalancing or even
> unexpected data loss.
>
> I propose to deprecate and eventually discontinue the embedded mode.
> Luckily, standalone mode is the default one, so we can simply print out a
> clear warning if one switches to embedded mode, and also mention this in
> the docs.
>
> Thoughts? If there are no objections, I will create a ticket and make the
> change.
>
> -Val
>
--
Twitter: https://twitter.com/holdenkarau
Re: Deprecate IgniteRDD in embedded mode
Posted by Valentin Kulichenko <va...@gmail.com>.
Ticket created: https://issues.apache.org/jira/browse/IGNITE-7092
-Val
On Thu, Nov 30, 2017 at 5:01 PM, Denis Magda <dm...@apache.org> wrote:
> Val,
>
> Sounds reasonable to me. The fewer useless and potentially harmful
> features or “switches” we have in Ignite the clear it will be for the user
> how to use us in a right way.
>
> +1 for the deprecation and further removal.
>
> —
> Denis
>
> > On Nov 30, 2017, at 3:07 PM, Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
> >
> > Igniters,
> >
> > Currently we claim to support IgniteRDD in two modes: standalone and
> > embedded. Standalone means there is a separately running Ignite cluster,
> > and Spark start client node(s) to interact with it. In embedded node
> > everything runs within Spark, including Ignite server nodes that are
> > started embedded into Spark executors.
> >
> > The latter case doesn't really work, mainly because the lifecycle of
> Spark
> > executors is not very predictable - Spark can start and stop them while
> > application is running. In case Ignite cluster is used to store data
> (which
> > is usually the case), this causes unnecessary rebalancing or even
> > unexpected data loss.
> >
> > I propose to deprecate and eventually discontinue the embedded mode.
> > Luckily, standalone mode is the default one, so we can simply print out a
> > clear warning if one switches to embedded mode, and also mention this in
> > the docs.
> >
> > Thoughts? If there are no objections, I will create a ticket and make the
> > change.
> >
> > -Val
>
>
Re: Deprecate IgniteRDD in embedded mode
Posted by Denis Magda <dm...@apache.org>.
Val,
Sounds reasonable to me. The fewer useless and potentially harmful features or “switches” we have in Ignite the clear it will be for the user how to use us in a right way.
+1 for the deprecation and further removal.
—
Denis
> On Nov 30, 2017, at 3:07 PM, Valentin Kulichenko <va...@gmail.com> wrote:
>
> Igniters,
>
> Currently we claim to support IgniteRDD in two modes: standalone and
> embedded. Standalone means there is a separately running Ignite cluster,
> and Spark start client node(s) to interact with it. In embedded node
> everything runs within Spark, including Ignite server nodes that are
> started embedded into Spark executors.
>
> The latter case doesn't really work, mainly because the lifecycle of Spark
> executors is not very predictable - Spark can start and stop them while
> application is running. In case Ignite cluster is used to store data (which
> is usually the case), this causes unnecessary rebalancing or even
> unexpected data loss.
>
> I propose to deprecate and eventually discontinue the embedded mode.
> Luckily, standalone mode is the default one, so we can simply print out a
> clear warning if one switches to embedded mode, and also mention this in
> the docs.
>
> Thoughts? If there are no objections, I will create a ticket and make the
> change.
>
> -Val