Apache Spark with Ignite

[datta <ta...@gmail.com>]

Hi,
I am using ignite 2.5.0 with spark 2.2.0

i have setup single node of ignite with data region size of max 60gb and
initsize of 10gb with jvm heap size of 32g

I am streaming data from kafka and writing to ignite table.

is spark a good solution to write realtime to ignite or should i use the
datastreamer option of ignite.

also when i am using spark my master is yarn 12 clients node of ignite get
started with the spark workers.

since my job is running in streaming mode is it safe if so many clients
remain connected to ignite.

Also how do i safely shutdown a ignite cluster. currently i am using kill
-SIGTERM pid to shutdown the cluster.
but many this does not work and cluster remains running mode . 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Apache Spark with Ignite

[Denis Magda <dm...@apache.org>]

Hi, let me try to answer some of your questions.

>>> is spark a good solution to write realtime to ignite or should i use the
datastreamer option of ignite.

Yes, Spark Streaming should work fine for you as long as you still will be
using JavaSharedRdd.savePairs [1] method that uses IgniteDataStreamer
internally [2].

>>> also when i am using spark my master is yarn 12 clients node of ignite
get
started with the spark workers.

I don't see any issues with this. Anyway, each Spark worker needs to
establish a connection to the cluster.

Also how do i safely shutdown a ignite cluster. currently i am using kill
> -SIGTERM pid to shutdown the cluster.
> but many this does not work and cluster remains running mode .


You can use Ignite.close programmatically, use JMX Beans or GridGain
WebConsole to stop the cluster gradually.

[1]
https://github.com/dmagda/IgniteSparkIoT/blob/master/src/main/java/org/apache/ignite/iot/SparkStreamerStartup.java#L162
[2]
https://github.com/apache/ignite/blob/2e7553aa469a679c8a297e49f50b5464d9d76488/modules/spark/src/main/scala/org/apache/ignite/spark/IgniteRDD.scala#L232
-
Denis


On Fri, Feb 28, 2020 at 8:22 PM datta <ta...@gmail.com> wrote:

> Hi,
> I am using ignite 2.5.0 with spark 2.2.0
>
> i have setup single node of ignite with data region size of max 60gb and
> initsize of 10gb with jvm heap size of 32g
>
> I am streaming data from kafka and writing to ignite table.
>
> is spark a good solution to write realtime to ignite or should i use the
> datastreamer option of ignite.
>
> also when i am using spark my master is yarn 12 clients node of ignite get
> started with the spark workers.
>
> since my job is running in streaming mode is it safe if so many clients
> remain connected to ignite.
>
> Also how do i safely shutdown a ignite cluster. currently i am using kill
> -SIGTERM pid to shutdown the cluster.
> but many this does not work and cluster remains running mode .
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>