How to safely shut down an app with shared RDD without lossing data?

[baozipu <jo...@gmail.com>]

Hello, 

I wrote two spark + ignite app, with shared RDD. 

*The first Application app1, create the Shared RDD and savePairs 1:100: *

    val sharedRDD: IgniteRDD[Int, Int] = igniteContext.fromCache[Int,
Int]("sharedRDD")

    sharedRDD.savePairs(ss.sparkContext.parallelize(1 to 100, 64).map(i =>
(i, i)))


*The second application app2, get the Shared RDD and save Pairs 101:200:*

    val cacheRDD: IgniteRDD[Int, Int] = igniteContext.fromCache[Int,
Int]("sharedRDD")
   
    cacheRDD.savePairs(ss.sparkContext.parallelize(101 to 200, 64).map(i =>
(i, i)))


Now, after the app2 saved the pairs,  I want the app2 shut down itself.

I would like to know, after the app2 exits, if app1 holds all the values
from 1:200? 

is there any potential data loss? How to prevent it? Launching app2 as a
client instead of server? 






--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-to-safely-shut-down-an-app-with-shared-RDD-without-lossing-data-tp13012.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: How to safely shut down an app with shared RDD without lossing data?

[dkarachentsev <dk...@gridgain.com>]

Hi,

You're supplying function that creates IgniteConfiguration:
CreateConfiguration(), modify it that it will set client mode when need:
IgniteConfiguration.setClientMode().

-Dmitry



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-to-safely-shut-down-an-app-with-shared-RDD-without-lossing-data-tp13012p13088.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: How to safely shut down an app with shared RDD without lossing data?

[baozipu <jo...@gmail.com>]

Thanks a lot for your help. Now I get much clear.

Could you help me to understand what are the "client nodes". Do I need to
change configuration of the cluster, or I can simply set up "client nodes"
by changing codes within the spark application code? Currently, I am running
ignite with embedded mode in spark application.   

The only thing I know the different between server and client is by setting
ignite context:

*Server*
val igniteContext = new IgniteContext(ss.sparkContext, () =>
CreateConfiguration(), false)


*Client*
val igniteContext = new IgniteContext(ss.sparkContext, () =>
CreateConfiguration(), true)


What else do I need to do, in order to setup client node?



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-to-safely-shut-down-an-app-with-shared-RDD-without-lossing-data-tp13012p13031.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: How to safely shut down an app with shared RDD without lossing data?

[dkarachentsev <dk...@gridgain.com>]

Hi,

It's usually better approach to run client code on client nodes, because
they don't keep cache data and, as a consequence, shutdown or restart
doesn't change cache state. 

Another way is to configure cache with backups, f.e. in your case you have 2
server nodes, if you set CacheConfiguration.setBackups(1), each node
additionally will keep copies of data from another one, so killing one of
them doesn't hurt the data. But additionally you need to set
CacheConfiguration.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC)
to make sure that all backups synced after each cache update.

-Dmitry



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-to-safely-shut-down-an-app-with-shared-RDD-without-lossing-data-tp13012p13019.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.