You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@ignite.apache.org by percent620 <pe...@163.com> on 2016/08/11 05:09:11 UTC

Embedded mode ignite on spark

Hello, I want to integrate ignite with spark on Embedded mode , and here is
my detailed steps for this.

1、added maven dependencies for pom.xml
 
        <dependency>
            <groupId>org.apache.ignite</groupId>
            <artifactId>ignite-core</artifactId>
            <version>1.6.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.ignite</groupId>
            <artifactId>ignite-indexing</artifactId>
            <version>1.6.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.ignite</groupId>
            <artifactId>ignite-visor-console</artifactId>
            <version>1.6.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.ignite</groupId>
            <artifactId>ignite-spring</artifactId>
            <version>1.6.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.ignite</groupId>
            <artifactId>ignite-spark</artifactId>
            <version>1.6.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.ignite</groupId>
            <artifactId>ignite-yarn</artifactId>
            <version>1.6.0</version>
        </dependency>

2、Write spark code and submit these code to yarn application, and the code
is 
val igniteContext = new IgniteContext[String, BaseLine](sc,() => new
IgniteConfiguration(),false)

false: is embedded mode for ignite, right?
tempDStream.foreachRDD(rdd =>{
      val rddCounter = rdd.count().toInt
totallyCounter += rddCounter
println(DateUtil.getNowDate() + " invoked! count=" + rddCounter +
";totallyCounter= => " + totallyCounter)
if(!rdd.isEmpty()) {
        //==================================start======
        val cacheRdd = igniteContext.fromCache("partitioned")

       //fianlLatestBaselineRDD: total rdd that need to cache on ignite
       cacheRdd.savePairs(fianlLatestBaselineRDD)

println("xx=> " + xxx() +"\t " +
          " xxx => " + xxx() + "\t " +
          " cacheRdd.counter=> " + cacheRdd.count())
}
      


}
3、the code is running on yarn successfully and I found that the ignite can't
cache all the datas,and it discard some datas, such as
2016-08-11 11:28:00:es write invoked! count=10;totally totallyCounter= => 10
cacheRdd.start.counter=> 0                                           
2、cacheRdd.start.fianlLatestBaselineRDD.counter=> 10                   
xxx.count=> 140       yyy.count => 10	  cacheRdd.counter=> 5(need to be 10)
=======================================================================================================================
2016-08-11 11:28:48:write invoked! count= 9;totallyCounter= => 19
cacheRdd.start.counter=> 5                                           
xxxx.count=> 126       yyy.count => 9	  cacheRdd.counter=> 9(need to be 19)
=========================================================================================================================


Can anyone help me on this issues? thanks again!!!!





--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Embedded-mode-ignite-on-spark-tp6942.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.