Getting an exception when listing partitions of IgniteDataFrame

[Ramakrishna Nalam <nr...@gmail.com>]

Hi,

Seeking help with someone on the group who has seen this error before, or
can help me figure out what it issue is.

I'm trying to get ignite's spark integration working with Spark 2.2.1.
The ignite version I'm using is 2.5.0. I'm running Ignite co-located with
the spark workers.

I have been able to successfully load data into Ignite using the snippet:

val igniteDf = spark.read .
    format(IgniteDataFrameSettings.FORMAT_IGNITE) .
    option(IgniteDataFrameSettings.OPTION_TABLE, "tmp_table") .
    option(IgniteDataFrameSettings.OPTION_CONFIG_FILE,
"/home/hadoop/ignite-config.xml") .
load;

the command:

igniteDf.select("_c1").show()

works as expected. Other simple commands / functions like count(), max()
etc. also work well.

But when I try:

igniteDf.rdd.partitions

There is an exception thrown, with the following stacktrace:

java.lang.IllegalArgumentException: requirement failed:
partitions(0).partition == 4, but it should equal 0
  at scala.Predef$.require(Predef.scala:224)
  at
org.apache.spark.rdd.RDD$$anonfun$partitions$2$$anonfun$apply$3.apply(RDD.scala:254)
  at
org.apache.spark.rdd.RDD$$anonfun$partitions$2$$anonfun$apply$3.apply(RDD.scala:253)
  at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
  at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
  at
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
  at
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
  at
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
  at
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
  ... 58 elided

It's a bit surprising, because looks like the code that is invoked is
the getPartitions
function of IgniteRDD
<https://github.com/apache/ignite/blob/master/modules/spark/src/main/scala/org/apache/ignite/spark/IgniteRDD.scala#L78>,
which should just return Array(IgnitePartition(0), IgnitePartition(1)...
and so on).

I have compiled ignite-spark module from github using the latest commit and
used that as well, but the same error persists.

Will be happy to provide any additional information that will help zero in
on the issue!


Thanks,
Rama.

Re: Getting an exception when listing partitions of IgniteDataFrame

[Ramakrishna Nalam <nr...@gmail.com>]

Hi Ray,

I could not find a solution to the problem.

I moved away from Ignite for now, so did not dig into it further.


Regards,
Rama.


On Wed, Aug 8, 2018 at 2:55 PM Ray <ra...@cisco.com> wrote:

> Hi Rama,
>
> Did you solve this problem?
> Please let me know your solution if you have solved this problem.
>
> Thanks
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Getting an exception when listing partitions of IgniteDataFrame

[Ray <ra...@cisco.com>]

Hi Rama,

Did you solve this problem?
Please let me know your solution if you have solved this problem.

Thanks



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Getting an exception when listing partitions of IgniteDataFrame

[akurbanov <an...@gmail.com>]

Hi,

Check out this answer and try to apply given advice, seems that it helped:
https://groups.google.com/forum/#!topic/bigdl-user-group/cZEBADHhhaE

>> /at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)/
That's Spark RDD, not Ignite.

Regards,
Anton



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/