IgniteRDD type mapping

[Taras Ledkov <tl...@gridgain.com>]

Igniters,

Lets discuss the IgniteRDD type mapping features.
You can see the problem discovery at the issue: 
https://issues.apache.org/jira/browse/IGNITE-3175

* Description:
The current implementation IgniteRDD supports only types from 
conjunction of the H2 types
wrappers & spark type set.

I've created the test to illustrate the problem with type mapping. The 
test try to select the non-primitive field from the IgniteRDD
(please take a look the pull request: 
https://github.com/apache/ignite/pull/784).
The exception scala.MatchError is thrown at the 
IgniteRDDTypeMappingTest.java:111

scala.MatchError : 
org.apache.ignite.spark.IgniteRDDTypeMappingTest$StructObj@68afa21 (of 
class org.apache.ignite.spark.IgniteRDDTypeMappingTest$StructObj)

* Background:
Please take a look at the H2 type mapping & the Spark type mapping specs:
http://www.h2database.com/html/datatypes.html
https://spark.apache.org/docs/1.3.0/sql-programming-guide.html#data-types

* Root cause:
The information about class of the TestObj.structVal field is lost in 
Ignite. IgniteRDD uses the
QueryCursor.fieldsMeta to discover type of the query fields. Ignite 
returns java.lang.Object in according with
H2 type mapping. But AFAIK Spark doesn't support plain java objects.

-- 
Taras Ledkov
Mail-To: tledkov@gridgain.com

Re: IgniteRDD type mapping

[Taras Ledkov <tl...@gridgain.com>]

Alexey,

According to the Spark type mapping StructType is mapped to the
org.apache.spark.sql.Row class. Please take a look at the Spark docs
(https://spark.apache.org/docs/1.3.0/sql-programming-guide.html#data-types)

or at the source code:
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala#L251

You filed the issue ignite-3077 against this.

I mistook  about the root cause. The information about the class of the 
object is helpless.

Spark supports only primitive types and some popular classes to 
represent decimal & dates, Map, List and the its own class the Row to 
represent complex objects.

09.06.2016 3:43, Alexey Goncharuk \u043f\u0438\u0448\u0435\u0442:
> Taras,
>
> According to IgniteRDD non-predefined types (including Object) are mapped
> to StructType with no fields, which should be recognizable by Spark. Can
> you explain why MatchError happens?
> \u200b
>

-- 
Taras Ledkov
Mail-To: tledkov@gridgain.com

Re: IgniteRDD type mapping

[Alexey Goncharuk <al...@gmail.com>]

Taras,

According to IgniteRDD non-predefined types (including Object) are mapped
to StructType with no fields, which should be recognizable by Spark. Can
you explain why MatchError happens?
​