[jira] [Resolved] (SPARK-16743) converter and access code out of sync: createDataFrame on RDD[Option[C]] fails with MatchError

["Hyukjin Kwon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-16743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-16743.
----------------------------------
    Resolution: Incomplete

> converter and access code out of sync: createDataFrame on RDD[Option[C]] fails with MatchError
> ----------------------------------------------------------------------------------------------
>
>                 Key: SPARK-16743
>                 URL: https://issues.apache.org/jira/browse/SPARK-16743
>             Project: Spark
>          Issue Type: Improvement
>    Affects Versions: 1.6.2
>            Reporter: Daniel Barclay
>            Priority: Minor
>              Labels: bulk-closed
>
> Calling {{SqlContext}}'s {{createDataFrame}} on an RDD of type {{RDD\[Option\[SomeUserClass]]}} leads to an internal error.
> For example, if the first field of {{SomeUserClass}} is of type {{String}}, evaluating the RDD yields a {{MatchError}} referring an instance of {{SomeUserClass}} in  {{org.apache.spark.sql.catalyst.CatalystTypeConverters$StringConverter$.toCatalystImpl}} (which should have been passed only certain kinds of representations of strings).
> The problem seems to be in {{ExistingRDD.scala}}'s {{RDDConversions.productToRowAdd(...)}}:
> It has a list of converters that reflects the list of members of {{SomeUserClass}} (looking past the {{Option}} part of the RDD record type {{Option\[SomeUserClass]}}).
> However, the data-access code ({{r.productElement\(i)}}) does not seem to look past the {{Option}} part correspondingly.  (It does not seem to also traverse from the Some instance to the {{SomeUserClass)}}.)
> Therefore, it ends up passing the instance of {{SomeUserClass}} to the converter intended for the first member field of {{SomeUserClass}} (e.g., a String converter), yielding an internal error.
> (If {{RDD\[Option\[...]]}} doesn't make sense in the first place, it should be rejected with a "conscious" error rather than failing with an internal inconsistency.) 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org