[jira] [Comment Edited] (SPARK-15491) JSON serializations fails for JDBC DataFrames

["Marc Prud'hommeaux (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-15491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15297466#comment-15297466 ] 

Marc Prud'hommeaux edited comment on SPARK-15491 at 5/24/16 1:01 AM:
---------------------------------------------------------------------

I'm guessing the problem is coming from the JDBCRelation having a transient constructor argument:

{code}
private[sql] case class JDBCRelation(
    url: String,
    table: String,
    parts: Array[Partition],
    properties: Properties = new Properties())(@transient val sparkSession: SparkSession)
{code}

It appears that getConstructorParameterNames is returning the 5 constructor arguments, but the productIterator is returning only the 4 non-transient properties.

{code}
      val fieldNames = getConstructorParameterNames(p.getClass)
      val fieldValues = p.productIterator.toSeq
      assert(fieldNames.length == fieldValues.length)
{code}



was (Author: mprudhom):
I'm guessing the problem is coming from the JDBCRelation having a transient constructor argument:

{code}
private[sql] case class JDBCRelation(
    url: String,
    table: String,
    parts: Array[Partition],
    properties: Properties = new Properties())(@transient val sparkSession: SparkSession)
code}

It appears that getConstructorParameterNames is returning the 5 constructor arguments, but the productIterator is returning only the 4 non-transient properties.

{code}
      val fieldNames = getConstructorParameterNames(p.getClass)
      val fieldValues = p.productIterator.toSeq
      assert(fieldNames.length == fieldValues.length)
{code}


> JSON serializations fails for JDBC DataFrames
> ---------------------------------------------
>
>                 Key: SPARK-15491
>                 URL: https://issues.apache.org/jira/browse/SPARK-15491
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.0.0
>         Environment: MacOS 10.11.5, Spark 2.0.0-preview
>            Reporter: Marc Prud'hommeaux
>
> The TreeNode.toJSON feature implemented in SPARK-12321 fails with an assertion error on DataFrames that use JDBC in spark 2.0.0-preview:
> {code}
> scala> sqlContext.read.json("examples/src/main/resources/people.json").select("name", "age").agg(avg("age"), count("name")).filter(avg("age") > 10).queryExecution.logical.toJSON
> res113: String = [{"class":"org.apache.spark.sql.catalyst.plans.logical.Filter","num-children":1,"condition":[{"class":"org.apache.spark.sql.catalyst.expressions.GreaterThan","num-children":2,"left":0,"right":1},{"class":"org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression","num-children":1,...
> scala> sqlContext.read.format("jdbc").options(db + ("dbtable" -> "categories")).load().queryExecution.logical.simpleString
> res120: String = Relation[category#2148,categoryname#2149] JDBCRelation(categories)
> scala> sqlContext.read.format("jdbc").options(db + ("dbtable" -> "categories")).load().queryExecution.logical.toJSON
> java.lang.AssertionError: assertion failed
>   at scala.Predef$.assert(Predef.scala:156)
>   at org.apache.spark.sql.catalyst.trees.TreeNode.org$apache$spark$sql$catalyst$trees$TreeNode$$parseToJson(TreeNode.scala:598)
>   at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$jsonFields$2.apply(TreeNode.scala:562)
>   at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$jsonFields$2.apply(TreeNode.scala:553)
>   at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at scala.collection.immutable.List.foreach(List.scala:381)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
>   at scala.collection.immutable.List.map(List.scala:285)
>   at org.apache.spark.sql.catalyst.trees.TreeNode.jsonFields(TreeNode.scala:553)
>   at org.apache.spark.sql.catalyst.trees.TreeNode.org$apache$spark$sql$catalyst$trees$TreeNode$$collectJsonValue$1(TreeNode.scala:538)
>   at org.apache.spark.sql.catalyst.trees.TreeNode.jsonValue(TreeNode.scala:543)
>   at org.apache.spark.sql.catalyst.trees.TreeNode.toJSON(TreeNode.scala:529)
>   ... 48 elided
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org