You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2017/03/07 16:29:38 UTC
[jira] [Assigned] (SPARK-19849) Support ArrayType in to_json
function/expression
[ https://issues.apache.org/jira/browse/SPARK-19849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-19849:
------------------------------------
Assignee: (was: Apache Spark)
> Support ArrayType in to_json function/expression
> ------------------------------------------------
>
> Key: SPARK-19849
> URL: https://issues.apache.org/jira/browse/SPARK-19849
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Hyukjin Kwon
>
> After SPARK-19595, we could
> {code}
> import org.apache.spark.sql.functions._
> import org.apache.spark.sql.types._
> val schema = ArrayType(StructType(StructField("a", IntegerType) :: Nil))
> Seq(("""[{"a": 1}, {"a": 2}]""")).toDF("array").select(from_json(col("array"), schema)).show()
> {code}
> Maybe, it'd be better if we provide a way to read it back via {{to_json}} as below:
> {code}
> import org.apache.spark.sql.functions._
> import org.apache.spark.sql.types._
> val schema = ArrayType(StructType(StructField("a", IntegerType) :: Nil))
> val df = Seq("""[{"a":1}, {"a":2}]""").toDF("json").select(from_json($"json", schema).as("array"))
> df.show()
> // Read back.
> df.select(to_json($"array").as("json")).show()
> {code}
> {code}
> +----------+
> | array|
> +----------+
> |[[1], [2]]|
> +----------+
> +-----------------+
> | json|
> +-----------------+
> |[{"a":1},{"a":2}]|
> +-----------------+
> {code}
> Currently, it throws an exception as below:
> {code}
> org.apache.spark.sql.AnalysisException: cannot resolve 'structtojson(`array`)' due to data type mismatch: structtojson requires that the expression is a struct expression.;;
> 'Project [structtojson(array#30, Some(Asia/Seoul)) AS structtojson(array)#45]
> +- Project [jsontostruct(ArrayType(StructType(StructField(a,IntegerType,true)),true), array#27, Some(Asia/Seoul)) AS array#30]
> +- Project [value#25 AS array#27]
> +- LocalRelation [value#25]
> at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
> at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:80)
> at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:72)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org