[jira] [Commented] (SPARK-18246) Throws an exception before execution for unsupported types in Json, CSV and text functionailities

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-18246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15632343#comment-15632343 ] 

Apache Spark commented on SPARK-18246:
--------------------------------------

User 'HyukjinKwon' has created a pull request for this issue:
https://github.com/apache/spark/pull/15751

> Throws an exception before execution for unsupported types in Json, CSV and text functionailities
> -------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-18246
>                 URL: https://issues.apache.org/jira/browse/SPARK-18246
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Hyukjin Kwon
>
> * Case 1 - {{read.json(rdd)}}
> {code}
> val rdd = spark.sparkContext.parallelize(1 to 100).map(i => s"""{"a": "str$i"}""")
> val schema = new StructType().add("a", CalendarIntervalType)
> spark.read.schema(schema).option("mode", "FAILFAST").json(rdd).show()
> {code}
> should throw an exception before the execution.
> * Case 2 - {{read.json(path}}
> {code}
> val path = "/tmp/a"
> val rdd = spark.sparkContext.parallelize(1 to 100).map(i => s"""{"a": "str$i"}""").saveAsTextFile(path)
> val schema = new StructType().add("a", CalendarIntervalType)
> spark.read.schema(schema).option("mode", "FAILFAST").json(path).show()
> {code}
> should throw an exception before the execution.
> * Case 3 - {{read.csv(path)}}
> {code}
> val path = "/tmp/b"
> val rdd = spark.sparkContext.parallelize(1 to 100).saveAsTextFile(path)
> val schema = new StructType().add("a", CalendarIntervalType)
> spark.read.schema(schema).option("mode", "FAILFAST").csv(path).show()
> {code}
> should throw an exception before the execution.
> * Case 4 - {{read.text(path)}}
> {code}
> val path = "/tmp/c"
> val rdd = spark.sparkContext.parallelize(1 to 100).saveAsTextFile(path)
> val schema = new StructType().add("a", LongType)
> spark.read.schema(schema).text(path).show()
> {code}
> should throw an exception before the execution rather than printing incorrect values.
> {code}
> +-----------+
> |          a|
> +-----------+
> |68719476738|
> |68719476738|
> |68719476738|
> |68719476738|
> |68719476738|
> |68719476738|
> |68719476738|
> |68719476738|
> |68719476738|
> |68719476738|
> |68719476738|
> |68719476738|
> |68719476739|
> |68719476738|
> |68719476738|
> |68719476738|
> |68719476738|
> |68719476738|
> |68719476738|
> |68719476738|
> +-----------+
> {code}
> * Case 5 - {{from_json}}
> {code}
> import org.apache.spark.sql.types._
> import org.apache.spark.sql.functions._
> import spark.implicits._
> val df = Seq("""{"a" 1}""").toDS()
> val schema = new StructType().add("a", CalendarIntervalType)
> df.select(from_json($"value", schema)).show()
> {code}
> prints
> {code}
> +-------------------+
> |jsontostruct(value)|
> +-------------------+
> |               null|
> +-------------------+
> {code}
> This should throw analysis exception as {{CalendarIntervalType}} is not supported.
> Likewise {{to_json}} throws an analysis error, for example,
> {code}
> val df = Seq(Tuple1(Tuple1("interval -3 month 7 hours"))).toDF("a")
>   .select(struct($"a._1".cast(CalendarIntervalType).as("a")).as("c"))
> df.select(to_json($"c")).collect()
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org