[jira] [Created] (SPARK-16924) DataStreamReader can not support option("inferSchema", true/false) for csv and json file source

["Xin Wu (JIRA)" <ji...@apache.org>]

Xin Wu created SPARK-16924:
------------------------------

             Summary: DataStreamReader can not support option("inferSchema", true/false) for csv and json file source
                 Key: SPARK-16924
                 URL: https://issues.apache.org/jira/browse/SPARK-16924
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.0.0
            Reporter: Xin Wu


Currently DataStreamReader can not support option("inferSchema", true|false) for csv and json file source. It only takes SQLConf setting "spark.sql.streaming.schemaInference", which needs to be set at session level. 

For example:
{code}
scala> val in = spark.readStream.format("json").option("inferSchema", true).load("/Users/xinwu/spark-test/data/json/t1")
java.lang.IllegalArgumentException: Schema must be specified when creating a streaming source DataFrame. If some files already exist in the directory, then depending on the file format you may be able to create a static DataFrame on that directory with 'spark.read.load(directory)' and infer schema from it.
  at org.apache.spark.sql.execution.datasources.DataSource.sourceSchema(DataSource.scala:223)
  at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo$lzycompute(DataSource.scala:80)
  at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo(DataSource.scala:80)
  at org.apache.spark.sql.execution.streaming.StreamingRelation$.apply(StreamingRelation.scala:30)
  at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:142)
  at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:153)
  ... 48 elided

scala> val in = spark.readStream.format("csv").option("inferSchema", true).load("/Users/xinwu/spark-test/data/csv")
java.lang.IllegalArgumentException: Schema must be specified when creating a streaming source DataFrame. If some files already exist in the directory, then depending on the file format you may be able to create a static DataFrame on that directory with 'spark.read.load(directory)' and infer schema from it.
  at org.apache.spark.sql.execution.datasources.DataSource.sourceSchema(DataSource.scala:223)
  at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo$lzycompute(DataSource.scala:80)
  at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo(DataSource.scala:80)
  at org.apache.spark.sql.execution.streaming.StreamingRelation$.apply(StreamingRelation.scala:30)
  at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:142)
  at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:153)
  ... 48 elided

{code}
In the example, even though users specify the option("inferSchema", true), it does not take it. But for batch data, DataFrameReader can take it:
{code}
scala> val in = spark.read.format("csv").option("header", true).option("inferSchema", true).load("/Users/xinwu/spark-test/data/csv1")
in: org.apache.spark.sql.DataFrame = [signal: string, flash: int]
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org