You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2016/05/23 08:05:13 UTC
[jira] [Assigned] (SPARK-15476) Support for reading text data
source without specifying schema
[ https://issues.apache.org/jira/browse/SPARK-15476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-15476:
------------------------------------
Assignee: (was: Apache Spark)
> Support for reading text data source without specifying schema
> --------------------------------------------------------------
>
> Key: SPARK-15476
> URL: https://issues.apache.org/jira/browse/SPARK-15476
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Hyukjin Kwon
> Priority: Minor
>
> Currently, Text data source requires a schema.
> So the codes below:
> {code}
> emptyDf.write
> .format("text")
> .save(path.getCanonicalPath)
> val copyEmptyDf = spark.read
> .format("text")
> .load(path.getCanonicalPath)
> {code}
> throws an exception below:
> {code}
> key not found: dataSchema
> java.util.NoSuchElementException: key not found: dataSchema
> at scala.collection.MapLike$class.default(MapLike.scala:228)
> at org.apache.spark.sql.execution.datasources.CaseInsensitiveMap.default(ddl.scala:139)
> at scala.collection.MapLike$class.apply(MapLike.scala:141)
> at org.apache.spark.sql.execution.datasources.CaseInsensitiveMap.apply(ddl.scala:139)
> at org.apache.spark.sql.sources.SimpleTextSource.inferSchema(SimpleTextRelation.scala:43)
> at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$15.apply(DataSource.scala:347)
> at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$15.apply(DataSource.scala:347)
> at scala.Option.orElse(Option.scala:289)
> {code}
> It might be nicer if this option is not required. CSV is doing this for CSV data which do not have header or empty string for fields in header.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org