You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2016/04/22 07:32:12 UTC
[jira] [Updated] (SPARK-14839) Support for other types as option in
OPTIONS clause
[ https://issues.apache.org/jira/browse/SPARK-14839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-14839:
---------------------------------
Description:
This was found in https://github.com/apache/spark/pull/12494.
Currently, Spark SQL does not support other types and {{null}} as a value of an options.
For example,
{code}
CREATE ...
USING csv
OPTIONS (path "your-path", quote null)
{code}
throws an exception below
{code}
Unsupported SQL statement
== SQL ==
CREATE TEMPORARY TABLE carsTable (yearMade double, makeName string, modelName string, comments string, grp string) USING csv OPTIONS (path "your-path", quote null)
org.apache.spark.sql.catalyst.parser.ParseException:
Unsupported SQL statement
== SQL ==
CREATE TEMPORARY TABLE carsTable (yearMade double, makeName string, modelName string, comments string, grp string) USING csv OPTIONS (path "your-path", quote null)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.nativeCommand(ParseDriver.scala:66)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:56)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:53)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:86)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:53)
at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:195)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:764)
...
{code}
Currently, Scala API supports to take options with the types, {{String}}, {{Long}}, {{Double}} and {{Boolean}} and Python API also supports other types. I think in this way we can support data sources in a consistent way.
It looks it is okay to to provide other types as arguments just like [Microsoft SQL|https://msdn.microsoft.com/en-us/library/ms190322.aspx] because [SQL-1992|http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt] standard mentions options as below:
{quote}
An implementation remains conforming even if it provides user op-
tions to process nonconforming SQL language or to process conform-
ing SQL language in a nonconforming manner.
{quote}
was:
This was found in https://github.com/apache/spark/pull/12494.
Currently, Spark SQL does not support other types and {{null}} as a value of an options.
For example,
{code}
...
CREATE ...
USING csv
OPTIONS (path "your-path", quote null)
{code}
throws an exception below
{code}
Unsupported SQL statement
== SQL ==
CREATE TEMPORARY TABLE carsTable (yearMade double, makeName string, modelName string, comments string, grp string) USING csv OPTIONS (path "your-path", quote null)
org.apache.spark.sql.catalyst.parser.ParseException:
Unsupported SQL statement
== SQL ==
CREATE TEMPORARY TABLE carsTable (yearMade double, makeName string, modelName string, comments string, grp string) USING csv OPTIONS (path "your-path", quote null)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.nativeCommand(ParseDriver.scala:66)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:56)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:53)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:86)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:53)
at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:195)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:764)
...
{code}
Currently, Scala API supports to take options with the types, {{String}}, {{Long}}, {{Double}} and {{Boolean}} and Python API also supports other types. I think in this way we can support data sources in a consistent way.
It looks it is okay to to provide other types as arguments just like [Microsoft SQL|https://msdn.microsoft.com/en-us/library/ms190322.aspx] because [SQL-1992|http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt] standard mentions options as below:
{quote}
An implementation remains conforming even if it provides user op-
tions to process nonconforming SQL language or to process conform-
ing SQL language in a nonconforming manner.
{quote}
> Support for other types as option in OPTIONS clause
> ---------------------------------------------------
>
> Key: SPARK-14839
> URL: https://issues.apache.org/jira/browse/SPARK-14839
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Hyukjin Kwon
> Priority: Minor
>
> This was found in https://github.com/apache/spark/pull/12494.
> Currently, Spark SQL does not support other types and {{null}} as a value of an options.
> For example,
> {code}
> CREATE ...
> USING csv
> OPTIONS (path "your-path", quote null)
> {code}
> throws an exception below
> {code}
> Unsupported SQL statement
> == SQL ==
> CREATE TEMPORARY TABLE carsTable (yearMade double, makeName string, modelName string, comments string, grp string) USING csv OPTIONS (path "your-path", quote null)
> org.apache.spark.sql.catalyst.parser.ParseException:
> Unsupported SQL statement
> == SQL ==
> CREATE TEMPORARY TABLE carsTable (yearMade double, makeName string, modelName string, comments string, grp string) USING csv OPTIONS (path "your-path", quote null)
> at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.nativeCommand(ParseDriver.scala:66)
> at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:56)
> at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:53)
> at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:86)
> at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:53)
> at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:195)
> at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:764)
> ...
> {code}
> Currently, Scala API supports to take options with the types, {{String}}, {{Long}}, {{Double}} and {{Boolean}} and Python API also supports other types. I think in this way we can support data sources in a consistent way.
> It looks it is okay to to provide other types as arguments just like [Microsoft SQL|https://msdn.microsoft.com/en-us/library/ms190322.aspx] because [SQL-1992|http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt] standard mentions options as below:
> {quote}
> An implementation remains conforming even if it provides user op-
> tions to process nonconforming SQL language or to process conform-
> ing SQL language in a nonconforming manner.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org