[GitHub] HyukjinKwon commented on issue #20125: [SPARK-17967][SQL] Support for array as an option in SQL parser

[GitBox <gi...@apache.org>]

HyukjinKwon commented on issue #20125: [SPARK-17967][SQL] Support for array as an option in SQL parser
URL: https://github.com/apache/spark/pull/20125#issuecomment-449186094
 
 
   1.. Any behavior change in the parser?
   
   I believe there's no behaviour changes since option clause itself does not support `[` and ` ]` tokens:
   
   ```
   CREATE TEMPORARY TABLE tableA USING csv
   OPTIONS (nullValue [2012, 1.1, 'null'], ...)
   ```
   
   Currently, option value takes, string`, integer, decimals, and bools. I believe it's not ambiguous or it doesn't introduce a behaviour change in our parser.
   
   2.. What is the corresponding interface for DataFrameReader and DataFrameWriter APIs?
   
   I wsa thinking about the interfaces as below:
   
   Scala - `Seq[String]`
   
   ```
   spark.read.format("csv")
     .option("nullValue", Seq("2012", "Tesla", "null"))
     ...
   ```
   
   Java - `String[]`
   
   ```
   spark.read().format("csv")
     .option("nullValue", new String[]{"", "null", "NA"})
     ...
   ```
   
   Previous PR includes that APIs https://github.com/apache/spark/pull/16611/files
   
   
   One concern is that:
   
   ```
   OPTIONS (nullValue "[2012, 1.1, 'null']", ...)
   ```
   
   ```
   option("[2012, 1.1, 'null']")
   ```
   
   could work in the same way .. which is a bit ugly.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org