You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Mohit Jaggi (JIRA)" <ji...@apache.org> on 2016/01/16 22:11:39 UTC
[jira] [Commented] (SPARK-12669) Organize options for default
values
[ https://issues.apache.org/jira/browse/SPARK-12669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15103441#comment-15103441 ]
Mohit Jaggi commented on SPARK-12669:
-------------------------------------
Based on my experience working with CSV files, I think the following set of options make sense. What do people think? Also, what is a good way to organize these options? I like https://github.com/typesafehub/config
refer: https://github.com/databricks/spark-csv/pull/124/files
Option by Categories:
1. Line parsing Options
a. Bad line handling: skip the line, fail completely or repair the line
b. Line repairing methods: fill with "filler value" which can be configured per data type
2. Real Number parsing
There are defaults that can be overridden or augmented
a. NaN value: default "NaN", "Double.NaN"
b. Infinity: default "Inf"
c. -Infinity: default "-Inf"
d. nulls: default "null"
3. Integer Parsing
a. nulls: default "null"
4. String Parsing
a. nulls: default "null"
b. empty strings: default ""
5. Formatting
a. field delimiter: default comma
b. record delimiter: default new line...due to Hadoop Input Format's behavior we probably can't allow arbitrary record delimiters
c. escape character: default backslash
d. quote character: default quote
e. ignore leading white space: default true
f. ignore trailing white space: default true
> Organize options for default values
> -----------------------------------
>
> Key: SPARK-12669
> URL: https://issues.apache.org/jira/browse/SPARK-12669
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Hossein Falaki
>
> CSV data source in SparkSQL should be able to differentiate empty string, null, NaN, “N/A” (maybe data type dependent).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org