You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by HyukjinKwon <gi...@git.apache.org> on 2016/05/01 03:33:51 UTC
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/12817
[SPARK-13425][SQL] Documentation for CSV datasource options
## What changes were proposed in this pull request?
This PR adds the explanation and documentation for CSV options for reading and writing.
## How was this patch tested?
Style tests with `./build/sbt scalastyle`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark SPARK-13425
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12817.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12817
----
commit 4471e2a40a1cb8a6815d075fe65a410af08f9083
Author: hyukjinkwon <gu...@gmail.com>
Date: 2016-05-01T03:31:20Z
Add CSV documentation
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216025375
**[Test build #57471 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57471/consoleFull)** for PR 12817 at commit [`8201f23`](https://github.com/apache/spark/commit/8201f234b4d3563239e59f900ce48812cc198c0e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216025415
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57471/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216040991
**[Test build #57479 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57479/consoleFull)** for PR 12817 at commit [`54f58d3`](https://github.com/apache/spark/commit/54f58d381b47762543278fffa33fb5fd13f64b91).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216021442
**[Test build #57471 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57471/consoleFull)** for PR 12817 at commit [`8201f23`](https://github.com/apache/spark/commit/8201f234b4d3563239e59f900ce48812cc198c0e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216015221
@rxin BTW, I found two todos, `TODO: Remove this one in Spark 2.0.` at `DataFrameReader` and `DataFrameWriter` added in https://github.com/apache/spark/pull/9945. Do you mind if I submit another PR to do this? (or I can do that in this PR if this does not break any tests)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216012673
**[Test build #57463 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57463/consoleFull)** for PR 12817 at commit [`4471e2a`](https://github.com/apache/spark/commit/4471e2a40a1cb8a6815d075fe65a410af08f9083).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216084433
@rxin I am so sorry, I think I totally misunderstood your initial comments before. I just addressed your comments later. Thank you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216015018
**[Test build #57466 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57466/consoleFull)** for PR 12817 at commit [`34b52fa`](https://github.com/apache/spark/commit/34b52fa8ae942d5f84049fa7c788a761b459d973).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216041028
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216014925
cc @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216036699
**[Test build #57479 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57479/consoleFull)** for PR 12817 at commit [`54f58d3`](https://github.com/apache/spark/commit/54f58d381b47762543278fffa33fb5fd13f64b91).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61691181
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -393,6 +393,45 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
* This function goes through the input once to determine the input schema. To avoid going
* through the entire data once, specify the schema explicitly using [[schema]].
*
+ * You can set the following CSV-specific options to deal with CSV files:
+ * <li>`delimiter` (default `,`): sets the single character as a delimiter for each
--- End diff --
sep is the main one isn't it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216084669
**[Test build #57491 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57491/consoleFull)** for PR 12817 at commit [`ab70b6d`](https://github.com/apache/spark/commit/ab70b6d05db5744fce3fecd37a28647fd6622411).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216092342
**[Test build #57491 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57491/consoleFull)** for PR 12817 at commit [`ab70b6d`](https://github.com/apache/spark/commit/ab70b6d05db5744fce3fecd37a28647fd6622411).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216012741
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61691200
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -282,6 +282,45 @@ def csv(self, paths):
:param paths: string, or list of strings, for input path(s).
+ You can set the following CSV-specific options to deal with CSV files:
+ * ``delimiter`` (default ``,``): sets the single character as a delimiter \
--- End diff --
sep
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61691203
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -282,6 +282,45 @@ def csv(self, paths):
:param paths: string, or list of strings, for input path(s).
+ You can set the following CSV-specific options to deal with CSV files:
+ * ``delimiter`` (default ``,``): sets the single character as a delimiter \
+ for each field and value.
+ * ``charset`` (default ``UTF-8``): decodes the CSV files by the given \
+ encoding type.
+ * ``quote`` (default ``"``): sets the single character used for escaping \
+ quoted values where the delimiter can be part of the value.
+ * ``escape`` (default ``\``): sets the single character used for escaping quotes \
+ inside an already quoted value.
+ * ``comment`` (default empty string): sets the single character used for skipping \
+ lines beginning with this character. By default, it is disabled.
+ * ``header`` (default ``false``): uses the first line as names of columns.
+ * ``ignoreLeadingWhiteSpace`` (default ``false``): defines whether or not leading \
+ whitespaces from values being read should be skipped.
+ * ``ignoreTrailingWhiteSpace`` (default ``false``): defines whether or not trailing \
+ whitespaces from values being read should be skipped.
+ * ``nullValue`` (default empty string): sets the string representation of a null value.
+ * ``nanValue`` (default ``NaN``): sets the string representation of a non-number \
+ value.
+ * ``positiveInf`` (default ``Inf``): sets the string representation of a positive \
+ infinity value.
+ * ``negativeInf`` (default ``-Inf``): sets the string representation of a negative \
+ infinity value.
+ * ``dateFormat`` (default ``null``): sets the string that indicates a date format. \
--- End diff --
null => None?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216013835
Hm.. It passes a local style test for Python and I did not edit `./python/pyspark/sql/__init__.py`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216012742
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57463/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61691201
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -282,6 +282,45 @@ def csv(self, paths):
:param paths: string, or list of strings, for input path(s).
+ You can set the following CSV-specific options to deal with CSV files:
+ * ``delimiter`` (default ``,``): sets the single character as a delimiter \
+ for each field and value.
+ * ``charset`` (default ``UTF-8``): decodes the CSV files by the given \
--- End diff --
encoding
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61679408
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -663,6 +700,18 @@ def csv(self, path, mode=None, compression=None):
known case-insensitive shorten names (none, bzip2, gzip, lz4,
snappy and deflate).
+ You can set the following CSV-specific options to deal with CSV files:
+ * ``sep`` or ``delimiter`` (default ``,``): sets the single character as a delimiter \
+ for each field and value.
+ * ``encoding`` or ``charset`` (default ``UTF-8``): decodes the CSV files by the given \
+ encoding type.
+ * ``quote`` (default ``"``): sets the single character used for escaping \
+ quoted values where the delimiter can be part of the value.
+ * ``escape`` (default ``\``): sets the single character used for escaping quotes \
+ inside an already quoted value.
+ * ``header`` (default ``false``): writes the names of columns as the first line.
+ * ``nullValue`` (default empty string): sets the string representation of a null value.
--- End diff --
I am less sure that `empty string` is okay.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216090565
Don't worry about it. We can just build it on top of this (we should still document them, just better as function arguments).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216017667
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216014991
**[Test build #57466 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57466/consoleFull)** for PR 12817 at commit [`34b52fa`](https://github.com/apache/spark/commit/34b52fa8ae942d5f84049fa7c788a761b459d973).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216014355
Hm.. this gives me a pass of Python style test at local.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61682274
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -393,6 +393,45 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
* This function goes through the input once to determine the input schema. To avoid going
* through the entire data once, specify the schema explicitly using [[schema]].
*
+ * You can set the following CSV-specific options to deal with CSV files:
+ * <li>`sep` or `delimiter` (default `,`): sets the single character as a delimiter for each
+ * field and value.</li>
+ * <li>`encoding` or `charset` (default `UTF-8`): decodes the CSV files by the given encoding
--- End diff --
Sure. Thank you. Let me correct them.
Also, I will add a document as soon as R csv API is added.
I tried this in https://github.com/apache/spark/pull/11457 but had to ask someone to do this as I am not really familiar with R.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61679234
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -393,6 +393,45 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
* This function goes through the input once to determine the input schema. To avoid going
* through the entire data once, specify the schema explicitly using [[schema]].
*
+ * You can set the following CSV-specific options to deal with CSV files:
+ * <li>`sep` or `delimiter` (default `,`): sets the single character as a delimiter for each
+ * field and value.</li>
+ * <li>`encoding` or `charset` (default `UTF-8`): decodes the CSV files by the given encoding
+ * type.</li>
+ * <li>`quote` (default `"`): sets the single character used for escaping quoted values where
+ * the delimiter can be part of the value.</li>
+ * <li>`escape` (default `\`): sets the single character used for escaping quotes inside
+ * an already quoted value.</li>
+ * <li>`comment` (default empty string): sets the single character used for skipping lines beginning
--- End diff --
I am less sure of `empty string` is okay.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216090207
Sure. Thank you. Do you want me to remove Python documentation here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216090703
@rxin I see. Thank you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12817
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216092409
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216012739
**[Test build #57463 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57463/consoleFull)** for PR 12817 at commit [`4471e2a`](https://github.com/apache/spark/commit/4471e2a40a1cb8a6815d075fe65a410af08f9083).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216013503
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216015022
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57466/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61691199
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -663,6 +702,19 @@ def csv(self, path, mode=None, compression=None):
known case-insensitive shorten names (none, bzip2, gzip, lz4,
snappy and deflate).
+ You can set the following CSV-specific options to deal with CSV files:
+ * ``delimiter`` (default ``,``): sets the single character as a delimiter \
--- End diff --
sep
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216025414
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216064461
Thanks - this is really close. Let's fix the minor issues and then we can merge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216092582
Thanks - merging in master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216017668
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57467/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216015119
**[Test build #57467 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57467/consoleFull)** for PR 12817 at commit [`b9aeac1`](https://github.com/apache/spark/commit/b9aeac1b592c6f7af6658fea5550632f0deda79c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61691183
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -393,6 +393,45 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
* This function goes through the input once to determine the input schema. To avoid going
* through the entire data once, specify the schema explicitly using [[schema]].
*
+ * You can set the following CSV-specific options to deal with CSV files:
+ * <li>`delimiter` (default `,`): sets the single character as a delimiter for each
+ * field and value.</li>
+ * <li>`charset` (default `UTF-8`): decodes the CSV files by the given encoding
--- End diff --
encoding is the main one
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61691196
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
@@ -606,6 +606,14 @@ final class DataFrameWriter private[sql](df: DataFrame) {
* }}}
*
* You can set the following CSV-specific option(s) for writing CSV files:
+ * <li>`delimiter` (default `,`): sets the single character as a delimiter for each
--- End diff --
sep here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61691211
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -282,6 +282,45 @@ def csv(self, paths):
:param paths: string, or list of strings, for input path(s).
+ You can set the following CSV-specific options to deal with CSV files:
+ * ``delimiter`` (default ``,``): sets the single character as a delimiter \
+ for each field and value.
+ * ``charset`` (default ``UTF-8``): decodes the CSV files by the given \
+ encoding type.
+ * ``quote`` (default ``"``): sets the single character used for escaping \
+ quoted values where the delimiter can be part of the value.
+ * ``escape`` (default ``\``): sets the single character used for escaping quotes \
+ inside an already quoted value.
+ * ``comment`` (default empty string): sets the single character used for skipping \
+ lines beginning with this character. By default, it is disabled.
+ * ``header`` (default ``false``): uses the first line as names of columns.
+ * ``ignoreLeadingWhiteSpace`` (default ``false``): defines whether or not leading \
+ whitespaces from values being read should be skipped.
+ * ``ignoreTrailingWhiteSpace`` (default ``false``): defines whether or not trailing \
+ whitespaces from values being read should be skipped.
+ * ``nullValue`` (default empty string): sets the string representation of a null value.
+ * ``nanValue`` (default ``NaN``): sets the string representation of a non-number \
+ value.
+ * ``positiveInf`` (default ``Inf``): sets the string representation of a positive \
+ infinity value.
+ * ``negativeInf`` (default ``-Inf``): sets the string representation of a negative \
+ infinity value.
+ * ``dateFormat`` (default ``null``): sets the string that indicates a date format. \
+ Custom date formats follow the formats at ``java.text.SimpleDateFormat``. This \
+ applies to both date type and timestamp type By default, it is `null` which means \
--- End diff --
null => None. And I would not quote NOne here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216015021
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216089939
The scala changes lgtm.
One thing I just realized -- for Python, I think we should turn those into named arguments, rather than just options. Can you do that in a separate pr?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216013507
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57464/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216013493
**[Test build #57464 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57464/consoleFull)** for PR 12817 at commit [`52efce1`](https://github.com/apache/spark/commit/52efce1d775ea5f2028b90c9c0d8ac4ceac1cd5f).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216092410
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57491/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61682060
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -393,6 +393,45 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
* This function goes through the input once to determine the input schema. To avoid going
* through the entire data once, specify the schema explicitly using [[schema]].
*
+ * You can set the following CSV-specific options to deal with CSV files:
+ * <li>`sep` or `delimiter` (default `,`): sets the single character as a delimiter for each
+ * field and value.</li>
+ * <li>`encoding` or `charset` (default `UTF-8`): decodes the CSV files by the given encoding
--- End diff --
BTW a really good reference for these documentation options is R: https://stat.ethz.ch/R-manual/R-devel/library/utils/html/read.table.html
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61682039
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -393,6 +393,45 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
* This function goes through the input once to determine the input schema. To avoid going
* through the entire data once, specify the schema explicitly using [[schema]].
*
+ * You can set the following CSV-specific options to deal with CSV files:
+ * <li>`sep` or `delimiter` (default `,`): sets the single character as a delimiter for each
+ * field and value.</li>
+ * <li>`encoding` or `charset` (default `UTF-8`): decodes the CSV files by the given encoding
--- End diff --
can we remove all the aliases? it'd be great to just have the primary one here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216092645
I also created a follow-up ticket for moving the options to function arguments: https://issues.apache.org/jira/browse/SPARK-15050
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216012967
**[Test build #57464 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57464/consoleFull)** for PR 12817 at commit [`52efce1`](https://github.com/apache/spark/commit/52efce1d775ea5f2028b90c9c0d8ac4ceac1cd5f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216041029
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57479/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12817#issuecomment-216017641
**[Test build #57467 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57467/consoleFull)** for PR 12817 at commit [`b9aeac1`](https://github.com/apache/spark/commit/b9aeac1b592c6f7af6658fea5550632f0deda79c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13425][SQL] Documentation for CSV datas...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12817#discussion_r61678681
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -393,6 +393,45 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
* This function goes through the input once to determine the input schema. To avoid going
* through the entire data once, specify the schema explicitly using [[schema]].
*
+ * You can set the following CSV-specific options to deal with CSV files:
+ * <li>`sep` or `delimiter` (default `,`): sets the single character as a delimiter for each
+ * field and value.</li>
+ * <li>`encoding` or `charset` (default `UTF-8`): decodes the CSV files by the given encoding
--- End diff --
Aliases were added for `codec`, `charset` and `delimiter` as `compression`, `charset` and `sep`.
I remember It was decided that`codec` is not documented but supported for backword compatibility because it looks not a good name.
As I thought both `sep` and `delimiter` or `encoding` or `charset` are good names, I just added but I am happy to get rid of them if any thinks not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org