You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gaborgsomogyi <gi...@git.apache.org> on 2017/11/24 15:21:13 UTC
[GitHub] spark pull request #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv...
GitHub user gaborgsomogyi opened a pull request:
https://github.com/apache/spark/pull/19814
[SPARK-22484][DOC] Document PySpark DataFrame csv writer behavior whe…
## What changes were proposed in this pull request?
In PySpark API Document, DataFrame.write.csv() says that setting the quote parameter to an empty string should turn off quoting. Instead, it uses the [null character](https://en.wikipedia.org/wiki/Null_character) as the quote.
This PR fixes the doc.
## How was this patch tested?
Manual.
```
cd python/docs
make html
open _build/html/pyspark.sql.html
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gaborgsomogyi/spark SPARK-22484
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19814.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19814
----
commit 821ee1993e26289bd5cc93be14543a8dbe913ff8
Author: gaborgsomogyi <ga...@gmail.com>
Date: 2017-11-24T14:50:53Z
[SPARK-22484][DOC] Document PySpark DataFrame csv writer behavior when empty string used as quote
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv...
Posted by gaborgsomogyi <gi...@git.apache.org>.
Github user gaborgsomogyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/19814#discussion_r153067049
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -828,8 +828,7 @@ def csv(self, path, mode=None, compression=None, sep=None, quote=None, escape=No
set, it uses the default value, ``,``.
:param quote: sets the single character used for escaping quoted values where the
separator can be part of the value. If None is set, it uses the default
- value, ``"``. If you would like to turn off quotations, you need to set an
- empty string.
+ value, ``"``. If empty string is set, it uses ``u0000``.
--- End diff --
Are you referring to the read part? If so then I left it intentionally there. In the read part there is a test which covers this functionality and works as expected.
CSVSuite.scala:test("SPARK-15585 turn off quotations")
If you mean something else please highlight it allowing me to learn from it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19814
Can we turn off it as documented? We could try to open a JIRA in Univocity if this functionality is not there and incorporate the change in Spark.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19814
**[Test build #84220 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84220/testReport)** for PR 19814 at commit [`46efe32`](https://github.com/apache/spark/commit/46efe32e23389d02babffb5fbec1b6d0c63c25cd).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19814
**[Test build #84219 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84219/testReport)** for PR 19814 at commit [`4bf4876`](https://github.com/apache/spark/commit/4bf4876223d6a0a859177ba7ede3aec4951b3497).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19814
I think we should document this in read side too if we are going to do and actually I think it's not intended as the thirdparty one does not allow empty string IIRC. Will take a closer look tomorrow.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19814
Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19814#discussion_r153062402
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -828,8 +828,7 @@ def csv(self, path, mode=None, compression=None, sep=None, quote=None, escape=No
set, it uses the default value, ``,``.
:param quote: sets the single character used for escaping quoted values where the
separator can be part of the value. If None is set, it uses the default
- value, ``"``. If you would like to turn off quotations, you need to set an
- empty string.
+ value, ``"``. If empty string is set, it uses ``u0000``.
--- End diff --
If there are doc changes to be done for options here, let's make sure chaning all. Quick and easy way will be just `grep` and replace.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19814
**[Test build #3992 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3992/testReport)** for PR 19814 at commit [`821ee19`](https://github.com/apache/spark/commit/821ee1993e26289bd5cc93be14543a8dbe913ff8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19814
Okay, I am fine with the current change for now. This PR fix the lie BTW and seems it's difficult to fix the behaviour as intended.
Will probably make a followup after the small talk with the author.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by gaborgsomogyi <gi...@git.apache.org>.
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19814
Good point, I'll speak with the Univocity guys... Then the question comes what should we do with this PR and Jira ticket? I'm a newbie so every help/advise is highly appreciated :)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19814
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by gaborgsomogyi <gi...@git.apache.org>.
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19814
OK. Explicit description not found. I've just tested it manually and taken a look at the source. It tries to match the character with the configured quote and it never matches because u0000 is end of string character.
All in all I believe we have multiple problems:
1. The documentation lies. This PR meant to solve it.
2. There are exception thrown or not producing the same output when u0000 used for writing and reading. For this we could turn off the u0000 usage.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19814
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84219/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19814
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84200/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by gaborgsomogyi <gi...@git.apache.org>.
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19814
Seems like the job died because of jvm internal issue:
> *** glibc detected *** /usr/java/jdk1.8.0_60/bin/java: double free or corruption (out): 0x0000000100038720 ***
Is it possible to restart it?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19814
**[Test build #84200 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84200/testReport)** for PR 19814 at commit [`821ee19`](https://github.com/apache/spark/commit/821ee1993e26289bd5cc93be14543a8dbe913ff8).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19814
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by gaborgsomogyi <gi...@git.apache.org>.
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19814
As I've seen the univocity library does not support explicit quote turn off feature on the write side.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19814
Will ask this with the author.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19814#discussion_r153199039
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -828,8 +828,7 @@ def csv(self, path, mode=None, compression=None, sep=None, quote=None, escape=No
set, it uses the default value, ``,``.
:param quote: sets the single character used for escaping quoted values where the
separator can be part of the value. If None is set, it uses the default
- value, ``"``. If you would like to turn off quotations, you need to set an
- empty string.
+ value, ``"``. If empty string is set, it uses ``u0000``.
--- End diff --
and maybe small additional comments for ``u0000`` would be nicer, e.g., it's null character.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19814
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19814
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19814
**[Test build #84220 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84220/testReport)** for PR 19814 at commit [`46efe32`](https://github.com/apache/spark/commit/46efe32e23389d02babffb5fbec1b6d0c63c25cd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv...
Posted by gaborgsomogyi <gi...@git.apache.org>.
Github user gaborgsomogyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/19814#discussion_r153199400
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -828,8 +828,7 @@ def csv(self, path, mode=None, compression=None, sep=None, quote=None, escape=No
set, it uses the default value, ``,``.
:param quote: sets the single character used for escaping quoted values where the
separator can be part of the value. If None is set, it uses the default
- value, ``"``. If you would like to turn off quotations, you need to set an
- empty string.
+ value, ``"``. If empty string is set, it uses ``u0000``.
--- End diff --
Fixing...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19814
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19814
Yup, seems unrelated. It's fine. Let me back soon with another look.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19814
**[Test build #84219 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84219/testReport)** for PR 19814 at commit [`4bf4876`](https://github.com/apache/spark/commit/4bf4876223d6a0a859177ba7ede3aec4951b3497).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19814
**[Test build #84200 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84200/testReport)** for PR 19814 at commit [`821ee19`](https://github.com/apache/spark/commit/821ee1993e26289bd5cc93be14543a8dbe913ff8).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19814
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84220/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19814
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19814
Please see the small discussion in `https://github.com/uniVocity/univocity-parsers/issues/215`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by gaborgsomogyi <gi...@git.apache.org>.
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19814
Suggested fix added. Happy to contribute in the followups if there are possibilities. Thanks :)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19814#discussion_r153198556
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -828,8 +828,7 @@ def csv(self, path, mode=None, compression=None, sep=None, quote=None, escape=No
set, it uses the default value, ``,``.
:param quote: sets the single character used for escaping quoted values where the
separator can be part of the value. If None is set, it uses the default
- value, ``"``. If you would like to turn off quotations, you need to set an
- empty string.
+ value, ``"``. If empty string is set, it uses ``u0000``.
--- End diff --
`If empty string` -> `If an empty string `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by gaborgsomogyi <gi...@git.apache.org>.
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19814
Seems like there is a good reason why this is not supported. Here is their opinion:
> The writer will only put quotes when absolutely required (i.e. the value has a delimiter, line ending, or another quote in it). If you don't want to output quotes in these cases the parser won't be able to read the data back properly. It makes no sense to force the CSV writer to not use quotes.
> If you really want to do this you'd be better off just writing your data directly to an instance of java.io.Writer (FileWriter, OutputStreamWriter, StringWriter, etc)
This case I tend to suggest the original idea. Suggestions/ideas?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19814
I believe Univocity parser does not describe this behaviour, ignoring quotes when it's set to `\u0000` when paring it in?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19814
**[Test build #3992 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3992/testReport)** for PR 19814 at commit [`821ee19`](https://github.com/apache/spark/commit/821ee1993e26289bd5cc93be14543a8dbe913ff8).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19814: [SPARK-22484][DOC] Document PySpark DataFrame csv writer...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19814
To be honest, I would like to suggest disallow it. I just ran few tests and looks we are still not able to read it back:
Empty `quote` (\u0000):
```scala
Seq(Tuple2("a\n a", "b \nb"), Tuple2("c\n c", "d \nd")).toDF.write.mode("overwrite").option("quote", "").csv("tmp.csv")
spark.read.option("multiLine", true).option("quote", "").csv("tmp.csv").collect()
```
```
Array[org.apache.spark.sql.Row] = Array([ a?,??b ], [b?,null], [ c?,??d ], [d?,null])
```
If \u0000 really disables quoting when read, I think it should give the same results when `quote` is another character for example:
```
Seq(Tuple2("a\n a", "b \nb"), Tuple2("c\n c", "d \nd")).toDF.write.mode("overwrite").option("quote", "").csv("tmp.csv")
spark.read.option("multiLine", true).option("quote", "^").csv("tmp.csv").collect()
```
```
Array[org.apache.spark.sql.Row] = Array([ a?,?b ], [b?,null], [ c?,?d ], [d?,null])
```
but the output is different as above.
It's Array(0, 98, 32) vs Array(0, 0, 98, 32) in `"?b "` vs `"??b "`
Default `quote`:
```
Seq(Tuple2("a\n a", "b \nb"), Tuple2("c\n c", "d \nd")).toDF.write.mode("overwrite").csv("tmp.csv")
spark.read.option("multiLine", true).csv("tmp.csv").collect()
```
```
Array[org.apache.spark.sql.Row] =
Array([a
a,b
b], [c
c,d
d])
```
Another `quote`:
```scala
Seq(Tuple2("a\n a", "b \nb"), Tuple2("c\n c", "d \nd")).toDF.write.mode("overwrite").option("quote", "!").csv("tmp.csv")
spark.read.option("multiLine", true).option("quote", "!").csv("tmp.csv").collect()
```
```
Array[org.apache.spark.sql.Row] =
Array([a
a,b
b], [c
c,d
d])
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org