You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2021/01/11 06:04:00 UTC
[jira] [Commented] (SPARK-34050) Parquet 2 CSV conversion wrong
quoting
[ https://issues.apache.org/jira/browse/SPARK-34050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262400#comment-17262400 ]
Hyukjin Kwon commented on SPARK-34050:
--------------------------------------
[~laszlo.torok] are you able to provide a self-contained reproducer?
> Parquet 2 CSV conversion wrong quoting
> ---------------------------------------
>
> Key: SPARK-34050
> URL: https://issues.apache.org/jira/browse/SPARK-34050
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.4.0
> Reporter: Laszlo Torok
> Priority: Minor
>
> Hi Experts,
> I work for GE Corporate. We have a Backup+Restore+extras project with AWS.
> I faced with incompatibility issues when tried to convert back parquet files to CSV.
> Our original sources (GreenPlum first) cannot process those backward converted files because of unproper quoting.
>
> We work on several kinds of ERPs and TechDatabses and there are:
> # multiline (CR,CRLF,LF) text fields
> # mixed quoting inside the fields or just one double quote in a text field
> # we have text field where EmptyString and Null values can be placed and has different meaning
> Our last option combination is:
> df.write.format("com.databricks.spark.csv").options(header='false',sep ='\013' ,multiLine ='true',escapeQuotes='true',quote = '"',nullValue ='
> N', encoding='UTF-8').option("quoteAll", 'false').option("compression","gzip").mode('overwrite').save(s3_csv)
> If I do not use escapeQuotes='true' it wont quote those fields where mixed or once occures a double quote.
> If I use this it will escape emptyString double quotes [sep]\"\"[sep] . ==> Our Greenplum reader cannot read (Restoration) this format for emptyString.
> It should be [sep]""[sep] or [sep][sep].
> Can you help our project with proper quote and escape combination where data looks like this:
> "2607 - CREDIT MEMO - SOCIETATEA NATIONALA 'NUCLEARELECTRICA\" S.A. | 13-APR-20 "
> "290208407
> |INT. RIEL DIN 2X32A 230/400V "|
> ""
> I found an earlier option what you moved out: quoteMode.Non_Numeric.
> Thank you in advance!
>
> Regards,
> Laszlo Torok
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org