You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/07/23 01:46:00 UTC

[jira] [Resolved] (SPARK-28431) CSV datasource throw com.univocity.parsers.common.TextParsingException with large size message

     [ https://issues.apache.org/jira/browse/SPARK-28431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-28431.
----------------------------------
       Resolution: Fixed
    Fix Version/s: 3.0.0

Issue resolved by pull request 25184
[https://github.com/apache/spark/pull/25184]

> CSV datasource throw com.univocity.parsers.common.TextParsingException with large size message 
> -----------------------------------------------------------------------------------------------
>
>                 Key: SPARK-28431
>                 URL: https://issues.apache.org/jira/browse/SPARK-28431
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 2.4.3
>            Reporter: Weichen Xu
>            Assignee: Weichen Xu
>            Priority: Minor
>             Fix For: 3.0.0
>
>
> CSV datasource throw com.univocity.parsers.common.TextParsingException with large size message, which will make log output consume large disk space.
> Reproduce code
> {code:java}
> val s = "a" * 40 * 1000000
> Seq(s).toDF.write.mode("overwrite").csv("/tmp/bogdan/es4196.csv")
> spark.read .option("maxCharsPerColumn", 30000000) .csv("/tmp/bogdan/es4196.csv").count{code}
> Because of maxCharsPerColumn limit of 30M, there will be a TextParsingException. The message of this exception actually includes what was parsed so far, in this case 30M chars.
>  
> This issue is troublesome when sometimes we need parse CSV with large column.
> We should truncate the large size message in the TextParsingException.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org