You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/07/23 01:46:00 UTC
[jira] [Resolved] (SPARK-28431) CSV datasource throw
com.univocity.parsers.common.TextParsingException with large size message
[ https://issues.apache.org/jira/browse/SPARK-28431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-28431.
----------------------------------
Resolution: Fixed
Fix Version/s: 3.0.0
Issue resolved by pull request 25184
[https://github.com/apache/spark/pull/25184]
> CSV datasource throw com.univocity.parsers.common.TextParsingException with large size message
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-28431
> URL: https://issues.apache.org/jira/browse/SPARK-28431
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.4.3
> Reporter: Weichen Xu
> Assignee: Weichen Xu
> Priority: Minor
> Fix For: 3.0.0
>
>
> CSV datasource throw com.univocity.parsers.common.TextParsingException with large size message, which will make log output consume large disk space.
> Reproduce code
> {code:java}
> val s = "a" * 40 * 1000000
> Seq(s).toDF.write.mode("overwrite").csv("/tmp/bogdan/es4196.csv")
> spark.read .option("maxCharsPerColumn", 30000000) .csv("/tmp/bogdan/es4196.csv").count{code}
> Because of maxCharsPerColumn limit of 30M, there will be a TextParsingException. The message of this exception actually includes what was parsed so far, in this case 30M chars.
>
> This issue is troublesome when sometimes we need parse CSV with large column.
> We should truncate the large size message in the TextParsingException.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org