[jira] [Created] (SPARK-28431) CSV datasource throw com.univocity.parsers.common.TextParsingException with large size message

["Weichen Xu (JIRA)" <ji...@apache.org>]

Weichen Xu created SPARK-28431:
----------------------------------

             Summary: CSV datasource throw com.univocity.parsers.common.TextParsingException with large size message 
                 Key: SPARK-28431
                 URL: https://issues.apache.org/jira/browse/SPARK-28431
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 2.4.3
            Reporter: Weichen Xu


CSV datasource throw com.univocity.parsers.common.TextParsingException with large size message, which will make log output consume large disk space.

Reproduce code
{code:java}
val s = "a" * 40 * 1000000
Seq(s).toDF.write.mode("overwrite").csv("/tmp/bogdan/es4196.csv")

spark.read .option("maxCharsPerColumn", 30000000) .csv("/tmp/bogdan/es4196.csv").count{code}
Because of maxCharsPerColumn limit of 30M, there will be a TextParsingException. The message of this exception actually includes what was parsed so far, in this case 30M chars.

 

This issue is troublesome when sometimes we need parse CSV with large column.

We should truncate the large size message in the TextParsingException.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org