You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2018/05/21 08:31:00 UTC
[jira] [Commented] (SPARK-24329) Remove comments filtering before
parsing of CSV files
[ https://issues.apache.org/jira/browse/SPARK-24329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16482295#comment-16482295 ]
Apache Spark commented on SPARK-24329:
--------------------------------------
User 'MaxGekk' has created a pull request for this issue:
https://github.com/apache/spark/pull/21380
> Remove comments filtering before parsing of CSV files
> -----------------------------------------------------
>
> Key: SPARK-24329
> URL: https://issues.apache.org/jira/browse/SPARK-24329
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Maxim Gekk
> Priority: Minor
>
> Comments and whitespace filtering has been performed by uniVocity parser already according to parser settings:
> https://github.com/apache/spark/blob/branch-2.3/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala#L178-L180
> It is not necessary to do the same before parsing. Need to inspect all places where the filterCommentAndEmpty method is called, and remove the former one if it duplicates filtering of uniVocity parser.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org