You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2016/05/05 06:33:12 UTC

[jira] [Commented] (SPARK-15148) Upgrade Univocity library from 2.0.2 to 2.1.0

    [ https://issues.apache.org/jira/browse/SPARK-15148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271964#comment-15271964 ] 

Apache Spark commented on SPARK-15148:
--------------------------------------

User 'HyukjinKwon' has created a pull request for this issue:
https://github.com/apache/spark/pull/12923

>  Upgrade Univocity library from 2.0.2 to 2.1.0
> ----------------------------------------------
>
>                 Key: SPARK-15148
>                 URL: https://issues.apache.org/jira/browse/SPARK-15148
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 2.0.0
>            Reporter: Hyukjin Kwon
>            Priority: Minor
>
> It looks a new release of Univocity CSV library was published, https://github.com/uniVocity/univocity-parsers/releases.
> This contains some improvements as below:
> {quote}
> 1. Performance improvements for parsing/writing CSV and TSV. CSV writing and parsing got 30-40% faster.
> 2. Deprecated methods setParseUnescapedQuotes and setParseUnescapedQuotesUntilDelimiter class CsvParserSettings in favor of the new setUnescapedQuoteHandling method that takes values from the UnescapedQuoteHandling enumeration.
> 3. Default behavior of the CSV parser when unescaped quotes are found on the input changed to parse until a delimiter character is found, i.e. UnescapedQuoteHandling.STOP_AT_DELIMITER. The old default of trying to find a closing quote (i.e. UnescapedQuoteHandling.STOP_AT_CLOSING_QUOTE) can be problematic when no closing quote is found, making the parser accumulate all characters into the same value, until the end of the input.
> {quote}
> With Spark,
> Firstly, It uses this library for CSV data source. This will affect the performance.
> Secondly, Spark uses {{setParseUnescapedQuotesUntilDelimiter}} which is deprecated in this version because It seems there are some more functionalities for parsing unescaped quotes. This seems not directly related with Spark but we might have to consider using this in the future.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org