Support for uniVocity in Spark 2.x

[Jean Georges Perrin <jg...@jgp.net>]

The original CSV parser from Databricks had support for uniVocity in Spark 1.x. Can someone confirm it has disappeared (per: https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/streaming/DataStreamReader.html#csv(java.lang.String) <https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/streaming/DataStreamReader.html#csv(java.lang.String)> ) in 2.0.x?

Thanks!

jg

Re: Support for uniVocity in Spark 2.x

[Hyukjin Kwon <gu...@gmail.com>]

Yeap, there is an option to switch Apache common CSV to Univocity in
external CSV library but it become univocity by default and Apache common
CSV was removed during porting it into Spark 2.0.

On 7 Oct 2016 2:53 a.m., "Sean Owen" <so...@cloudera.com> wrote:

> It still uses univocity, but this is an implementation detail, so I don't
> think that has amounted to supporting or not supporting it.
>
> On Thu, Oct 6, 2016 at 4:00 PM Jean Georges Perrin <jg...@jgp.net> wrote:
>
>>
>> The original CSV parser from Databricks had support for uniVocity in
>> Spark 1.x. Can someone confirm it has disappeared (per:
>> https://spark.apache.org/docs/latest/api/java/org/
>> apache/spark/sql/streaming/DataStreamReader.html#csv(java.lang.String) )
>> in 2.0.x?
>>
>> Thanks!
>>
>> jg
>>
>>
>>
>>

Re: Support for uniVocity in Spark 2.x

[Sean Owen <so...@cloudera.com>]

It still uses univocity, but this is an implementation detail, so I don't
think that has amounted to supporting or not supporting it.

On Thu, Oct 6, 2016 at 4:00 PM Jean Georges Perrin <jg...@jgp.net> wrote:

>
> The original CSV parser from Databricks had support for uniVocity in Spark
> 1.x. Can someone confirm it has disappeared (per:
> https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/streaming/DataStreamReader.html#csv(java.lang.String) )
> in 2.0.x?
>
> Thanks!
>
> jg
>
>
>
>