Re: Does spark support Apache Arrow

[Nirav Patel <np...@xactlycorp.com>]

Kwon,

Isn't that JIRA is part of integration with Arrow.  As far as arrow as
in-memory store goes it probably conflicts with spark's own tungsten memory
representation, right?

Thanks
Nir

On Thu, May 19, 2016 at 8:03 PM, Hyukjin Kwon <gu...@gmail.com> wrote:

> FYI, there is a JIRA for this, https://issues.apache.
> org/jira/browse/SPARK-13534
>
> I hope this link is helpful.
>
> Thanks!
>
>
> 2016-05-20 11:18 GMT+09:00 Sun Rui <su...@163.com>:
>
>> 1. I don’t think so
>> 2. Arrow is for in-memory columnar execution. While cache is for
>> in-memory columnar storage
>>
>> On May 20, 2016, at 10:16, Todd <bi...@163.com> wrote:
>>
>> From the official site http://arrow.apache.org/, Apache Arrow is used
>> for Columnar In-Memory storage. I have two quick questions:
>> 1. Does spark support Apache Arrow?
>> 2. When dataframe is cached in memory, the data are saved in columnar
>> in-memory style. What is the relationship between this feature and Apache
>> Arrow,that is,
>> when the data is in Apache Arrow format，does spark still need the effort
>> to cache the dataframe in columnar in-memory?
>>
>>
>>
>

-- 


[image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>

<https://www.nyse.com/quote/XNYS:XTLY>  [image: LinkedIn] 
<https://www.linkedin.com/company/xactly-corporation>  [image: Twitter] 
<https://twitter.com/Xactly>  [image: Facebook] 
<https://www.facebook.com/XactlyCorp>  [image: YouTube] 
<http://www.youtube.com/xactlycorporation>