You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@arrow.apache.org by Bryan Cutler <cu...@gmail.com> on 2017/09/05 19:33:43 UTC

Re: how to use arrow in project?

Hello,

At the moment for the upcoming release of Spark 2.3, Arrow support is
limited to using PySpark DataFrame.toPandas(), so reading Parquet data from
Spark does not use Arrow.

-Bryan

On Wed, Aug 30, 2017 at 2:47 AM, big data <bi...@outlook.com> wrote:

> I want to use Arrow as a middle layer between spark and parquet data in
> HDFS, but I don't find any docs about how to load parquet data to arrow
> in memory, and how spark read arrow data format. Does anyone provide
> some examples or manuals to describe it?
>
> thanks.
>
>