You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Tóth Zoltán <tz...@looper.hu> on 2015/11/04 14:11:43 UTC
Looking for the method executors uses to write to HDFS
Hi,
I'd like to write a parquet file from the driver. I could use the HDFS API
but I am worried that it won't work on a secure cluster. I assume that the
method the executors use to write to HDFS takes care of managing Hadoop
security. However, I can't find the place where HDFS write happens in the
spark source.
Please help me:
1.How to write parquet from the driver using the Spark API?
2. If this wouldn't possible, where can I find the method executors use to
write to HDFS?
Thanks,
Zoltan
Re: Looking for the method executors uses to write to HDFS
Posted by Reynold Xin <rx...@databricks.com>.
Are you looking for this?
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRelation.scala#L69
On Wed, Nov 4, 2015 at 5:11 AM, Tóth Zoltán <tz...@looper.hu> wrote:
> Hi,
>
> I'd like to write a parquet file from the driver. I could use the HDFS API
> but I am worried that it won't work on a secure cluster. I assume that the
> method the executors use to write to HDFS takes care of managing Hadoop
> security. However, I can't find the place where HDFS write happens in the
> spark source.
>
> Please help me:
> 1.How to write parquet from the driver using the Spark API?
> 2. If this wouldn't possible, where can I find the method executors use to
> write to HDFS?
>
> Thanks,
> Zoltan
>
>
Re: Looking for the method executors uses to write to HDFS
Posted by Reynold Xin <rx...@databricks.com>.
Are you looking for this?
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRelation.scala#L69
On Wed, Nov 4, 2015 at 5:11 AM, Tóth Zoltán <tz...@looper.hu> wrote:
> Hi,
>
> I'd like to write a parquet file from the driver. I could use the HDFS API
> but I am worried that it won't work on a secure cluster. I assume that the
> method the executors use to write to HDFS takes care of managing Hadoop
> security. However, I can't find the place where HDFS write happens in the
> spark source.
>
> Please help me:
> 1.How to write parquet from the driver using the Spark API?
> 2. If this wouldn't possible, where can I find the method executors use to
> write to HDFS?
>
> Thanks,
> Zoltan
>
>