Dataframe to parquet using hdfs or parquet block size

[bdev <bu...@gmail.com>]

I need to save the dataframe to parquet format and need some input on
choosing the appropriate block size to help efficiently parallelize/localize
the data to the executors. Should I be using parquet block size or hdfs
block size and what is the optimal block size to use on a 100 node cluster?


Thanks!



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Dataframe-to-parquet-using-hdfs-or-parquet-block-size-tp26693.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Dataframe to parquet using hdfs or parquet block size

[Buntu Dev <bu...@gmail.com>]

I tried setting both the hdfs and parquet block size but write to parquet
did not seem to have had any effect on the total number of blocks or the
average block size. Here is what I did:

~~~~
sqlContext.setConf("dfs.blocksize", "134217728")
sqlContext.setConf("parquet.block.size", "134217728")
sqlContext.setConf("spark.broadcast.blockSize", "134217728")
df.write.parquet("/path/to/dest")
~~~~

I tried the same with different block sizes but none had any effect. Is
this the right way to set the properties using setConf()?


Thanks!


On Wed, Apr 6, 2016 at 11:28 PM, bdev <bu...@gmail.com> wrote:

> I need to save the dataframe to parquet format and need some input on
> choosing the appropriate block size to help efficiently
> parallelize/localize
> the data to the executors. Should I be using parquet block size or hdfs
> block size and what is the optimal block size to use on a 100 node cluster?
>
>
> Thanks!
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Dataframe-to-parquet-using-hdfs-or-parquet-block-size-tp26693.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>