You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Masf <ma...@gmail.com> on 2015/04/10 08:44:37 UTC

Increase partitions reading Parquet File

Hi

I have this statement:

val file =
SqlContext.parquetfile("hdfs://node1/user/hive/warehouse/file.parquet")

This code generates as many partitions as files are. So, I want to increase
the number of partitions.
I've tested coalesce (file.coalesce(100)) but the number of partitions
doesn't change.

How can I increase the number of partitions?

Thanks

-- 


Regards.
Miguel Ángel

Re: Increase partitions reading Parquet File

Posted by Ignacio Blasco <el...@gmail.com>.

To increase partitions you have to use repartition but this will force a
shuffle write
El 10/4/2015 8:45 a. m., "Masf" <ma...@gmail.com> escribió:

> Hi
>
> I have this statement:
>
> val file =
> SqlContext.parquetfile("hdfs://node1/user/hive/warehouse/file.parquet")
>
> This code generates as many partitions as files are. So, I want to
> increase the number of partitions.
> I've tested coalesce (file.coalesce(100)) but the number of partitions
> doesn't change.
>
> How can I increase the number of partitions?
>
> Thanks
>
> --
>
>
> Regards.
> Miguel Ángel
>