You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Rishi Shah <ri...@gmail.com> on 2019/05/22 03:29:16 UTC

[pyspark 2.3+] how to dynamically determine DataFrame partitions while writing

Hi All,

What is the best way to determine partitions of a dataframe dynamically
before writing to disk?

1) statically determine based on data and use coalesce or repartition while
writing
2) somehow determine count of records for entire dataframe and divide that
number to determine partition - however how to determine total count
without having to risk computing dataframe twice (if dataframe is not
cached, and count() is used)

-- 
Regards,

Rishi Shah

Re: [pyspark 2.3+] how to dynamically determine DataFrame partitions while writing

Posted by Rishi Shah <ri...@gmail.com>.

Hi All,

Any idea about this?

Thanks,
Rishi

On Tue, May 21, 2019 at 11:29 PM Rishi Shah <ri...@gmail.com>
wrote:

> Hi All,
>
> What is the best way to determine partitions of a dataframe dynamically
> before writing to disk?
>
> 1) statically determine based on data and use coalesce or repartition
> while writing
> 2) somehow determine count of records for entire dataframe and divide that
> number to determine partition - however how to determine total count
> without having to risk computing dataframe twice (if dataframe is not
> cached, and count() is used)
>
> --
> Regards,
>
> Rishi Shah
>


-- 
Regards,

Rishi Shah