Spark partitioned By

[venkatesh bandaru <ve...@gmail.com>]

Hi Team,

I have asked this question in our stackoverflow group

pyspark - Apache Spark partition by output path - Stack Overflow
<https://stackoverflow.com/questions/74089582/apache-spark-partition-by-output-path>


*Requirement*

1. I have huge data coming from source and loaded into Azure Data lake in
csv format.
2. One of the column in csv file is tenant Id.
3. I need to partition this csv on tenantId and store in ADLS in this
directory {*tenantId*}\(tenanid}.csv  --> bolder part is storage container
4. There is one more challenge. my source csv file can have more than
50,000 unique tenant Id's.  Maximum In one storage account i want to keep
only 25000 tenant data in this format  {*tenantId*}\(tenanid}.csv  ,
remaining 25000 should go to another storage account.


I want to know how do i customize or write custom code for PartitionBy , so
that i can have more control on this method, i can write my own logic to
map tenant data into their respective storage.


Need your help in this regard.


Thanks in advance.

Regards
Venkatesh