You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Dark Crusader <re...@gmail.com> on 2020/07/07 18:06:53 UTC

Mocking pyspark read writes

Hi everyone,

I have a function which reads and writes a parquet file from HDFS. When I'm
writing a unit test for this function, I want to mock this read & write.

How do you achieve this?
Any help would be appreciated. Thank you.

Re: Mocking pyspark read writes

Posted by Jörn Franke <jo...@gmail.com>.
Write to a local temp directory via file:// ?

> Am 07.07.2020 um 20:07 schrieb Dark Crusader <re...@gmail.com>:
> 
> 
> Hi everyone,
> 
> I have a function which reads and writes a parquet file from HDFS. When I'm writing a unit test for this function, I want to mock this read & write.
> 
> How do you achieve this? 
> Any help would be appreciated. Thank you.
> 

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org