You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Ramandeep Singh Nanda <ra...@gmail.com> on 2018/11/08 15:38:11 UTC

Is Dataframe write blocking?

HI,

I have some futures setup to operate in stages, where I expect one stage to
complete before another begins. I was hoping that dataframe write call is
blocking, whereas the behavior i see is that the call returns before data
is persisted. This can cause unintended consequences. I am also using fair
scheduler so that independent jobs can run in parallel.

Spark 2.3.1.

-- 
Regards,
Ramandeep Singh
http://orastack.com