You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Pietro Gentile <pi...@gmail.com> on 2016/05/20 09:34:52 UTC

Spark CacheManager Thread-safety

Hi all,

I have a series of doubts about CacheManager used by SQLContext to cache
DataFrame.

My use case requires different threads persisting/reading dataframes
cuncurrently. I realized using spark that persistence really does not work
in parallel mode.

I would like it if I'm persisting a data frame, another user should be able
to persist a different DF and not wait for the first to finish.

Is there any way to implement this scenario?

Thank in advance,
Pietro.

Fwd: Spark CacheManager Thread-safety

Posted by Pietro Gentile <pi...@gmail.com>.

Hi all,

I have a series of doubts about CacheManager used by SQLContext to cache
DataFrame.

My use case requires different threads persisting/reading dataframes
cuncurrently. I realized using spark that persistence really does not work
in parallel mode.

I would like it if I'm persisting a data frame, another user should be able
to persist a different DF and not wait for the first to finish.

Is there any way to implement this scenario?

Thank in advance,
Pietro.