You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Alex Sulimanov <as...@tremorvideodsp.com> on 2017/10/31 03:10:11 UTC

Question regarding cached partitions

Hi,

I started Spark Streaming job with 96 executors which reads from 96 Kafka partitions and applies mapWithState on the incoming DStream.

Why would it cache only 77 partitions? Do I have to allocate more memory? Currently each executor gets 10 GB and it is not clear why it can't cache all 96 partitions.
[cid:5DD89010-AB5E-43B3-88F6-486FDB58CD75]

Thank you!