You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Hrishikesh Mishra <sd...@gmail.com> on 2020/05/05 14:03:25 UTC

Any impact on Driver out memory with long batch queue.

Hi All,

Has anyone observed that having long streaming queues impact Driver memory.
Our Spark Driver is getting killed due Java heap out memory (4GB). We are
not doing any computation / aggregation in Driver just consuming from Kafka
topics. But there around 100-500 queued batches.

Spark Version: 2.4
Kafka Direct stream.

[image: image.png]


Regards
Hrishi