[Streaming]Executor OOM

[Yana Kadiyska <ya...@gmail.com>]

Hi all,

I'm running a very basic program: updateStateByKey over a directory that
receives NO new entries.
Number of key-value pairs in the RDD is 440,568

The program runs fine for about an hour and then tasks start failing as the
executors start failing w/ OOM.

I am not giving a ton of memory (512m per executor) but it's also puzzling
to me that it works for an hour since I'm guessing no work is actually done
on the RDD after the initial reduce? (i.e. the memory should be close to
flat?)

RDD is checkpointed with the default frequency (batchSize=10sec). I am
attaching a heap dump of the OOM executor. I see a large byte[] but cannot
make much sense of where it comes from.

Does anyone have long-running executors (I'd imagine the answer is yes for
streaming). Any tips on what I'm doing wrong?

Here is the full program I'm running:

https://github.com/yanakad/streaming_code/blob/59a028d9093d45150cd50acd46cc467d29474421/src/main/scala/com/akamai/test/UpdateStream.scala