akka.FrameSize

[Chen Jin <ka...@gmail.com>]

Hi all,

I have run into a very interesting bug which is not exactly as same as
Spark-1112.

Here is how to reproduce the bug, I have one input csv file and use
partitionBy function to create an RDD, say repartitionedRDD. The
partitionBy function takes the number of partitions as a parameter
such that we can vary the serialized size per partition easily for the
following experiments. At the end, I just simply call
repartitionedRDD.collect().

1) spark.akka.frameSize = 10
If one of the partition size is very close to 10MB, say 9.97MB, the
execution blocks without any exception or warning. Worker finished the
task to send the serialized result, and then throw exception saying
hadoop IPC client connection stops (changing the logging to debug
level). However, the master never receives the results and the program
just hangs.
But if sizes for all the partitions less than some number btw 9.96MB
amd 9.97MB, the program works fine.
2) spark.akka.frameSize = 9
when the partition size is just a little bit smaller than 9MB, it fails as well.
This bug behavior is not exactly what spark-1112 is about, could you
please guide me how to open a separate bug when the serialization size
is very close to 10MB.

I googled around and haven't found anything which relates to the
behavior we have found. Any insights or suggestions would be greatly
appreciated.

Thanks! :-)

-chen