Spark YARN Shuffle service wire compatibility

[Jong Wook Kim <jo...@nyu.edu>]

Hi, I’d like to know if there is a guarantee that Spark YARN shuffle service has wire compatibility between 1.x versions.

I could run Spark 1.5 job with YARN nodemanagers having shuffle service 1.4, but it might’ve been just a coincidence.

Now we’re upgrading CDH to 5.3 to 5.4, whose NodeManager already have shuffle service of 1.3 in its classpath, and concerned that it might not be always compatible with 1.5 jobs expecting shuffle service.


Jong Wook
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org