Suggestion for SPARK-1825

[innowireless TaeYun Kim <ta...@innowireless.co.kr>]

Hi,

 

A couple of month ago, I made a pull request to fix
https://issues.apache.org/jira/browse/SPARK-1825.

My pull request is here: https://github.com/apache/spark/pull/899 

 

But that pull request has problems:

l  It is Hadoop 2.4.0+ only. It won't compile on the versions below it.

l  The related Hadoop API is marked as '@Unstable'.

 

Here is an idea to remedy the problems: a new Spark configuration variable.

Maybe it can be named as "spark.yarn.submit.crossplatform".

If it is set to "true"(default is false), the related Spark code can use the
hard-coded strings that is the same as the Hadoop API provides, thus
avoiding compile error on the Hadoop versions below 2.4.0.

 

Can someone implement this feature, if this idea is acceptable?

Currently my knowledge on Spark source code and Scala is limited to
implement it myself.

To the right person, the modification should be trivial.

You can refer to the source code changes of my pull request.

 

Thanks.