[GitHub] [spark] dbtsai commented on pull request #29843: [SPARK-29250][test-maven][test-hadoop2.7] Upgrade to Hadoop 3.2.1 and move to shaded client

[GitBox <gi...@apache.org>]

dbtsai commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-706035486


   +1 on considering to deprecate Hadoop 2.7
   
   As @HeartSaVioR mentioned, if we are able to upgrade to Hadoop 3.2.1, we can remove workaround via SPARK-28025. However, without deprecating Hadoop 2.7, we'll have to maintain two code paths and we choose different implements depending on which Hadoop client is used.
   
   New Hadoop 3.2 client works really well with older Hadoop clusters such as 2.7. In fact, we deploy Spark with bundled Hadoop 3.2 on Hadoop 2.7 Yarn clusters in production with great success.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org