You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "panbingkun (via GitHub)" <gi...@apache.org> on 2023/05/18 02:53:42 UTC

[GitHub] [spark] panbingkun opened a new pull request, #41209: [SPARK-43548][SS] Remove workaround for HADOOP-16255

panbingkun opened a new pull request, #41209:
URL: https://github.com/apache/spark/pull/41209

   ### What changes were proposed in this pull request?
   The pr aims to remove workaround for HADOOP-16255.
   
   ### Why are the changes needed?
   - Because HADOOP-16255 has been fix after hadoop version  3.1.2. Spark support hadoop version: >= 3.2.2 or >= 3.3.1
   - Make code clean.
   
   ### Does this PR introduce _any_ user-facing change?
   No.
   
   ### How was this patch tested?
   Pass GA.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #41209: [SPARK-43548][SS] Remove workaround for HADOOP-16255

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #41209:
URL: https://github.com/apache/spark/pull/41209#issuecomment-1552365417

   Hmm .. does that mean Hadoop 3.2.0 won't work with this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #41209: [SPARK-43548][SS] Remove workaround for HADOOP-16255

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #41209:
URL: https://github.com/apache/spark/pull/41209#issuecomment-1552366068

   Hm, we currently build Spark w/ Hadoop 3.3.0 by default it might be fine but I would also ask some more looks e.g., @srowen @mridulm @tgravescs @dongjoon-hyun 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on pull request #41209: [SPARK-43548][SS] Remove workaround for HADOOP-16255

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #41209:
URL: https://github.com/apache/spark/pull/41209#issuecomment-1552385793

   Spark currently only works with Hadoop 3.3.1+ at the moment, as we discovered in https://github.com/apache/spark/pull/40847. We can potentially make it work with Hadoop 3.2.2+ if there's a workaround for https://issues.apache.org/jira/browse/SPARK-40039 which uses Hadoop API that only exist in Hadoop 3.3.1+. It definitely won't work with Hadoop 3.2.0 due to some shaded client related issues.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] panbingkun commented on pull request #41209: [SPARK-43548][SS] Remove workaround for HADOOP-16255

Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #41209:
URL: https://github.com/apache/spark/pull/41209#issuecomment-1552341297

   <img width="893" alt="image" src="https://github.com/apache/spark/assets/15246973/6da74b5d-4e71-440e-bb47-d17ba7f7de1e">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #41209: [SPARK-43548][SS] Remove workaround for HADOOP-16255

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #41209:
URL: https://github.com/apache/spark/pull/41209#issuecomment-1552395877

   Thanks for clarification. Lgtm2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun closed pull request #41209: [SPARK-43548][SS] Remove workaround for HADOOP-16255

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #41209: [SPARK-43548][SS] Remove workaround for HADOOP-16255
URL: https://github.com/apache/spark/pull/41209


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] LuciferYang commented on pull request #41209: [SPARK-43548][SS] Remove workaround for HADOOP-16255

Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #41209:
URL: https://github.com/apache/spark/pull/41209#issuecomment-1552334558

   cc @attilapiros @viirya @sunchao @pan3793 FYI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org