[GitHub] [hudi] a0x edited a comment on issue #4442: [SUPPORT] PySpark(3.1.2) with Hudi(0.10.0) failed when querying spark sql

[GitBox <gi...@apache.org>]

a0x edited a comment on issue #4442:
URL: https://github.com/apache/hudi/issues/4442#issuecomment-1004486507


   > I have the same issue when running hudi on emr. This issue seems to have the same root cause as in this one: #4474 . The solution is to shade and relocate aws dependencies introduced in hudi-aws:
   > 
   > > For our internal hudi version, we shade aws dependencies, you can add new relocation and build a new bundle package:
   > > For example, to shade aws dependencies in spark, add following codes in **packaging/hudi-spark-bundle/pom.xml**
   > > ```
   > > <!-- line 185-->
   > > <relocation>
   > >  <pattern>com.amazonaws.</pattern>
   > >  <shadedPattern>${spark.bundle.spark.shade.prefix}com.amazonaws.</shadedPattern>
   > > </relocation>
   > > ```
   > 
   > @xushiyan should this relocation be added to the official hudi release to avoid such conflicts?
   
   @kazdy Thank you! This should work.
   
   But shall we shade all aws deps in Spark? I'm worrying about the side effict, but let me have a try before replying in that issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org