You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/09/24 04:26:52 UTC

[GitHub] [hudi] xushiyan commented on issue #3635: [SUPPORT] java.lang.NoSuchMethodError while launching Spark 2.3.1 in stand alone cluster mode.

xushiyan commented on issue #3635:
URL: https://github.com/apache/hudi/issues/3635#issuecomment-926332253


   @deshpandeanoop noticed that you used a different version `"spark-avro" % "2.4.4"` rather than your main spark version 2.3.1
   
   A more important matter: it has been discussed long time back where we only support spark 2.4+
   
   https://lists.apache.org/thread.html/r19ec206e33f8b63e95a840bfba519ab89d1b7af790adef0bc369d618%40%3Cdev.hudi.apache.org%3E
   
   Strongly suggest that you make efforts to upgrade Spark in some way. there are important avro updates in newer version. Spark 2.3.1 uses avro 1.7.7 which can be very problematic
   https://github.com/apache/spark/blob/30aaa5a3a1076ca52439a905274b1fcf498bc562/pom.xml#L142


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org