You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@hudi.apache.org by "Li, Adam" <ad...@amazon.com.INVALID> on 2025/01/14 15:57:00 UTC
Questions Regarding Compatability of Hudi v0.15, Hadoop v3.4.x and Spark 3.5.x
Hi,
I am using trying to setup a PySpark Jupyter notebook on an AWS EMR cluster to read Hudi datasets. I am using the latest settings:
* Emr7.6, Hudi v0.15, Hadoop v3.4.x and Spark 3.5.x
However, I obtained an error shown below. I have a few questions:
1. I could not find where the API may have changed, but I am wondering if this is due to a version incomptability? I realize I have not linked any code, but I’m using some custom JAR files and setups.
2. Is there a matrix somewhere showing compatability of Hudi with different Hadoop versions?
```
25/01/14 15:38:08 WARN SparkSession: Cannot use org.apache.spark.sql.hudi.HoodieSparkSessionExtension to configure session extensions.
java.lang.NoClassDefFoundError: org/apache/spark/rdd/SecureRDD
```
Note that my setup does work with Emr7.0, Hudi v0.15, Hadoop v3.3.x and Spark 3.5.x, but I am trying to understand the scope of this issue, and if the `SecureRDD` class was deprecated or removed. I could not find any information online, but I may have been looking in the wrong places.
Thanks!
Re: Questions Regarding Compatability of Hudi v0.15, Hadoop v3.4.x and Spark 3.5.x
Posted by Sagar Sumit <co...@apache.org>.
Hi Adam,
Could you please share the full stacktrace? Also, if you could open an issue in Github, it would be much easier to followup there.
Regards,
Sagar
On 2025/01/14 15:57:00 "Li, Adam" wrote:
> Hi,
>
> I am using trying to setup a PySpark Jupyter notebook on an AWS EMR cluster to read Hudi datasets. I am using the latest settings:
>
>
> * Emr7.6, Hudi v0.15, Hadoop v3.4.x and Spark 3.5.x
>
> However, I obtained an error shown below. I have a few questions:
>
>
> 1. I could not find where the API may have changed, but I am wondering if this is due to a version incomptability? I realize I have not linked any code, but I’m using some custom JAR files and setups.
> 2. Is there a matrix somewhere showing compatability of Hudi with different Hadoop versions?
>
> ```
> 25/01/14 15:38:08 WARN SparkSession: Cannot use org.apache.spark.sql.hudi.HoodieSparkSessionExtension to configure session extensions.
> java.lang.NoClassDefFoundError: org/apache/spark/rdd/SecureRDD
> ```
>
> Note that my setup does work with Emr7.0, Hudi v0.15, Hadoop v3.3.x and Spark 3.5.x, but I am trying to understand the scope of this issue, and if the `SecureRDD` class was deprecated or removed. I could not find any information online, but I may have been looking in the wrong places.
>
> Thanks!
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@hudi.apache.org
For additional commands, e-mail: users-help@hudi.apache.org