You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Pani Dhakshnamurthy <dp...@gmail.com> on 2023/04/28 19:28:13 UTC
Help needed with Iceberg on Databricks platform
Hi all,
I am currently attempting to make Iceberg work on the Databricks platform.
Without adding IcebergSparkSessionExtensions, I am able to perform both
data and metadata reads. However, when I add IcebergSparkSessionExtensions
with spark.sql.extensions, Databricks throws the following error:
Caused by: java.lang.NoClassDefFoundError:
org/apache/spark/sql/catalyst/analysis/ResolveProcedures
... 41 more
Caused by: java.lang.ClassNotFoundException:
org.apache.spark.sql.catalyst.analysis.ResolveProcedures
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
... 41 more
If I don't add IcebergSparkSessionExtensions, I receive the following error
from DBR when I execute UPDATE SQL: Error in SQL statement:
DeltaAnalysisException: UPDATE destination only supports Delta sources.
The Databricks 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12) runtime
comes with a shaded Iceberg jar with version 0.11. For this testing, I
removed it and added iceberg-spark-runtime-3.3_2.12-1.2.1.jar to the
classpath.
Has anyone successfully used Databricks to perform all Iceberg supported
features? Any help would be greatly appreciated. Thank you in advance.
Best regards,
Pani
Re: Help needed with Iceberg on Databricks platform
Posted by Anton Okolnychyi <ao...@apple.com.INVALID>.
It looks like you are still pulling in some 0.11 version classes. That version is very old and may not contain `ResolveProcedures`. I assume you have classpath first flag enabled?
- Anton
> On Apr 28, 2023, at 3:18 PM, Pani Dhakshnamurthy <dp...@gmail.com> wrote:
>
> Hi John,
> I tried using both the "set spark.sql.extensions" command and "spark.conf.get("spark.sql.extensions")" method, but both returned "undefined."
>
> To resolve this issue, I attempted to set the following values:
>
> spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
>
> spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
>
> However, these attempts were unsuccessful.
>
> I'm not sure if Databricks allows users to override the spark.sql.extensions configuration.
>
> Thank you for your help.
>
> Thanks
> Pani
>
>
> On Fri, Apr 28, 2023 at 3:04 PM John Zhuge <jzhuge@apache.org <ma...@apache.org>> wrote:
> Have you tried this?
>
> Copy the value of `spark.sql.extensions` (a comma separated list) from an existing Spark job on Databricks
> Append `org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions` to the comma separated list
> Submit a new job with `spark.sql.extensions` set to the modified value
>
> On Fri, Apr 28, 2023 at 12:28 PM Pani Dhakshnamurthy <dpaani@gmail.com <ma...@gmail.com>> wrote:
> Hi all,
>
> I am currently attempting to make Iceberg work on the Databricks platform. Without adding IcebergSparkSessionExtensions, I am able to perform both data and metadata reads. However, when I add IcebergSparkSessionExtensions with spark.sql.extensions, Databricks throws the following error:
>
> Caused by: java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/analysis/ResolveProcedures
> ... 41 more
> Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.catalyst.analysis.ResolveProcedures
> at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
> ... 41 more
>
> If I don't add IcebergSparkSessionExtensions, I receive the following error from DBR when I execute UPDATE SQL: Error in SQL statement: DeltaAnalysisException: UPDATE destination only supports Delta sources.
>
> The Databricks 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12) runtime comes with a shaded Iceberg jar with version 0.11. For this testing, I removed it and added iceberg-spark-runtime-3.3_2.12-1.2.1.jar to the classpath.
>
> Has anyone successfully used Databricks to perform all Iceberg supported features? Any help would be greatly appreciated. Thank you in advance.
>
> Best regards,
> Pani
>
>
> --
> John Zhuge
Re: Help needed with Iceberg on Databricks platform
Posted by Pani Dhakshnamurthy <dp...@gmail.com>.
Hi John,
I tried using both the "set spark.sql.extensions" command and
"spark.conf.get("spark.sql.extensions")" method, but both returned
"undefined."
To resolve this issue, I attempted to set the following values:
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
However, these attempts were unsuccessful.
I'm not sure if Databricks allows users to override the
spark.sql.extensions configuration.
Thank you for your help.
Thanks
Pani
On Fri, Apr 28, 2023 at 3:04 PM John Zhuge <jz...@apache.org> wrote:
> Have you tried this?
>
>
> 1. Copy the value of `spark.sql.extensions` (a comma separated
> list) from an existing Spark job on Databricks
> 2. Append
> `org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions` to the
> comma separated list
> 3. Submit a new job with `spark.sql.extensions` set to the modified
> value
>
>
> On Fri, Apr 28, 2023 at 12:28 PM Pani Dhakshnamurthy <dp...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I am currently attempting to make Iceberg work on the Databricks
>> platform. Without adding IcebergSparkSessionExtensions, I am able to
>> perform both data and metadata reads. However, when I add
>> IcebergSparkSessionExtensions with spark.sql.extensions, Databricks throws
>> the following error:
>>
>> Caused by: java.lang.NoClassDefFoundError:
>> org/apache/spark/sql/catalyst/analysis/ResolveProcedures
>> ... 41 more
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.spark.sql.catalyst.analysis.ResolveProcedures
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
>> ... 41 more
>>
>> If I don't add IcebergSparkSessionExtensions, I receive the following
>> error from DBR when I execute UPDATE SQL: Error in SQL statement:
>> DeltaAnalysisException: UPDATE destination only supports Delta sources.
>>
>> The Databricks 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12) runtime
>> comes with a shaded Iceberg jar with version 0.11. For this testing, I
>> removed it and added iceberg-spark-runtime-3.3_2.12-1.2.1.jar to the
>> classpath.
>>
>> Has anyone successfully used Databricks to perform all Iceberg supported
>> features? Any help would be greatly appreciated. Thank you in advance.
>>
>> Best regards,
>> Pani
>>
>
>
> --
> John Zhuge
>
Re: Help needed with Iceberg on Databricks platform
Posted by John Zhuge <jz...@apache.org>.
Have you tried this?
1. Copy the value of `spark.sql.extensions` (a comma separated
list) from an existing Spark job on Databricks
2. Append
`org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions` to the
comma separated list
3. Submit a new job with `spark.sql.extensions` set to the modified value
On Fri, Apr 28, 2023 at 12:28 PM Pani Dhakshnamurthy <dp...@gmail.com>
wrote:
> Hi all,
>
> I am currently attempting to make Iceberg work on the Databricks platform.
> Without adding IcebergSparkSessionExtensions, I am able to perform both
> data and metadata reads. However, when I add IcebergSparkSessionExtensions
> with spark.sql.extensions, Databricks throws the following error:
>
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/spark/sql/catalyst/analysis/ResolveProcedures
> ... 41 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.spark.sql.catalyst.analysis.ResolveProcedures
> at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
> ... 41 more
>
> If I don't add IcebergSparkSessionExtensions, I receive the following
> error from DBR when I execute UPDATE SQL: Error in SQL statement:
> DeltaAnalysisException: UPDATE destination only supports Delta sources.
>
> The Databricks 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12) runtime
> comes with a shaded Iceberg jar with version 0.11. For this testing, I
> removed it and added iceberg-spark-runtime-3.3_2.12-1.2.1.jar to the
> classpath.
>
> Has anyone successfully used Databricks to perform all Iceberg supported
> features? Any help would be greatly appreciated. Thank you in advance.
>
> Best regards,
> Pani
>
--
John Zhuge