You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Aaron Grubb <aa...@kaden.ai> on 2025/01/13 18:49:37 UTC

Storing a JDBC-based table in a catalog for direct use in Spark SQL

Hi all,

I'm trying to figure out how to persist a table definition in a catalog that can be used from different sessions. Something along the lines
of 

-------------------
CREATE TABLE spark_catalog.default.test_table (
    name string
)
USING jdbc
OPTIONS (
    driver 'com.mysql.cj.jdbc.Driver',
    url 'jdbc:mysql://example.com:3306/db',
    user 'user',
    password 'pass',
    query 'SELECT name FROM test_table WHERE type_id = 2'
)
--------------------

and then from another session, directly calling

--------------------
SparkSession.builder.getOrCreate().sql('SELECT * FROM spark_catalog.default.test_table').show()
--------------------

However, I'm unable to figure out how to accomplish this. I initially tried to define the table in a Nessie catalog but it won't accept
"USING JDBC". I have also tried setting spark.sql.warehouse.dir to a location in S3 and setting enableHiveSupport() on the session, however
creating this table under these circumstances only creates an empty directory and the table doesn't show up in the next session. Would I need
to set up a Hive Metastore to accomplish this or what other options do I have?

Thanks,
Aaron

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Storing a JDBC-based table in a catalog for direct use in Spark SQL

Posted by Aaron Grubb <aa...@kaden.ai>.
In case anyone comes across this, I managed to accomplish this using the Hive Standalone Metastore, with help from an article [1]. Required
configurations:

spark.sql.warehouse.dir=s3a://<BUCKET>/
spark.sql.catalogImplementation=hive
hive.metastore.uris=thrift://<URL>:9083

This enables me to run the example successfully. I would still be interested in knowing if there's another way to accomplish this,
particularly without an external service.

[1]
https://christiaan-viljoen.medium.com/how-to-deploy-a-minimal-apache-hive-standalone-metastore-to-enable-delta-lake-support-for-trino-on-6b03964882a2

On Mon, 2025-01-13 at 18:49 +0000, Aaron Grubb wrote:
> Hi all,
> 
> I'm trying to figure out how to persist a table definition in a catalog that can be used from different sessions. Something along the lines
> of 
> 
> -------------------
> CREATE TABLE spark_catalog.default.test_table (
>     name string
> )
> USING jdbc
> OPTIONS (
>     driver 'com.mysql.cj.jdbc.Driver',
>     url 'jdbc:mysql://example.com:3306/db',
>     user 'user',
>     password 'pass',
>     query 'SELECT name FROM test_table WHERE type_id = 2'
> )
> --------------------
> 
> and then from another session, directly calling
> 
> --------------------
> SparkSession.builder.getOrCreate().sql('SELECT * FROM spark_catalog.default.test_table').show()
> --------------------
> 
> However, I'm unable to figure out how to accomplish this. I initially tried to define the table in a Nessie catalog but it won't accept
> "USING JDBC". I have also tried setting spark.sql.warehouse.dir to a location in S3 and setting enableHiveSupport() on the session, however
> creating this table under these circumstances only creates an empty directory and the table doesn't show up in the next session. Would I
> need
> to set up a Hive Metastore to accomplish this or what other options do I have?
> 
> Thanks,
> Aaron
> 
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> 


---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Incorrect Results and SIGSEGV on Read with Iceberg + PySpark + Nessie

Posted by Aaron Grubb <aa...@kaden.ai>.
Someone just replied to the bug, it was already known about and will be fixed in the upcoming Iceberg 1.7.2 release.

On Thu, 2025-02-06 at 09:35 +0000, Aaron Grubb wrote:
> Hi all,
> 
> I filed a bug with the Iceberg team [1] but I'm not sure that it's 100% specific to Iceberg (I assume it is as data in the related parquet
> file is correct and session.read.parquet always returns correct results) so I figured I would flag it here in case anyone has some insight.
> Currently planning on trying different versions of Iceberg to see if I can get correct results but this is a major blocker to my company's
> infrastructure planning so any help would be appreciated. Maybe it's wiser to try different Java versions or is it possible this is related
> to using aarch64 somehow? All JARs are precompiled versions.
> 
> Thanks,
> Aaron
> 
> [1] https://github.com/apache/iceberg/issues/12178
> 
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> 


---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Incorrect Results and SIGSEGV on Read with Iceberg + PySpark + Nessie

Posted by Aaron Grubb <aa...@kaden.ai>.
Hi all,

I filed a bug with the Iceberg team [1] but I'm not sure that it's 100% specific to Iceberg (I assume it is as data in the related parquet
file is correct and session.read.parquet always returns correct results) so I figured I would flag it here in case anyone has some insight.
Currently planning on trying different versions of Iceberg to see if I can get correct results but this is a major blocker to my company's
infrastructure planning so any help would be appreciated. Maybe it's wiser to try different Java versions or is it possible this is related
to using aarch64 somehow? All JARs are precompiled versions.

Thanks,
Aaron

[1] https://github.com/apache/iceberg/issues/12178

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org