You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "alexeyegorov (via GitHub)" <gi...@apache.org> on 2025/10/23 21:10:56 UTC

[I] Databricks + DBT + Sedona compatibility [sedona]

alexeyegorov opened a new issue, #2445:
URL: https://github.com/apache/sedona/issues/2445

   At our startup, we use the combination of Sedona and DBT on Databricks. However, the compatibility of latest Sedona versions with Databricks Runtime is not easy or clear to understand. 
   
   Following the documentation I though 1.7.2 and above should be compatible with DBR 16.4 LTS. I was able to run the Sedona spatial functions, but I could not the resulting dataset as Delta file. Using DBT I ran into a following error:
   
   <img width="1616" height="226" alt="Image" src="https://github.com/user-attachments/assets/3d95427b-9840-4559-af75-1698b9e7188d" />
   
   I have tried the following combinations:
   
   - Sedona 1.7.2 + DBR 16.4 (above error)
   - Sedona 1.8.0 + DBR 16.4 (above error)
   - Sedona 1.8.0 + DBR 17.X (some Spark 4.0 incompatibility)
   
   Does anyone here have experience with the setup? If there is something in development that we might test using our setup, I would be open to support that.
   
   Until then, it seems that the Sedona 1.7.0 and DBR 15.4 is just the one stable solution.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Databricks + DBT + Sedona compatibility [sedona]

Posted by "alexeyegorov (via GitHub)" <gi...@apache.org>.
alexeyegorov commented on issue #2445:
URL: https://github.com/apache/sedona/issues/2445#issuecomment-3441501698

   Thanks for your reply @jiayuasu 
   
   We were expecting that Sedona will add support for Delta's new geometry/geography types with the newest release? Or is it possible to achieve this using Iceberg? In a later case, we could hope that Databricks allows users not only to create the metadata for other formats, but to simply write Iceberg tables.
   
   In case we would need to switch to read/write WKB types, that would mean rewriting a lot of code. But it would be possible though while taking a lot of convenience. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Databricks + DBT + Sedona compatibility [sedona]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2445:
URL: https://github.com/apache/sedona/issues/2445#issuecomment-3439210097

   Thank you for your interest in Apache Sedona! We appreciate you opening your first issue. Contributions like yours help make Apache Sedona better.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Databricks + DBT + Sedona compatibility [sedona]

Posted by "alexeyegorov (via GitHub)" <gi...@apache.org>.
alexeyegorov commented on issue #2445:
URL: https://github.com/apache/sedona/issues/2445#issuecomment-4351431889

   Update for anyone after some new tests @jiayuasu:
   
   - combine DBR 16.4 LTS with Sedona 1.8.1 and 1.9.0.
   	- DBR 17.1+ introduced Databricks' ST_ functions making the usage of Sedona more complex
   - set `spark.sedona.enableNativeGeometryType false` to disable Sedona saving the geometry as a native type
   
   After the update I realized that we might need to re-ingest data that was saved as previous UDT binary format using Sedona 1.7.0 using the new version. We also decided to use the EWKB format for that such that the processing with DBT later works the same and it standardized.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Databricks + DBT + Sedona compatibility [sedona]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on issue #2445:
URL: https://github.com/apache/sedona/issues/2445#issuecomment-3439600447

   @alexeyegorov DeltaLake never officially supports `Geometry` data type of Sedona. So even if something that happens to work before, there is no guarantee that it will work in the future. Especially given that the Databricks DeltaLake is not 100% identical to OSS DeltaLake, we can only speculate.
   
   I would recommend one of the following 2 options
   
   1. Read / write geometries via WKB Binary data type in DeltaLake.
   2. Read / write geometries via `geoparquet` data source using Sedona 1.8.0. You will get native geo type experience and spatial filter pushdown too.
   
   
   DBR 17.X is known to have issues with Sedona 1.8.0 and we are waiting for a LTS version of DBR 17 to fix it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org