You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "BruAPAHE (via GitHub)" <gi...@apache.org> on 2025/05/21 15:59:06 UTC

[I] RS_FromNetCDF fails on NetCDF4/HDF5 with filter type=32015 (e.g. ZSTD) [sedona]

BruAPAHE opened a new issue, #1951:
URL: https://github.com/apache/sedona/issues/1951

   I encountered a runtime failure when using RS_FromNetCDF(content, 'v') on a valid NetCDF4 file stored in S3.
   
   The error is:
   
   ```Exception occurred while evaluating expression RS_FromNetCDF - inputs: [[B@xxxx, v], cause: Unknown filter type=32015```
   
   This happens because the file uses an HDF5 compression filter (likely BLOSC, filter ID 32015), which is not supported by the version of netcdf-java (4.6.11) used in netcdfAll.
   
   Steps to reproduce:
   
       Load .nc file containing BLOSC/ZSTD compression into Spark via binaryFile.
   
       Try to evaluate RS_FromNetCDF(content, 'variable').
   
   A workaround is to re-encode the file using nccopy or xarray to remove unsupported filters, but this limitation is undocumented and leads to runtime crashes.
   
   Expected:
   
       An informative exception or warning.
   
       Ideally: support for filter 32015 
   
   Environment:
   
       Sedona: 1.7.1
   
       Spark: 3.5.0
   
       File: NetCDF4/HDF5 with ZSTD filter


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] RS_FromNetCDF fails on NetCDF4/HDF5 with filter type=32015 (e.g. ZSTD) [sedona]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1951:
URL: https://github.com/apache/sedona/issues/1951#issuecomment-2898481500

   Thank you for your interest in Apache Sedona! We appreciate you opening your first issue. Contributions like yours help make Apache Sedona better.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] RS_FromNetCDF fails on NetCDF4/HDF5 with filter type=32015 (e.g. ZSTD) [sedona]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu closed issue #1951: RS_FromNetCDF fails on NetCDF4/HDF5 with filter type=32015 (e.g. ZSTD)
URL: https://github.com/apache/sedona/issues/1951


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] RS_FromNetCDF fails on NetCDF4/HDF5 with filter type=32015 (e.g. ZSTD) [sedona]

Posted by "minhpham1810 (via GitHub)" <gi...@apache.org>.
minhpham1810 commented on issue #1951:
URL: https://github.com/apache/sedona/issues/1951#issuecomment-5689106134

   Hi, I'd like to work on this issue. Filter 32015 is Zstandard (BLOSC is 32001). Upgrading netcdf-java won't fix this: even the latest release, 5.10.0, has no Zstandard filter. It only adds BLOSC, through the separate `libblosc2-jna` module.
   
   Plan:
   1. Rethrow `UnknownFilterException` with a clear message that names the filter ID and the `nccopy -F none` workaround, and document the supported filters.
   2. Add Zstandard support by registering a netcdf-java `FilterProvider` for ID 32015, backed by `zstd-jni`, which Spark already ships.
   
   Let me know if that works.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] RS_FromNetCDF fails on NetCDF4/HDF5 with filter type=32015 (e.g. ZSTD) [sedona]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on issue #1951:
URL: https://github.com/apache/sedona/issues/1951#issuecomment-2898537256

   @BruAPAHE Thanks for reporting. I believe recently netcdf-java introduced this support. Do you want to create a PR to upgrade the dependency?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] RS_FromNetCDF fails on NetCDF4/HDF5 with filter type=32015 (e.g. ZSTD) [sedona]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on issue #1951:
URL: https://github.com/apache/sedona/issues/1951#issuecomment-5691627803

   ok


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org