[I] DataFrame.to_parquet can't write to remote URLs (s3://, gs://) while COPY TO can [sedona-db]

["jiayuasu (via GitHub)" <gi...@apache.org>]

jiayuasu opened a new issue, #1440:
URL: https://github.com/apache/sedona-db/issues/1440

   `DataFrame.to_parquet()` can't write to a remote URL (`s3://`, `gs://`, `https://`), while SQL `COPY ... TO` can. Reported on #1403 (`sedonadb-geopandas`'s `to_parquet`, which calls it).
   
   - `to_parquet("s3://bucket/out.parquet")` fails with `No suitable object store found for s3:%2F. See RuntimeEnv::register_object_store`. Nothing registers an object store for the output URL, unlike reads (`ensure_object_store_registered_with_options` in `rust/sedona/src/context.rs`) and SQL `COPY ... TO` (`register_object_store_and_config_extensions` in `rust/sedona/src/exec.rs`).
   - There is no way to pass store options: `to_parquet(options=...)` takes Parquet writer options only (`Config value "aws" not found on ParquetOptions`).
   - Even with a store already registered for the bucket by an earlier `read_parquet(..., options=...)`, the write still fails the same way. The URL seems to be mangled on the way (`s3://...` reported as `s3:%2F`).
   
   ```python
   import sedonadb
   
   sd = sedonadb.connect()
   df = sd.sql("SELECT 1 AS a")
   df.to_view("t")
   bucket = "s3://sedona-nonexistent-write-probe-7f3a9c"  # does not exist, so nothing is written
   options = {"aws.skip_signature": True, "aws.region": "us-west-2"}
   
   # SQL COPY TO registers the store and reaches S3 (the PUT fails only because
   # the bucket does not exist)
   try:
       sd.sql(
           f"COPY t TO '{bucket}/out.parquet' "
           "OPTIONS ('aws.skip_signature' 'true', 'aws.region' 'us-west-2')"
       ).to_arrow_table()
   except Exception as e:
       print(str(e).splitlines()[0][:100])  # ... Error performing PUT https://s3.us-west-2.amazonaws.com/...
   
   # DataFrame.to_parquet does not
   try:
       df.to_parquet(f"{bucket}/out.parquet")
   except Exception as e:
       print(str(e).splitlines()[0])  # No suitable object store found for s3:%2F. ...
   
   # its options are Parquet writer options only
   try:
       df.to_parquet(f"{bucket}/out.parquet", options=options)
   except Exception as e:
       print(str(e).splitlines()[0])  # Config value "aws" not found on ParquetOptions
   
   # and a store registered by a read of the same bucket is not found either
   try:
       sd.read_parquet(f"{bucket}/in/", options=options)
   except Exception:
       pass
   try:
       df.to_parquet(f"{bucket}/out.parquet")
   except Exception as e:
       print(str(e).splitlines()[0])  # No suitable object store found for s3:%2F. ...
   ```
   
   Registering the output URL's store as `COPY ... TO` does, and accepting store options (for example a separate `storage_options` argument, since `options` already means writer options), would let `to_parquet()` write to S3 and other object stores.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] DataFrame.to_parquet can't write to remote URLs (s3://, gs://) while COPY TO can [sedona-db]

["jiayuasu (via GitHub)" <gi...@apache.org>]

jiayuasu closed issue #1440: DataFrame.to_parquet can't write to remote URLs (s3://, gs://) while COPY TO can
URL: https://github.com/apache/sedona-db/issues/1440


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org