You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kevinjqliu (via GitHub)" <gi...@apache.org> on 2024/08/11 17:33:12 UTC
[I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
kevinjqliu opened a new issue, #1041:
URL: https://github.com/apache/iceberg-python/issues/1041
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
PyIceberg assumes the same FS implementation is used for reading both metadata and data.
However, I want to use a catalog with local FS as the warehouse while referencing S3 files as data.
The `fs` implementation is determined by metadata location, which is then passed down to the function which reads the data file.
https://github.com/apache/iceberg-python/blob/d8b5c17cadbc99e53d08ade6109283ee73f0d83e/pyiceberg/io/pyarrow.py#L1428-L1430
See this [example Jupyter notebook](https://gist.github.com/kevinjqliu/647808faba256855639e91dd58243082) to reproduce
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1041:
URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324750207
Thanks for taking a look at this @TiansuYu
> why we are implementing a custom
I think custom scheme parsing avoids picking one library over another (`fsspec` vs `pyarrow`). `fsspec.core.get_fs_token_paths` seems like an interesting replacement when using `fsspec`.
> (On a side note: this looks a bit confusing to me, as why for gs, FSSPEC_FILE_IO is not added as a viable io method as we have added gcfs into extras.)
Good catch, `fsspec` should be included since GCS is supported
https://github.com/apache/iceberg-python/blob/dc6d2429aafbffc626cba53aaac3f6198fc37eb3/pyiceberg/io/fsspec.py#L161-L176
> (Also I am not sure why pyarrow is not using fsspec as io layer but implement things on their own.)
`pyarrow` is using its own native fs implementations
https://github.com/apache/iceberg-python/blob/dc6d2429aafbffc626cba53aaac3f6198fc37eb3/pyiceberg/io/pyarrow.py#L348-L403
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
Posted by "TiansuYu (via GitHub)" <gi...@apache.org>.
TiansuYu commented on issue #1041:
URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324181879
I will have a look this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1041:
URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2557842068
assigned to you @jiakai-li
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1041:
URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324804328
My preference is resolving `fs` at the file level. It's more flexible and the performance difference _should be_ negligible.
Another reason is to be able to write data across clouds. Technically, I can write to multiple clouds, across AWS, Azure, and GCP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
Posted by "TiansuYu (via GitHub)" <gi...@apache.org>.
TiansuYu commented on issue #1041:
URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324512692
A generic question: why we are implementing a custom
https://github.com/apache/iceberg-python/blob/dc6d2429aafbffc626cba53aaac3f6198fc37eb3/pyiceberg/io/__init__.py#L320
instead of using `fsspec.get_fs_token_path(file_path)[0]`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #1041:
URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2298248947
This is a good point, I've heard that folks store their metadata on HDFS, and the data itself on S3.
I don't think the example with the add-files is the best, it would be better to support the `write.data.path` and `write.metadata.path` configuration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
Posted by "TiansuYu (via GitHub)" <gi...@apache.org>.
TiansuYu commented on issue #1041:
URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324566016
Read my comment [here](https://gist.github.com/kevinjqliu/647808faba256855639e91dd58243082?permalink_comment_id=5175413#gistcomment-5175413) for the cause of the issue.
I dont think fixing `SqlCatalog` alone is the proper answer to this bug. The io layer seems to me ill written and has to be fixed somewhere in the uppper level (e.g. `FsspecInputFile` or `InputFile`).
Let me know what do you think, then we can come up with a way to properly address this. 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
Posted by "jiakai-li (via GitHub)" <gi...@apache.org>.
jiakai-li commented on issue #1041:
URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2557828231
Hey guys, I can pick this up together with #1279 if no one is currently working on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1041:
URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324758344
> I dont think fixing SqlCatalog alone is the proper answer to this bug. The io layer seems to me ill written and has to be fixed somewhere in the uppper level (e.g. FsspecInputFile or InputFile).
yea, the main issue is the assumption that the same `io` (and `fs` implementation) is used for reading both data and metadata files. The example you pointed to pass in the `io` parameter
https://github.com/apache/iceberg-python/blob/dc6d2429aafbffc626cba53aaac3f6198fc37eb3/pyiceberg/table/__init__.py#L655-L657
Instead, we would want to recreate `io`/`fs` based on the file currently being processed.
Here's another example of passing in the `io` parameter on the write path
https://github.com/apache/iceberg-python/blob/dc6d2429aafbffc626cba53aaac3f6198fc37eb3/pyiceberg/table/__init__.py#L530-L532
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
Posted by "TiansuYu (via GitHub)" <gi...@apache.org>.
TiansuYu commented on issue #1041:
URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324857080
Reading on table spec, I just realised that there is a field `location` in https://iceberg.apache.org/spec/#table-metadata-fields that specifies a base location of the table. Does Iceberg Java actually allows the split between metadata and data location?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
Posted by "jiakai-li (via GitHub)" <gi...@apache.org>.
jiakai-li commented on issue #1041:
URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2574308740
@kevinjqliu I guess we can close this issue now? At the meantime, I'm keen to work on the `write.data.path` and `write.metadata.path` if that's something we want to enable and no one else is currently working on it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1041:
URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324865049
Its configurable via the write properties. See this comment https://github.com/apache/iceberg-python/issues/1041#issuecomment-2323380629
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1041:
URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324831712
yep! There's definitely opportunities to consolidate the two. I opened #310 with some details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
Posted by "TiansuYu (via GitHub)" <gi...@apache.org>.
TiansuYu commented on issue #1041:
URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324794360
@kevinjqliu I think resolving fs at file level should make the API cleaner. I would say one benefit one might choose fs on table level is to reuse that fs instance for performance boost. Otherwise, I would say we need to make two io configs, one for metadata, one for data on the `MetastoreCatalog` or `Catalog` level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
Posted by "Sairam90 (via GitHub)" <gi...@apache.org>.
Sairam90 commented on issue #1041:
URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2557215917
Any updates on this issue , I face a similar issue when creating a table on S3 as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
Posted by "TiansuYu (via GitHub)" <gi...@apache.org>.
TiansuYu commented on issue #1041:
URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324809059
I will make a PR according to this:
> My preference is resolving fs at the file level. It's more flexible and the performance difference should be negligible.
> Another reason is to be able to write data across clouds. Technically, I can write to multiple clouds, across AWS, Azure, and GCP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu closed issue #1041: [Bug] Cannot use PyIceberg with multiple FS
URL: https://github.com/apache/iceberg-python/issues/1041
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1041:
URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2323380629
Oh interesting, thanks!
Here's the config definition for `write.data.path` and `write.metadata.path`
https://iceberg.apache.org/docs/latest/configuration/#write-properties
I also found an old issue referencing this #161
Another example usage is to use both local FS and S3(minio), which might be easier to set up and test against
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1041:
URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324762218
Generally, this problem should go away if we re-evaluate `fs` and `io` each time a file is read and written. Or other words, we should stop passing the `io` parameter around.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
Posted by "TiansuYu (via GitHub)" <gi...@apache.org>.
TiansuYu commented on issue #1041:
URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2324824803
Also reading on here:
https://arrow.apache.org/docs/python/filesystems.html#using-arrow-filesystems-with-fsspec
There might be some opportunity that we simplify the split between arrow and fsspec file_system.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1041:
URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2575575091
@jiakai-li we can close this issue!
> At the meantime, I'm keen to work on the write.data.path and write.metadata.path if that's something we want to enable and no one else is currently working on it?
Let's open a new issue for those
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org