You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kevinjqliu (via GitHub)" <gi...@apache.org> on 2024/11/13 21:34:25 UTC
[I] [Investigate] Whether `data_files` metadata table requires both pyarrow and s3fs [iceberg-python]
kevinjqliu opened a new issue, #1317:
URL: https://github.com/apache/iceberg-python/issues/1317
### Apache Iceberg version
None
### Please describe the bug 🐞
From devlist, https://lists.apache.org/thread/lvtx80c2b47ckpfrylo4ot5o00nfb5r4
```
When attempting to use data_files metadata tables, I needed to install both pyarrow and s3fs. I would have expected that I would just need pyarrow if that was the FileIO implementation.
```
Lets try to reproduce. Seems like `s3fs` is only used for fsspec https://grep.app/search?q=s3fs&filter[repo][0]=apache/iceberg-python&filter[lang][0]=Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Investigate] Whether `data_files` metadata table requires both pyarrow and s3fs [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu closed issue #1317: [Investigate] Whether `data_files` metadata table requires both pyarrow and s3fs
URL: https://github.com/apache/iceberg-python/issues/1317
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Investigate] Whether `data_files` metadata table requires both pyarrow and s3fs [iceberg-python]
Posted by "jiakai-li (via GitHub)" <gi...@apache.org>.
jiakai-li commented on issue #1317:
URL: https://github.com/apache/iceberg-python/issues/1317#issuecomment-2549089683
Cool, thanks @kevinjqliu . Let's close this one then.
At the mean time, as you mentioned, I'll verify the `fsspec` and open another issue if I found any problem for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Investigate] Whether `data_files` metadata table requires both pyarrow and s3fs [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1317:
URL: https://github.com/apache/iceberg-python/issues/1317#issuecomment-2549081163
> From the posted meesage, I see pyarrow is an expected dependency but s3fs is the problematic one
Oh you're right. Sorry for the confusion. From the original message "`I would have expected that I would just need pyarrow if that was the FileIO implementation`".
Looking into `s3fs` usage, its imported with `fsspec` https://github.com/search?q=repo%3Aapache%2Ficeberg-python+s3fs+language%3APython+path%3A%2F%5Epyiceberg%5C%2Fio%5C%2F%2F&type=code
This means that depending on which FileIO is chosen, the required library is different.
* `fsspec` requires `s3fs` to read from s3
* `pyarrow` requires only `pyarrow` and not `s3fs`
Your comment above verified the `pyarrow` path, without `s3fs`.
And I would assume `fsspec` doesn't use `pyarrow` at all (we can verify this).
I think we can close this investigation based on the above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Investigate] Whether `data_files` metadata table requires both pyarrow and s3fs [iceberg-python]
Posted by "jiakai-li (via GitHub)" <gi...@apache.org>.
jiakai-li commented on issue #1317:
URL: https://github.com/apache/iceberg-python/issues/1317#issuecomment-2549030114
Thank you @kevinjqliu . I might need some more help understanding the issue I'm trying to reproduce.
From the posted meesage, I see `pyarrow` is an expected dependency but `s3fs` is the problematic one. Also from the `InspectTable.data_files` method I see it actually returns a `pa.Table` object. I think in this case the `pyarrow` cannot be bypassed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Investigate] Whether `data_files` metadata table requires both pyarrow and s3fs [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1317:
URL: https://github.com/apache/iceberg-python/issues/1317#issuecomment-2541919224
Hi @jiakai-li thanks for looking into this!
> It was run with the s3fs module removed from the environment, which runs ok:
I think you'd want to remove `pyarrow` instead. `data_files` is a metadata table that only requires reading metadata files from the object store using s3fs. `pyarrow` should not be necessary to read metadata files
I think the issue is related to the import here
https://github.com/apache/iceberg-python/blob/a97d13c17cd03f86252b9df2c65532ec45fb05da/pyiceberg/table/inspect.py#L476-L480
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Investigate] Whether `data_files` metadata table requires both pyarrow and s3fs [iceberg-python]
Posted by "jiakai-li (via GitHub)" <gi...@apache.org>.
jiakai-li commented on issue #1317:
URL: https://github.com/apache/iceberg-python/issues/1317#issuecomment-2525310075
I was trying to identify the issue, but I can't seem to reproduce it, not sure if I've done it wrong (sorry that I'm very new to the community and this is the second issue I tried to work with). Below is the code I tried in order to reproduce the issue. It was run with the `s3fs` module removed from the environment, which runs ok:
```python
from pyiceberg.catalog.sql import SqlCatalog
warehouse_path = r"/tmp/warehouse"
catalog = SqlCatalog(
"default",
**{
"uri": f"sqlite:///{warehouse_path}/pyiceberg_catalog.db",
"warehouse": f"file://{warehouse_path}",
},
)
import pyarrow.parquet as pq
df = pq.read_table(r"/tmp/warehouse/yellow_tripdata_2024-01.parquet")
table = catalog.load_table('default.taxi_dataset')
table.append(df)
from pyiceberg.table import InspectTable
inspect = InspectTable(table)
inspect.data_files().to_pandas()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org