You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/09/20 07:56:26 UTC
[GitHub] [iceberg] Fokko opened a new issue, #5801: Read iceberg table into a PyArrow Dataset
Fokko opened a new issue, #5801:
URL: https://github.com/apache/iceberg/issues/5801
### Feature Request / Improvement
Convert an Iceberg table into a PyArrow dataset. This would unlock all the users that use Arrow and Pandas. This way users can read data that fits into memory from a huge iceberg table, and iterate on it quickly.
### Query engine
Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on issue #5801: Read iceberg table into a PyArrow Dataset
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #5801:
URL: https://github.com/apache/iceberg/issues/5801#issuecomment-1448936674
@ianmcook that is correct. If you have a petabyte Iceberg table, and you'll select a subset of the rows, only the relevant files are being read. This is directly read into a `pa.table`, and not a `pa.dataset`. If we want to make this truly lazy, the Iceberg primitives need to be implemented on the PyArrow side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ianmcook commented on issue #5801: Read iceberg table into a PyArrow Dataset
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on issue #5801:
URL: https://github.com/apache/iceberg/issues/5801#issuecomment-1448931543
Nit: the term _Dataset_ has a [specific meaning in PyArrow](https://arrow.apache.org/docs/python/dataset.html). It refers to a lazily read set of files on disk. As opposed to a _Table_ in PyArrow which is materialized in memory. I believe what has been implemented in #6233 is converting an Iceberg Table to a PyArrow _Table_.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] djouallah commented on issue #5801: Read iceberg table into a PyArrow Dataset
Posted by "djouallah (via GitHub)" <gi...@apache.org>.
djouallah commented on issue #5801:
URL: https://github.com/apache/iceberg/issues/5801#issuecomment-1454645179
@Fokko that's fine, the confusion is in the title of the bug report, Pyarrow dataset means a view, but currently to_duckdb() load a table that's two different thing, that's the same point raised by @ianmcook
the row_filter does not work anyway in 0.3, so basically we end up with crashing the local session as it loads the whole table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on issue #5801: Read iceberg table into a PyArrow Dataset
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #5801:
URL: https://github.com/apache/iceberg/issues/5801#issuecomment-1454632236
@djouallah you are right on that.
Currently, it is still a two-stage rocket, you first need to prune the data using the PyIceberg `row_filter`:
```python
catalog = load_catalog('default')
tbl = catalog.load_table('nyc.taxis')
sc = tbl.scan(row_filter=And(
GreaterThanOrEqual("tpep_pickup_datetime", "2022-01-01T00:00:00.000000+00:00")
LessThan("tpep_pickup_datetime", "2022-02-01T00:00:00.000000+00:00")
).to_arrow() # Same for .to_duckdb()
```
This will load all the data from January. If you don't pass in a `row_filter`, it will pull in the current version of the table. Right now, it won't push the predicate down to the Iceberg layer, unfortunately. This is something that we're working on but requires tighter integration with PyArrow and DuckDB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dungdm93 commented on issue #5801: Read iceberg table into a PyArrow Dataset
Posted by GitBox <gi...@apache.org>.
dungdm93 commented on issue #5801:
URL: https://github.com/apache/iceberg/issues/5801#issuecomment-1324546747
Implemented in #6233
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] djouallah commented on issue #5801: Read iceberg table into a PyArrow Dataset
Posted by GitBox <gi...@apache.org>.
djouallah commented on issue #5801:
URL: https://github.com/apache/iceberg/issues/5801#issuecomment-1264314702
Looking forward to use it with DuckDB :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] djouallah commented on issue #5801: Read iceberg table into a PyArrow Dataset
Posted by GitBox <gi...@apache.org>.
djouallah commented on issue #5801:
URL: https://github.com/apache/iceberg/issues/5801#issuecomment-1324555305
@dungdm93 trust me, I made sure every one knows about it :) , please don't close till the python binary is released :)
https://twitter.com/mim_djo/status/1594821677904715776
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] djouallah commented on issue #5801: Read iceberg table into a PyArrow Dataset
Posted by "djouallah (via GitHub)" <gi...@apache.org>.
djouallah commented on issue #5801:
URL: https://github.com/apache/iceberg/issues/5801#issuecomment-1454346679
either there is a bug or it is really loading a table not a view , testing it in colab and it is Out of memory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on issue #5801: Read iceberg table into a PyArrow Table
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #5801:
URL: https://github.com/apache/iceberg/issues/5801#issuecomment-1454675576
> the row_filter does not work anyway in 0.3
Can you elaborate? Or even better, create a ticket for this?
> do you mind changing the title of the bug and replace dataset with table.
No problem at all
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on issue #5801: Read iceberg table into a PyArrow Dataset
Posted by GitBox <gi...@apache.org>.
Fokko commented on issue #5801:
URL: https://github.com/apache/iceberg/issues/5801#issuecomment-1329067464
Since this is in, I'm going to close this to clean up the milestone for the next release. This way we can see what needs to be done before creating the Release Candidate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko closed issue #5801: Read iceberg table into a PyArrow Dataset
Posted by GitBox <gi...@apache.org>.
Fokko closed issue #5801: Read iceberg table into a PyArrow Dataset
URL: https://github.com/apache/iceberg/issues/5801
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org