You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by "ikrommyd (via GitHub)" <gi...@apache.org> on 2025/10/28 20:00:45 UTC
[D] Iterating over parquet dataset in batches [arrow]
GitHub user ikrommyd created a discussion: Iterating over parquet dataset in batches
Hello,
I would like to iterate over a parquet dataset in batches.
I get my parquet dataset like this
```py
import pyarrow.parquet as pq
dataset = pq.ParquetDataset(source_path, filters=cat_dict[cat]["cat_filter"])
```
However, the `pyarrow.parquet.ParquetDataset` class doesn't seem to have a batched iteration method.
After very briefly looking at its source code, I found that I can access the underlying `pyarrow.dataset.Dataset` by with the `_dataset` attribute.
Therefore I'm doing this to iterate
```py
for batch in dataset._dataset.to_batches(filter=dataset._filter_expression):
...
```
That feels a bit hacky to me because I'm depending on internals like `dataset._dataset` and `dataset._filter_expression`. Is this the proper way to do this? Is there a better API that I couldn't find that users should be using?
Thanks in advance!
GitHub link: https://github.com/apache/arrow/discussions/47988
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] Iterating over parquet dataset in batches [arrow]
Posted by "sidneymau (via GitHub)" <gi...@apache.org>.
GitHub user sidneymau edited a comment on the discussion: Iterating over parquet dataset in batches
I believe the `dataset` module is the preferred way to do this:
```python
import pyarrow.dataset as ds
dataset = ds.dataset(source_path, filters=cat_dict[cat]["cat_filter"]))
for batch in dataset.to_batches():
...
```
See https://arrow.apache.org/docs/python/dataset.html for some more docs
GitHub link: https://github.com/apache/arrow/discussions/47988#discussioncomment-14807877
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] Iterating over parquet dataset in batches [arrow]
Posted by "ikrommyd (via GitHub)" <gi...@apache.org>.
GitHub user ikrommyd edited a comment on the discussion: Iterating over parquet dataset in batches
> I believe the `dataset` module is the preferred way to do this:
>
> ```python
> import pyarrow.dataset as ds
>
> dataset = ds.dataset(source_path)
>
> for batch in dataset.to_batches(filter=cat_dict[cat]["cat_filter"]):
> ...
> ```
>
> See https://arrow.apache.org/docs/python/dataset.html for some more docs
Yeah I had seen that. The problem I had encountered was this
```py
In [11]: filters = [("pt", ">", -1.0)]
In [12]: dataset = pq.ParquetDataset("storage/NTuples/BBHto2G_M-125/nominal/", filters=filters)
In [13]: batch = next(dataset._dataset.to_batches(filter=dataset._filter_expression))
In [14]: type(batch)
Out[14]: pyarrow.lib.RecordBatch
In [15]: dataset = ds.dataset("storage/NTuples/BBHto2G_M-125/nominal/")
In [16]: type(dataset)
Out[16]: pyarrow._dataset.FileSystemDataset
In [17]: batch = next(dataset.to_batches(filter=filters))
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[17], line 1
----> 1 batch = next(dataset.to_batches(filter=filters))
TypeError: Argument 'filter' has incorrect type (expected pyarrow._compute.Expression, got list)
```
`ParquetDataset` can get a list for its `filters` argument while `to_batches` expects an `Expression` as it's `filter` argument.
Is there a public API to compile a list into an `Expression`?
GitHub link: https://github.com/apache/arrow/discussions/47988#discussioncomment-14808010
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] Iterating over parquet dataset in batches [arrow]
Posted by "ikrommyd (via GitHub)" <gi...@apache.org>.
GitHub user ikrommyd added a comment to the discussion: Iterating over parquet dataset in batches
That looks like it solves my problem and I can use "public" API only but maybe `filters_to_expression` should be added to the API reference in that case?
GitHub link: https://github.com/apache/arrow/discussions/47988#discussioncomment-14808059
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] Iterating over parquet dataset in batches [arrow]
Posted by "ikrommyd (via GitHub)" <gi...@apache.org>.
GitHub user ikrommyd edited a discussion: Iterating over parquet dataset in batches
Hello,
I would like to iterate over a parquet dataset in batches.
I get my parquet dataset like this
```py
import pyarrow.parquet as pq
dataset = pq.ParquetDataset(source_path, filters=cat_dict[cat]["cat_filter"])
```
However, the `pyarrow.parquet.ParquetDataset` class doesn't seem to have a batched iteration method.
After very briefly looking at its source code, I found that I can access the underlying `pyarrow.dataset.Dataset` by with the `_dataset` attribute and that has a `to_batches` method.
Therefore I'm doing this to iterate
```py
for batch in dataset._dataset.to_batches(filter=dataset._filter_expression):
...
```
That feels a bit hacky to me because I'm depending on internals like `dataset._dataset` and `dataset._filter_expression`. Is this the proper way to do this? Is there a better API that I couldn't find that users should be using?
Thanks in advance!
GitHub link: https://github.com/apache/arrow/discussions/47988
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] Iterating over parquet dataset in batches [arrow]
Posted by "sidneymau (via GitHub)" <gi...@apache.org>.
GitHub user sidneymau added a comment to the discussion: Iterating over parquet dataset in batches
I believe the `dataset` module is the preferred way to do this:
```
import pyarrow.dataset as ds
dataset = ds.dataset(source_path, filters=cat_dict[cat]["cat_filter"]))
for batch in dataset.to_batches():
...
```
See https://arrow.apache.org/docs/python/dataset.html for some more docs
GitHub link: https://github.com/apache/arrow/discussions/47988#discussioncomment-14807877
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] Iterating over parquet dataset in batches [arrow]
Posted by "sidneymau (via GitHub)" <gi...@apache.org>.
GitHub user sidneymau edited a comment on the discussion: Iterating over parquet dataset in batches
I believe the `dataset` module is the preferred way to do this:
```python
import pyarrow.dataset as ds
dataset = ds.dataset(source_path)
for batch in dataset.to_batches(filter=cat_dict[cat]["cat_filter"])):
...
```
See https://arrow.apache.org/docs/python/dataset.html for some more docs
GitHub link: https://github.com/apache/arrow/discussions/47988#discussioncomment-14807877
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] Iterating over parquet dataset in batches [arrow]
Posted by "ikrommyd (via GitHub)" <gi...@apache.org>.
GitHub user ikrommyd added a comment to the discussion: Iterating over parquet dataset in batches
> I believe the `dataset` module is the preferred way to do this:
>
> ```python
> import pyarrow.dataset as ds
>
> dataset = ds.dataset(source_path)
>
> for batch in dataset.to_batches(filter=cat_dict[cat]["cat_filter"]):
> ...
> ```
>
> See https://arrow.apache.org/docs/python/dataset.html for some more docs
Yeah I had seen that. The problem I had encountered was this
```py
In [11]: filters = [("pt", ">", -1.0)]
In [12]: dataset = pq.ParquetDataset("storage/NTuples/BBHto2G_M-125/nominal/", filters=filters)
In [13]: batch = next(dataset._dataset.to_batches(filter=dataset._filter_expression))
In [14]: type(batch)
Out[14]: pyarrow.lib.RecordBatch
In [15]: dataset = ds.dataset("storage/NTuples/BBHto2G_M-125/nominal/")
In [16]: type(dataset)
Out[16]: pyarrow._dataset.FileSystemDataset
In [17]: batch = next(dataset.to_batches(filter=filters))
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[17], line 1
----> 1 batch = next(dataset.to_batches(filter=filters))
TypeError: Argument 'filter' has incorrect type (expected pyarrow._compute.Expression, got list)
```
`ParquetDataset` can get a list for its `filters` argument while `to_batches` expects an `Expression` as it's `filter` argument.
Is there a way to compile the list into an `Expression`?
GitHub link: https://github.com/apache/arrow/discussions/47988#discussioncomment-14808010
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] Iterating over parquet dataset in batches [arrow]
Posted by "ikrommyd (via GitHub)" <gi...@apache.org>.
GitHub user ikrommyd edited a comment on the discussion: Iterating over parquet dataset in batches
Ah....looking at the source code, I found this: https://github.com/apache/arrow/blob/7623887f0aaef84c3f1fc209fd3f054feeeba753/python/pyarrow/parquet/core.py#L134 It's not in the documentation, but looks "public" enough as it is accessible like `pyarrow.parquet.filters_to_expression`
GitHub link: https://github.com/apache/arrow/discussions/47988#discussioncomment-14808041
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] Iterating over parquet dataset in batches [arrow]
Posted by "ikrommyd (via GitHub)" <gi...@apache.org>.
GitHub user ikrommyd added a comment to the discussion: Iterating over parquet dataset in batches
Great! Closing the discussion then. Will open an issue regarding the `filters_to_expression` being part of the API reference. Thanks!
GitHub link: https://github.com/apache/arrow/discussions/47988#discussioncomment-14808288
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] Iterating over parquet dataset in batches [arrow]
Posted by "ikrommyd (via GitHub)" <gi...@apache.org>.
GitHub user ikrommyd edited a discussion: Iterating over parquet dataset in batches
Hello,
I would like to iterate over a parquet dataset in batches.
I get my parquet dataset like this
```py
import pyarrow.parquet as pq
dataset = pq.ParquetDataset(source_path, filters=cat_dict[cat]["cat_filter"])
```
However, the `pyarrow.parquet.ParquetDataset` class doesn't seem to have a batched iteration method.
After very briefly looking at its source code, I found that I can access the underlying `pyarrow.dataset.Dataset` by with the `_dataset` attribute and that has a `to_baches` method.
Therefore I'm doing this to iterate
```py
for batch in dataset._dataset.to_batches(filter=dataset._filter_expression):
...
```
That feels a bit hacky to me because I'm depending on internals like `dataset._dataset` and `dataset._filter_expression`. Is this the proper way to do this? Is there a better API that I couldn't find that users should be using?
Thanks in advance!
GitHub link: https://github.com/apache/arrow/discussions/47988
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] Iterating over parquet dataset in batches [arrow]
Posted by "ikrommyd (via GitHub)" <gi...@apache.org>.
GitHub user ikrommyd added a comment to the discussion: Iterating over parquet dataset in batches
Ah....looking at the source code, I found this: https://github.com/apache/arrow/blob/7623887f0aaef84c3f1fc209fd3f054feeeba753/python/pyarrow/parquet/core.py#L134. It's not in the documentation but looks "public" enough as it is accessible like `pyarrow.parquet.filters_to_expression`
GitHub link: https://github.com/apache/arrow/discussions/47988#discussioncomment-14808041
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] Iterating over parquet dataset in batches [arrow]
Posted by "ikrommyd (via GitHub)" <gi...@apache.org>.
GitHub user ikrommyd closed a discussion: Iterating over parquet dataset in batches
Hello,
I would like to iterate over a parquet dataset in batches.
I get my parquet dataset like this
```py
import pyarrow.parquet as pq
dataset = pq.ParquetDataset(source_path, filters=cat_dict[cat]["cat_filter"])
```
However, the `pyarrow.parquet.ParquetDataset` class doesn't seem to have a batched iteration method.
After very briefly looking at its source code, I found that I can access the underlying `pyarrow.dataset.Dataset` by with the `_dataset` attribute and that has a `to_batches` method.
Therefore I'm doing this to iterate
```py
for batch in dataset._dataset.to_batches(filter=dataset._filter_expression):
...
```
That feels a bit hacky to me because I'm depending on internals like `dataset._dataset` and `dataset._filter_expression`. Is this the proper way to do this? Is there a better API that I couldn't find that users should be using?
Thanks in advance!
GitHub link: https://github.com/apache/arrow/discussions/47988
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] Iterating over parquet dataset in batches [arrow]
Posted by "ikrommyd (via GitHub)" <gi...@apache.org>.
GitHub user ikrommyd edited a comment on the discussion: Iterating over parquet dataset in batches
> I believe the `dataset` module is the preferred way to do this:
>
> ```python
> import pyarrow.dataset as ds
>
> dataset = ds.dataset(source_path)
>
> for batch in dataset.to_batches(filter=cat_dict[cat]["cat_filter"]):
> ...
> ```
>
> See https://arrow.apache.org/docs/python/dataset.html for some more docs
Yeah I had seen that. The problem I had encountered was this
```py
In [11]: filters = [("pt", ">", -1.0)]
In [12]: dataset = pq.ParquetDataset("storage/NTuples/BBHto2G_M-125/nominal/", filters=filters)
In [13]: batch = next(dataset._dataset.to_batches(filter=dataset._filter_expression))
In [14]: type(batch)
Out[14]: pyarrow.lib.RecordBatch
In [15]: dataset = ds.dataset("storage/NTuples/BBHto2G_M-125/nominal/")
In [16]: type(dataset)
Out[16]: pyarrow._dataset.FileSystemDataset
In [17]: batch = next(dataset.to_batches(filter=filters))
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[17], line 1
----> 1 batch = next(dataset.to_batches(filter=filters))
TypeError: Argument 'filter' has incorrect type (expected pyarrow._compute.Expression, got list)
```
`ParquetDataset` can get a list for its `filters` argument while `to_batches` expects an `Expression` as it's `filter` argument.
Is there a public API to compile a list into an `Expression`? There definitely is a private one since `ParquetDataset` can do it.
GitHub link: https://github.com/apache/arrow/discussions/47988#discussioncomment-14808010
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] Iterating over parquet dataset in batches [arrow]
Posted by "sidneymau (via GitHub)" <gi...@apache.org>.
GitHub user sidneymau added a comment to the discussion: Iterating over parquet dataset in batches
Yes, `pq.filters_to_expression` is the only way I am aware for converting filters written in that way to compute expressions, and I have used it to do similar things. I think the more idiomatic solution would be to directly write the compute expression (e.g., filter = `ds.field("pt") > -1.0`, but I agree that `pq.filters_to_expression` is useful and ought to be documented
GitHub link: https://github.com/apache/arrow/discussions/47988#discussioncomment-14808174
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] Iterating over parquet dataset in batches [arrow]
Posted by "sidneymau (via GitHub)" <gi...@apache.org>.
GitHub user sidneymau edited a comment on the discussion: Iterating over parquet dataset in batches
I believe the `dataset` module is the preferred way to do this:
```python
import pyarrow.dataset as ds
dataset = ds.dataset(source_path)
for batch in dataset.to_batches(filter=cat_dict[cat]["cat_filter"]):
...
```
See https://arrow.apache.org/docs/python/dataset.html for some more docs
GitHub link: https://github.com/apache/arrow/discussions/47988#discussioncomment-14807877
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org