You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jorisvandenbossche (via GitHub)" <gi...@apache.org> on 2026/09/03 13:09:55 UTC
[PR] MINOR: [Python][Parquet] Clarify code around restoring pandas metadata in ParquetDataset.read() [arrow]
jorisvandenbossche opened a new pull request, #51151:
URL: https://github.com/apache/arrow/pull/51151
### Rationale for this change
This might not strictly speaking be a valid "minor" issue since it changes code, but it should keep the logic the same and (for me) clarify the intent of it.
### Are these changes tested?
By existing tests
### Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: [Python][Parquet] Clarify code around restoring pandas metadata in ParquetDataset.read() [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #51151:
URL: https://github.com/apache/arrow/pull/51151#issuecomment-6085455796
I was wondering how this could actually have caused such behaviour change, because essentially before this PR it was replacing _identical_ pandas metadata.
But it seems replacing the schema metadata at the moment impacts the table:
```python
>>> table = pa.table({"a": [1, 2, 3]}).select([])
>>> table.num_rows
3
>>> table = table.replace_schema_metadata({"test": "value"})
>>> table.num_rows
0
```
That seems like an actual bug in pyarrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: [Python][Parquet] Clarify code around restoring pandas metadata in ParquetDataset.read() [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #51151:
URL: https://github.com/apache/arrow/pull/51151#discussion_r3924881092
##########
python/pyarrow/parquet/core.py:
##########
@@ -1592,13 +1593,12 @@ def read(self, columns=None, use_threads=True, use_pandas_metadata=False):
use_threads=use_threads
)
- # if use_pandas_metadata, restore the pandas metadata (which gets
- # lost if doing a specific `columns` selection in to_table)
- if use_pandas_metadata:
- if metadata and b"pandas" in metadata:
Review Comment:
If `metadata` would come from `metadata = self.schema.metadata`, it means it already had a "pandas" key, and there is no need to add it back here (`to_table()` does not (or no longer) loose the metadata)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: [Python][Parquet] Clarify code around restoring pandas metadata in ParquetDataset.read() [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF merged PR #51151:
URL: https://github.com/apache/arrow/pull/51151
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: [Python][Parquet] Clarify code around restoring pandas metadata in ParquetDataset.read() [arrow]
Posted by "TomAugspurger (via GitHub)" <gi...@apache.org>.
TomAugspurger commented on PR #51151:
URL: https://github.com/apache/arrow/pull/51151#issuecomment-6084791916
FYI @jorisvandenbossche, this did cause a behavior change in pandas when reading zero columns. With pyarrow==25.x, that returned a `(0, 0)` shape dataframe. With pyarrow==26, it returns a `(nrows, 0)` shape dataframe
```python
import io
import pandas as pd
df = pd.DataFrame({"a": [1, 2, 3]})
b = io.BytesIO()
df.to_parquet(b)
empty = pd.read_parquet(b, columns=[])
print(empty.shape) # (0, 0) for pyarrow=25.x; (3, 0) for pyarrow=26.x
```
I think that `(nrows, 0)` is the shape I'd expect, but leaving a note here in case anyone else stumbles across this (xref https://github.com/NVIDIA/cudf/pull/24520).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: [Python][Parquet] Clarify code around restoring pandas metadata in ParquetDataset.read() [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #51151:
URL: https://github.com/apache/arrow/pull/51151#issuecomment-6085293052
Damn, I should not have called it "minor" .. But relieved that the behaviour change was not too bad. Agreed that the new behaviour seems fine (and we should maybe add a test for that in pandas and pyarrow as well, to cover that case)
I suppose when the dataframe has a non-range index, it already gave the non-zero shape.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org