You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Nishuuzz (via GitHub)" <gi...@apache.org> on 2026/10/05 08:50:26 UTC
[PR] MINOR: [Python] Fix the codec list in show_info() [arrow]
Nishuuzz opened a new pull request, #51753:
URL: https://github.com/apache/arrow/pull/51753
`pa.show_info()` prints its Compression Codecs section from a hardcoded list:
```python
codecs = ["brotli", "bz2", "gzip", "lz4_frame", "lz4", "snappy", "zstd"]
```
That list has drifted from `_ensure_compression()` in io.pxi, which is what actually decides which names work. Two things fall out of it:
- `lz4` and `lz4_frame` both map to `CCompressionType_LZ4_FRAME`, so the same codec is reported twice under two names.
- `lz4_raw` maps to `CCompressionType_LZ4` and is a separate codec, but it is not listed at all, even though `pa.Codec.is_available('lz4_raw')` is True.
So seven lines get printed for six codecs, and the one codec that is missing is a real one. Before:
```
Compression Codecs:
brotli : Enabled
bz2 : Enabled
gzip : Enabled
lz4_frame : Enabled
lz4 : Enabled
snappy : Enabled
zstd : Enabled
```
After:
```
Compression Codecs:
brotli : Enabled
bz2 : Enabled
gzip : Enabled
lz4 : Enabled
lz4_raw : Enabled
snappy : Enabled
zstd : Enabled
```
`lz4` is kept as the name for the frame codec since that is the one the docstrings lead with (`'lz4' (or 'lz4_frame')`).
I checked this by applying the same change to an installed pyarrow and reading the output back, and by walking every name `_ensure_compression()` accepts to confirm which map onto the same C++ enum value. `show_info()` has no test coverage today and its output is a diagnostic print, so I have not added one, but I am happy to if you would prefer it pinned.
Found while comparing hardcoded Python lists against their C++ counterparts, the same way GH-50847 turned up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-52024: [Python] Fix the codec list in show_info() [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #51753:
URL: https://github.com/apache/arrow/pull/51753
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-52024: [Python] Fix the codec list in show_info() [arrow]
Posted by "Nishuuzz (via GitHub)" <gi...@apache.org>.
Nishuuzz commented on PR #51753:
URL: https://github.com/apache/arrow/pull/51753#issuecomment-6036312721
Both of those are done — filed #52024 for the bug and retitled this accordingly, and the description now follows the template.
The remaining question is yours: deriving the list programmatically rather than hardcoding it. I agree that is the better fix, and the obstacle is that the canonical mapping lives in `_ensure_compression()` in `io.pxi` and nothing currently exposes the supported set — neither `Codec` nor the C++ `compression.h` — so it needs a new accessor on the Cython side for `show_info()` to read. I am happy to write that here; the caveat is that I cannot compile the Cython extension in my environment, so I would be leaning on CI to verify it. Otherwise I can leave this as the minimal fix and open a separate issue for the programmatic version. Either is fine by me, whichever you prefer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-52024: [Python] Fix the codec list in show_info() [arrow]
Posted by "Nishuuzz (via GitHub)" <gi...@apache.org>.
Nishuuzz commented on PR #51753:
URL: https://github.com/apache/arrow/pull/51753#issuecomment-6145933211
Both done.
The list now says `lz4_frame` rather than `lz4`, so the output reads:
```
Compression Codecs:
brotli : Enabled
bz2 : Enabled
gzip : Enabled
lz4_frame : Enabled
lz4_raw : Enabled
snappy : Enabled
zstd : Enabled
```
`Codec.is_available("lz4_frame")` works the same as `"lz4"`, since the alias is resolved in `_ensure_compression()`, so this is only a change to what gets printed.
For the test I went with `capsys` and checked the codec labels and the section headings rather than their status, since as you say the enabled set depends on the build. The codec list is compared exactly, which is what pins the bug: before this, that list contained `lz4` and `lz4_frame` as separate entries and no `lz4_raw`.
One caveat on how far I verified it: I can't build the Cython extension here, so I could not run `test_misc.py` itself. What I did instead was reproduce `show_info()`'s printing exactly as this branch has it, against the installed pyarrow, capture it and run the test's assertions over that text — they pass, and the parsed labels come out as the seven above. The real run will have to be CI.
Still happy to do the programmatic version you raised originally, which needs a new accessor on the Cython side since nothing currently exposes the supported codec set. Given I can't compile here, I'd rather not write that blind in this PR — I'd suggest landing this and doing it as a follow-up, but say the word if you'd prefer it all together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-52024: [Python] Fix the codec list in show_info() [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #51753:
URL: https://github.com/apache/arrow/pull/51753#issuecomment-6143863627
> Compression Codecs:
> brotli : Enabled
> bz2 : Enabled
> gzip : Enabled
> lz4 : Enabled
> lz4_raw : Enabled
> snappy : Enabled
> zstd : Enabled
Can we perhaps have the names be "lz4_frame" and "lz4_raw" to avoid any further ambiguity?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-52024: [Python] Fix the codec list in show_info() [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #51753:
URL: https://github.com/apache/arrow/pull/51753#issuecomment-6036318635
:warning: GitHub issue #52024 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-52024: [Python] Fix the codec list in show_info() [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #51753:
URL: https://github.com/apache/arrow/pull/51753#issuecomment-6143879265
> Not by a new test. `show_info()` has no existing test coverage and its output is
> a diagnostic print, so I did not want to pin the exact text without knowing
> whether you would want that — happy to add one if you do.
Perhaps you could add a test that captures stdout/stderr? Though the exact outcome will depend on how PyArrow was compiled, so the test will have to be a bit flexible in that regard.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org