You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "douglas-raillard-arm (via GitHub)" <gi...@apache.org> on 2024/05/28 13:12:54 UTC
[I] Support LZ4_RAW for parquet writing [arrow]
douglas-raillard-arm opened a new issue, #41863:
URL: https://github.com/apache/arrow/issues/41863
### Describe the enhancement requested
`pyarrow.dataset.write_dataset(compression='lz4_raw')` currently fails with:
```
Traceback (most recent call last):
File "/work/projects/lisa/testpyarrow.py", line 3, in <module>
_reencode_parquet('sched_switch.lz4.parquet', 'updated.parquet', compression='lz4_raw')#, row_group_size=128*1024*1024, compression='LZ4')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "x.py", line 1, in my_write_parquet
options = pyarrow.dataset.ParquetFileFormat().make_write_options(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/_dataset_parquet.pyx", line 206, in pyarrow._dataset_parquet.ParquetFileFormat.make_write_options
File "pyarrow/_dataset_parquet.pyx", line 594, in pyarrow._dataset_parquet.ParquetFileWriteOptions.update
File "pyarrow/_dataset_parquet.pyx", line 599, in pyarrow._dataset_parquet.ParquetFileWriteOptions._set_properties
File "pyarrow/_parquet.pyx", line 1855, in pyarrow._parquet._create_writer_properties
File "pyarrow/_parquet.pyx", line 1369, in pyarrow._parquet.check_compression_name
pyarrow.lib.ArrowException: Unsupported compression: lz4_raw
```
And indeed, no mention of `lz4_raw` is to be found in `python/pyarrow/_parquet.pyx`.
Would it be possible to add support for LZ4_RAW codec when writing parquet files, particularly using the dataset API ?
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][Parquet] Support LZ4_RAW for parquet writing [arrow]
Posted by "nwoolmer (via GitHub)" <gi...@apache.org>.
nwoolmer commented on issue #41863:
URL: https://github.com/apache/arrow/issues/41863#issuecomment-3842318325
Bumping this issue. On the parquet spec, `LZ4` and `LZ4_RAW` are specified. The choice to make `lz4` equivalent to `lz4_raw` makes sense, due to the Hadoop format being deprecated. That being said, it'd be good to provide `lz4_raw` as an additional alias for those users who are familiar with this detail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support LZ4_RAW for parquet writing [arrow]
Posted by "douglas-raillard-arm (via GitHub)" <gi...@apache.org>.
douglas-raillard-arm commented on issue #41863:
URL: https://github.com/apache/arrow/issues/41863#issuecomment-2137413096
Thanks everyone, so I'll go ahead and assume that the legacy lz4 is now more-or-less out of the picture nowadays and that "lz4" these days usually means "lz4_raw" (at least in the Python and Rust ecosystems)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support LZ4_RAW for parquet writing [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #41863:
URL: https://github.com/apache/arrow/issues/41863#issuecomment-2137605032
> I think recent versions of parquet-java support reading LZ4_RAW data. @wgtmac Can you confirm?
Yes, I can confirm that it is supported since 1.13.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support LZ4_RAW for parquet writing [arrow]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #41863:
URL: https://github.com/apache/arrow/issues/41863#issuecomment-2137531963
Perhaps you could remove as an option `"lz4"` and replace it with `"lz4_hadoop"`, that way it would still be unambiguous which encoding is being used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support LZ4_RAW for parquet writing [arrow]
Posted by "douglas-raillard-arm (via GitHub)" <gi...@apache.org>.
douglas-raillard-arm commented on issue #41863:
URL: https://github.com/apache/arrow/issues/41863#issuecomment-2137087777
Ok I think I confused myself, pyarrow is indeed consistent and uses "lz4" everywhere with the meaning of "lz4_raw", same as `polars` it seems. Are you aware of any reader library that would require the legacy lz4 in their current version ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support LZ4_RAW for parquet writing [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #41863:
URL: https://github.com/apache/arrow/issues/41863#issuecomment-2137374179
That said, perhaps it would be easy and convenient to allow "lz4_raw" as a synonym of "lz4" in the PyArrow bindings for Parquet. What do you think, @AlenkaF @jorisvandenbossche ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][Parquet] Support LZ4_RAW for parquet writing [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #41863:
URL: https://github.com/apache/arrow/issues/41863#issuecomment-3547517473
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support LZ4_RAW for parquet writing [arrow]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #41863:
URL: https://github.com/apache/arrow/issues/41863#issuecomment-2137430917
The Rust ecosystem uses "lz4" to refer to the Hadoop codec, as per the parquet specification - https://github.com/apache/parquet-format/blob/master/Compression.md
I think it would be quite confusing to use `"lz4"` to refer to the `"lz4_raw"` codec
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][Parquet] Support LZ4_RAW for parquet writing [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #41863:
URL: https://github.com/apache/arrow/issues/41863#issuecomment-3852083071
Issue resolved by pull request 49135
https://github.com/apache/arrow/pull/49135
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support LZ4_RAW for parquet writing [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #41863:
URL: https://github.com/apache/arrow/issues/41863#issuecomment-2137524142
> I think it would be quite confusing to use `"lz4"` to refer to the `"lz4_raw"` codec, but if parquet-cpp already diverges from the specification here, perhaps that ship has sailed.
We want to discourage the use of the old and ill-specified "Hadoop LZ4" codec, and for that we have to ensure that asking for "lz4" selects the new "Raw LZ4" codec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support LZ4_RAW for parquet writing [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #41863:
URL: https://github.com/apache/arrow/issues/41863#issuecomment-2137583974
> Perhaps you could remove as an option `"lz4"` and replace it with `"lz4_hadoop"`, that way it would still be unambiguous which encoding is being used.
I'm rather lukewarm towards this.
1) We want to ensure backwards compatibility with code that passes `"lz4"`.
2) We don't want to stifle user-friendliness. "lz4" is easy to remember, while "lz4_raw" and "lz4_hadoop" probably have to be looked up every time by non-specialists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][Parquet] Support LZ4_RAW for parquet writing [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF closed issue #41863: [Python][Parquet] Support LZ4_RAW for parquet writing
URL: https://github.com/apache/arrow/issues/41863
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][Parquet] Support LZ4_RAW for parquet writing [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #41863:
URL: https://github.com/apache/arrow/issues/41863#issuecomment-3842429470
> That being said, it'd be good to provide `lz4_raw` as an additional alias for those users who are familiar with this detail.
>
> In my case, it broke an automation which expected `lz4_raw` to be accepted by several tools.
That sounds reasonable, do you want to submit a PR for it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support LZ4_RAW for parquet writing [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #41863:
URL: https://github.com/apache/arrow/issues/41863#issuecomment-2135308074
IIUC you just have to pass `compression='lz4'`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support LZ4_RAW for parquet writing [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #41863:
URL: https://github.com/apache/arrow/issues/41863#issuecomment-2137391527
> That said, perhaps it would be easy and convenient to allow "lz4_raw" as a synonym of "lz4" in the PyArrow bindings for Parquet
I agree, we can allow "lz4_raw" as a synonym with a note in the [user guide](https://arrow.apache.org/docs/dev/python/parquet.html#compression-encoding-and-file-compatibility).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support LZ4_RAW for parquet writing [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #41863:
URL: https://github.com/apache/arrow/issues/41863#issuecomment-2137309589
I think recent versions of parquet-java support reading LZ4_RAW data. @wgtmac Can you confirm?
I think parquet-rs also supports it. @tustvold Am I right?
DuckDB might also support it: https://duckdb.org/docs/data/parquet/overview.html#writing-to-parquet-files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support LZ4_RAW for parquet writing [arrow]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #41863:
URL: https://github.com/apache/arrow/issues/41863#issuecomment-2137363772
parquet-rs supports all 3 LZ4 flavored encodings
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][Parquet] Support LZ4_RAW for parquet writing [arrow]
Posted by "nwoolmer (via GitHub)" <gi...@apache.org>.
nwoolmer commented on issue #41863:
URL: https://github.com/apache/arrow/issues/41863#issuecomment-3842631208
Absolutely, wilco.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org