[jira] [Commented] (ARROW-17046) [Python] pyarrow.parquet.write_to_dataset fails to pass kwargs to write_table function

["Alenka Frim (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-17046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565268#comment-17565268 ] 

Alenka Frim commented on ARROW-17046:
-------------------------------------

The default option for {{write_to_dataset}} has changed from legacy implementation to new dataset implementation in https://issues.apache.org/jira/browse/ARROW-16122.

To use the legacy implementation you need to add {{use_legacy_dataset=True}} to {{{}write_to_dataset{}}}. But note that this option is deprecated, and the legacy implementation will be removed in a future version.

Also see: https://issues.apache.org/jira/browse/ARROW-16119.

As for the documentation - **kwargs part should be clearer due to the change.
I will move this issue from bug to improvement.
Contributions are welcome!

> [Python] pyarrow.parquet.write_to_dataset fails to pass kwargs to write_table function
> --------------------------------------------------------------------------------------
>
>                 Key: ARROW-17046
>                 URL: https://issues.apache.org/jira/browse/ARROW-17046
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>    Affects Versions: 8.0.0
>            Reporter: Amir Khosroshahi
>            Priority: Minor
>
> According to PyArrow 8.0.0 [documentation|https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_to_dataset.html] {{kwargs}} is "Additional kwargs for {{write_table}} function." However when I try to pass additional arguments, for example {{{}flavor{}}}, to the underlying write_table I get the following error
> {code:java}
> TypeError: unexpected parquet write option: flavor{code}
> This used to work in PyArrow versions as late as 7.0.0 but started to break in 8.0.0.
> Minimal example to reproduce the error
> {code:java}
> import pyarrow as pa
> import pandas as pd
> import pyarrow.parquet as pq
> df = pd.DataFrame({'a': [1, 2], 'b': [3, 4]})
> tb = pa.Table.from_pandas(df)
> pq.write_to_dataset(tb, "test.parquet", flavor="spark") {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)