You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Ying Zhou <yz...@gmail.com> on 2021/03/12 08:43:11 UTC

[Python] Best practices when exposing options

Hi,

Currently I’m working on ARROW-11297 https://github.com/mathyingzhou/arrow/tree/ARROW-11297 <https://github.com/mathyingzhou/arrow/tree/ARROW-11297>) which will be filed as soon as the current PR is merged. 

I managed to reimplement orc::WriterOptions in Arrow (with naming conventions Arrow-ized) as arrow::adapters::orc::WriterOptions (which is necessary since we do not allow third party headers to be included in our public headers) and finished the C++ part of the work. Now I’m trying to expose WriterOptions in Python. I do wonder how this is supposed to be done in general. After reading the code in array.pxi I think maybe this is the way I want to do it:

1. The end user will see individual ORC writer options (e.g. CompressionKind, that is, whether we use ZLIB, LZ0 or some other form of compression or none at all) as keyword arguments.
2. These keyword arguments will be processed in _orc.pyx first as a dictionary and then using an adapter they will be converted into an arrow::adapters::orc::WriterOptions. 

Is this the right way?

Moreover I do wonder how we should convert the enums. Shall I use a series of if/elif or a mapping dict to force people to use one of the correct strings or get a ValueError?

e.g.

compression_kind_mapping = {’snappy’:CompressionKind._CompressionKind_SNAPPY,
						’zl0’:CompressionKind._CompressionKind_ZL0}} #There are other options, this is just an example
If compression_kind not in compression_kind_mapping.keys():
	raise ValueError(“Unknown compression_kind”)
c_compression_kind = compression_kind_mapping[compression_kind]

Ying

Re: [Python] Best practices when exposing options

Posted by Neal Richardson <ne...@gmail.com>.
Hi Ying,
I'd suggest looking at how the other file readers and writers (CSV,
Parquet, etc.) expose their options. I don't know pyarrow well enough
myself to tell you what the answer is, but the answer is probably following
whatever model is already there for those options.

Neal

On Fri, Mar 12, 2021 at 12:43 AM Ying Zhou <yz...@gmail.com> wrote:

> Hi,
>
> Currently I’m working on ARROW-11297
> https://github.com/mathyingzhou/arrow/tree/ARROW-11297 <
> https://github.com/mathyingzhou/arrow/tree/ARROW-11297>) which will be
> filed as soon as the current PR is merged.
>
> I managed to reimplement orc::WriterOptions in Arrow (with naming
> conventions Arrow-ized) as arrow::adapters::orc::WriterOptions (which is
> necessary since we do not allow third party headers to be included in our
> public headers) and finished the C++ part of the work. Now I’m trying to
> expose WriterOptions in Python. I do wonder how this is supposed to be done
> in general. After reading the code in array.pxi I think maybe this is the
> way I want to do it:
>
> 1. The end user will see individual ORC writer options (e.g.
> CompressionKind, that is, whether we use ZLIB, LZ0 or some other form of
> compression or none at all) as keyword arguments.
> 2. These keyword arguments will be processed in _orc.pyx first as a
> dictionary and then using an adapter they will be converted into an
> arrow::adapters::orc::WriterOptions.
>
> Is this the right way?
>
> Moreover I do wonder how we should convert the enums. Shall I use a series
> of if/elif or a mapping dict to force people to use one of the correct
> strings or get a ValueError?
>
> e.g.
>
> compression_kind_mapping =
> {’snappy’:CompressionKind._CompressionKind_SNAPPY,
>
> ’zl0’:CompressionKind._CompressionKind_ZL0}} #There are other options, this
> is just an example
> If compression_kind not in compression_kind_mapping.keys():
>         raise ValueError(“Unknown compression_kind”)
> c_compression_kind = compression_kind_mapping[compression_kind]
>
> Ying