Creating dictionary encoded string in C++

[Li Jin <ic...@gmail.com>]

Hello,

I am working on converting some internal data sources to Arrow data. One
particularly sets of data we have contains many string columns that can be
dictionary-encoded (basically string enums)

The current internal C++ API I am using gives me an iterator of "row"
objects, for each string column, the row object exposes a method
"getStringField(index)" that return me a "string_view" and I want to
construct a dictionary-encoded Arrow string column from it.

My question is:
(1) Is there a way to do this using the Arrow C++ API?
(2) Does the internal C++ API need to return something other than a
"string_view" to support this? Internally the string column is already
dictionary-encoded (although not in Arrow format) and it might already know
the dictionary and the encoded (int) value for each string field, but it
doesn't expose it now.

Thanks,
Li

Re: Creating dictionary encoded string in C++

[Rok Mihevc <ro...@gmail.com>]

Hi Li,

If it's practical for you to create an index and a dictionary array from
your source you could use those to create a DictionaryArray as seen here
[1].
Another option that might fit your situation is to use a dictionary builder
[2].

Best,
Rok

[1]
https://github.com/apache/arrow/blob/master/cpp/src/arrow/array/array_dict_test.cc#L1128-L1160
[2]
https://github.com/apache/arrow/blob/master/cpp/src/arrow/array/array_dict_test.cc#L218-L249

On Thu, Nov 3, 2022 at 3:50 PM Li Jin <ic...@gmail.com> wrote:

> Hello,
>
> I am working on converting some internal data sources to Arrow data. One
> particularly sets of data we have contains many string columns that can be
> dictionary-encoded (basically string enums)
>
> The current internal C++ API I am using gives me an iterator of "row"
> objects, for each string column, the row object exposes a method
> "getStringField(index)" that return me a "string_view" and I want to
> construct a dictionary-encoded Arrow string column from it.
>
> My question is:
> (1) Is there a way to do this using the Arrow C++ API?
> (2) Does the internal C++ API need to return something other than a
> "string_view" to support this? Internally the string column is already
> dictionary-encoded (although not in Arrow format) and it might already know
> the dictionary and the encoded (int) value for each string field, but it
> doesn't expose it now.
>
> Thanks,
> Li
>