[Rust] [Datafusion] DictionaryArray querying

[Olo Sawyerr <os...@outlook.com>]

Hi there,

Hope you're well. A couple of questions:

In the query below, if col1 is a DictionaryArray:

select count(*), col1 from some_table
where col1 = 'foo'
group by col1

  *   Is 'foo' converted to a key using the Dictionary before the filter is performed?
  *   Is col1 evaluated in the group by as keys and then converted to the value just before the results are returned?

OR

  *   Is col1 just converted to a StringArray before anything is evalulated?

Regards,


Olo

Re: [Rust] [Datafusion] DictionaryArray querying

[Andrew Lamb <al...@influxdata.com>]

Hi

> Is col1 just converted to a StringArray before anything is evalulated?
No

> Is 'foo' converted to a key using the Dictionary before the filter is
performed?

DataFusion uses the arrow-rs kernels, so I believe in this case the filter
is applied on the dictionary values to find the matching entries, though I
may be mistaken. The code is [1].

> Is col1 evaluated in the group by as keys and then converted to the value
just before the results are returned?

I hope so , but I am not sure if we have implemented that optimization yet.



[1]
https://github.com/apache/arrow-rs/blob/f24afc8c57141734a14a9c46d6b8dc1610f936f3/arrow-ord/src/comparison.rs#L1893


On Fri, Apr 14, 2023 at 7:22 PM Olo Sawyerr <os...@outlook.com> wrote:

> Hi there,
>
> Hope you're well. A couple of questions:
>
> In the query below, if col1 is a DictionaryArray:
>
> select count(*), col1 from some_table
> where col1 = 'foo'
> group by col1
>
>
>    - Is 'foo' converted to a key using the Dictionary before the filter
>    is performed?
>    - Is col1 evaluated in the group by as *keys* and then converted to
>    the *value* just before the results are returned?
>
> OR
>
>    - Is col1 just converted to a StringArray before anything is
>    evalulated?
>
>
> Regards,
>
>
> Olo
>
>
>