Limit for IN clause?

[Roberto Diaz <rd...@gmail.com>]

Hello dear impala users.

Recently I have been trying to debug a performance problem in one of our
microservices.. at the end It turns out is due to an SQL query containing
thousands of elements into an IN clause

Now I am not sure whether the problem is due to sql query compilation or
the query execution itself.

Do you know if there is some limit in impala for an IN clause? And if so
how can I overcome this problem?

Thank you very much in avance!

BR

Roberto

Re: Limit for IN clause?

[Quanlong Huang <hu...@gmail.com>]

Hi Roberto,

In the query profile, there is a section of "Query Timeline". It will show
you whether the time is spent in query compilation or the query execution
itself.

Impala has a 10,000 hard limit on the width of the expression tree:
https://impala.apache.org/docs/build/html/topics/impala_scaling_limits.html
So the items of IN clause should be lower than that. Otherwise, the planner
will fail the query.

If you do need such a large IN clause, maybe you can try dumping the
contents into a table and use it instead.

Regards,
Quanlong

On Mon, Dec 26, 2022 at 7:51 PM Roberto Diaz <rd...@gmail.com> wrote:

> Hello dear impala users.
>
> Recently I have been trying to debug a performance problem in one of our
> microservices.. at the end It turns out is due to an SQL query containing
> thousands of elements into an IN clause
>
> Now I am not sure whether the problem is due to sql query compilation or
> the query execution itself.
>
> Do you know if there is some limit in impala for an IN clause? And if so
> how can I overcome this problem?
>
> Thank you very much in avance!
>
> BR
>
> Roberto
>