You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Soheil Pourbafrani <so...@gmail.com> on 2020/01/29 20:02:27 UTC

fliter and flatMap operation VS only a flatMap operation

Hi,

In case we need to filter operation followed by a transformation, which one
is more efficient in Flink, applying the filter operation first and then a
flatMap operation separately OR using only a flatMap operation that
internally includes the filter logic, too?

best
Soheil

Re: fliter and flatMap operation VS only a flatMap operation

Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.

Hi,

If your filter and flatMap operators are chained, then the performance
difference should not be noticeable.
If a shuffle (i.e. a keyBy operation) occurs after the filter and before
the flatMap, then applying the filter first will be more efficient.

Cheers,
Gordon

On Thu, Jan 30, 2020 at 4:03 AM Soheil Pourbafrani <so...@gmail.com>
wrote:

> Hi,
>
> In case we need to filter operation followed by a transformation, which
> one is more efficient in Flink, applying the filter operation first and
> then a flatMap operation separately OR using only a flatMap operation that
> internally includes the filter logic, too?
>
> best
> Soheil
>