You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Tao Li <ta...@zillow.com> on 2021/07/08 20:49:14 UTC
Re: Beam Calcite SQL LIMIT Clause Processing All Rows
Thanks @Yuchu Cao<ma...@trulia.com>. I think this is a good catch.
+ @Alexey Romanenko<ma...@gmail.com>
From: Yuchu Cao <yu...@trulia.com>
Date: Tuesday, July 6, 2021 at 6:24 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Cc: Tao Li <ta...@zillow.com>
Subject: Beam Calcite SQL LIMIT Clause Processing All Rows
Hi Beam community,
We are testing a Beam SQL query like below against 35 GB parquet files on SparkRunner, and it appears that LIMIT Clause is not honored, and Beam is processing the entire datasets:
SELECT auction, 0.82*(price) as euro, bidder, datetime FROM nexmark_bid limit 1000
CalCiteQueryPlanner output:
[Graphical user interface, text, application Description automatically generated]
From Spark History server, it seems Beam is processing all the rows, and Beam query would choke because certain executor starts to fail with Out of Memory Error “ExecutorLostFailure (executor 13 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 17.7 GB of 17.6 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead”.
Could you please advise why this would occur? Thank you!
[Table Description automatically generated]
Re: Beam Calcite SQL LIMIT Clause Processing All Rows
Posted by Andrew Pilloud <ap...@google.com>.
Limit is only partially implemented in Beam, it uses the Top transform and
still processes the entire dataset. I don't believe there is a way for a
transform to indicate that it is done producing output, so we have to
process the entire input dataset even after hitting the limit. If we fixed
that we would need something like limit pushdown to make this work well.
On Thu, Jul 8, 2021 at 1:49 PM Tao Li <ta...@zillow.com> wrote:
> Thanks @Yuchu Cao <yu...@trulia.com>. I think this is a good catch.
>
>
>
> + @Alexey Romanenko <ar...@gmail.com>
>
>
>
> *From: *Yuchu Cao <yu...@trulia.com>
> *Date: *Tuesday, July 6, 2021 at 6:24 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Cc: *Tao Li <ta...@zillow.com>
> *Subject: *Beam Calcite SQL LIMIT Clause Processing All Rows
>
>
>
> Hi Beam community,
>
>
>
> We are testing a Beam SQL query like below against 35 GB parquet files on
> SparkRunner, and it appears that LIMIT Clause is not honored, and Beam is
> processing the entire datasets:
>
> *SELECT auction, 0.82*(price) as euro, bidder, datetime FROM nexmark_bid
> limit 1000 *
>
>
>
> CalCiteQueryPlanner output:
>
>
>
> [image: Graphical user interface, text, application Description
> automatically generated]
>
> From Spark History server, it seems Beam is processing all the rows, and
> Beam query would choke because certain executor starts to fail with Out of
> Memory Error “ExecutorLostFailure (executor 13 exited caused by one of the
> running tasks) Reason: Container killed by YARN for exceeding memory
> limits. 17.7 GB of 17.6 GB physical memory used. Consider boosting
> spark.yarn.executor.memoryOverhead”.
>
>
>
> Could you please advise why this would occur? Thank you!
>
>
>
> [image: Table Description automatically generated]
>
>
>