You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Joe <jo...@net2020.org> on 2022/04/21 14:09:47 UTC
Why is spark running multiple stages with the same code line?
Hi,
When looking at application UI (in Amazon EMR) I'm seeing one job for
my particular line of code, for example:
64 Running count at MySparkJob.scala:540
When I click into the job and go to stages I can see over a 100 stages
running the same line of code (stages are active, pending or
completed):
190 Pending count at MySparkJob.scala:540
...
162 Active count at MySparkJob.scala:540
...
108 Completed count at MySparkJob.scala:540
...
I'm not sure what that means, I thought that stage was a logical
operation boundary and you could have only one stage in the job (unless
you executed the same dataset+action many times on purpose) and tasks
were the ones that were replicated across partitions. But here I'm
seeing many stages running, each with the same line of code?
I don't have a situation where my code is re-processing the same set of
data many times, all intermediate sets are persisted.
I'm not sure if EMR UI display is wrong or if spark stages are not what
I thought they were?
Thanks,
Joe
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Why is spark running multiple stages with the same code line?
Posted by Sean Owen <sr...@gmail.com>.
The line of code triggers a job, the job triggers stages. You should see
they are different operations, all supporting execution of the action on
that line.
On Thu, Apr 21, 2022 at 9:24 AM Joe <jo...@net2020.org> wrote:
> Hi Sean,
> Thanks for replying but my question was about multiple stages running
> the same line of code, not about multiple stages in general. Yes single
> job can have multiple stages, but they should not be repeated, as far
> as I know, if you're caching/persisting your intermediate outputs.
>
> My question is why am I seeing multiple stages running the same line of
> code? As I understand it stage is a grouping of operations that can be
> executed without shuffling data or invoking a new action and they are
> divided into tasks, and tasks are the ones that are executed in
> parallel and can have the same line of code running on different
> executors. Or is this assumption wrong?
> Thanks,
>
> Joe
>
>
> On Thu, 2022-04-21 at 09:14 -0500, Sean Owen wrote:
> > A job can have multiple stages for sure. One action triggers a job.
> > This seems normal.
> >
> > On Thu, Apr 21, 2022, 9:10 AM Joe <jo...@net2020.org> wrote:
> > > Hi,
> > > When looking at application UI (in Amazon EMR) I'm seeing one job
> > > for
> > > my particular line of code, for example:
> > > 64 Running count at MySparkJob.scala:540
> > >
> > > When I click into the job and go to stages I can see over a 100
> > > stages
> > > running the same line of code (stages are active, pending or
> > > completed):
> > > 190 Pending count at MySparkJob.scala:540
> > > ...
> > > 162 Active count at MySparkJob.scala:540
> > > ...
> > > 108 Completed count at MySparkJob.scala:540
> > > ...
> > >
> > > I'm not sure what that means, I thought that stage was a logical
> > > operation boundary and you could have only one stage in the job
> > > (unless
> > > you executed the same dataset+action many times on purpose) and
> > > tasks
> > > were the ones that were replicated across partitions. But here I'm
> > > seeing many stages running, each with the same line of code?
> > >
> > > I don't have a situation where my code is re-processing the same
> > > set of
> > > data many times, all intermediate sets are persisted.
> > > I'm not sure if EMR UI display is wrong or if spark stages are not
> > > what
> > > I thought they were?
> > > Thanks,
> > >
> > > Joe
> > >
> > >
> > >
> > > -------------------------------------------------------------------
> > > --
> > > To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> > >
>
>
>
Re: Why is spark running multiple stages with the same code line?
Posted by Joe <jo...@net2020.org>.
Hi Sean,
Thanks for replying but my question was about multiple stages running
the same line of code, not about multiple stages in general. Yes single
job can have multiple stages, but they should not be repeated, as far
as I know, if you're caching/persisting your intermediate outputs.
My question is why am I seeing multiple stages running the same line of
code? As I understand it stage is a grouping of operations that can be
executed without shuffling data or invoking a new action and they are
divided into tasks, and tasks are the ones that are executed in
parallel and can have the same line of code running on different
executors. Or is this assumption wrong?
Thanks,
Joe
On Thu, 2022-04-21 at 09:14 -0500, Sean Owen wrote:
> A job can have multiple stages for sure. One action triggers a job.
> This seems normal.
>
> On Thu, Apr 21, 2022, 9:10 AM Joe <jo...@net2020.org> wrote:
> > Hi,
> > When looking at application UI (in Amazon EMR) I'm seeing one job
> > for
> > my particular line of code, for example:
> > 64 Running count at MySparkJob.scala:540
> >
> > When I click into the job and go to stages I can see over a 100
> > stages
> > running the same line of code (stages are active, pending or
> > completed):
> > 190 Pending count at MySparkJob.scala:540
> > ...
> > 162 Active count at MySparkJob.scala:540
> > ...
> > 108 Completed count at MySparkJob.scala:540
> > ...
> >
> > I'm not sure what that means, I thought that stage was a logical
> > operation boundary and you could have only one stage in the job
> > (unless
> > you executed the same dataset+action many times on purpose) and
> > tasks
> > were the ones that were replicated across partitions. But here I'm
> > seeing many stages running, each with the same line of code?
> >
> > I don't have a situation where my code is re-processing the same
> > set of
> > data many times, all intermediate sets are persisted.
> > I'm not sure if EMR UI display is wrong or if spark stages are not
> > what
> > I thought they were?
> > Thanks,
> >
> > Joe
> >
> >
> >
> > -------------------------------------------------------------------
> > --
> > To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> >
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Why is spark running multiple stages with the same code line?
Posted by Sean Owen <sr...@gmail.com>.
A job can have multiple stages for sure. One action triggers a job. This
seems normal.
On Thu, Apr 21, 2022, 9:10 AM Joe <jo...@net2020.org> wrote:
> Hi,
> When looking at application UI (in Amazon EMR) I'm seeing one job for
> my particular line of code, for example:
> 64 Running count at MySparkJob.scala:540
>
> When I click into the job and go to stages I can see over a 100 stages
> running the same line of code (stages are active, pending or
> completed):
> 190 Pending count at MySparkJob.scala:540
> ...
> 162 Active count at MySparkJob.scala:540
> ...
> 108 Completed count at MySparkJob.scala:540
> ...
>
> I'm not sure what that means, I thought that stage was a logical
> operation boundary and you could have only one stage in the job (unless
> you executed the same dataset+action many times on purpose) and tasks
> were the ones that were replicated across partitions. But here I'm
> seeing many stages running, each with the same line of code?
>
> I don't have a situation where my code is re-processing the same set of
> data many times, all intermediate sets are persisted.
> I'm not sure if EMR UI display is wrong or if spark stages are not what
> I thought they were?
> Thanks,
>
> Joe
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
Re: Why is spark running multiple stages with the same code line?
Posted by Russell Spitzer <ru...@gmail.com>.
There are a few things going on here.
1. Spark is lazy, so nothing happens until a result is collected back to the driver or data is written to a sink. So the 1 line you see
is most likely just that trigger. Once triggered, all of the work required to make that final result happen occurs. If the final collect depends on 100 joins for example, you would run one line on the final result and it would trigger 100 stages.
2. A stage is not a logical boundary, it's a physical boundary between shuffle exchanges. In practice this means any time an operation requires a portion from all of the data in the previous operation. For example, doing a global sort may require data that is in every task from the previous operation to complete successfully. A job may have a theoretically unlimited number of stages although this does have some technical limits.
> On Apr 21, 2022, at 9:09 AM, Joe <jo...@net2020.org> wrote:
>
> Hi,
> When looking at application UI (in Amazon EMR) I'm seeing one job for
> my particular line of code, for example:
> 64 Running count at MySparkJob.scala:540
>
> When I click into the job and go to stages I can see over a 100 stages
> running the same line of code (stages are active, pending or
> completed):
> 190 Pending count at MySparkJob.scala:540
> ...
> 162 Active count at MySparkJob.scala:540
> ...
> 108 Completed count at MySparkJob.scala:540
> ...
>
> I'm not sure what that means, I thought that stage was a logical
> operation boundary and you could have only one stage in the job (unless
> you executed the same dataset+action many times on purpose) and tasks
> were the ones that were replicated across partitions. But here I'm
> seeing many stages running, each with the same line of code?
>
> I don't have a situation where my code is re-processing the same set of
> data many times, all intermediate sets are persisted.
> I'm not sure if EMR UI display is wrong or if spark stages are not what
> I thought they were?
> Thanks,
>
> Joe
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org