You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Yuhao Yang <hh...@gmail.com> on 2015/11/29 15:12:31 UTC
Need suggestions on monitor Spark progress
Hi all,
I got a simple processing job for 20000 accounts on 8 partitions. It's
roughly 2500 accounts on each partition. Each account will take about 1s to
complete the computation. That means each partition will take about 2500
seconds to finish the batch.
My question is how can I get the detailed progress of how many accounts has
been processed for each partition during the computation. An ideal solution
would allow me to know how many accounts has been processed periodically
(like every minute) so I can monitor and take action to save some time.
Right now on UI I can only get that task is running.
I know one solution is to split the data horizontally on driver and submit
to spark in mini batches, yet I think that would waste some cluster
resource and create extra complexity for result handling.
Any experience or best practice is welcome. Thanks a lot.
Regards,
Yuhao
Re: Need suggestions on monitor Spark progress
Posted by Alex Rovner <al...@magnetic.com>.
In these scenarios it's fairly standard to report the metrics either
directly or through accumulators (
http://spark.apache.org/docs/latest/programming-guide.html#accumulators-a-nameaccumlinka)
to a time series database such as Graphite (http://graphite.wikidot.com/)
or OpenTSDB (http://opentsdb.net/) and monitor the progress through the UI
provided by the DB.
*Alex Rovner*
*Director, Data Engineering *
*o:* 646.759.0052
* <http://www.magnetic.com/>*
On Mon, Nov 30, 2015 at 1:43 PM, Jacek Laskowski <ja...@japila.pl> wrote:
> Hi,
>
> My limited understanding of Spark tells me that a task is the least
> possible working unit and Spark itself won't give you much. It
> wouldn't expect so since "acount" is a business entity not Spark's
> one.
>
> What about using mapPartitions* to know the details of partitions and
> do whatever you want (log to stdout or whatever)? Just a thought.
>
> Pozdrawiam,
> Jacek
>
> --
> Jacek Laskowski | https://medium.com/@jaceklaskowski/ |
> http://blog.jaceklaskowski.pl
> Mastering Spark https://jaceklaskowski.gitbooks.io/mastering-apache-spark/
> Follow me at https://twitter.com/jaceklaskowski
> Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski
>
>
> On Sun, Nov 29, 2015 at 3:12 PM, Yuhao Yang <hh...@gmail.com> wrote:
> > Hi all,
> >
> > I got a simple processing job for 20000 accounts on 8 partitions. It's
> > roughly 2500 accounts on each partition. Each account will take about 1s
> to
> > complete the computation. That means each partition will take about 2500
> > seconds to finish the batch.
> >
> > My question is how can I get the detailed progress of how many accounts
> has
> > been processed for each partition during the computation. An ideal
> solution
> > would allow me to know how many accounts has been processed periodically
> > (like every minute) so I can monitor and take action to save some time.
> > Right now on UI I can only get that task is running.
> >
> > I know one solution is to split the data horizontally on driver and
> submit
> > to spark in mini batches, yet I think that would waste some cluster
> resource
> > and create extra complexity for result handling.
> >
> > Any experience or best practice is welcome. Thanks a lot.
> >
> > Regards,
> > Yuhao
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: Need suggestions on monitor Spark progress
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,
My limited understanding of Spark tells me that a task is the least
possible working unit and Spark itself won't give you much. It
wouldn't expect so since "acount" is a business entity not Spark's
one.
What about using mapPartitions* to know the details of partitions and
do whatever you want (log to stdout or whatever)? Just a thought.
Pozdrawiam,
Jacek
--
Jacek Laskowski | https://medium.com/@jaceklaskowski/ |
http://blog.jaceklaskowski.pl
Mastering Spark https://jaceklaskowski.gitbooks.io/mastering-apache-spark/
Follow me at https://twitter.com/jaceklaskowski
Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski
On Sun, Nov 29, 2015 at 3:12 PM, Yuhao Yang <hh...@gmail.com> wrote:
> Hi all,
>
> I got a simple processing job for 20000 accounts on 8 partitions. It's
> roughly 2500 accounts on each partition. Each account will take about 1s to
> complete the computation. That means each partition will take about 2500
> seconds to finish the batch.
>
> My question is how can I get the detailed progress of how many accounts has
> been processed for each partition during the computation. An ideal solution
> would allow me to know how many accounts has been processed periodically
> (like every minute) so I can monitor and take action to save some time.
> Right now on UI I can only get that task is running.
>
> I know one solution is to split the data horizontally on driver and submit
> to spark in mini batches, yet I think that would waste some cluster resource
> and create extra complexity for result handling.
>
> Any experience or best practice is welcome. Thanks a lot.
>
> Regards,
> Yuhao
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org