You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by on 2016/05/13 17:05:52 UTC
Tracking / estimating job progress
I provide a RESTful API interface from scalatra for launching Spark jobs
- part of the functionality is tracking these jobs. What API is
available to track the progress of a particular spark application? How
about estimating where in the total job progress the job is?
Thanks!
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Tracking / estimating job progress
Posted by .
On 5/13/2016 10:39 AM, Anthony May wrote:
> It looks like it might only be available via REST,
> http://spark.apache.org/docs/latest/monitoring.html#rest-api
Nice, thanks!
>
> On Fri, 13 May 2016 at 11:24 Dood@ODDO <oddodaoddo@gmail.com
> <ma...@gmail.com>> wrote:
>
> On 5/13/2016 10:16 AM, Anthony May wrote:
> >
> http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.SparkStatusTracker
> >
> > Might be useful
>
> How do you use it? You cannot instantiate the class - is the
> constructor
> private? Thanks!
>
> >
> > On Fri, 13 May 2016 at 11:11 Ted Yu <yuzhihong@gmail.com
> <ma...@gmail.com>
> > <mailto:yuzhihong@gmail.com <ma...@gmail.com>>> wrote:
> >
> > Have you looked
> > at
> core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala
> > ?
> >
> > Cheers
> >
> > On Fri, May 13, 2016 at 10:05 AM, Dood@ODDO
> <oddodaoddo@gmail.com <ma...@gmail.com>
> > <mailto:oddodaoddo@gmail.com <ma...@gmail.com>>>
> wrote:
> >
> > I provide a RESTful API interface from scalatra for
> launching
> > Spark jobs - part of the functionality is tracking these
> jobs.
> > What API is available to track the progress of a particular
> > spark application? How about estimating where in the
> total job
> > progress the job is?
> >
> > Thanks!
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> user-unsubscribe@spark.apache.org
> <ma...@spark.apache.org>
> > <mailto:user-unsubscribe@spark.apache.org
> <ma...@spark.apache.org>>
> > For additional commands, e-mail:
> user-help@spark.apache.org <ma...@spark.apache.org>
> > <mailto:user-help@spark.apache.org
> <ma...@spark.apache.org>>
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> <ma...@spark.apache.org>
> For additional commands, e-mail: user-help@spark.apache.org
> <ma...@spark.apache.org>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Tracking / estimating job progress
Posted by Anthony May <an...@gmail.com>.
It looks like it might only be available via REST,
http://spark.apache.org/docs/latest/monitoring.html#rest-api
On Fri, 13 May 2016 at 11:24 Dood@ODDO <od...@gmail.com> wrote:
> On 5/13/2016 10:16 AM, Anthony May wrote:
> >
> http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.SparkStatusTracker
> >
> > Might be useful
>
> How do you use it? You cannot instantiate the class - is the constructor
> private? Thanks!
>
> >
> > On Fri, 13 May 2016 at 11:11 Ted Yu <yuzhihong@gmail.com
> > <ma...@gmail.com>> wrote:
> >
> > Have you looked
> > at
> core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala
> > ?
> >
> > Cheers
> >
> > On Fri, May 13, 2016 at 10:05 AM, Dood@ODDO <oddodaoddo@gmail.com
> > <ma...@gmail.com>> wrote:
> >
> > I provide a RESTful API interface from scalatra for launching
> > Spark jobs - part of the functionality is tracking these jobs.
> > What API is available to track the progress of a particular
> > spark application? How about estimating where in the total job
> > progress the job is?
> >
> > Thanks!
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > <ma...@spark.apache.org>
> > For additional commands, e-mail: user-help@spark.apache.org
> > <ma...@spark.apache.org>
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Tracking / estimating job progress
Posted by .
On 5/13/2016 10:16 AM, Anthony May wrote:
> http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.SparkStatusTracker
>
> Might be useful
How do you use it? You cannot instantiate the class - is the constructor
private? Thanks!
>
> On Fri, 13 May 2016 at 11:11 Ted Yu <yuzhihong@gmail.com
> <ma...@gmail.com>> wrote:
>
> Have you looked
> at core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala
> ?
>
> Cheers
>
> On Fri, May 13, 2016 at 10:05 AM, Dood@ODDO <oddodaoddo@gmail.com
> <ma...@gmail.com>> wrote:
>
> I provide a RESTful API interface from scalatra for launching
> Spark jobs - part of the functionality is tracking these jobs.
> What API is available to track the progress of a particular
> spark application? How about estimating where in the total job
> progress the job is?
>
> Thanks!
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> <ma...@spark.apache.org>
> For additional commands, e-mail: user-help@spark.apache.org
> <ma...@spark.apache.org>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Tracking / estimating job progress
Posted by Anthony May <an...@gmail.com>.
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.SparkStatusTracker
Might be useful
On Fri, 13 May 2016 at 11:11 Ted Yu <yu...@gmail.com> wrote:
> Have you looked
> at core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ?
>
> Cheers
>
> On Fri, May 13, 2016 at 10:05 AM, Dood@ODDO <od...@gmail.com> wrote:
>
>> I provide a RESTful API interface from scalatra for launching Spark jobs
>> - part of the functionality is tracking these jobs. What API is available
>> to track the progress of a particular spark application? How about
>> estimating where in the total job progress the job is?
>>
>> Thanks!
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: Tracking / estimating job progress
Posted by Ted Yu <yu...@gmail.com>.
Have you looked
at core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ?
Cheers
On Fri, May 13, 2016 at 10:05 AM, Dood@ODDO <od...@gmail.com> wrote:
> I provide a RESTful API interface from scalatra for launching Spark jobs -
> part of the functionality is tracking these jobs. What API is available to
> track the progress of a particular spark application? How about estimating
> where in the total job progress the job is?
>
> Thanks!
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>