Monitoring Url

["Winkelman, Kyle G" <ky...@optum.com>]

Hello,

I am interested in exposing a monitoring url through the PipelineResult object. All of the documentation pages in the runners tab of beam.apache.org have a section dedicated to ‘Monitoring your job’ and most of those reference a web interface. If we were able to get a link to this web interface, we would be able to lead non-technical users of our application to a place where they can get some understanding of the progress of the job.

An example of this in the SparkPipelineResult would be returning the value of the javaSparkContext.sc().uiWebUrl().get(). I haven’t looked into other runners but if most of them have web interfaces it would make sense that they would be able to programmatically get a url to that web interface.

Looking for feedback on if this would be accepted and ways this can be accomplished in other runners.

Thanks,
Kyle Winkelman


This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.

Re: Monitoring Url

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi

I think it's something I proposed while ago. Maybe some of you remember the discussion and use of Karaf Decanter like.

The idea was to collect (collector) metrics and data and dispatch to appender. The UI could be on the appender side.

With kind of Karaf Decanter approach we can have UI (leveraging other tools) covering monitoring, bam, auditing, etc.

Just my €0.01 😄😄

Regards
JB

Le 9 oct. 2018 à 22:34, à 22:34, Scott Wegner <sc...@apache.org> a écrit:
>IMO Beam would greatly benefit from having a monitoring UI. Individual
>runners (Spark, Dataflow) have their own UI's, which could be exposed
>through an API as you suggest.
>
>I also think there would be value in having a Beam-centric monitoring
>UI
>which focuses on concepts in the Beam model (PTransforms/PCollections,
>windows/triggers, bundles, metrics, state/timers, etc.) Dataflow's UI
>is
>pretty good, but I believe most users develop and debug locally (i.e.
>DirectRunner) before submitting to a cluster.
>
>On Mon, Oct 8, 2018 at 2:37 PM Winkelman, Kyle G
><ky...@optum.com>
>wrote:
>
>> Hello,
>>
>>
>>
>> I am interested in exposing a monitoring url through the
>PipelineResult
>> object. All of the documentation pages in the runners tab of
>> beam.apache.org have a section dedicated to ‘Monitoring your job’ and
>> most of those reference a web interface. If we were able to get a
>link to
>> this web interface, we would be able to lead non-technical users of
>our
>> application to a place where they can get some understanding of the
>> progress of the job.
>>
>>
>>
>> An example of this in the SparkPipelineResult would be returning the
>value
>> of the javaSparkContext.sc().uiWebUrl().get(). I haven’t looked into
>other
>> runners but if most of them have web interfaces it would make sense
>that
>> they would be able to programmatically get a url to that web
>interface.
>>
>>
>>
>> Looking for feedback on if this would be accepted and ways this can
>be
>> accomplished in other runners.
>>
>>
>>
>> Thanks,
>>
>> Kyle Winkelman
>>
>>
>>
>>
>> This e-mail, including attachments, may include confidential and/or
>> proprietary information, and may be used only by the person or entity
>> to which it is addressed. If the reader of this e-mail is not the
>intended
>> recipient or his or her authorized agent, the reader is hereby
>notified
>> that any dissemination, distribution or copying of this e-mail is
>> prohibited. If you have received this e-mail in error, please notify
>the
>> sender by replying to this message and delete this e-mail
>immediately.
>>
>
>
>-- 
>
>
>
>
>Got feedback? tinyurl.com/swegner-feedback

Re: Monitoring Url

[Scott Wegner <sc...@apache.org>]

IMO Beam would greatly benefit from having a monitoring UI. Individual
runners (Spark, Dataflow) have their own UI's, which could be exposed
through an API as you suggest.

I also think there would be value in having a Beam-centric monitoring UI
which focuses on concepts in the Beam model (PTransforms/PCollections,
windows/triggers, bundles, metrics, state/timers, etc.) Dataflow's UI is
pretty good, but I believe most users develop and debug locally (i.e.
DirectRunner) before submitting to a cluster.

On Mon, Oct 8, 2018 at 2:37 PM Winkelman, Kyle G <ky...@optum.com>
wrote:

> Hello,
>
>
>
> I am interested in exposing a monitoring url through the PipelineResult
> object. All of the documentation pages in the runners tab of
> beam.apache.org have a section dedicated to ‘Monitoring your job’ and
> most of those reference a web interface. If we were able to get a link to
> this web interface, we would be able to lead non-technical users of our
> application to a place where they can get some understanding of the
> progress of the job.
>
>
>
> An example of this in the SparkPipelineResult would be returning the value
> of the javaSparkContext.sc().uiWebUrl().get(). I haven’t looked into other
> runners but if most of them have web interfaces it would make sense that
> they would be able to programmatically get a url to that web interface.
>
>
>
> Looking for feedback on if this would be accepted and ways this can be
> accomplished in other runners.
>
>
>
> Thanks,
>
> Kyle Winkelman
>
>
>
>
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or entity
> to which it is addressed. If the reader of this e-mail is not the intended
> recipient or his or her authorized agent, the reader is hereby notified
> that any dissemination, distribution or copying of this e-mail is
> prohibited. If you have received this e-mail in error, please notify the
> sender by replying to this message and delete this e-mail immediately.
>


-- 




Got feedback? tinyurl.com/swegner-feedback