user notification upon application error

[Manuel Sopena Ballesteros <ma...@garvan.org.au>]

Dear Hadoop community,

I have a Hadoop cluster with yarn, spark and hdfs. Users don't have access to the yarn web interface and I would like to as what options I have to notify users when a job fails and the error message?

Is there anything I can setup for that?

Thank you very much

Manuel Sopena Ballesteros

Big Data Engineer | Kinghorn Centre for Clinical Genomics

 [cid:image001.png@01D4C835.ED3C2230] <https://www.garvan.org.au/>

a: 384 Victoria Street, Darlinghurst NSW 2010
p: +61 2 9355 5760  |  +61 4 12 123 123
e: manuel.sb@garvan.org.au<ma...@garvan.org.au>

Like us on Facebook<http://www.facebook.com/garvaninstitute> | Follow us on Twitter<http://twitter.com/GarvanInstitute> and LinkedIn<http://www.linkedin.com/company/garvan-institute-of-medical-research>

NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.

Re: user notification upon application error

[Prabhu Josephraj <pj...@cloudera.com.INVALID>]

MapReduce Application can be configured to notify the status on completion
through mapreduce.job.end-notification.url.
Need to write a webservice to collect the status and send email to users.
Below has examples

https://hadoopi.wordpress.com/2013/09/18/hadoop-get-a-callback-on-mapreduce-job-completion/
https://amalgjose.com/2015/07/21/notification-on-completion-of-mapreduce-jobs/

Other applications Spark, TEZ does not have similar way. YARN also does not
have a generic way to notify.
Will suggest you to write a service which checks job status periodically
using YARN REST Api and notify
users with status and application logs. The interval should be higher else
it will overload YARN.

https://community.cloudera.com/t5/Support-Questions/Hi-I-have-a-query-in-Spark-How-to-set-up-email-notification/td-p/117924






On Thu, Dec 5, 2019 at 10:07 AM Manuel Sopena Ballesteros <
manuel.sb@garvan.org.au> wrote:

> Dear Hadoop community,
>
>
>
> I have a Hadoop cluster with yarn, spark and hdfs. Users don’t have access
> to the yarn web interface and I would like to as what options I have to
> notify users when a job fails and the error message?
>
>
>
> Is there anything I can setup for that?
>
>
>
> Thank you very much
>
>
>
> Manuel Sopena Ballesteros
>
> Big Data Engineer | Kinghorn Centre for Clinical Genomics
>
>  [image: cid:image001.png@01D4C835.ED3C2230] <https://www.garvan.org.au/>
>
>
> *a:* 384 Victoria Street, Darlinghurst NSW 2010
> *p:* +61 2 9355 5760  |  +61 4 12 123 123
> *e:* manuel.sb@garvan.org.au
>
> Like us on Facebook <http://www.facebook.com/garvaninstitute> | Follow us
> on Twitter <http://twitter.com/GarvanInstitute> and LinkedIn
> <http://www.linkedin.com/company/garvan-institute-of-medical-research>
>
>
> NOTICE
> Please consider the environment before printing this email. This message
> and any attachments are intended for the addressee named and may contain
> legally privileged/confidential/copyright information. If you are not the
> intended recipient, you should not read, use, disclose, copy or distribute
> this communication. If you have received this message in error please
> notify us at once by return email and then delete both messages. We accept
> no liability for the distribution of viruses or similar in electronic
> communications. This notice should not be removed.
>