You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Shivam Sharma <28...@gmail.com> on 2017/12/26 08:29:29 UTC
Need to restart flink job on yarn as supervisord does
Hi,
My Flink job fails due to external activity like when Kafka goes down. I
want to restart my Flink job after certain time interval.
*I need to know best practices in this. How to restart Flink job
automatically.*
Thanks
--
Shivam Sharma
Data Engineer @ Goibibo
Indian Institute Of Information Technology, Design and Manufacturing
Jabalpur
Mobile No- (+91) 8882114744
Email:- 28shivamsharma@gmail.com
LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
<https://www.linkedin.com/in/28shivamsharma>*
Re: Need to restart flink job on yarn as supervisord does
Posted by Till Rohrmann <tr...@apache.org>.
Hi Shivam,
could you elaborate a little bit on the OutOfMemory issue you're observing?
Maybe you could provide the logs.
Cheers,
Till
On Tue, Jan 2, 2018 at 2:12 PM, Shivam Sharma <28...@gmail.com>
wrote:
> Hi,
>
> I am using below restart strategy
>
> // Retry always
> env.setRestartStrategy(RestartStrategies.failureRateRestart(
> Integer.MAX_VALUE, // max failures per unit
> Time.of(20, TimeUnit.MINUTES), //time interval for measuring failure
> rate
> Time.of(10, TimeUnit.MINUTES) // delay
> ))
>
> But I am facing OutOfMemory issue.
>
> On Wed, Dec 27, 2017 at 1:23 PM, Ufuk Celebi <uc...@apache.org> wrote:
>
> > Hey Shivam,
> >
> > check this out:
> > https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/restart_
> > strategies.html
> >
> > Does it answer your questions?
> >
> > – Ufuk
> >
> > On Tue, Dec 26, 2017 at 9:30 AM, Shivam Sharma <28shivamsharma@gmail.com
> >
> > wrote:
> > > I am submitting my Flink Job on Yarn(Amazon EMR)
> > >
> > > On Tue, Dec 26, 2017 at 1:59 PM, Shivam Sharma <
> 28shivamsharma@gmail.com
> > >
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> My Flink job fails due to external activity like when Kafka goes
> down. I
> > >> want to restart my Flink job after certain time interval.
> > >>
> > >> *I need to know best practices in this. How to restart Flink job
> > >> automatically.*
> > >>
> > >> Thanks
> > >>
> > >> --
> > >> Shivam Sharma
> > >> Data Engineer @ Goibibo
> > >> Indian Institute Of Information Technology, Design and Manufacturing
> > >> Jabalpur
> > >> Mobile No- (+91) 8882114744
> > >> Email:- 28shivamsharma@gmail.com
> > >> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
> > >> <https://www.linkedin.com/in/28shivamsharma>*
> > >>
> > >
> > >
> > >
> > > --
> > > Shivam Sharma
> > > Data Engineer @ Goibibo
> > > Indian Institute Of Information Technology, Design and Manufacturing
> > > Jabalpur
> > > Mobile No- (+91) 8882114744
> > > Email:- 28shivamsharma@gmail.com
> > > LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
> > > <https://www.linkedin.com/in/28shivamsharma>*
> >
>
>
>
> --
> Shivam Sharma
> Data Engineer @ Goibibo
> Indian Institute Of Information Technology, Design and Manufacturing
> Jabalpur
> Mobile No- (+91) 8882114744
> Email:- 28shivamsharma@gmail.com
> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
> <https://www.linkedin.com/in/28shivamsharma>*
>
Re: Need to restart flink job on yarn as supervisord does
Posted by Shivam Sharma <28...@gmail.com>.
Hi,
I am using below restart strategy
// Retry always
env.setRestartStrategy(RestartStrategies.failureRateRestart(
Integer.MAX_VALUE, // max failures per unit
Time.of(20, TimeUnit.MINUTES), //time interval for measuring failure rate
Time.of(10, TimeUnit.MINUTES) // delay
))
But I am facing OutOfMemory issue.
On Wed, Dec 27, 2017 at 1:23 PM, Ufuk Celebi <uc...@apache.org> wrote:
> Hey Shivam,
>
> check this out:
> https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/restart_
> strategies.html
>
> Does it answer your questions?
>
> – Ufuk
>
> On Tue, Dec 26, 2017 at 9:30 AM, Shivam Sharma <28...@gmail.com>
> wrote:
> > I am submitting my Flink Job on Yarn(Amazon EMR)
> >
> > On Tue, Dec 26, 2017 at 1:59 PM, Shivam Sharma <28shivamsharma@gmail.com
> >
> > wrote:
> >
> >> Hi,
> >>
> >> My Flink job fails due to external activity like when Kafka goes down. I
> >> want to restart my Flink job after certain time interval.
> >>
> >> *I need to know best practices in this. How to restart Flink job
> >> automatically.*
> >>
> >> Thanks
> >>
> >> --
> >> Shivam Sharma
> >> Data Engineer @ Goibibo
> >> Indian Institute Of Information Technology, Design and Manufacturing
> >> Jabalpur
> >> Mobile No- (+91) 8882114744
> >> Email:- 28shivamsharma@gmail.com
> >> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
> >> <https://www.linkedin.com/in/28shivamsharma>*
> >>
> >
> >
> >
> > --
> > Shivam Sharma
> > Data Engineer @ Goibibo
> > Indian Institute Of Information Technology, Design and Manufacturing
> > Jabalpur
> > Mobile No- (+91) 8882114744
> > Email:- 28shivamsharma@gmail.com
> > LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
> > <https://www.linkedin.com/in/28shivamsharma>*
>
--
Shivam Sharma
Data Engineer @ Goibibo
Indian Institute Of Information Technology, Design and Manufacturing
Jabalpur
Mobile No- (+91) 8882114744
Email:- 28shivamsharma@gmail.com
LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
<https://www.linkedin.com/in/28shivamsharma>*
Re: Need to restart flink job on yarn as supervisord does
Posted by Ufuk Celebi <uc...@apache.org>.
Hey Shivam,
check this out:
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/restart_strategies.html
Does it answer your questions?
– Ufuk
On Tue, Dec 26, 2017 at 9:30 AM, Shivam Sharma <28...@gmail.com> wrote:
> I am submitting my Flink Job on Yarn(Amazon EMR)
>
> On Tue, Dec 26, 2017 at 1:59 PM, Shivam Sharma <28...@gmail.com>
> wrote:
>
>> Hi,
>>
>> My Flink job fails due to external activity like when Kafka goes down. I
>> want to restart my Flink job after certain time interval.
>>
>> *I need to know best practices in this. How to restart Flink job
>> automatically.*
>>
>> Thanks
>>
>> --
>> Shivam Sharma
>> Data Engineer @ Goibibo
>> Indian Institute Of Information Technology, Design and Manufacturing
>> Jabalpur
>> Mobile No- (+91) 8882114744
>> Email:- 28shivamsharma@gmail.com
>> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
>> <https://www.linkedin.com/in/28shivamsharma>*
>>
>
>
>
> --
> Shivam Sharma
> Data Engineer @ Goibibo
> Indian Institute Of Information Technology, Design and Manufacturing
> Jabalpur
> Mobile No- (+91) 8882114744
> Email:- 28shivamsharma@gmail.com
> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
> <https://www.linkedin.com/in/28shivamsharma>*
Re: Need to restart flink job on yarn as supervisord does
Posted by Shivam Sharma <28...@gmail.com>.
I am submitting my Flink Job on Yarn(Amazon EMR)
On Tue, Dec 26, 2017 at 1:59 PM, Shivam Sharma <28...@gmail.com>
wrote:
> Hi,
>
> My Flink job fails due to external activity like when Kafka goes down. I
> want to restart my Flink job after certain time interval.
>
> *I need to know best practices in this. How to restart Flink job
> automatically.*
>
> Thanks
>
> --
> Shivam Sharma
> Data Engineer @ Goibibo
> Indian Institute Of Information Technology, Design and Manufacturing
> Jabalpur
> Mobile No- (+91) 8882114744
> Email:- 28shivamsharma@gmail.com
> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
> <https://www.linkedin.com/in/28shivamsharma>*
>
--
Shivam Sharma
Data Engineer @ Goibibo
Indian Institute Of Information Technology, Design and Manufacturing
Jabalpur
Mobile No- (+91) 8882114744
Email:- 28shivamsharma@gmail.com
LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
<https://www.linkedin.com/in/28shivamsharma>*