You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Robert Schmidtke <ro...@gmail.com> on 2016/04/18 22:50:48 UTC
Gracefully stop long running streaming job
Hi everyone,
I am running a streaming benchmark which involves a potentially infinitely
running Flink Streaming Job. I run it blocking on YARN using ./bin/flink
run ... and then send the command into background, remembering its PID to
kill it later on. While this gets the work done, the job always ends up in
the FAILED state. I imagine it would be the same if I used ./bin/flink
cancel ... to cancel the job? It's not that pressing but it would be nice
to shut down a streaming job properly.
Thanks
Robert
--
My GPG Key ID: 336E2680
Re: Gracefully stop long running streaming job
Posted by Maximilian Michels <mx...@apache.org>.
I have to warn you that the Storm SpoutWrapper and the TwitterSource
are currently the only stoppable sources. However, we could make more
stoppable, e.g. the KafkaConsumer.
On Tue, Apr 19, 2016 at 12:38 AM, Robert Schmidtke
<ro...@gmail.com> wrote:
> I'm on 0.10.2 which seems to be still lacking this feature. Anyway I'm happy
> to see it'll be in future releases, so I'll get to enjoy it once I upgrade
> :) I'm using a FlinkKafkaConsumer081 for the record.
>
> Anyway, thanks a bunch
> Robert
>
> On Tue, Apr 19, 2016 at 12:14 AM, Matthias J. Sax <mj...@apache.org> wrote:
>>
>> If all your sources implements Stoppable interface, you can STOP a job.
>>
>> ./bin/flink stop JobID
>>
>> STOP is however quite new and it is ongoing work to make available
>> sources stoppable (some are already). Not sure what kind of sources you
>> are using right now.
>>
>> -Matthias
>>
>>
>> On 04/18/2016 10:50 PM, Robert Schmidtke wrote:
>> > Hi everyone,
>> >
>> > I am running a streaming benchmark which involves a potentially
>> > infinitely running Flink Streaming Job. I run it blocking on YARN using
>> > ./bin/flink run ... and then send the command into background,
>> > remembering its PID to kill it later on. While this gets the work done,
>> > the job always ends up in the FAILED state. I imagine it would be the
>> > same if I used ./bin/flink cancel ... to cancel the job? It's not that
>> > pressing but it would be nice to shut down a streaming job properly.
>> >
>> > Thanks
>> >
>> > Robert
>> >
>> > --
>> > My GPG Key ID: 336E2680
>>
>
>
>
> --
> My GPG Key ID: 336E2680
Re: Gracefully stop long running streaming job
Posted by Robert Schmidtke <ro...@gmail.com>.
I'm on 0.10.2 which seems to be still lacking this feature. Anyway I'm
happy to see it'll be in future releases, so I'll get to enjoy it once I
upgrade :) I'm using a FlinkKafkaConsumer081 for the record.
Anyway, thanks a bunch
Robert
On Tue, Apr 19, 2016 at 12:14 AM, Matthias J. Sax <mj...@apache.org> wrote:
> If all your sources implements Stoppable interface, you can STOP a job.
>
> ./bin/flink stop JobID
>
> STOP is however quite new and it is ongoing work to make available
> sources stoppable (some are already). Not sure what kind of sources you
> are using right now.
>
> -Matthias
>
>
> On 04/18/2016 10:50 PM, Robert Schmidtke wrote:
> > Hi everyone,
> >
> > I am running a streaming benchmark which involves a potentially
> > infinitely running Flink Streaming Job. I run it blocking on YARN using
> > ./bin/flink run ... and then send the command into background,
> > remembering its PID to kill it later on. While this gets the work done,
> > the job always ends up in the FAILED state. I imagine it would be the
> > same if I used ./bin/flink cancel ... to cancel the job? It's not that
> > pressing but it would be nice to shut down a streaming job properly.
> >
> > Thanks
> >
> > Robert
> >
> > --
> > My GPG Key ID: 336E2680
>
>
--
My GPG Key ID: 336E2680
Re: Gracefully stop long running streaming job
Posted by "Matthias J. Sax" <mj...@apache.org>.
If all your sources implements Stoppable interface, you can STOP a job.
./bin/flink stop JobID
STOP is however quite new and it is ongoing work to make available
sources stoppable (some are already). Not sure what kind of sources you
are using right now.
-Matthias
On 04/18/2016 10:50 PM, Robert Schmidtke wrote:
> Hi everyone,
>
> I am running a streaming benchmark which involves a potentially
> infinitely running Flink Streaming Job. I run it blocking on YARN using
> ./bin/flink run ... and then send the command into background,
> remembering its PID to kill it later on. While this gets the work done,
> the job always ends up in the FAILED state. I imagine it would be the
> same if I used ./bin/flink cancel ... to cancel the job? It's not that
> pressing but it would be nice to shut down a streaming job properly.
>
> Thanks
>
> Robert
>
> --
> My GPG Key ID: 336E2680