You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Bhupesh Chawda <bh...@datatorrent.com> on 2017/01/18 06:15:06 UTC
Shutdown of an Apex app
Hi All,
Currently we can shutdown an Apex app in the following ways:
1. Throw ShutdownException() from *all* the input operators
2. Use Apex CLI to shutdown an app using the YARN App Id
I think we should have some way of shutting down an application from within
an operator. It is not always true that the trigger for shutdown is sent by
the input operator only. Sometimes, an end condition may be detected by
some operator in the DAG which wants the processing to end. Such a
shutdown, although triggered from some intermediate operator in the DAG,
should guarantee graceful shut down of the application.
Thoughts?
~ Bhupesh
Re: Shutdown of an Apex app
Posted by Amol Kekre <am...@datatorrent.com>.
Agreed on input adapter way.
Enforcing all input adapters send shutdown on the same window cannot be
guaranteed, so there will be a corner case where windows are not aligned.
An ask from Stram to send shutdown way in the future reduces the scope of
the corner case, but then defeats the purpose. Even for a single logical
input adapter, all its partitions will need to issue shutdown control tuple.
So a best case attempt to align within at a window should be done.
Thks
Amol
On Wed, Jan 18, 2017 at 12:14 AM, Bhupesh Chawda <bh...@datatorrent.com>
wrote:
> Yes, Amol. That will be the case for a kill type of shutdown.
> But for graceful shutdown, it is important that each operator in the DAG
> processes exactly the same amount of data. That is the reason the shutdown
> needs to start at input operators and stop their functioning as the first
> step. Thereafter the control tuple can propagate down the DAG and shutdown
> operators as and when they encounter the control tuple.
>
> ~ Bhupesh
>
> On Wed, Jan 18, 2017 at 1:10 PM, Amol Kekre <am...@datatorrent.com> wrote:
>
> > Can be done by sending "shutdown" message via heartbeat to Stram. Then on
> > stram can shutdown the entire app
> >
> > Thks
> > Amol
> >
> >
> > On Tue, Jan 17, 2017 at 11:05 PM, Bhupesh Chawda <
> bhupesh@datatorrent.com>
> > wrote:
> >
> > > Yes Ajay, for a graceful shutdown, the data sent out should be
> processed.
> > >
> > > On Wed, Jan 18, 2017 at 12:19 PM, AJAY GUPTA <aj...@gmail.com>
> > wrote:
> > >
> > > > +1 to idea.
> > > >
> > > > Will this ensure downstream operators to process all data received
> > before
> > > > shutdown is called?
> > > > Also, how do we plan to handle cases where 2 sub-DAGs merge to a
> single
> > > > operator somewhere downstream, and an operator in one of the sub-DAGs
> > > sends
> > > > ShutdownException.
> > > >
> > > >
> > > > Ajay
> > > >
> > > > On Wed, Jan 18, 2017 at 12:00 PM, Bhupesh Chawda <
> > > bhupesh@datatorrent.com>
> > > > wrote:
> > > >
> > > > > This JIRA is to stop the DAG in a crude manner, based on an error
> > > > > condition. I think this might also need similar functionality as an
> > > error
> > > > > condition can occur anywhere in the DAG.
> > > > >
> > > > > Perhaps we can modify the same JIRA to include a graceful +
> > ungraceful
> > > > > (kill) shutdown from any operator in the DAG.
> > > > >
> > > > > ~ Bhupesh
> > > > >
> > > > > On Wed, Jan 18, 2017 at 11:55 AM, Tushar Gosavi <
> > > tushar@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > I think this would be a great addition for batch use cases or use
> > > > > > cases were DAG needs to be shutdown after detecting some
> > > > > > completion/error condition through the operator. We have one Jira
> > > > > > Opened for such functionality
> > > > > > https://issues.apache.org/jira/browse/APEXCORE-503.
> > > > > >
> > > > > > - Tushar.
> > > > > >
> > > > > >
> > > > > > On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda
> > > > > > <bh...@datatorrent.com> wrote:
> > > > > > > Hi All,
> > > > > > >
> > > > > > > Currently we can shutdown an Apex app in the following ways:
> > > > > > > 1. Throw ShutdownException() from *all* the input operators
> > > > > > > 2. Use Apex CLI to shutdown an app using the YARN App Id
> > > > > > >
> > > > > > > I think we should have some way of shutting down an application
> > > from
> > > > > > within
> > > > > > > an operator. It is not always true that the trigger for
> shutdown
> > is
> > > > > sent
> > > > > > by
> > > > > > > the input operator only. Sometimes, an end condition may be
> > > detected
> > > > by
> > > > > > > some operator in the DAG which wants the processing to end.
> Such
> > a
> > > > > > > shutdown, although triggered from some intermediate operator in
> > the
> > > > > DAG,
> > > > > > > should guarantee graceful shut down of the application.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > ~ Bhupesh
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Shutdown of an Apex app
Posted by Bhupesh Chawda <bh...@datatorrent.com>.
Yes, Amol. That will be the case for a kill type of shutdown.
But for graceful shutdown, it is important that each operator in the DAG
processes exactly the same amount of data. That is the reason the shutdown
needs to start at input operators and stop their functioning as the first
step. Thereafter the control tuple can propagate down the DAG and shutdown
operators as and when they encounter the control tuple.
~ Bhupesh
On Wed, Jan 18, 2017 at 1:10 PM, Amol Kekre <am...@datatorrent.com> wrote:
> Can be done by sending "shutdown" message via heartbeat to Stram. Then on
> stram can shutdown the entire app
>
> Thks
> Amol
>
>
> On Tue, Jan 17, 2017 at 11:05 PM, Bhupesh Chawda <bh...@datatorrent.com>
> wrote:
>
> > Yes Ajay, for a graceful shutdown, the data sent out should be processed.
> >
> > On Wed, Jan 18, 2017 at 12:19 PM, AJAY GUPTA <aj...@gmail.com>
> wrote:
> >
> > > +1 to idea.
> > >
> > > Will this ensure downstream operators to process all data received
> before
> > > shutdown is called?
> > > Also, how do we plan to handle cases where 2 sub-DAGs merge to a single
> > > operator somewhere downstream, and an operator in one of the sub-DAGs
> > sends
> > > ShutdownException.
> > >
> > >
> > > Ajay
> > >
> > > On Wed, Jan 18, 2017 at 12:00 PM, Bhupesh Chawda <
> > bhupesh@datatorrent.com>
> > > wrote:
> > >
> > > > This JIRA is to stop the DAG in a crude manner, based on an error
> > > > condition. I think this might also need similar functionality as an
> > error
> > > > condition can occur anywhere in the DAG.
> > > >
> > > > Perhaps we can modify the same JIRA to include a graceful +
> ungraceful
> > > > (kill) shutdown from any operator in the DAG.
> > > >
> > > > ~ Bhupesh
> > > >
> > > > On Wed, Jan 18, 2017 at 11:55 AM, Tushar Gosavi <
> > tushar@datatorrent.com>
> > > > wrote:
> > > >
> > > > > I think this would be a great addition for batch use cases or use
> > > > > cases were DAG needs to be shutdown after detecting some
> > > > > completion/error condition through the operator. We have one Jira
> > > > > Opened for such functionality
> > > > > https://issues.apache.org/jira/browse/APEXCORE-503.
> > > > >
> > > > > - Tushar.
> > > > >
> > > > >
> > > > > On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda
> > > > > <bh...@datatorrent.com> wrote:
> > > > > > Hi All,
> > > > > >
> > > > > > Currently we can shutdown an Apex app in the following ways:
> > > > > > 1. Throw ShutdownException() from *all* the input operators
> > > > > > 2. Use Apex CLI to shutdown an app using the YARN App Id
> > > > > >
> > > > > > I think we should have some way of shutting down an application
> > from
> > > > > within
> > > > > > an operator. It is not always true that the trigger for shutdown
> is
> > > > sent
> > > > > by
> > > > > > the input operator only. Sometimes, an end condition may be
> > detected
> > > by
> > > > > > some operator in the DAG which wants the processing to end. Such
> a
> > > > > > shutdown, although triggered from some intermediate operator in
> the
> > > > DAG,
> > > > > > should guarantee graceful shut down of the application.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > ~ Bhupesh
> > > > >
> > > >
> > >
> >
>
Re: Shutdown of an Apex app
Posted by Amol Kekre <am...@datatorrent.com>.
Can be done by sending "shutdown" message via heartbeat to Stram. Then on
stram can shutdown the entire app
Thks
Amol
On Tue, Jan 17, 2017 at 11:05 PM, Bhupesh Chawda <bh...@datatorrent.com>
wrote:
> Yes Ajay, for a graceful shutdown, the data sent out should be processed.
>
> On Wed, Jan 18, 2017 at 12:19 PM, AJAY GUPTA <aj...@gmail.com> wrote:
>
> > +1 to idea.
> >
> > Will this ensure downstream operators to process all data received before
> > shutdown is called?
> > Also, how do we plan to handle cases where 2 sub-DAGs merge to a single
> > operator somewhere downstream, and an operator in one of the sub-DAGs
> sends
> > ShutdownException.
> >
> >
> > Ajay
> >
> > On Wed, Jan 18, 2017 at 12:00 PM, Bhupesh Chawda <
> bhupesh@datatorrent.com>
> > wrote:
> >
> > > This JIRA is to stop the DAG in a crude manner, based on an error
> > > condition. I think this might also need similar functionality as an
> error
> > > condition can occur anywhere in the DAG.
> > >
> > > Perhaps we can modify the same JIRA to include a graceful + ungraceful
> > > (kill) shutdown from any operator in the DAG.
> > >
> > > ~ Bhupesh
> > >
> > > On Wed, Jan 18, 2017 at 11:55 AM, Tushar Gosavi <
> tushar@datatorrent.com>
> > > wrote:
> > >
> > > > I think this would be a great addition for batch use cases or use
> > > > cases were DAG needs to be shutdown after detecting some
> > > > completion/error condition through the operator. We have one Jira
> > > > Opened for such functionality
> > > > https://issues.apache.org/jira/browse/APEXCORE-503.
> > > >
> > > > - Tushar.
> > > >
> > > >
> > > > On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda
> > > > <bh...@datatorrent.com> wrote:
> > > > > Hi All,
> > > > >
> > > > > Currently we can shutdown an Apex app in the following ways:
> > > > > 1. Throw ShutdownException() from *all* the input operators
> > > > > 2. Use Apex CLI to shutdown an app using the YARN App Id
> > > > >
> > > > > I think we should have some way of shutting down an application
> from
> > > > within
> > > > > an operator. It is not always true that the trigger for shutdown is
> > > sent
> > > > by
> > > > > the input operator only. Sometimes, an end condition may be
> detected
> > by
> > > > > some operator in the DAG which wants the processing to end. Such a
> > > > > shutdown, although triggered from some intermediate operator in the
> > > DAG,
> > > > > should guarantee graceful shut down of the application.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > ~ Bhupesh
> > > >
> > >
> >
>
Re: Shutdown of an Apex app
Posted by Bhupesh Chawda <bh...@datatorrent.com>.
Yes Ajay, for a graceful shutdown, the data sent out should be processed.
On Wed, Jan 18, 2017 at 12:19 PM, AJAY GUPTA <aj...@gmail.com> wrote:
> +1 to idea.
>
> Will this ensure downstream operators to process all data received before
> shutdown is called?
> Also, how do we plan to handle cases where 2 sub-DAGs merge to a single
> operator somewhere downstream, and an operator in one of the sub-DAGs sends
> ShutdownException.
>
>
> Ajay
>
> On Wed, Jan 18, 2017 at 12:00 PM, Bhupesh Chawda <bh...@datatorrent.com>
> wrote:
>
> > This JIRA is to stop the DAG in a crude manner, based on an error
> > condition. I think this might also need similar functionality as an error
> > condition can occur anywhere in the DAG.
> >
> > Perhaps we can modify the same JIRA to include a graceful + ungraceful
> > (kill) shutdown from any operator in the DAG.
> >
> > ~ Bhupesh
> >
> > On Wed, Jan 18, 2017 at 11:55 AM, Tushar Gosavi <tu...@datatorrent.com>
> > wrote:
> >
> > > I think this would be a great addition for batch use cases or use
> > > cases were DAG needs to be shutdown after detecting some
> > > completion/error condition through the operator. We have one Jira
> > > Opened for such functionality
> > > https://issues.apache.org/jira/browse/APEXCORE-503.
> > >
> > > - Tushar.
> > >
> > >
> > > On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda
> > > <bh...@datatorrent.com> wrote:
> > > > Hi All,
> > > >
> > > > Currently we can shutdown an Apex app in the following ways:
> > > > 1. Throw ShutdownException() from *all* the input operators
> > > > 2. Use Apex CLI to shutdown an app using the YARN App Id
> > > >
> > > > I think we should have some way of shutting down an application from
> > > within
> > > > an operator. It is not always true that the trigger for shutdown is
> > sent
> > > by
> > > > the input operator only. Sometimes, an end condition may be detected
> by
> > > > some operator in the DAG which wants the processing to end. Such a
> > > > shutdown, although triggered from some intermediate operator in the
> > DAG,
> > > > should guarantee graceful shut down of the application.
> > > >
> > > > Thoughts?
> > > >
> > > > ~ Bhupesh
> > >
> >
>
Re: Shutdown of an Apex app
Posted by AJAY GUPTA <aj...@gmail.com>.
+1 to idea.
Will this ensure downstream operators to process all data received before
shutdown is called?
Also, how do we plan to handle cases where 2 sub-DAGs merge to a single
operator somewhere downstream, and an operator in one of the sub-DAGs sends
ShutdownException.
Ajay
On Wed, Jan 18, 2017 at 12:00 PM, Bhupesh Chawda <bh...@datatorrent.com>
wrote:
> This JIRA is to stop the DAG in a crude manner, based on an error
> condition. I think this might also need similar functionality as an error
> condition can occur anywhere in the DAG.
>
> Perhaps we can modify the same JIRA to include a graceful + ungraceful
> (kill) shutdown from any operator in the DAG.
>
> ~ Bhupesh
>
> On Wed, Jan 18, 2017 at 11:55 AM, Tushar Gosavi <tu...@datatorrent.com>
> wrote:
>
> > I think this would be a great addition for batch use cases or use
> > cases were DAG needs to be shutdown after detecting some
> > completion/error condition through the operator. We have one Jira
> > Opened for such functionality
> > https://issues.apache.org/jira/browse/APEXCORE-503.
> >
> > - Tushar.
> >
> >
> > On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda
> > <bh...@datatorrent.com> wrote:
> > > Hi All,
> > >
> > > Currently we can shutdown an Apex app in the following ways:
> > > 1. Throw ShutdownException() from *all* the input operators
> > > 2. Use Apex CLI to shutdown an app using the YARN App Id
> > >
> > > I think we should have some way of shutting down an application from
> > within
> > > an operator. It is not always true that the trigger for shutdown is
> sent
> > by
> > > the input operator only. Sometimes, an end condition may be detected by
> > > some operator in the DAG which wants the processing to end. Such a
> > > shutdown, although triggered from some intermediate operator in the
> DAG,
> > > should guarantee graceful shut down of the application.
> > >
> > > Thoughts?
> > >
> > > ~ Bhupesh
> >
>
Re: Shutdown of an Apex app
Posted by Bhupesh Chawda <bh...@datatorrent.com>.
Hi Thomas,
The part of initiating the shutdown from another operator was the original
intention of the email.
~ Bhupesh
On Wed, Jan 18, 2017 at 10:35 PM, Thomas Weise <th...@apache.org> wrote:
> Bhupesh,
>
> If your intention is to let processing complete, the sources should
> terminate the stream (that's already implemented, except that you want to
> initiate it from another operator). It does not really matter if they all
> stop at the same streaming window.
>
> If you want to just stop the application from an operator (similar to
> kill), it needs to be signaled to the master and the master needs to
> terminate. One way of doing it could be the stats listener.
>
> Thomas
>
>
> On Tue, Jan 17, 2017 at 10:30 PM, Bhupesh Chawda <bh...@datatorrent.com>
> wrote:
>
> > This JIRA is to stop the DAG in a crude manner, based on an error
> > condition. I think this might also need similar functionality as an error
> > condition can occur anywhere in the DAG.
> >
> > Perhaps we can modify the same JIRA to include a graceful + ungraceful
> > (kill) shutdown from any operator in the DAG.
> >
> > ~ Bhupesh
> >
> > On Wed, Jan 18, 2017 at 11:55 AM, Tushar Gosavi <tu...@datatorrent.com>
> > wrote:
> >
> > > I think this would be a great addition for batch use cases or use
> > > cases were DAG needs to be shutdown after detecting some
> > > completion/error condition through the operator. We have one Jira
> > > Opened for such functionality
> > > https://issues.apache.org/jira/browse/APEXCORE-503.
> > >
> > > - Tushar.
> > >
> > >
> > > On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda
> > > <bh...@datatorrent.com> wrote:
> > > > Hi All,
> > > >
> > > > Currently we can shutdown an Apex app in the following ways:
> > > > 1. Throw ShutdownException() from *all* the input operators
> > > > 2. Use Apex CLI to shutdown an app using the YARN App Id
> > > >
> > > > I think we should have some way of shutting down an application from
> > > within
> > > > an operator. It is not always true that the trigger for shutdown is
> > sent
> > > by
> > > > the input operator only. Sometimes, an end condition may be detected
> by
> > > > some operator in the DAG which wants the processing to end. Such a
> > > > shutdown, although triggered from some intermediate operator in the
> > DAG,
> > > > should guarantee graceful shut down of the application.
> > > >
> > > > Thoughts?
> > > >
> > > > ~ Bhupesh
> > >
> >
>
Re: Shutdown of an Apex app
Posted by Tushar Gosavi <tu...@datatorrent.com>.
Hi Thomas,
Handling application shutdown from an operator with help of stats
listener can be lot of work, instead we could provide an shortcut api
for
user to terminate the application. It could be based on response code
from statsListener as well as throwing an special exception from
operator thread or setting a special request through Context.
As far as I know the statsListener approach would involve
- Adding a shared stats listener between operator detecting
termination condition and with all input operators.
- stats listener sending an OperatorRequest whilch will throw
ShutdowException in execute method.
- Tushar.
On Wed, Jan 18, 2017 at 10:35 PM, Thomas Weise <th...@apache.org> wrote:
> Bhupesh,
>
> If your intention is to let processing complete, the sources should
> terminate the stream (that's already implemented, except that you want to
> initiate it from another operator). It does not really matter if they all
> stop at the same streaming window.
>
> If you want to just stop the application from an operator (similar to
> kill), it needs to be signaled to the master and the master needs to
> terminate. One way of doing it could be the stats listener.
>
> Thomas
>
>
> On Tue, Jan 17, 2017 at 10:30 PM, Bhupesh Chawda <bh...@datatorrent.com>
> wrote:
>
>> This JIRA is to stop the DAG in a crude manner, based on an error
>> condition. I think this might also need similar functionality as an error
>> condition can occur anywhere in the DAG.
>>
>> Perhaps we can modify the same JIRA to include a graceful + ungraceful
>> (kill) shutdown from any operator in the DAG.
>>
>> ~ Bhupesh
>>
>> On Wed, Jan 18, 2017 at 11:55 AM, Tushar Gosavi <tu...@datatorrent.com>
>> wrote:
>>
>> > I think this would be a great addition for batch use cases or use
>> > cases were DAG needs to be shutdown after detecting some
>> > completion/error condition through the operator. We have one Jira
>> > Opened for such functionality
>> > https://issues.apache.org/jira/browse/APEXCORE-503.
>> >
>> > - Tushar.
>> >
>> >
>> > On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda
>> > <bh...@datatorrent.com> wrote:
>> > > Hi All,
>> > >
>> > > Currently we can shutdown an Apex app in the following ways:
>> > > 1. Throw ShutdownException() from *all* the input operators
>> > > 2. Use Apex CLI to shutdown an app using the YARN App Id
>> > >
>> > > I think we should have some way of shutting down an application from
>> > within
>> > > an operator. It is not always true that the trigger for shutdown is
>> sent
>> > by
>> > > the input operator only. Sometimes, an end condition may be detected by
>> > > some operator in the DAG which wants the processing to end. Such a
>> > > shutdown, although triggered from some intermediate operator in the
>> DAG,
>> > > should guarantee graceful shut down of the application.
>> > >
>> > > Thoughts?
>> > >
>> > > ~ Bhupesh
>> >
>>
Re: Shutdown of an Apex app
Posted by Thomas Weise <th...@apache.org>.
Bhupesh,
If your intention is to let processing complete, the sources should
terminate the stream (that's already implemented, except that you want to
initiate it from another operator). It does not really matter if they all
stop at the same streaming window.
If you want to just stop the application from an operator (similar to
kill), it needs to be signaled to the master and the master needs to
terminate. One way of doing it could be the stats listener.
Thomas
On Tue, Jan 17, 2017 at 10:30 PM, Bhupesh Chawda <bh...@datatorrent.com>
wrote:
> This JIRA is to stop the DAG in a crude manner, based on an error
> condition. I think this might also need similar functionality as an error
> condition can occur anywhere in the DAG.
>
> Perhaps we can modify the same JIRA to include a graceful + ungraceful
> (kill) shutdown from any operator in the DAG.
>
> ~ Bhupesh
>
> On Wed, Jan 18, 2017 at 11:55 AM, Tushar Gosavi <tu...@datatorrent.com>
> wrote:
>
> > I think this would be a great addition for batch use cases or use
> > cases were DAG needs to be shutdown after detecting some
> > completion/error condition through the operator. We have one Jira
> > Opened for such functionality
> > https://issues.apache.org/jira/browse/APEXCORE-503.
> >
> > - Tushar.
> >
> >
> > On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda
> > <bh...@datatorrent.com> wrote:
> > > Hi All,
> > >
> > > Currently we can shutdown an Apex app in the following ways:
> > > 1. Throw ShutdownException() from *all* the input operators
> > > 2. Use Apex CLI to shutdown an app using the YARN App Id
> > >
> > > I think we should have some way of shutting down an application from
> > within
> > > an operator. It is not always true that the trigger for shutdown is
> sent
> > by
> > > the input operator only. Sometimes, an end condition may be detected by
> > > some operator in the DAG which wants the processing to end. Such a
> > > shutdown, although triggered from some intermediate operator in the
> DAG,
> > > should guarantee graceful shut down of the application.
> > >
> > > Thoughts?
> > >
> > > ~ Bhupesh
> >
>
Re: Shutdown of an Apex app
Posted by Bhupesh Chawda <bh...@datatorrent.com>.
This JIRA is to stop the DAG in a crude manner, based on an error
condition. I think this might also need similar functionality as an error
condition can occur anywhere in the DAG.
Perhaps we can modify the same JIRA to include a graceful + ungraceful
(kill) shutdown from any operator in the DAG.
~ Bhupesh
On Wed, Jan 18, 2017 at 11:55 AM, Tushar Gosavi <tu...@datatorrent.com>
wrote:
> I think this would be a great addition for batch use cases or use
> cases were DAG needs to be shutdown after detecting some
> completion/error condition through the operator. We have one Jira
> Opened for such functionality
> https://issues.apache.org/jira/browse/APEXCORE-503.
>
> - Tushar.
>
>
> On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda
> <bh...@datatorrent.com> wrote:
> > Hi All,
> >
> > Currently we can shutdown an Apex app in the following ways:
> > 1. Throw ShutdownException() from *all* the input operators
> > 2. Use Apex CLI to shutdown an app using the YARN App Id
> >
> > I think we should have some way of shutting down an application from
> within
> > an operator. It is not always true that the trigger for shutdown is sent
> by
> > the input operator only. Sometimes, an end condition may be detected by
> > some operator in the DAG which wants the processing to end. Such a
> > shutdown, although triggered from some intermediate operator in the DAG,
> > should guarantee graceful shut down of the application.
> >
> > Thoughts?
> >
> > ~ Bhupesh
>
Re: Shutdown of an Apex app
Posted by Bhupesh Chawda <bh...@datatorrent.com>.
If it is not an error condition, I would like for the data that is already
sent out by the upstream operators to get processed, before the shutdown of
an operator is initiated. Imagine something like a control tuple that
starts from the input operators and flows down the stream and stops the
containers in the DAG.
However, in case of an error condition, we have no choice but to kill the
DAG.
~ Bhupesh
On Wed, Jan 18, 2017 at 12:01 PM, Chinmay Kolhatkar <chinmay@datatorrent.com
> wrote:
> +1 for the feature in general.
>
> Though I have a question about it: Lets say 10th operator in the DAG
> initiates the shutdown.. And operator being in quite downstream as compared
> to Input Operator, it might be fairly behind in window it is processing.
> What happens to the data being already processed by upstream operators?
>
> Is there a possibility where Idempotency might get affected because
> termination action is taken in downstream operators?
>
> -Chinmay.
>
>
> On Wed, Jan 18, 2017 at 11:55 AM, Tushar Gosavi <tu...@datatorrent.com>
> wrote:
>
> > I think this would be a great addition for batch use cases or use
> > cases were DAG needs to be shutdown after detecting some
> > completion/error condition through the operator. We have one Jira
> > Opened for such functionality
> > https://issues.apache.org/jira/browse/APEXCORE-503.
> >
> > - Tushar.
> >
> >
> > On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda
> > <bh...@datatorrent.com> wrote:
> > > Hi All,
> > >
> > > Currently we can shutdown an Apex app in the following ways:
> > > 1. Throw ShutdownException() from *all* the input operators
> > > 2. Use Apex CLI to shutdown an app using the YARN App Id
> > >
> > > I think we should have some way of shutting down an application from
> > within
> > > an operator. It is not always true that the trigger for shutdown is
> sent
> > by
> > > the input operator only. Sometimes, an end condition may be detected by
> > > some operator in the DAG which wants the processing to end. Such a
> > > shutdown, although triggered from some intermediate operator in the
> DAG,
> > > should guarantee graceful shut down of the application.
> > >
> > > Thoughts?
> > >
> > > ~ Bhupesh
> >
>
Re: Shutdown of an Apex app
Posted by Bhupesh Chawda <bh...@datatorrent.com>.
Hi Sanjay,
The task of the shutdown is to stop the processing from the input operator
itself. Once we reach a consistent state (endWindow() for example), we stop
the input operator. Thereafter, the next operators in the DAG would only
have to process the data which is already emitted by the input operators.
I am not sure of the implementation though; just discussing the need and
requirements.
~ Bhupesh
On Wed, Jan 18, 2017 at 12:13 PM, Sanjay Pujare <sa...@datatorrent.com>
wrote:
> I have similar question/concerns. When a downstream operator initiates a
> shutdown, how is that trigger signaled to and co-ordinated with the input
> operators stopping their input reading activity? Or idempotency etc don’t
> matter after shutdown?
>
>
>
> On 1/17/17, 10:31 PM, "Chinmay Kolhatkar" <ch...@datatorrent.com> wrote:
>
> +1 for the feature in general.
>
> Though I have a question about it: Lets say 10th operator in the DAG
> initiates the shutdown.. And operator being in quite downstream as
> compared
> to Input Operator, it might be fairly behind in window it is
> processing.
> What happens to the data being already processed by upstream operators?
>
> Is there a possibility where Idempotency might get affected because
> termination action is taken in downstream operators?
>
> -Chinmay.
>
>
> On Wed, Jan 18, 2017 at 11:55 AM, Tushar Gosavi <
> tushar@datatorrent.com>
> wrote:
>
> > I think this would be a great addition for batch use cases or use
> > cases were DAG needs to be shutdown after detecting some
> > completion/error condition through the operator. We have one Jira
> > Opened for such functionality
> > https://issues.apache.org/jira/browse/APEXCORE-503.
> >
> > - Tushar.
> >
> >
> > On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda
> > <bh...@datatorrent.com> wrote:
> > > Hi All,
> > >
> > > Currently we can shutdown an Apex app in the following ways:
> > > 1. Throw ShutdownException() from *all* the input operators
> > > 2. Use Apex CLI to shutdown an app using the YARN App Id
> > >
> > > I think we should have some way of shutting down an application
> from
> > within
> > > an operator. It is not always true that the trigger for shutdown
> is sent
> > by
> > > the input operator only. Sometimes, an end condition may be
> detected by
> > > some operator in the DAG which wants the processing to end. Such a
> > > shutdown, although triggered from some intermediate operator in
> the DAG,
> > > should guarantee graceful shut down of the application.
> > >
> > > Thoughts?
> > >
> > > ~ Bhupesh
> >
>
>
>
>
Re: Shutdown of an Apex app
Posted by Sanjay Pujare <sa...@datatorrent.com>.
I have similar question/concerns. When a downstream operator initiates a shutdown, how is that trigger signaled to and co-ordinated with the input operators stopping their input reading activity? Or idempotency etc don’t matter after shutdown?
On 1/17/17, 10:31 PM, "Chinmay Kolhatkar" <ch...@datatorrent.com> wrote:
+1 for the feature in general.
Though I have a question about it: Lets say 10th operator in the DAG
initiates the shutdown.. And operator being in quite downstream as compared
to Input Operator, it might be fairly behind in window it is processing.
What happens to the data being already processed by upstream operators?
Is there a possibility where Idempotency might get affected because
termination action is taken in downstream operators?
-Chinmay.
On Wed, Jan 18, 2017 at 11:55 AM, Tushar Gosavi <tu...@datatorrent.com>
wrote:
> I think this would be a great addition for batch use cases or use
> cases were DAG needs to be shutdown after detecting some
> completion/error condition through the operator. We have one Jira
> Opened for such functionality
> https://issues.apache.org/jira/browse/APEXCORE-503.
>
> - Tushar.
>
>
> On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda
> <bh...@datatorrent.com> wrote:
> > Hi All,
> >
> > Currently we can shutdown an Apex app in the following ways:
> > 1. Throw ShutdownException() from *all* the input operators
> > 2. Use Apex CLI to shutdown an app using the YARN App Id
> >
> > I think we should have some way of shutting down an application from
> within
> > an operator. It is not always true that the trigger for shutdown is sent
> by
> > the input operator only. Sometimes, an end condition may be detected by
> > some operator in the DAG which wants the processing to end. Such a
> > shutdown, although triggered from some intermediate operator in the DAG,
> > should guarantee graceful shut down of the application.
> >
> > Thoughts?
> >
> > ~ Bhupesh
>
Re: Shutdown of an Apex app
Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
+1 for the feature in general.
Though I have a question about it: Lets say 10th operator in the DAG
initiates the shutdown.. And operator being in quite downstream as compared
to Input Operator, it might be fairly behind in window it is processing.
What happens to the data being already processed by upstream operators?
Is there a possibility where Idempotency might get affected because
termination action is taken in downstream operators?
-Chinmay.
On Wed, Jan 18, 2017 at 11:55 AM, Tushar Gosavi <tu...@datatorrent.com>
wrote:
> I think this would be a great addition for batch use cases or use
> cases were DAG needs to be shutdown after detecting some
> completion/error condition through the operator. We have one Jira
> Opened for such functionality
> https://issues.apache.org/jira/browse/APEXCORE-503.
>
> - Tushar.
>
>
> On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda
> <bh...@datatorrent.com> wrote:
> > Hi All,
> >
> > Currently we can shutdown an Apex app in the following ways:
> > 1. Throw ShutdownException() from *all* the input operators
> > 2. Use Apex CLI to shutdown an app using the YARN App Id
> >
> > I think we should have some way of shutting down an application from
> within
> > an operator. It is not always true that the trigger for shutdown is sent
> by
> > the input operator only. Sometimes, an end condition may be detected by
> > some operator in the DAG which wants the processing to end. Such a
> > shutdown, although triggered from some intermediate operator in the DAG,
> > should guarantee graceful shut down of the application.
> >
> > Thoughts?
> >
> > ~ Bhupesh
>
Re: Shutdown of an Apex app
Posted by Tushar Gosavi <tu...@datatorrent.com>.
I think this would be a great addition for batch use cases or use
cases were DAG needs to be shutdown after detecting some
completion/error condition through the operator. We have one Jira
Opened for such functionality
https://issues.apache.org/jira/browse/APEXCORE-503.
- Tushar.
On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda
<bh...@datatorrent.com> wrote:
> Hi All,
>
> Currently we can shutdown an Apex app in the following ways:
> 1. Throw ShutdownException() from *all* the input operators
> 2. Use Apex CLI to shutdown an app using the YARN App Id
>
> I think we should have some way of shutting down an application from within
> an operator. It is not always true that the trigger for shutdown is sent by
> the input operator only. Sometimes, an end condition may be detected by
> some operator in the DAG which wants the processing to end. Such a
> shutdown, although triggered from some intermediate operator in the DAG,
> should guarantee graceful shut down of the application.
>
> Thoughts?
>
> ~ Bhupesh