You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by "Doyle, Austin O." <Au...@leidos.com> on 2016/08/10 15:56:03 UTC
Apex Finished Status
When would an Apex Application return a FINISHED status from YARN as opposed to continually being in the RUNNING state?
Is there a way to force the YARN status to FINISHED from within the application when processing is complete?
RE: Apex Finished Status
Posted by "Doyle, Austin O." <Au...@leidos.com>.
The exception is thrown and the data stops its flow, but the state doesn’t change. I’ve also tried throwing the shutdown exception at the very beginning of the input operator so that the application would shut down to start with, however it continues to have the state RUNNING.
From: Ashwin Chandra Putta [mailto:ashwinchandrap@gmail.com]
Sent: Thursday, August 11, 2016 11:53 AM
To: users@apex.apache.org
Subject: RE: Apex Finished Status
No, that should be it. Do you see any operator in this dag still running with lagging windows. The shutdown will happen in all operators in the same window the exception is thrown.
On Aug 11, 2016 8:44 AM, "Doyle, Austin O." <Au...@leidos.com>> wrote:
>
> Yes I put it in my input operator, do I need to do anything or make sure there’s anything specific in the other operators?
>
>
>
> From: Ashwin Chandra Putta [mailto:ashwinchandrap@gmail.com<ma...@gmail.com>]
> Sent: Thursday, August 11, 2016 11:43 AM
> To: users@apex.apache.org<ma...@apex.apache.org>
> Subject: RE: Apex Finished Status
>
>
>
> Doyle,
>
> Have you thrown it in all input operators? It starts from the operator you throw it in and propagates downstream. All operators need to shutdown before application changes state.
>
> Regards,
> Ashwin.
>
>
>
> On Aug 11, 2016 5:01 AM, "Doyle, Austin O." <Au...@leidos.com>> wrote:
>
> The ShutdownException does stop the operator but doesn’t seem to update the state of the application, still in the RUNNING state. Is there something specific I need to put in the endwindows or teardowns?
>
>
>
> From: Chinmay Kolhatkar [mailto:chinmay@datatorrent.com<ma...@datatorrent.com>]
> Sent: Thursday, August 11, 2016 2:12 AM
> To: users@apex.apache.org<ma...@apex.apache.org>
> Subject: Re: Apex Finished Status
>
>
>
> Just one additional information to what Sandesh said. ShutdownException shuts down the operator from which its thrown and all downstream operators to that operator.
>
> Hence if you want the DAG to shutdown completely, you need to throw ShutdownException from all *input* operators.
>
>
>
> -Chinmay.
>
>
>
>
>
> On Thu, Aug 11, 2016 at 9:58 AM, Sandesh Hegde <sa...@datatorrent.com>> wrote:
>
> Hello Austin,
>
>
>
> Throwing ShutdownException from the operators will shutdown the Apex Application.
>
>
>
>
>
> On Wed, Aug 10, 2016 at 8:56 AM Doyle, Austin O. <Au...@leidos.com>> wrote:
>>
>> When would an Apex Application return a FINISHED status from YARN as opposed to continually being in the RUNNING state?
>>
>> Is there a way to force the YARN status to FINISHED from within the application when processing is complete?
>
>
RE: Apex Finished Status
Posted by Ashwin Chandra Putta <as...@gmail.com>.
No, that should be it. Do you see any operator in this dag still running
with lagging windows. The shutdown will happen in all operators in the same
window the exception is thrown.
On Aug 11, 2016 8:44 AM, "Doyle, Austin O." <Au...@leidos.com>
wrote:
>
> Yes I put it in my input operator, do I need to do anything or make sure
there’s anything specific in the other operators?
>
>
>
> From: Ashwin Chandra Putta [mailto:ashwinchandrap@gmail.com]
> Sent: Thursday, August 11, 2016 11:43 AM
> To: users@apex.apache.org
> Subject: RE: Apex Finished Status
>
>
>
> Doyle,
>
> Have you thrown it in all input operators? It starts from the operator
you throw it in and propagates downstream. All operators need to shutdown
before application changes state.
>
> Regards,
> Ashwin.
>
>
>
> On Aug 11, 2016 5:01 AM, "Doyle, Austin O." <Au...@leidos.com>
wrote:
>
> The ShutdownException does stop the operator but doesn’t seem to update
the state of the application, still in the RUNNING state. Is there
something specific I need to put in the endwindows or teardowns?
>
>
>
> From: Chinmay Kolhatkar [mailto:chinmay@datatorrent.com]
> Sent: Thursday, August 11, 2016 2:12 AM
> To: users@apex.apache.org
> Subject: Re: Apex Finished Status
>
>
>
> Just one additional information to what Sandesh said. ShutdownException
shuts down the operator from which its thrown and all downstream operators
to that operator.
>
> Hence if you want the DAG to shutdown completely, you need to throw
ShutdownException from all *input* operators.
>
>
>
> -Chinmay.
>
>
>
>
>
> On Thu, Aug 11, 2016 at 9:58 AM, Sandesh Hegde <sa...@datatorrent.com>
wrote:
>
> Hello Austin,
>
>
>
> Throwing ShutdownException from the operators will shutdown the Apex
Application.
>
>
>
>
>
> On Wed, Aug 10, 2016 at 8:56 AM Doyle, Austin O. <
Austin.O.Doyle-2@leidos.com> wrote:
>>
>> When would an Apex Application return a FINISHED status from YARN as
opposed to continually being in the RUNNING state?
>>
>> Is there a way to force the YARN status to FINISHED from within the
application when processing is complete?
>
>
RE: Apex Finished Status
Posted by "Doyle, Austin O." <Au...@leidos.com>.
Yes I put it in my input operator, do I need to do anything or make sure there’s anything specific in the other operators?
From: Ashwin Chandra Putta [mailto:ashwinchandrap@gmail.com]
Sent: Thursday, August 11, 2016 11:43 AM
To: users@apex.apache.org
Subject: RE: Apex Finished Status
Doyle,
Have you thrown it in all input operators? It starts from the operator you throw it in and propagates downstream. All operators need to shutdown before application changes state.
Regards,
Ashwin.
On Aug 11, 2016 5:01 AM, "Doyle, Austin O." <Au...@leidos.com>> wrote:
The ShutdownException does stop the operator but doesn’t seem to update the state of the application, still in the RUNNING state. Is there something specific I need to put in the endwindows or teardowns?
From: Chinmay Kolhatkar [mailto:chinmay@datatorrent.com<ma...@datatorrent.com>]
Sent: Thursday, August 11, 2016 2:12 AM
To: users@apex.apache.org<ma...@apex.apache.org>
Subject: Re: Apex Finished Status
Just one additional information to what Sandesh said. ShutdownException shuts down the operator from which its thrown and all downstream operators to that operator.
Hence if you want the DAG to shutdown completely, you need to throw ShutdownException from all *input* operators.
-Chinmay.
On Thu, Aug 11, 2016 at 9:58 AM, Sandesh Hegde <sa...@datatorrent.com>> wrote:
Hello Austin,
Throwing ShutdownException from the operators will shutdown the Apex Application.
On Wed, Aug 10, 2016 at 8:56 AM Doyle, Austin O. <Au...@leidos.com>> wrote:
When would an Apex Application return a FINISHED status from YARN as opposed to continually being in the RUNNING state?
Is there a way to force the YARN status to FINISHED from within the application when processing is complete?
RE: Apex Finished Status
Posted by Ashwin Chandra Putta <as...@gmail.com>.
Doyle,
Have you thrown it in all input operators? It starts from the operator you
throw it in and propagates downstream. All operators need to shutdown
before application changes state.
Regards,
Ashwin.
On Aug 11, 2016 5:01 AM, "Doyle, Austin O." <Au...@leidos.com>
wrote:
> The ShutdownException does stop the operator but doesn’t seem to update
> the state of the application, still in the RUNNING state. Is there
> something specific I need to put in the endwindows or teardowns?
>
>
>
> *From:* Chinmay Kolhatkar [mailto:chinmay@datatorrent.com]
> *Sent:* Thursday, August 11, 2016 2:12 AM
> *To:* users@apex.apache.org
> *Subject:* Re: Apex Finished Status
>
>
>
> Just one additional information to what Sandesh said. ShutdownException
> shuts down the operator from which its thrown and all downstream operators
> to that operator.
>
> Hence if you want the DAG to shutdown completely, you need to throw
> ShutdownException from all *input* operators.
>
>
>
> -Chinmay.
>
>
>
>
>
> On Thu, Aug 11, 2016 at 9:58 AM, Sandesh Hegde <sa...@datatorrent.com>
> wrote:
>
> Hello Austin,
>
>
>
> Throwing ShutdownException from the operators will shutdown the Apex
> Application.
>
>
>
>
>
> On Wed, Aug 10, 2016 at 8:56 AM Doyle, Austin O. <
> Austin.O.Doyle-2@leidos.com> wrote:
>
> When would an Apex Application return a FINISHED status from YARN as
> opposed to continually being in the RUNNING state?
>
> Is there a way to force the YARN status to FINISHED from within the
> application when processing is complete?
>
>
>
RE: Apex Finished Status
Posted by "Doyle, Austin O." <Au...@leidos.com>.
The ShutdownException does stop the operator but doesn’t seem to update the state of the application, still in the RUNNING state. Is there something specific I need to put in the endwindows or teardowns?
From: Chinmay Kolhatkar [mailto:chinmay@datatorrent.com]
Sent: Thursday, August 11, 2016 2:12 AM
To: users@apex.apache.org
Subject: Re: Apex Finished Status
Just one additional information to what Sandesh said. ShutdownException shuts down the operator from which its thrown and all downstream operators to that operator.
Hence if you want the DAG to shutdown completely, you need to throw ShutdownException from all *input* operators.
-Chinmay.
On Thu, Aug 11, 2016 at 9:58 AM, Sandesh Hegde <sa...@datatorrent.com>> wrote:
Hello Austin,
Throwing ShutdownException from the operators will shutdown the Apex Application.
On Wed, Aug 10, 2016 at 8:56 AM Doyle, Austin O. <Au...@leidos.com>> wrote:
When would an Apex Application return a FINISHED status from YARN as opposed to continually being in the RUNNING state?
Is there a way to force the YARN status to FINISHED from within the application when processing is complete?
Re: Apex Finished Status
Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
Just one additional information to what Sandesh said. ShutdownException
shuts down the operator from which its thrown and all downstream operators
to that operator.
Hence if you want the DAG to shutdown completely, you need to throw
ShutdownException from all *input* operators.
-Chinmay.
On Thu, Aug 11, 2016 at 9:58 AM, Sandesh Hegde <sa...@datatorrent.com>
wrote:
> Hello Austin,
>
> Throwing ShutdownException from the operators will shutdown the Apex
> Application.
>
>
> On Wed, Aug 10, 2016 at 8:56 AM Doyle, Austin O. <
> Austin.O.Doyle-2@leidos.com> wrote:
>
>> When would an Apex Application return a FINISHED status from YARN as
>> opposed to continually being in the RUNNING state?
>>
>> Is there a way to force the YARN status to FINISHED from within the
>> application when processing is complete?
>>
>
Re: Apex Finished Status
Posted by Sandesh Hegde <sa...@datatorrent.com>.
Hello Austin,
Throwing ShutdownException from the operators will shutdown the Apex
Application.
On Wed, Aug 10, 2016 at 8:56 AM Doyle, Austin O. <
Austin.O.Doyle-2@leidos.com> wrote:
> When would an Apex Application return a FINISHED status from YARN as
> opposed to continually being in the RUNNING state?
>
> Is there a way to force the YARN status to FINISHED from within the
> application when processing is complete?
>