You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by chiranjeevi vasupilli <ch...@gmail.com> on 2017/01/03 14:10:56 UTC
DT job not getting finished
Hi Team,
In my use case , i have two dags to completed the processing. And iam using
curl command to shutdown the application.
But some times we are facing an issue to shutdown the first dag . From DT
console we are able to see that the application has released all the
resources but it remains in RUNNING state for infinite time.
Please suggest .
[image: Inline image 1]
--
ur's
chiru
Re: DT job not getting finished
Posted by Sanjay Pujare <sa...@datatorrent.com>.
You can also check out this blog https://www.datatorrent.com/blog/getting-stack-traces-apache-apex-applications/ for help with getting stack traces (provided you have the correct versions).
From: Tushar Gosavi <tu...@datatorrent.com>
Reply-To: <us...@apex.apache.org>
Date: Wednesday, January 4, 2017 at 3:05 AM
To: <us...@apex.apache.org>
Subject: Re: DT job not getting finished
Hi Chiranjeevi,
When shutdown is not working can you check stack trace of all containers which are active using jstack. If your operator has created threads then needs to be cleanly terminated for JVM to terminate. Also try shutting down the application again using shutdown-app command.
- Tushar.
On Wed, Jan 4, 2017 at 3:48 PM, chiranjeevi vasupilli <ch...@gmail.com> wrote:
Hi Sanjay,
thanks for your response.
These days im getting the same issue more frequently. And the suggested solution to kill the job is not a good idea.
As in our flow Once the first App shutdown successfully,next app will launch and proceed further.
Please suggest how we can dealt If an operator is stuck in “teardown()” it can prevent successful shutdown of the application and it could show up as RUNNING indefinitely.
Thanks
Chiranjeevi V
On Wed, Jan 4, 2017 at 3:27 AM, Sanjay Pujare <sa...@datatorrent.com> wrote:
Chiranjeevi
Shutdown requires that an application’s all containers/operators are “torn down” gracefully as part of processing the latest window. If an operator is stuck in “teardown()” it can prevent successful shutdown of the application and it could show up as RUNNING indefinitely.
If you don’t want to debug these issues in teardown() you can just kill the application thru the Yarn REST API
Sanjay
From: chiranjeevi vasupilli <ch...@gmail.com>
Reply-To: <us...@apex.apache.org>
Date: Tuesday, January 3, 2017 at 6:10 AM
To: <us...@apex.apache.org>
Subject: DT job not getting finished
Hi Team,
In my use case , i have two dags to completed the processing. And iam using curl command to shutdown the application.
But some times we are facing an issue to shutdown the first dag . From DT console we are able to see that the application has released all the resources but it remains in RUNNING state for infinite time.
Please suggest .
--
ur's
chiru
--
ur's
chiru
Re: DT job not getting finished
Posted by Tushar Gosavi <tu...@datatorrent.com>.
Hi Chiranjeevi,
When shutdown is not working can you check stack trace of all containers
which are active using jstack. If your operator has created threads then
needs to be cleanly terminated for JVM to terminate. Also try shutting down
the application again using shutdown-app command.
- Tushar.
On Wed, Jan 4, 2017 at 3:48 PM, chiranjeevi vasupilli <ch...@gmail.com>
wrote:
> Hi Sanjay,
>
> thanks for your response.
>
> These days im getting the same issue more frequently. And the suggested
> solution to kill the job is not a good idea.
> As in our flow Once the first App shutdown successfully,next app will
> launch and proceed further.
>
> Please suggest how we can dealt *If an operator is stuck in “teardown()”
> it can prevent successful shutdown of the application and it could show up
> as RUNNING indefinitely*.
>
> Thanks
> Chiranjeevi V
>
> On Wed, Jan 4, 2017 at 3:27 AM, Sanjay Pujare <sa...@datatorrent.com>
> wrote:
>
>> Chiranjeevi
>>
>>
>>
>> Shutdown requires that an application’s all containers/operators are
>> “torn down” gracefully as part of processing the latest window. If an
>> operator is stuck in “teardown()” it can prevent successful shutdown of the
>> application and it could show up as RUNNING indefinitely.
>>
>>
>>
>> If you don’t want to debug these issues in teardown() you can just kill
>> the application thru the Yarn REST API
>>
>>
>>
>> Sanjay
>>
>>
>>
>> *From: *chiranjeevi vasupilli <ch...@gmail.com>
>> *Reply-To: *<us...@apex.apache.org>
>> *Date: *Tuesday, January 3, 2017 at 6:10 AM
>> *To: *<us...@apex.apache.org>
>> *Subject: *DT job not getting finished
>>
>>
>>
>> Hi Team,
>>
>>
>>
>> In my use case , i have two dags to completed the processing. And iam
>> using curl command to shutdown the application.
>>
>>
>>
>> But some times we are facing an issue to shutdown the first dag . From DT
>> console we are able to see that the application has released all the
>> resources but it remains in RUNNING state for infinite time.
>>
>>
>>
>> Please suggest .
>>
>>
>> [image: nline image 1]
>>
>>
>>
>> --
>>
>> ur's
>> chiru
>>
>
>
>
> --
> ur's
> chiru
>
Re: DT job not getting finished
Posted by chiranjeevi vasupilli <ch...@gmail.com>.
Hi Sanjay,
thanks for your response.
These days im getting the same issue more frequently. And the suggested
solution to kill the job is not a good idea.
As in our flow Once the first App shutdown successfully,next app will
launch and proceed further.
Please suggest how we can dealt *If an operator is stuck in “teardown()” it
can prevent successful shutdown of the application and it could show up as
RUNNING indefinitely*.
Thanks
Chiranjeevi V
On Wed, Jan 4, 2017 at 3:27 AM, Sanjay Pujare <sa...@datatorrent.com>
wrote:
> Chiranjeevi
>
>
>
> Shutdown requires that an application’s all containers/operators are “torn
> down” gracefully as part of processing the latest window. If an operator is
> stuck in “teardown()” it can prevent successful shutdown of the application
> and it could show up as RUNNING indefinitely.
>
>
>
> If you don’t want to debug these issues in teardown() you can just kill
> the application thru the Yarn REST API
>
>
>
> Sanjay
>
>
>
> *From: *chiranjeevi vasupilli <ch...@gmail.com>
> *Reply-To: *<us...@apex.apache.org>
> *Date: *Tuesday, January 3, 2017 at 6:10 AM
> *To: *<us...@apex.apache.org>
> *Subject: *DT job not getting finished
>
>
>
> Hi Team,
>
>
>
> In my use case , i have two dags to completed the processing. And iam
> using curl command to shutdown the application.
>
>
>
> But some times we are facing an issue to shutdown the first dag . From DT
> console we are able to see that the application has released all the
> resources but it remains in RUNNING state for infinite time.
>
>
>
> Please suggest .
>
>
> [image: nline image 1]
>
>
>
> --
>
> ur's
> chiru
>
--
ur's
chiru
Re: DT job not getting finished
Posted by Sanjay Pujare <sa...@datatorrent.com>.
Chiranjeevi
Shutdown requires that an application’s all containers/operators are “torn down” gracefully as part of processing the latest window. If an operator is stuck in “teardown()” it can prevent successful shutdown of the application and it could show up as RUNNING indefinitely.
If you don’t want to debug these issues in teardown() you can just kill the application thru the Yarn REST API
Sanjay
From: chiranjeevi vasupilli <ch...@gmail.com>
Reply-To: <us...@apex.apache.org>
Date: Tuesday, January 3, 2017 at 6:10 AM
To: <us...@apex.apache.org>
Subject: DT job not getting finished
Hi Team,
In my use case , i have two dags to completed the processing. And iam using curl command to shutdown the application.
But some times we are facing an issue to shutdown the first dag . From DT console we are able to see that the application has released all the resources but it remains in RUNNING state for infinite time.
Please suggest .
--
ur's
chiru