You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by "Raja.Aravapalli" <Ra...@target.com> on 2016/07/22 20:14:04 UTC
hdfs output file operator
Hi,
I have File output operator which writes to hdfs files!!
Application is trying to deploy the operator which writes to hdfs files in many different containers for a long time… but is not succeeding!!! Status is showing as PENDING_DEPLOY
In the logs of the container which the Application is trying to deploy hdfs write operator, I can only see, path corrupted!!
Can someone please guide or suggest me on this ?
Regards,
Raja.
Re: hdfs output file operator
Posted by "Raja.Aravapalli" <Ra...@target.com>.
Thanks for the response Chinmay.
Yes, this issue is during restart of a failed application.
As per my observation, my dag/application failed during Resource Manager failover!! And during failover something went wrong to one of the files writing to hdfs!!! Application tried many a times to restore the file by launching the operator which is writing to hdfs on many containers and failed!!
When I restarted the application, the application again tried many times to restore the hdfs file… and still launched many containers to recover….. App took really a very long time say 4 – 5hrs to successfully launch those hdfs operators and resume!!!
Regards,
Raja.
From: Chinmay Kolhatkar <ch...@datatorrent.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Saturday, July 23, 2016 at 12:11 AM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: hdfs output file operator
Hi Raja,
I can see such a log message in AbstractFileOutputOperator at line 455.
As this code is called from setup of the operator, the operator is getting deployed and then failing while restoring existing file because of mismatch in length of the file and the offset the operator has stored previously.
From the code it looks like it takes care of such cases and restores the file.
From what I understand either the file got changes by some other way or the offset management has a problem.
Are you restarting the application from previous application Id?
To narrow down the problem, can you please try to change the destination path and see if that works?
Thanks,
Chinmay.
On Sat, Jul 23, 2016 at 5:00 AM, Sandesh Hegde <sa...@datatorrent.com>> wrote:
Please check,
1. AppMaster logs
2. Cluster resources
On Fri, Jul 22, 2016 at 1:14 PM Raja.Aravapalli <Ra...@target.com>> wrote:
Hi,
I have File output operator which writes to hdfs files!!
Application is trying to deploy the operator which writes to hdfs files in many different containers for a long time… but is not succeeding!!! Status is showing as PENDING_DEPLOY
In the logs of the container which the Application is trying to deploy hdfs write operator, I can only see, path corrupted!!
Can someone please guide or suggest me on this ?
Regards,
Raja.
Re: hdfs output file operator
Posted by Teddy Rusli <te...@datatorrent.com>.
@Rahul
You can unsubscribe to users-unsubscribe@apex.apache.org for users@apex.
Nobody here is an admin that can do it for you.
On Sat, Jul 23, 2016 at 8:45 AM, Yogi Devendra <devendra.vyavahare@gmail.com
> wrote:
> @Rahul
> Send an email to dev-unsubscribe@apex.apache.org
> from your registered email.
>
> ~ Yogi
>
> On 23 July 2016 at 21:03, Rahul More <ra...@gmail.com> wrote:
>
>> Please stop the mails
>> On Jul 23, 2016 10:41 AM, "Chinmay Kolhatkar" <ch...@datatorrent.com>
>> wrote:
>>
>>> Hi Raja,
>>>
>>> I can see such a log message in AbstractFileOutputOperator at line 455.
>>>
>>> As this code is called from setup of the operator, the operator is
>>> getting deployed and then failing while restoring existing file because of
>>> mismatch in length of the file and the offset the operator has stored
>>> previously.
>>>
>>> From the code it looks like it takes care of such cases and restores the
>>> file.
>>>
>>> From what I understand either the file got changes by some other way or
>>> the offset management has a problem.
>>>
>>> Are you restarting the application from previous application Id?
>>>
>>> To narrow down the problem, can you please try to change the destination
>>> path and see if that works?
>>>
>>> Thanks,
>>> Chinmay.
>>>
>>>
>>>
>>> On Sat, Jul 23, 2016 at 5:00 AM, Sandesh Hegde <sa...@datatorrent.com>
>>> wrote:
>>>
>>>> Please check,
>>>> 1. AppMaster logs
>>>> 2. Cluster resources
>>>>
>>>> On Fri, Jul 22, 2016 at 1:14 PM Raja.Aravapalli <
>>>> Raja.Aravapalli@target.com> wrote:
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> I have File output operator which writes to hdfs files!!
>>>>>
>>>>> Application is trying to deploy the operator which writes to hdfs
>>>>> files in many different containers for a long time… but is not
>>>>> succeeding!!! Status is showing as PENDING_DEPLOY
>>>>>
>>>>> In the logs of the container which the Application is trying to deploy
>>>>> hdfs write operator, I can only see, path corrupted!!
>>>>>
>>>>>
>>>>> Can someone please guide or suggest me on this ?
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>> Raja.
>>>>>
>>>>
>>>
>
--
Regards,
Teddy Rusli
Re: hdfs output file operator
Posted by Yogi Devendra <de...@gmail.com>.
@Rahul
Send an email to dev-unsubscribe@apex.apache.org
from your registered email.
~ Yogi
On 23 July 2016 at 21:03, Rahul More <ra...@gmail.com> wrote:
> Please stop the mails
> On Jul 23, 2016 10:41 AM, "Chinmay Kolhatkar" <ch...@datatorrent.com>
> wrote:
>
>> Hi Raja,
>>
>> I can see such a log message in AbstractFileOutputOperator at line 455.
>>
>> As this code is called from setup of the operator, the operator is
>> getting deployed and then failing while restoring existing file because of
>> mismatch in length of the file and the offset the operator has stored
>> previously.
>>
>> From the code it looks like it takes care of such cases and restores the
>> file.
>>
>> From what I understand either the file got changes by some other way or
>> the offset management has a problem.
>>
>> Are you restarting the application from previous application Id?
>>
>> To narrow down the problem, can you please try to change the destination
>> path and see if that works?
>>
>> Thanks,
>> Chinmay.
>>
>>
>>
>> On Sat, Jul 23, 2016 at 5:00 AM, Sandesh Hegde <sa...@datatorrent.com>
>> wrote:
>>
>>> Please check,
>>> 1. AppMaster logs
>>> 2. Cluster resources
>>>
>>> On Fri, Jul 22, 2016 at 1:14 PM Raja.Aravapalli <
>>> Raja.Aravapalli@target.com> wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> I have File output operator which writes to hdfs files!!
>>>>
>>>> Application is trying to deploy the operator which writes to hdfs files
>>>> in many different containers for a long time… but is not succeeding!!!
>>>> Status is showing as PENDING_DEPLOY
>>>>
>>>> In the logs of the container which the Application is trying to deploy
>>>> hdfs write operator, I can only see, path corrupted!!
>>>>
>>>>
>>>> Can someone please guide or suggest me on this ?
>>>>
>>>>
>>>>
>>>> Regards,
>>>> Raja.
>>>>
>>>
>>
Re: hdfs output file operator
Posted by Rahul More <ra...@gmail.com>.
Please stop the mails
On Jul 23, 2016 10:41 AM, "Chinmay Kolhatkar" <ch...@datatorrent.com>
wrote:
> Hi Raja,
>
> I can see such a log message in AbstractFileOutputOperator at line 455.
>
> As this code is called from setup of the operator, the operator is getting
> deployed and then failing while restoring existing file because of mismatch
> in length of the file and the offset the operator has stored previously.
>
> From the code it looks like it takes care of such cases and restores the
> file.
>
> From what I understand either the file got changes by some other way or
> the offset management has a problem.
>
> Are you restarting the application from previous application Id?
>
> To narrow down the problem, can you please try to change the destination
> path and see if that works?
>
> Thanks,
> Chinmay.
>
>
>
> On Sat, Jul 23, 2016 at 5:00 AM, Sandesh Hegde <sa...@datatorrent.com>
> wrote:
>
>> Please check,
>> 1. AppMaster logs
>> 2. Cluster resources
>>
>> On Fri, Jul 22, 2016 at 1:14 PM Raja.Aravapalli <
>> Raja.Aravapalli@target.com> wrote:
>>
>>>
>>> Hi,
>>>
>>> I have File output operator which writes to hdfs files!!
>>>
>>> Application is trying to deploy the operator which writes to hdfs files
>>> in many different containers for a long time… but is not succeeding!!!
>>> Status is showing as PENDING_DEPLOY
>>>
>>> In the logs of the container which the Application is trying to deploy
>>> hdfs write operator, I can only see, path corrupted!!
>>>
>>>
>>> Can someone please guide or suggest me on this ?
>>>
>>>
>>>
>>> Regards,
>>> Raja.
>>>
>>
>
Re: hdfs output file operator
Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
Hi Raja,
I can see such a log message in AbstractFileOutputOperator at line 455.
As this code is called from setup of the operator, the operator is getting
deployed and then failing while restoring existing file because of mismatch
in length of the file and the offset the operator has stored previously.
From the code it looks like it takes care of such cases and restores the
file.
From what I understand either the file got changes by some other way or the
offset management has a problem.
Are you restarting the application from previous application Id?
To narrow down the problem, can you please try to change the destination
path and see if that works?
Thanks,
Chinmay.
On Sat, Jul 23, 2016 at 5:00 AM, Sandesh Hegde <sa...@datatorrent.com>
wrote:
> Please check,
> 1. AppMaster logs
> 2. Cluster resources
>
> On Fri, Jul 22, 2016 at 1:14 PM Raja.Aravapalli <
> Raja.Aravapalli@target.com> wrote:
>
>>
>> Hi,
>>
>> I have File output operator which writes to hdfs files!!
>>
>> Application is trying to deploy the operator which writes to hdfs files
>> in many different containers for a long time… but is not succeeding!!!
>> Status is showing as PENDING_DEPLOY
>>
>> In the logs of the container which the Application is trying to deploy
>> hdfs write operator, I can only see, path corrupted!!
>>
>>
>> Can someone please guide or suggest me on this ?
>>
>>
>>
>> Regards,
>> Raja.
>>
>
Re: hdfs output file operator
Posted by Sandesh Hegde <sa...@datatorrent.com>.
Please check,
1. AppMaster logs
2. Cluster resources
On Fri, Jul 22, 2016 at 1:14 PM Raja.Aravapalli <Ra...@target.com>
wrote:
>
> Hi,
>
> I have File output operator which writes to hdfs files!!
>
> Application is trying to deploy the operator which writes to hdfs files in
> many different containers for a long time… but is not succeeding!!! Status
> is showing as PENDING_DEPLOY
>
> In the logs of the container which the Application is trying to deploy
> hdfs write operator, I can only see, path corrupted!!
>
>
> Can someone please guide or suggest me on this ?
>
>
>
> Regards,
> Raja.
>