You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Satyam Singh <sa...@ericsson.com> on 2014/07/28 17:56:59 UTC
One datanode is down then write/read starts failing
Hello,
I have hadoop cluster setup of one namenode and two datanodes.
And i continuously write/read/delete through hdfs on namenode through
hadoop client.
Then i kill one of the datanode, still one is working but writing on
datanode is getting failed for all write requests.
I want to overcome this scenario because at live traffic scenario any of
datanode might get down then how do we handle those cases.
Can anybody face this issue or i am doing something wrong in my setup.
Thanx in advance.
Warm Regards,
Satyam
Re: One datanode is down then write/read starts failing
Posted by Satyam Singh <sa...@ericsson.com>.
Yes, there is lot of space available at that instant.
I am not sure but i have read somewhere that we must have datanodes live
>= replication factor given at namenode at any point of time. If live
datanodes get less than replication factor then this write/read failure
occurs.
In my case i have given replication factor 2 and initially i have 2 live
datanodes.
Then i killed one of the datanode and number of live DN becomes 1 and
still replication is 2.
Please comment.
On 07/28/2014 09:31 PM, Wellington Chevreuil wrote:
> Can you make sure you still have enough HDFS space once you kill this DN? If not, HDFS will automatically enter safemode if it detects there's no hdfs space available. The error message on the logs should have some hints on this.
>
> Cheers.
>
> On 28 Jul 2014, at 16:56, Satyam Singh <sa...@ericsson.com> wrote:
>
>> Hello,
>>
>>
>> I have hadoop cluster setup of one namenode and two datanodes.
>> And i continuously write/read/delete through hdfs on namenode through hadoop client.
>>
>> Then i kill one of the datanode, still one is working but writing on datanode is getting failed for all write requests.
>>
>> I want to overcome this scenario because at live traffic scenario any of datanode might get down then how do we handle those cases.
>>
>> Can anybody face this issue or i am doing something wrong in my setup.
>>
>> Thanx in advance.
>>
>>
>> Warm Regards,
>> Satyam
Re: One datanode is down then write/read starts failing
Posted by Satyam Singh <sa...@ericsson.com>.
Yes, there is lot of space available at that instant.
I am not sure but i have read somewhere that we must have datanodes live
>= replication factor given at namenode at any point of time. If live
datanodes get less than replication factor then this write/read failure
occurs.
In my case i have given replication factor 2 and initially i have 2 live
datanodes.
Then i killed one of the datanode and number of live DN becomes 1 and
still replication is 2.
Please comment.
On 07/28/2014 09:31 PM, Wellington Chevreuil wrote:
> Can you make sure you still have enough HDFS space once you kill this DN? If not, HDFS will automatically enter safemode if it detects there's no hdfs space available. The error message on the logs should have some hints on this.
>
> Cheers.
>
> On 28 Jul 2014, at 16:56, Satyam Singh <sa...@ericsson.com> wrote:
>
>> Hello,
>>
>>
>> I have hadoop cluster setup of one namenode and two datanodes.
>> And i continuously write/read/delete through hdfs on namenode through hadoop client.
>>
>> Then i kill one of the datanode, still one is working but writing on datanode is getting failed for all write requests.
>>
>> I want to overcome this scenario because at live traffic scenario any of datanode might get down then how do we handle those cases.
>>
>> Can anybody face this issue or i am doing something wrong in my setup.
>>
>> Thanx in advance.
>>
>>
>> Warm Regards,
>> Satyam
Re: One datanode is down then write/read starts failing
Posted by Satyam Singh <sa...@ericsson.com>.
Yes, there is lot of space available at that instant.
I am not sure but i have read somewhere that we must have datanodes live
>= replication factor given at namenode at any point of time. If live
datanodes get less than replication factor then this write/read failure
occurs.
In my case i have given replication factor 2 and initially i have 2 live
datanodes.
Then i killed one of the datanode and number of live DN becomes 1 and
still replication is 2.
Please comment.
On 07/28/2014 09:31 PM, Wellington Chevreuil wrote:
> Can you make sure you still have enough HDFS space once you kill this DN? If not, HDFS will automatically enter safemode if it detects there's no hdfs space available. The error message on the logs should have some hints on this.
>
> Cheers.
>
> On 28 Jul 2014, at 16:56, Satyam Singh <sa...@ericsson.com> wrote:
>
>> Hello,
>>
>>
>> I have hadoop cluster setup of one namenode and two datanodes.
>> And i continuously write/read/delete through hdfs on namenode through hadoop client.
>>
>> Then i kill one of the datanode, still one is working but writing on datanode is getting failed for all write requests.
>>
>> I want to overcome this scenario because at live traffic scenario any of datanode might get down then how do we handle those cases.
>>
>> Can anybody face this issue or i am doing something wrong in my setup.
>>
>> Thanx in advance.
>>
>>
>> Warm Regards,
>> Satyam
Re: One datanode is down then write/read starts failing
Posted by Satyam Singh <sa...@ericsson.com>.
Yes, there is lot of space available at that instant.
I am not sure but i have read somewhere that we must have datanodes live
>= replication factor given at namenode at any point of time. If live
datanodes get less than replication factor then this write/read failure
occurs.
In my case i have given replication factor 2 and initially i have 2 live
datanodes.
Then i killed one of the datanode and number of live DN becomes 1 and
still replication is 2.
Please comment.
On 07/28/2014 09:31 PM, Wellington Chevreuil wrote:
> Can you make sure you still have enough HDFS space once you kill this DN? If not, HDFS will automatically enter safemode if it detects there's no hdfs space available. The error message on the logs should have some hints on this.
>
> Cheers.
>
> On 28 Jul 2014, at 16:56, Satyam Singh <sa...@ericsson.com> wrote:
>
>> Hello,
>>
>>
>> I have hadoop cluster setup of one namenode and two datanodes.
>> And i continuously write/read/delete through hdfs on namenode through hadoop client.
>>
>> Then i kill one of the datanode, still one is working but writing on datanode is getting failed for all write requests.
>>
>> I want to overcome this scenario because at live traffic scenario any of datanode might get down then how do we handle those cases.
>>
>> Can anybody face this issue or i am doing something wrong in my setup.
>>
>> Thanx in advance.
>>
>>
>> Warm Regards,
>> Satyam
Re: One datanode is down then write/read starts failing
Posted by Wellington Chevreuil <we...@gmail.com>.
Can you make sure you still have enough HDFS space once you kill this DN? If not, HDFS will automatically enter safemode if it detects there's no hdfs space available. The error message on the logs should have some hints on this.
Cheers.
On 28 Jul 2014, at 16:56, Satyam Singh <sa...@ericsson.com> wrote:
> Hello,
>
>
> I have hadoop cluster setup of one namenode and two datanodes.
> And i continuously write/read/delete through hdfs on namenode through hadoop client.
>
> Then i kill one of the datanode, still one is working but writing on datanode is getting failed for all write requests.
>
> I want to overcome this scenario because at live traffic scenario any of datanode might get down then how do we handle those cases.
>
> Can anybody face this issue or i am doing something wrong in my setup.
>
> Thanx in advance.
>
>
> Warm Regards,
> Satyam
Re: One datanode is down then write/read starts failing
Posted by Wellington Chevreuil <we...@gmail.com>.
Can you make sure you still have enough HDFS space once you kill this DN? If not, HDFS will automatically enter safemode if it detects there's no hdfs space available. The error message on the logs should have some hints on this.
Cheers.
On 28 Jul 2014, at 16:56, Satyam Singh <sa...@ericsson.com> wrote:
> Hello,
>
>
> I have hadoop cluster setup of one namenode and two datanodes.
> And i continuously write/read/delete through hdfs on namenode through hadoop client.
>
> Then i kill one of the datanode, still one is working but writing on datanode is getting failed for all write requests.
>
> I want to overcome this scenario because at live traffic scenario any of datanode might get down then how do we handle those cases.
>
> Can anybody face this issue or i am doing something wrong in my setup.
>
> Thanx in advance.
>
>
> Warm Regards,
> Satyam
Re: One datanode is down then write/read starts failing
Posted by Wellington Chevreuil <we...@gmail.com>.
Can you make sure you still have enough HDFS space once you kill this DN? If not, HDFS will automatically enter safemode if it detects there's no hdfs space available. The error message on the logs should have some hints on this.
Cheers.
On 28 Jul 2014, at 16:56, Satyam Singh <sa...@ericsson.com> wrote:
> Hello,
>
>
> I have hadoop cluster setup of one namenode and two datanodes.
> And i continuously write/read/delete through hdfs on namenode through hadoop client.
>
> Then i kill one of the datanode, still one is working but writing on datanode is getting failed for all write requests.
>
> I want to overcome this scenario because at live traffic scenario any of datanode might get down then how do we handle those cases.
>
> Can anybody face this issue or i am doing something wrong in my setup.
>
> Thanx in advance.
>
>
> Warm Regards,
> Satyam
Re: One datanode is down then write/read starts failing
Posted by Wellington Chevreuil <we...@gmail.com>.
Can you make sure you still have enough HDFS space once you kill this DN? If not, HDFS will automatically enter safemode if it detects there's no hdfs space available. The error message on the logs should have some hints on this.
Cheers.
On 28 Jul 2014, at 16:56, Satyam Singh <sa...@ericsson.com> wrote:
> Hello,
>
>
> I have hadoop cluster setup of one namenode and two datanodes.
> And i continuously write/read/delete through hdfs on namenode through hadoop client.
>
> Then i kill one of the datanode, still one is working but writing on datanode is getting failed for all write requests.
>
> I want to overcome this scenario because at live traffic scenario any of datanode might get down then how do we handle those cases.
>
> Can anybody face this issue or i am doing something wrong in my setup.
>
> Thanx in advance.
>
>
> Warm Regards,
> Satyam