You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Sreenath <sr...@gmail.com> on 2014/05/22 08:23:12 UTC
Reducers Failing
Hi All,
We are running a hadoop cluster and many of our hive queries are failing in
the reduce phase with the following error
java.io.IOException: All datanodes *.*.*.*:50230 are bad. Aborting...
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3290)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2200(DFSClient.java:2783)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2987)
Any IDead why this might be happening ?
--
Sreenath S Kamath
Bangalore
Ph No:+91-9590989106
Re: Reducers Failing
Posted by Sreenath <sr...@gmail.com>.
Hi,
No the data nodes are not dead and the HDFS is almost 70% free.
Is it related to some network Issues?
On Thu, May 22, 2014 at 11:58 AM, Shengjun Xin <sx...@gopivotal.com> wrote:
> Are datanodes dead?
>
>
> On Thu, May 22, 2014 at 2:23 PM, Sreenath <sr...@gmail.com> wrote:
>
>> Hi All,
>>
>> We are running a hadoop cluster and many of our hive queries are failing
>> in the reduce phase with the following error
>>
>> java.io.IOException: All datanodes *.*.*.*:50230 are bad. Aborting...
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3290)
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2200(DFSClient.java:2783)
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2987)
>>
>> Any IDead why this might be happening ?
>> --
>> Sreenath S Kamath
>> Bangalore
>> Ph No:+91-9590989106
>>
>
>
>
> --
> Regards
> Shengjun
>
--
Sreenath S Kamath
Bangalore
Ph No:+91-9590989106
Re: Reducers Failing
Posted by Sreenath <sr...@gmail.com>.
Thats the summary of i'm getting when i run the command
DFS Used%: 27%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
On Thu, May 22, 2014 at 12:22 PM, Nagarjuna Vissarapu <
nagarjuna.viss@gmail.com> wrote:
> It displays the usage details
>
>
> On Thu, May 22, 2014 at 12:20 PM, Sreenath <sr...@gmail.com>wrote:
>
>> Ok what is the result you are expecting once i run this command ?
>>
>>
>> On Thu, May 22, 2014 at 12:17 PM, Nagarjuna Vissarapu <
>> nagarjuna.viss@gmail.com> wrote:
>>
>>> I think your hdfs memory is filled. Type the following command and check
>>> it once hadoop dfsadmin -report.
>>>
>>>
>>> On Thu, May 22, 2014 at 11:58 AM, Shengjun Xin <sx...@gopivotal.com>wrote:
>>>
>>>> Are datanodes dead?
>>>>
>>>>
>>>> On Thu, May 22, 2014 at 2:23 PM, Sreenath <sr...@gmail.com>wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> We are running a hadoop cluster and many of our hive queries are
>>>>> failing in the reduce phase with the following error
>>>>>
>>>>> java.io.IOException: All datanodes *.*.*.*:50230 are bad. Aborting...
>>>>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3290)
>>>>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2200(DFSClient.java:2783)
>>>>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2987)
>>>>>
>>>>> Any IDead why this might be happening ?
>>>>> --
>>>>> Sreenath S Kamath
>>>>> Bangalore
>>>>> Ph No:+91-9590989106
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards
>>>> Shengjun
>>>>
>>>
>>>
>>>
>>> --
>>> With Thanks & Regards
>>> Nagarjuna Vissarapu
>>> 9052179339
>>>
>>>
>>
>>
>> --
>> Sreenath S Kamath
>> Bangalore
>> Ph No:+91-9590989106
>>
>
>
>
> --
> With Thanks & Regards
> Nagarjuna Vissarapu
> 9052179339
>
>
--
Sreenath S Kamath
Bangalore
Ph No:+91-9590989106
Re: Reducers Failing
Posted by Nagarjuna Vissarapu <na...@gmail.com>.
It displays the usage details
On Thu, May 22, 2014 at 12:20 PM, Sreenath <sr...@gmail.com> wrote:
> Ok what is the result you are expecting once i run this command ?
>
>
> On Thu, May 22, 2014 at 12:17 PM, Nagarjuna Vissarapu <
> nagarjuna.viss@gmail.com> wrote:
>
>> I think your hdfs memory is filled. Type the following command and check
>> it once hadoop dfsadmin -report.
>>
>>
>> On Thu, May 22, 2014 at 11:58 AM, Shengjun Xin <sx...@gopivotal.com>wrote:
>>
>>> Are datanodes dead?
>>>
>>>
>>> On Thu, May 22, 2014 at 2:23 PM, Sreenath <sr...@gmail.com>wrote:
>>>
>>>> Hi All,
>>>>
>>>> We are running a hadoop cluster and many of our hive queries are
>>>> failing in the reduce phase with the following error
>>>>
>>>> java.io.IOException: All datanodes *.*.*.*:50230 are bad. Aborting...
>>>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3290)
>>>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2200(DFSClient.java:2783)
>>>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2987)
>>>>
>>>> Any IDead why this might be happening ?
>>>> --
>>>> Sreenath S Kamath
>>>> Bangalore
>>>> Ph No:+91-9590989106
>>>>
>>>
>>>
>>>
>>> --
>>> Regards
>>> Shengjun
>>>
>>
>>
>>
>> --
>> With Thanks & Regards
>> Nagarjuna Vissarapu
>> 9052179339
>>
>>
>
>
> --
> Sreenath S Kamath
> Bangalore
> Ph No:+91-9590989106
>
--
With Thanks & Regards
Nagarjuna Vissarapu
9052179339
Re: Reducers Failing
Posted by Sreenath <sr...@gmail.com>.
Ok what is the result you are expecting once i run this command ?
On Thu, May 22, 2014 at 12:17 PM, Nagarjuna Vissarapu <
nagarjuna.viss@gmail.com> wrote:
> I think your hdfs memory is filled. Type the following command and check
> it once hadoop dfsadmin -report.
>
>
> On Thu, May 22, 2014 at 11:58 AM, Shengjun Xin <sx...@gopivotal.com> wrote:
>
>> Are datanodes dead?
>>
>>
>> On Thu, May 22, 2014 at 2:23 PM, Sreenath <sr...@gmail.com>wrote:
>>
>>> Hi All,
>>>
>>> We are running a hadoop cluster and many of our hive queries are failing
>>> in the reduce phase with the following error
>>>
>>> java.io.IOException: All datanodes *.*.*.*:50230 are bad. Aborting...
>>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3290)
>>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2200(DFSClient.java:2783)
>>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2987)
>>>
>>> Any IDead why this might be happening ?
>>> --
>>> Sreenath S Kamath
>>> Bangalore
>>> Ph No:+91-9590989106
>>>
>>
>>
>>
>> --
>> Regards
>> Shengjun
>>
>
>
>
> --
> With Thanks & Regards
> Nagarjuna Vissarapu
> 9052179339
>
>
--
Sreenath S Kamath
Bangalore
Ph No:+91-9590989106
Re: Reducers Failing
Posted by Nagarjuna Vissarapu <na...@gmail.com>.
I think your hdfs memory is filled. Type the following command and check it
once hadoop dfsadmin -report.
On Thu, May 22, 2014 at 11:58 AM, Shengjun Xin <sx...@gopivotal.com> wrote:
> Are datanodes dead?
>
>
> On Thu, May 22, 2014 at 2:23 PM, Sreenath <sr...@gmail.com> wrote:
>
>> Hi All,
>>
>> We are running a hadoop cluster and many of our hive queries are failing
>> in the reduce phase with the following error
>>
>> java.io.IOException: All datanodes *.*.*.*:50230 are bad. Aborting...
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3290)
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2200(DFSClient.java:2783)
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2987)
>>
>> Any IDead why this might be happening ?
>> --
>> Sreenath S Kamath
>> Bangalore
>> Ph No:+91-9590989106
>>
>
>
>
> --
> Regards
> Shengjun
>
--
With Thanks & Regards
Nagarjuna Vissarapu
9052179339
Re: Reducers Failing
Posted by Shengjun Xin <sx...@gopivotal.com>.
Are datanodes dead?
On Thu, May 22, 2014 at 2:23 PM, Sreenath <sr...@gmail.com> wrote:
> Hi All,
>
> We are running a hadoop cluster and many of our hive queries are failing
> in the reduce phase with the following error
>
> java.io.IOException: All datanodes *.*.*.*:50230 are bad. Aborting...
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3290)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2200(DFSClient.java:2783)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2987)
>
> Any IDead why this might be happening ?
> --
> Sreenath S Kamath
> Bangalore
> Ph No:+91-9590989106
>
--
Regards
Shengjun