DiskUsage ('du -sk') probably hangs Datanode

[Sagar Naik <sn...@attributor.com>]

I see createBlockException and Abandoning block quite often
When I check the datanode, they are running. I can browse file system 
from that datanode:50075
However, I also notice tht a  du forked off from the DN. This 'du' run 
anywhere from 6mins to 30 mins.

During this time no logs are generated . DN appears in S1 state and the 
'du' in D state.

Is it possible tht jvm has bug or hdd is bad.
I m using "/usr/java/jdk1.6.0_07/bin/java" and planing to move onto 11

However, I start noticing this after DFS is 50% (on avg) full

Pl help me with some pointers

Hadoop version : .18.1

-Sagar

Re: DiskUsage ('du -sk') probably hangs Datanode

[Sagar Naik <sn...@attributor.com>]

Brian Bockelman wrote:
> Hey Sagar,
>
> If the 'du' is in the "D" state, then that probably means bad things 
> for your hardware.
>
> I recommend looking in dmesg and /var/log/messages for anything 
> interesting, as well as perform a hard-drive diagnostic test (may be 
> as simple as a SMART tests) to see if there's an issue.
>
> I can't say for sure, but the 'du' is probably not hanging the 
> Datanode; it's probably a symptom of larger problems.
>
Thanks Brian
I will start SMART tests
Pl tell me what direction I should look in case of "larger problems".


> Brian
>
> On Dec 17, 2008, at 8:29 PM, Sagar Naik wrote:
>
>>
>> I see createBlockException and Abandoning block quite often
>> When I check the datanode, they are running. I can browse file system 
>> from that datanode:50075
>> However, I also notice tht a  du forked off from the DN. This 'du' 
>> run anywhere from 6mins to 30 mins.
>>
>> During this time no logs are generated . DN appears in S1 state and 
>> the 'du' in D state.
>>
>> Is it possible tht jvm has bug or hdd is bad.
>> I m using "/usr/java/jdk1.6.0_07/bin/java" and planing to move onto 11
>>
>> However, I start noticing this after DFS is 50% (on avg) full
>>
>> Pl help me with some pointers
>>
>> Hadoop version : .18.1
>>
>> -Sagar
>>
>>
>>
>>
>>
>

Re: DiskUsage ('du -sk') probably hangs Datanode

[Brian Bockelman <bb...@cse.unl.edu>]

Hey Sagar,

If the 'du' is in the "D" state, then that probably means bad things  
for your hardware.

I recommend looking in dmesg and /var/log/messages for anything  
interesting, as well as perform a hard-drive diagnostic test (may be  
as simple as a SMART tests) to see if there's an issue.

I can't say for sure, but the 'du' is probably not hanging the  
Datanode; it's probably a symptom of larger problems.

Brian

On Dec 17, 2008, at 8:29 PM, Sagar Naik wrote:

>
> I see createBlockException and Abandoning block quite often
> When I check the datanode, they are running. I can browse file  
> system from that datanode:50075
> However, I also notice tht a  du forked off from the DN. This 'du'  
> run anywhere from 6mins to 30 mins.
>
> During this time no logs are generated . DN appears in S1 state and  
> the 'du' in D state.
>
> Is it possible tht jvm has bug or hdd is bad.
> I m using "/usr/java/jdk1.6.0_07/bin/java" and planing to move onto 11
>
> However, I start noticing this after DFS is 50% (on avg) full
>
> Pl help me with some pointers
>
> Hadoop version : .18.1
>
> -Sagar
>
>
>
>
>