You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Dennis Kubes <ku...@apache.org> on 2007/06/01 08:00:18 UTC
Upgrade of DFS - Urgent
All,
I upgraded to the most recent trunk of Hadoop and I started getting the
error below, where /d01/hadoop/dfs/name is our namenode directory:
org.apache.hadoop.dfs.InconsistentFSStateException: Directory
/d01/hadoop/dfs/name is in an inconsistent state:
/d01/hadoop/dfs/name/image does not exist.
The old configuration was under a directory structure like:
/d01/hadoop/dfs/name/current
After backup up the namenode and playing around a little I found that if
I reformatted the namenode and then copied over the old files that were
in the current directory back into the "new" current directory that the
namenode would start up.
We have quite a bit of data on this cluster (around 8T) and I am a
little nervous about starting up the entire cluster without a little
clarification. If I startup the cluster now, will any old data blocks
be deleted or will those data blocks remain because I copied over the
old configuration files into the new "current"?
Is there another way to upgrade this DFS cluster? Any help is appreciated.
Dennis Kubes
Re: Upgrade of DFS - Urgent
Posted by Doug Cutting <cu...@apache.org>.
Dhruba Borthakur wrote:
> You might have bumped into http://issues.apache.org/jira/browse/HADOOP-1443
>
> From the JIRA issue, there isn't an available patch for it yet:)
Do we need to fix this for 0.13? It sounds like we might...
Doug
Re: Upgrade of DFS - Urgent
Posted by Konstantin Shvachko <sh...@yahoo-inc.com>.
> Is there another way to upgrade this DFS cluster? Any help is
appreciated.
Denis, latest hadoop supports upgrade/rollback/finalize. You do not need
to manually backup data/image directories.
Do you run secondary name-node on your cluster? It is a good idea to
have it if you want an extra copy of your
namespace image.
--Konstantin
Dennis Kubes wrote:
> Yeah, that's it. It is caused by a zero length file (like index.done
> on the indexer in Nutch). When the zero length file is deleted then
> everything works fine. I don't think it can get block locations for a
> file that doesn't have any blocks and therefore throws the error. I
> will continue to look into it.
>
> Dennis Kubes
>
> Dhruba Borthakur wrote:
>
>> You might have bumped into
>> http://issues.apache.org/jira/browse/HADOOP-1443
>>
>> From the JIRA issue, there isn't an available patch for it yet:)
>>
>> Thanks,
>> dhruba
>>
>> -----Original Message-----
>> From: Dennis Kubes [mailto:kubes@apache.org] Sent: Thursday, May 31,
>> 2007 11:58 PM
>> To: hadoop-user@lucene.apache.org
>> Subject: Re: Upgrade of DFS - Urgent
>>
>> Okay. My procedure was: I backup old current directory, did a
>> namenode reformat, then copied old current files into the reformated
>> current directory. Then started up namenode (while praying very hard
>> and sweating profusely). Everything seems to have worked fine. I am
>> able to copy files to an from the dfs and all block reports look good
>> as does the fsck output.
>>
>> On another note I am noticing a bug in the getBlockLocations where a
>> zero length file will through an ArrayIndexOutOfBoundsException. I
>> am still tracking this one down in the code and will submit a patch
>> when I have found it.
>>
>> Dennis Kubes
>>
>> Raghu Angadi wrote:
>>
>>> This is the result of HADOOP-1242. I prefer if it did not require
>>> presence of this image directory.
>>>
>>> For now you could manually create image/fsimage file in name/
>>> directory. If you write random 4 bytes to fsimage, you have 50%
>>> chance of success. Basically readInt() from the file should be less
>>> than -3. Only first 4 bytes are important.
>>>
>>> Raghu.
>>>
>>> Dennis Kubes wrote:
>>>
>>>> All,
>>>>
>>>> I upgraded to the most recent trunk of Hadoop and I started getting
>>>> the error below, where /d01/hadoop/dfs/name is our namenode directory:
>>>>
>>>> org.apache.hadoop.dfs.InconsistentFSStateException: Directory
>>>> /d01/hadoop/dfs/name is in an inconsistent state:
>>>> /d01/hadoop/dfs/name/image does not exist.
>>>>
>>>> The old configuration was under a directory structure like:
>>>>
>>>> /d01/hadoop/dfs/name/current
>>>>
>>>> After backup up the namenode and playing around a little I found
>>>> that if I reformatted the namenode and then copied over the old
>>>> files that were in the current directory back into the "new"
>>>> current directory that the namenode would start up.
>>>>
>>>> We have quite a bit of data on this cluster (around 8T) and I am a
>>>> little nervous about starting up the entire cluster without a
>>>> little clarification. If I startup the cluster now, will any old
>>>> data blocks be deleted or will those data blocks remain because I
>>>> copied over the old configuration files into the new "current"?
>>>>
>>>> Is there another way to upgrade this DFS cluster? Any help is
>>>> appreciated.
>>>>
>>>> Dennis Kubes
>>>>
>>
>
Re: Upgrade of DFS - Urgent
Posted by Dennis Kubes <ku...@apache.org>.
Yeah, that's it. It is caused by a zero length file (like index.done on
the indexer in Nutch). When the zero length file is deleted then
everything works fine. I don't think it can get block locations for a
file that doesn't have any blocks and therefore throws the error. I
will continue to look into it.
Dennis Kubes
Dhruba Borthakur wrote:
> You might have bumped into http://issues.apache.org/jira/browse/HADOOP-1443
>
> From the JIRA issue, there isn't an available patch for it yet:)
>
> Thanks,
> dhruba
>
> -----Original Message-----
> From: Dennis Kubes [mailto:kubes@apache.org]
> Sent: Thursday, May 31, 2007 11:58 PM
> To: hadoop-user@lucene.apache.org
> Subject: Re: Upgrade of DFS - Urgent
>
> Okay. My procedure was: I backup old current directory, did a namenode
> reformat, then copied old current files into the reformated current
> directory. Then started up namenode (while praying very hard and
> sweating profusely). Everything seems to have worked fine. I am able
> to copy files to an from the dfs and all block reports look good as does
> the fsck output.
>
> On another note I am noticing a bug in the getBlockLocations where a
> zero length file will through an ArrayIndexOutOfBoundsException. I am
> still tracking this one down in the code and will submit a patch when I
> have found it.
>
> Dennis Kubes
>
> Raghu Angadi wrote:
>> This is the result of HADOOP-1242. I prefer if it did not require
>> presence of this image directory.
>>
>> For now you could manually create image/fsimage file in name/ directory.
>> If you write random 4 bytes to fsimage, you have 50% chance of success.
>> Basically readInt() from the file should be less than -3. Only first 4
>> bytes are important.
>>
>> Raghu.
>>
>> Dennis Kubes wrote:
>>> All,
>>>
>>> I upgraded to the most recent trunk of Hadoop and I started getting
>>> the error below, where /d01/hadoop/dfs/name is our namenode directory:
>>>
>>> org.apache.hadoop.dfs.InconsistentFSStateException: Directory
>>> /d01/hadoop/dfs/name is in an inconsistent state:
>>> /d01/hadoop/dfs/name/image does not exist.
>>>
>>> The old configuration was under a directory structure like:
>>>
>>> /d01/hadoop/dfs/name/current
>>>
>>> After backup up the namenode and playing around a little I found that
>>> if I reformatted the namenode and then copied over the old files that
>>> were in the current directory back into the "new" current directory
>>> that the namenode would start up.
>>>
>>> We have quite a bit of data on this cluster (around 8T) and I am a
>>> little nervous about starting up the entire cluster without a little
>>> clarification. If I startup the cluster now, will any old data blocks
>>> be deleted or will those data blocks remain because I copied over the
>>> old configuration files into the new "current"?
>>>
>>> Is there another way to upgrade this DFS cluster? Any help is
>>> appreciated.
>>>
>>> Dennis Kubes
>>>
>
RE: Upgrade of DFS - Urgent
Posted by Dhruba Borthakur <dh...@yahoo-inc.com>.
You might have bumped into http://issues.apache.org/jira/browse/HADOOP-1443
>From the JIRA issue, there isn't an available patch for it yet:)
Thanks,
dhruba
-----Original Message-----
From: Dennis Kubes [mailto:kubes@apache.org]
Sent: Thursday, May 31, 2007 11:58 PM
To: hadoop-user@lucene.apache.org
Subject: Re: Upgrade of DFS - Urgent
Okay. My procedure was: I backup old current directory, did a namenode
reformat, then copied old current files into the reformated current
directory. Then started up namenode (while praying very hard and
sweating profusely). Everything seems to have worked fine. I am able
to copy files to an from the dfs and all block reports look good as does
the fsck output.
On another note I am noticing a bug in the getBlockLocations where a
zero length file will through an ArrayIndexOutOfBoundsException. I am
still tracking this one down in the code and will submit a patch when I
have found it.
Dennis Kubes
Raghu Angadi wrote:
>
> This is the result of HADOOP-1242. I prefer if it did not require
> presence of this image directory.
>
> For now you could manually create image/fsimage file in name/ directory.
> If you write random 4 bytes to fsimage, you have 50% chance of success.
> Basically readInt() from the file should be less than -3. Only first 4
> bytes are important.
>
> Raghu.
>
> Dennis Kubes wrote:
>> All,
>>
>> I upgraded to the most recent trunk of Hadoop and I started getting
>> the error below, where /d01/hadoop/dfs/name is our namenode directory:
>>
>> org.apache.hadoop.dfs.InconsistentFSStateException: Directory
>> /d01/hadoop/dfs/name is in an inconsistent state:
>> /d01/hadoop/dfs/name/image does not exist.
>>
>> The old configuration was under a directory structure like:
>>
>> /d01/hadoop/dfs/name/current
>>
>> After backup up the namenode and playing around a little I found that
>> if I reformatted the namenode and then copied over the old files that
>> were in the current directory back into the "new" current directory
>> that the namenode would start up.
>>
>> We have quite a bit of data on this cluster (around 8T) and I am a
>> little nervous about starting up the entire cluster without a little
>> clarification. If I startup the cluster now, will any old data blocks
>> be deleted or will those data blocks remain because I copied over the
>> old configuration files into the new "current"?
>>
>> Is there another way to upgrade this DFS cluster? Any help is
>> appreciated.
>>
>> Dennis Kubes
>>
>
Re: Upgrade of DFS - Urgent
Posted by Dennis Kubes <ku...@apache.org>.
Okay. My procedure was: I backup old current directory, did a namenode
reformat, then copied old current files into the reformated current
directory. Then started up namenode (while praying very hard and
sweating profusely). Everything seems to have worked fine. I am able
to copy files to an from the dfs and all block reports look good as does
the fsck output.
On another note I am noticing a bug in the getBlockLocations where a
zero length file will through an ArrayIndexOutOfBoundsException. I am
still tracking this one down in the code and will submit a patch when I
have found it.
Dennis Kubes
Raghu Angadi wrote:
>
> This is the result of HADOOP-1242. I prefer if it did not require
> presence of this image directory.
>
> For now you could manually create image/fsimage file in name/ directory.
> If you write random 4 bytes to fsimage, you have 50% chance of success.
> Basically readInt() from the file should be less than -3. Only first 4
> bytes are important.
>
> Raghu.
>
> Dennis Kubes wrote:
>> All,
>>
>> I upgraded to the most recent trunk of Hadoop and I started getting
>> the error below, where /d01/hadoop/dfs/name is our namenode directory:
>>
>> org.apache.hadoop.dfs.InconsistentFSStateException: Directory
>> /d01/hadoop/dfs/name is in an inconsistent state:
>> /d01/hadoop/dfs/name/image does not exist.
>>
>> The old configuration was under a directory structure like:
>>
>> /d01/hadoop/dfs/name/current
>>
>> After backup up the namenode and playing around a little I found that
>> if I reformatted the namenode and then copied over the old files that
>> were in the current directory back into the "new" current directory
>> that the namenode would start up.
>>
>> We have quite a bit of data on this cluster (around 8T) and I am a
>> little nervous about starting up the entire cluster without a little
>> clarification. If I startup the cluster now, will any old data blocks
>> be deleted or will those data blocks remain because I copied over the
>> old configuration files into the new "current"?
>>
>> Is there another way to upgrade this DFS cluster? Any help is
>> appreciated.
>>
>> Dennis Kubes
>>
>
Re: Upgrade of DFS - Urgent
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
This is the result of HADOOP-1242. I prefer if it did not require
presence of this image directory.
For now you could manually create image/fsimage file in name/ directory.
If you write random 4 bytes to fsimage, you have 50% chance of success.
Basically readInt() from the file should be less than -3. Only first 4
bytes are important.
Raghu.
Dennis Kubes wrote:
> All,
>
> I upgraded to the most recent trunk of Hadoop and I started getting the
> error below, where /d01/hadoop/dfs/name is our namenode directory:
>
> org.apache.hadoop.dfs.InconsistentFSStateException: Directory
> /d01/hadoop/dfs/name is in an inconsistent state:
> /d01/hadoop/dfs/name/image does not exist.
>
> The old configuration was under a directory structure like:
>
> /d01/hadoop/dfs/name/current
>
> After backup up the namenode and playing around a little I found that if
> I reformatted the namenode and then copied over the old files that were
> in the current directory back into the "new" current directory that the
> namenode would start up.
>
> We have quite a bit of data on this cluster (around 8T) and I am a
> little nervous about starting up the entire cluster without a little
> clarification. If I startup the cluster now, will any old data blocks
> be deleted or will those data blocks remain because I copied over the
> old configuration files into the new "current"?
>
> Is there another way to upgrade this DFS cluster? Any help is appreciated.
>
> Dennis Kubes
>