You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "Peng, Wei" <We...@xerox.com> on 2011/09/21 05:30:04 UTC
java.io.IOException: Incorrect data format
I was not able to restart my name server because I the name server ran
out of space. Then I adjusted dfs.datanode.du.reserved to 0, and used
tune2fs -m to get some space, but I still could not restart the name
node.
I got the following error:
java.io.IOException: Incorrect data format. logVersion is -18 but
writables.length is 0.
Anyone knows how to resolve this issue?
Best,
Wei
RE: RE: java.io.IOException: Incorrect data format
Posted by "Peng, Wei" <We...@xerox.com>.
I just solved the problem by releasing more space on the related HD
partitions.
Thank you all for your help !
Wei
-----Original Message-----
From: Peng, Wei [mailto:Wei.Peng@xerox.com]
Sent: Tuesday, September 20, 2011 9:35 PM
To: common-user@hadoop.apache.org
Subject: RE: RE: java.io.IOException: Incorrect data format
After manually created /state/partition2/hadoop/dfs/tmp, the datanode
still could not be started and weirdly the
/state/partition2/hadoop/dfs/tmp file is removed somehow...
Wei
-----Original Message-----
From: Uma Maheswara Rao G 72686 [mailto:maheswara@huawei.com]
Sent: Tuesday, September 20, 2011 9:30 PM
To: common-user@hadoop.apache.org
Subject: Re: RE: java.io.IOException: Incorrect data format
Are you able to create the directory manually in the DataNode Machine?
#mkdirs /state/partition2/hadoop/dfs/tmp
Regards,
Uma
----- Original Message -----
From: "Peng, Wei" <We...@xerox.com>
Date: Wednesday, September 21, 2011 9:44 am
Subject: RE: java.io.IOException: Incorrect data format
To: common-user@hadoop.apache.org
> I modified edits so that hadoop namenode is restarted, however, I
> couldnot start my datanode. The datanode log shows
>
> 2011-09-20 21:07:10,068 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Mkdirs failed to create /state/partition2/hadoop/dfs/tmp
> at
>
org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.<init>(FSDatas
> et.java:394)
> at
>
org.apache.hadoop.hdfs.server.datanode.FSDataset.<init>(FSDataset.java:8
> 94)
> at
>
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.j
> ava:318)
> at
>
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:232
> )
> at
>
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.ja
> va:1363)
> at
>
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(Data
> Node.java:1318)
> at
>
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.
> java:1326)
> at
>
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1448)
> Wei
>
> -----Original Message-----
> From: Uma Maheswara Rao G 72686 [mailto:maheswara@huawei.com]
> Sent: Tuesday, September 20, 2011 9:10 PM
> To: common-user@hadoop.apache.org
> Subject: Re: java.io.IOException: Incorrect data format
>
> Can you check what is the value for command 'df -h'in NN machine.
> I think, one more possibility could be that while saving image
> itself it
> would have been currupted.
>
> To avoid such cases it has been already handled in trunk.For more
> details
> https://issues.apache.org/jira/browse/HDFS-1594
>
> Regards,
> Uma
> ----- Original Message -----
> From: "Peng, Wei" <We...@xerox.com>
> Date: Wednesday, September 21, 2011 9:01 am
> Subject: java.io.IOException: Incorrect data format
> To: common-user@hadoop.apache.org
>
> > I was not able to restart my name server because I the name
> server ran
> > out of space. Then I adjusted dfs.datanode.du.reserved to 0, and
> used> tune2fs -m to get some space, but I still could not restart
> the name
> > node.
> >
> >
> >
> > I got the following error:
> >
> > java.io.IOException: Incorrect data format. logVersion is -18 but
> > writables.length is 0.
> >
> >
> >
> > Anyone knows how to resolve this issue?
> >
> >
> >
> > Best,
> >
> > Wei
> >
> >
> >
> >
>
RE: RE: java.io.IOException: Incorrect data format
Posted by "Peng, Wei" <We...@xerox.com>.
After manually created /state/partition2/hadoop/dfs/tmp, the datanode
still could not be started and weirdly the
/state/partition2/hadoop/dfs/tmp file is removed somehow...
Wei
-----Original Message-----
From: Uma Maheswara Rao G 72686 [mailto:maheswara@huawei.com]
Sent: Tuesday, September 20, 2011 9:30 PM
To: common-user@hadoop.apache.org
Subject: Re: RE: java.io.IOException: Incorrect data format
Are you able to create the directory manually in the DataNode Machine?
#mkdirs /state/partition2/hadoop/dfs/tmp
Regards,
Uma
----- Original Message -----
From: "Peng, Wei" <We...@xerox.com>
Date: Wednesday, September 21, 2011 9:44 am
Subject: RE: java.io.IOException: Incorrect data format
To: common-user@hadoop.apache.org
> I modified edits so that hadoop namenode is restarted, however, I
> couldnot start my datanode. The datanode log shows
>
> 2011-09-20 21:07:10,068 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Mkdirs failed to create /state/partition2/hadoop/dfs/tmp
> at
>
org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.<init>(FSDatas
> et.java:394)
> at
>
org.apache.hadoop.hdfs.server.datanode.FSDataset.<init>(FSDataset.java:8
> 94)
> at
>
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.j
> ava:318)
> at
>
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:232
> )
> at
>
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.ja
> va:1363)
> at
>
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(Data
> Node.java:1318)
> at
>
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.
> java:1326)
> at
>
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1448)
> Wei
>
> -----Original Message-----
> From: Uma Maheswara Rao G 72686 [mailto:maheswara@huawei.com]
> Sent: Tuesday, September 20, 2011 9:10 PM
> To: common-user@hadoop.apache.org
> Subject: Re: java.io.IOException: Incorrect data format
>
> Can you check what is the value for command 'df -h'in NN machine.
> I think, one more possibility could be that while saving image
> itself it
> would have been currupted.
>
> To avoid such cases it has been already handled in trunk.For more
> details
> https://issues.apache.org/jira/browse/HDFS-1594
>
> Regards,
> Uma
> ----- Original Message -----
> From: "Peng, Wei" <We...@xerox.com>
> Date: Wednesday, September 21, 2011 9:01 am
> Subject: java.io.IOException: Incorrect data format
> To: common-user@hadoop.apache.org
>
> > I was not able to restart my name server because I the name
> server ran
> > out of space. Then I adjusted dfs.datanode.du.reserved to 0, and
> used> tune2fs -m to get some space, but I still could not restart
> the name
> > node.
> >
> >
> >
> > I got the following error:
> >
> > java.io.IOException: Incorrect data format. logVersion is -18 but
> > writables.length is 0.
> >
> >
> >
> > Anyone knows how to resolve this issue?
> >
> >
> >
> > Best,
> >
> > Wei
> >
> >
> >
> >
>
Re: RE: RE: java.io.IOException: Incorrect data format
Posted by Uma Maheswara Rao G 72686 <ma...@huawei.com>.
I would suggest you to clean some space and try.
Regards,
Uma
----- Original Message -----
From: "Peng, Wei" <We...@xerox.com>
Date: Wednesday, September 21, 2011 10:03 am
Subject: RE: RE: java.io.IOException: Incorrect data format
To: common-user@hadoop.apache.org
> Yes, I can. The datanode is not able to start after crashing without
> enough HD space.
>
> Wei
>
> -----Original Message-----
> From: Uma Maheswara Rao G 72686 [mailto:maheswara@huawei.com]
> Sent: Tuesday, September 20, 2011 9:30 PM
> To: common-user@hadoop.apache.org
> Subject: Re: RE: java.io.IOException: Incorrect data format
>
>
> Are you able to create the directory manually in the DataNode Machine?
> #mkdirs /state/partition2/hadoop/dfs/tmp
>
> Regards,
> Uma
> ----- Original Message -----
> From: "Peng, Wei" <We...@xerox.com>
> Date: Wednesday, September 21, 2011 9:44 am
> Subject: RE: java.io.IOException: Incorrect data format
> To: common-user@hadoop.apache.org
>
> > I modified edits so that hadoop namenode is restarted, however,
> I
> > couldnot start my datanode. The datanode log shows
> >
> > 2011-09-20 21:07:10,068 ERROR
> > org.apache.hadoop.hdfs.server.datanode.DataNode:
> java.io.IOException:> Mkdirs failed to create
> /state/partition2/hadoop/dfs/tmp> at
> >
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.<init>(FSDatas
> > et.java:394)
> > at
> >
> org.apache.hadoop.hdfs.server.datanode.FSDataset.<init>(FSDataset.java:8
> > 94)
> > at
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.j
> > ava:318)
> > at
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:232
> > )
> > at
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.ja
> > va:1363)
> > at
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(Data
> > Node.java:1318)
> > at
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.
> > java:1326)
> > at
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1448)
> > Wei
> >
> > -----Original Message-----
> > From: Uma Maheswara Rao G 72686 [mailto:maheswara@huawei.com]
> > Sent: Tuesday, September 20, 2011 9:10 PM
> > To: common-user@hadoop.apache.org
> > Subject: Re: java.io.IOException: Incorrect data format
> >
> > Can you check what is the value for command 'df -h'in NN machine.
> > I think, one more possibility could be that while saving image
> > itself it
> > would have been currupted.
> >
> > To avoid such cases it has been already handled in trunk.For more
> > details
> > https://issues.apache.org/jira/browse/HDFS-1594
> >
> > Regards,
> > Uma
> > ----- Original Message -----
> > From: "Peng, Wei" <We...@xerox.com>
> > Date: Wednesday, September 21, 2011 9:01 am
> > Subject: java.io.IOException: Incorrect data format
> > To: common-user@hadoop.apache.org
> >
> > > I was not able to restart my name server because I the name
> > server ran
> > > out of space. Then I adjusted dfs.datanode.du.reserved to 0,
> and
> > used> tune2fs -m to get some space, but I still could not
> restart
> > the name
> > > node.
> > >
> > >
> > >
> > > I got the following error:
> > >
> > > java.io.IOException: Incorrect data format. logVersion is -18 but
> > > writables.length is 0.
> > >
> > >
> > >
> > > Anyone knows how to resolve this issue?
> > >
> > >
> > >
> > > Best,
> > >
> > > Wei
> > >
> > >
> > >
> > >
> >
>
RE: RE: java.io.IOException: Incorrect data format
Posted by "Peng, Wei" <We...@xerox.com>.
Yes, I can. The datanode is not able to start after crashing without
enough HD space.
Wei
-----Original Message-----
From: Uma Maheswara Rao G 72686 [mailto:maheswara@huawei.com]
Sent: Tuesday, September 20, 2011 9:30 PM
To: common-user@hadoop.apache.org
Subject: Re: RE: java.io.IOException: Incorrect data format
Are you able to create the directory manually in the DataNode Machine?
#mkdirs /state/partition2/hadoop/dfs/tmp
Regards,
Uma
----- Original Message -----
From: "Peng, Wei" <We...@xerox.com>
Date: Wednesday, September 21, 2011 9:44 am
Subject: RE: java.io.IOException: Incorrect data format
To: common-user@hadoop.apache.org
> I modified edits so that hadoop namenode is restarted, however, I
> couldnot start my datanode. The datanode log shows
>
> 2011-09-20 21:07:10,068 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Mkdirs failed to create /state/partition2/hadoop/dfs/tmp
> at
>
org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.<init>(FSDatas
> et.java:394)
> at
>
org.apache.hadoop.hdfs.server.datanode.FSDataset.<init>(FSDataset.java:8
> 94)
> at
>
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.j
> ava:318)
> at
>
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:232
> )
> at
>
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.ja
> va:1363)
> at
>
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(Data
> Node.java:1318)
> at
>
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.
> java:1326)
> at
>
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1448)
> Wei
>
> -----Original Message-----
> From: Uma Maheswara Rao G 72686 [mailto:maheswara@huawei.com]
> Sent: Tuesday, September 20, 2011 9:10 PM
> To: common-user@hadoop.apache.org
> Subject: Re: java.io.IOException: Incorrect data format
>
> Can you check what is the value for command 'df -h'in NN machine.
> I think, one more possibility could be that while saving image
> itself it
> would have been currupted.
>
> To avoid such cases it has been already handled in trunk.For more
> details
> https://issues.apache.org/jira/browse/HDFS-1594
>
> Regards,
> Uma
> ----- Original Message -----
> From: "Peng, Wei" <We...@xerox.com>
> Date: Wednesday, September 21, 2011 9:01 am
> Subject: java.io.IOException: Incorrect data format
> To: common-user@hadoop.apache.org
>
> > I was not able to restart my name server because I the name
> server ran
> > out of space. Then I adjusted dfs.datanode.du.reserved to 0, and
> used> tune2fs -m to get some space, but I still could not restart
> the name
> > node.
> >
> >
> >
> > I got the following error:
> >
> > java.io.IOException: Incorrect data format. logVersion is -18 but
> > writables.length is 0.
> >
> >
> >
> > Anyone knows how to resolve this issue?
> >
> >
> >
> > Best,
> >
> > Wei
> >
> >
> >
> >
>
Re: RE: java.io.IOException: Incorrect data format
Posted by Uma Maheswara Rao G 72686 <ma...@huawei.com>.
Are you able to create the directory manually in the DataNode Machine?
#mkdirs /state/partition2/hadoop/dfs/tmp
Regards,
Uma
----- Original Message -----
From: "Peng, Wei" <We...@xerox.com>
Date: Wednesday, September 21, 2011 9:44 am
Subject: RE: java.io.IOException: Incorrect data format
To: common-user@hadoop.apache.org
> I modified edits so that hadoop namenode is restarted, however, I
> couldnot start my datanode. The datanode log shows
>
> 2011-09-20 21:07:10,068 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Mkdirs failed to create /state/partition2/hadoop/dfs/tmp
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.<init>(FSDatas
> et.java:394)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.<init>(FSDataset.java:8
> 94)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.j
> ava:318)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:232
> )
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.ja
> va:1363)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(Data
> Node.java:1318)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.
> java:1326)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1448)
> Wei
>
> -----Original Message-----
> From: Uma Maheswara Rao G 72686 [mailto:maheswara@huawei.com]
> Sent: Tuesday, September 20, 2011 9:10 PM
> To: common-user@hadoop.apache.org
> Subject: Re: java.io.IOException: Incorrect data format
>
> Can you check what is the value for command 'df -h'in NN machine.
> I think, one more possibility could be that while saving image
> itself it
> would have been currupted.
>
> To avoid such cases it has been already handled in trunk.For more
> details
> https://issues.apache.org/jira/browse/HDFS-1594
>
> Regards,
> Uma
> ----- Original Message -----
> From: "Peng, Wei" <We...@xerox.com>
> Date: Wednesday, September 21, 2011 9:01 am
> Subject: java.io.IOException: Incorrect data format
> To: common-user@hadoop.apache.org
>
> > I was not able to restart my name server because I the name
> server ran
> > out of space. Then I adjusted dfs.datanode.du.reserved to 0, and
> used> tune2fs -m to get some space, but I still could not restart
> the name
> > node.
> >
> >
> >
> > I got the following error:
> >
> > java.io.IOException: Incorrect data format. logVersion is -18 but
> > writables.length is 0.
> >
> >
> >
> > Anyone knows how to resolve this issue?
> >
> >
> >
> > Best,
> >
> > Wei
> >
> >
> >
> >
>
RE: java.io.IOException: Incorrect data format
Posted by "Peng, Wei" <We...@xerox.com>.
I modified edits so that hadoop namenode is restarted, however, I could
not start my datanode. The datanode log shows
2011-09-20 21:07:10,068 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
Mkdirs failed to create /state/partition2/hadoop/dfs/tmp
at
org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.<init>(FSDatas
et.java:394)
at
org.apache.hadoop.hdfs.server.datanode.FSDataset.<init>(FSDataset.java:8
94)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.j
ava:318)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:232
)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.ja
va:1363)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(Data
Node.java:1318)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.
java:1326)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1448)
Wei
-----Original Message-----
From: Uma Maheswara Rao G 72686 [mailto:maheswara@huawei.com]
Sent: Tuesday, September 20, 2011 9:10 PM
To: common-user@hadoop.apache.org
Subject: Re: java.io.IOException: Incorrect data format
Can you check what is the value for command 'df -h'in NN machine.
I think, one more possibility could be that while saving image itself it
would have been currupted.
To avoid such cases it has been already handled in trunk.For more
details
https://issues.apache.org/jira/browse/HDFS-1594
Regards,
Uma
----- Original Message -----
From: "Peng, Wei" <We...@xerox.com>
Date: Wednesday, September 21, 2011 9:01 am
Subject: java.io.IOException: Incorrect data format
To: common-user@hadoop.apache.org
> I was not able to restart my name server because I the name server ran
> out of space. Then I adjusted dfs.datanode.du.reserved to 0, and used
> tune2fs -m to get some space, but I still could not restart the name
> node.
>
>
>
> I got the following error:
>
> java.io.IOException: Incorrect data format. logVersion is -18 but
> writables.length is 0.
>
>
>
> Anyone knows how to resolve this issue?
>
>
>
> Best,
>
> Wei
>
>
>
>
Re: java.io.IOException: Incorrect data format
Posted by Uma Maheswara Rao G 72686 <ma...@huawei.com>.
Can you check what is the value for command 'df -h'in NN machine.
I think, one more possibility could be that while saving image itself it would have been currupted.
To avoid such cases it has been already handled in trunk.For more details
https://issues.apache.org/jira/browse/HDFS-1594
Regards,
Uma
----- Original Message -----
From: "Peng, Wei" <We...@xerox.com>
Date: Wednesday, September 21, 2011 9:01 am
Subject: java.io.IOException: Incorrect data format
To: common-user@hadoop.apache.org
> I was not able to restart my name server because I the name server ran
> out of space. Then I adjusted dfs.datanode.du.reserved to 0, and used
> tune2fs -m to get some space, but I still could not restart the name
> node.
>
>
>
> I got the following error:
>
> java.io.IOException: Incorrect data format. logVersion is -18 but
> writables.length is 0.
>
>
>
> Anyone knows how to resolve this issue?
>
>
>
> Best,
>
> Wei
>
>
>
>