You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by bo...@post.tau.ac.il on 2012/08/28 18:13:45 UTC
datanode has no storageID
Hi,
hope it's not a newby question...
I installed several versions of hadoop for testing,
(0.20.203, 0.21.0, and 1.0.3)
on various machines.
now I am using 1.0.3 on all the machines,
I face a problem that in some of the machhines, the datanode gets no
storageID from the namenode.
where it works, the datanode has the following lines in the log file:
(and current/VERSION has a storageID=< some ID > )
---------------------------------------------------------------
2012-08-28 19:04:31,415 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration =
DatanodeRegistration(datanode-works.cs.tau.ac.il:50010,
storageID=DS-996163017-<machines-ip>-50010-1342683478942,
infoPort=50075, ipcPort=50020)
2012-08-28 19:04:31,418 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Starting asynchronous
block report scan
2012-08-28 19:04:31,418 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(<machines-ip>:50010,
storageID=DS-996163017-<machines-ip>-50010-1342683478942,
infoPort=50075, ipcPort=50020)In DataNode.run, data =
FSDataset{dirpath='/var/cache/hdfs/hadoop-data-node/current'}
2012-08-28 19:04:31,419 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
-------------------------------------------------------------------------------
where it doesn't work I have only the first line and it hangs.
(and current/VERSION has a 'storageID=' empty value line )
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2012-08-28 18:42:01,297 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration =
DatanodeRegistration(machinename.cs.tau.ac.il:50010, storageID=,
infoPort=50075, ipcPort=50020)
2012-08-28 18:42:01,287 INFO org.apache.hadoop.ipc.Server: Starting
SocketReader
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1. any Ideas?
2. how/where does the namenode stores the datanodes's storageids ?
3. how can I get a new storageid for a datanode or it's old ID ?
4. can I format/reset the namenode to enable the datanode to reconnect ?
thanks!
-
Boaz Yarom
CS System Team
03-640-8961 / 7637
Re: datanode has no storageID
Posted by bo...@post.tau.ac.il.
hi,
thanks for the reply,
I stopped all the processes,
moved the hadoop-data-node folders from all,
moved hadoop-name-node folder on the namenode machine
and formatted the namenode.
still some machines, the same as before has no storageID in
current/VERSION file.
these machine are'nt shown in the web-GUI.
and this is the last line from the datanode log file:
2012-08-30 10:05:29,430 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration =
DatanodeRegistration(datanode-machine-name.cs.tau.ac.il:50010,
storageID=, infoPort=50075, ipcPort=50020)
any ideas ?
--
Boaz Yarom
CS System Team
03-640-8961 / 7637
Quoting Arpit Gupta <ar...@hortonworks.com>:
>> 2. how/where does the namenode stores the datanodes's storageids ?
>
> When the datanode connects with the namenode for the first time it
> will register with the namenode and during the registration the of
> the datanode the storage id for the datanode is generated.
>
>>
>> 4. can I format/reset the namenode to enable the datanode to reconnect ?
>
> If you do not care about the existing data you can certainly do
> that. Make sure you stop all processes, format the namenode, clean
> up the data directories on all your datanodes (dfs.data.dir prop in
> hdfs-site.xml) and then start the namenode and the datanodes.
>
> --
> Arpit Gupta
> Hortonworks Inc.
> http://hortonworks.com/
>
> On Aug 28, 2012, at 9:13 AM, boazya@post.tau.ac.il wrote:
>
>> Hi,
>> hope it's not a newby question...
>> I installed several versions of hadoop for testing,
>> (0.20.203, 0.21.0, and 1.0.3)
>> on various machines.
>> now I am using 1.0.3 on all the machines,
>> I face a problem that in some of the machhines, the datanode gets no
>> storageID from the namenode.
>> where it works, the datanode has the following lines in the log file:
>> (and current/VERSION has a storageID=< some ID > )
>> ---------------------------------------------------------------
>> 2012-08-28 19:04:31,415 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration =
>> DatanodeRegistration(datanode-works.cs.tau.ac.il:50010,
>> storageID=DS-996163017-<machines-ip>-50010-1342683478942,
>> infoPort=50075, ipcPort=50020)
>> 2012-08-28 19:04:31,418 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: Starting
>> asynchronous block report scan
>> 2012-08-28 19:04:31,418 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>> DatanodeRegistration(<machines-ip>:50010,
>> storageID=DS-996163017-<machines-ip>-50010-1342683478942,
>> infoPort=50075, ipcPort=50020)In DataNode.run, data =
>> FSDataset{dirpath='/var/cache/hdfs/hadoop-data-node/current'}
>> 2012-08-28 19:04:31,419 INFO org.apache.hadoop.ipc.Server: IPC
>> Server Responder: starting
>> -------------------------------------------------------------------------------
>>
>> where it doesn't work I have only the first line and it hangs.
>> (and current/VERSION has a 'storageID=' empty value line )
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> 2012-08-28 18:42:01,297 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration =
>> DatanodeRegistration(machinename.cs.tau.ac.il:50010, storageID=,
>> infoPort=50075, ipcPort=50020)
>> 2012-08-28 18:42:01,287 INFO org.apache.hadoop.ipc.Server: Starting
>> SocketReader
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> 1. any Ideas?
>> 2. how/where does the namenode stores the datanodes's storageids ?
>> 3. how can I get a new storageid for a datanode or it's old ID ?
>> 4. can I format/reset the namenode to enable the datanode to reconnect ?
>>
>> thanks!
>> -
>> Boaz Yarom
>> CS System Team
>> 03-640-8961 / 7637
>>
>>
Re: datanode has no storageID
Posted by bo...@post.tau.ac.il.
hi,
thanks for the reply,
I stopped all the processes,
moved the hadoop-data-node folders from all,
moved hadoop-name-node folder on the namenode machine
and formatted the namenode.
still some machines, the same as before has no storageID in
current/VERSION file.
these machine are'nt shown in the web-GUI.
and this is the last line from the datanode log file:
2012-08-30 10:05:29,430 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration =
DatanodeRegistration(datanode-machine-name.cs.tau.ac.il:50010,
storageID=, infoPort=50075, ipcPort=50020)
any ideas ?
--
Boaz Yarom
CS System Team
03-640-8961 / 7637
Quoting Arpit Gupta <ar...@hortonworks.com>:
>> 2. how/where does the namenode stores the datanodes's storageids ?
>
> When the datanode connects with the namenode for the first time it
> will register with the namenode and during the registration the of
> the datanode the storage id for the datanode is generated.
>
>>
>> 4. can I format/reset the namenode to enable the datanode to reconnect ?
>
> If you do not care about the existing data you can certainly do
> that. Make sure you stop all processes, format the namenode, clean
> up the data directories on all your datanodes (dfs.data.dir prop in
> hdfs-site.xml) and then start the namenode and the datanodes.
>
> --
> Arpit Gupta
> Hortonworks Inc.
> http://hortonworks.com/
>
> On Aug 28, 2012, at 9:13 AM, boazya@post.tau.ac.il wrote:
>
>> Hi,
>> hope it's not a newby question...
>> I installed several versions of hadoop for testing,
>> (0.20.203, 0.21.0, and 1.0.3)
>> on various machines.
>> now I am using 1.0.3 on all the machines,
>> I face a problem that in some of the machhines, the datanode gets no
>> storageID from the namenode.
>> where it works, the datanode has the following lines in the log file:
>> (and current/VERSION has a storageID=< some ID > )
>> ---------------------------------------------------------------
>> 2012-08-28 19:04:31,415 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration =
>> DatanodeRegistration(datanode-works.cs.tau.ac.il:50010,
>> storageID=DS-996163017-<machines-ip>-50010-1342683478942,
>> infoPort=50075, ipcPort=50020)
>> 2012-08-28 19:04:31,418 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: Starting
>> asynchronous block report scan
>> 2012-08-28 19:04:31,418 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>> DatanodeRegistration(<machines-ip>:50010,
>> storageID=DS-996163017-<machines-ip>-50010-1342683478942,
>> infoPort=50075, ipcPort=50020)In DataNode.run, data =
>> FSDataset{dirpath='/var/cache/hdfs/hadoop-data-node/current'}
>> 2012-08-28 19:04:31,419 INFO org.apache.hadoop.ipc.Server: IPC
>> Server Responder: starting
>> -------------------------------------------------------------------------------
>>
>> where it doesn't work I have only the first line and it hangs.
>> (and current/VERSION has a 'storageID=' empty value line )
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> 2012-08-28 18:42:01,297 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration =
>> DatanodeRegistration(machinename.cs.tau.ac.il:50010, storageID=,
>> infoPort=50075, ipcPort=50020)
>> 2012-08-28 18:42:01,287 INFO org.apache.hadoop.ipc.Server: Starting
>> SocketReader
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> 1. any Ideas?
>> 2. how/where does the namenode stores the datanodes's storageids ?
>> 3. how can I get a new storageid for a datanode or it's old ID ?
>> 4. can I format/reset the namenode to enable the datanode to reconnect ?
>>
>> thanks!
>> -
>> Boaz Yarom
>> CS System Team
>> 03-640-8961 / 7637
>>
>>
Re: datanode has no storageID
Posted by bo...@post.tau.ac.il.
hi,
thanks for the reply,
I stopped all the processes,
moved the hadoop-data-node folders from all,
moved hadoop-name-node folder on the namenode machine
and formatted the namenode.
still some machines, the same as before has no storageID in
current/VERSION file.
these machine are'nt shown in the web-GUI.
and this is the last line from the datanode log file:
2012-08-30 10:05:29,430 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration =
DatanodeRegistration(datanode-machine-name.cs.tau.ac.il:50010,
storageID=, infoPort=50075, ipcPort=50020)
any ideas ?
--
Boaz Yarom
CS System Team
03-640-8961 / 7637
Quoting Arpit Gupta <ar...@hortonworks.com>:
>> 2. how/where does the namenode stores the datanodes's storageids ?
>
> When the datanode connects with the namenode for the first time it
> will register with the namenode and during the registration the of
> the datanode the storage id for the datanode is generated.
>
>>
>> 4. can I format/reset the namenode to enable the datanode to reconnect ?
>
> If you do not care about the existing data you can certainly do
> that. Make sure you stop all processes, format the namenode, clean
> up the data directories on all your datanodes (dfs.data.dir prop in
> hdfs-site.xml) and then start the namenode and the datanodes.
>
> --
> Arpit Gupta
> Hortonworks Inc.
> http://hortonworks.com/
>
> On Aug 28, 2012, at 9:13 AM, boazya@post.tau.ac.il wrote:
>
>> Hi,
>> hope it's not a newby question...
>> I installed several versions of hadoop for testing,
>> (0.20.203, 0.21.0, and 1.0.3)
>> on various machines.
>> now I am using 1.0.3 on all the machines,
>> I face a problem that in some of the machhines, the datanode gets no
>> storageID from the namenode.
>> where it works, the datanode has the following lines in the log file:
>> (and current/VERSION has a storageID=< some ID > )
>> ---------------------------------------------------------------
>> 2012-08-28 19:04:31,415 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration =
>> DatanodeRegistration(datanode-works.cs.tau.ac.il:50010,
>> storageID=DS-996163017-<machines-ip>-50010-1342683478942,
>> infoPort=50075, ipcPort=50020)
>> 2012-08-28 19:04:31,418 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: Starting
>> asynchronous block report scan
>> 2012-08-28 19:04:31,418 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>> DatanodeRegistration(<machines-ip>:50010,
>> storageID=DS-996163017-<machines-ip>-50010-1342683478942,
>> infoPort=50075, ipcPort=50020)In DataNode.run, data =
>> FSDataset{dirpath='/var/cache/hdfs/hadoop-data-node/current'}
>> 2012-08-28 19:04:31,419 INFO org.apache.hadoop.ipc.Server: IPC
>> Server Responder: starting
>> -------------------------------------------------------------------------------
>>
>> where it doesn't work I have only the first line and it hangs.
>> (and current/VERSION has a 'storageID=' empty value line )
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> 2012-08-28 18:42:01,297 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration =
>> DatanodeRegistration(machinename.cs.tau.ac.il:50010, storageID=,
>> infoPort=50075, ipcPort=50020)
>> 2012-08-28 18:42:01,287 INFO org.apache.hadoop.ipc.Server: Starting
>> SocketReader
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> 1. any Ideas?
>> 2. how/where does the namenode stores the datanodes's storageids ?
>> 3. how can I get a new storageid for a datanode or it's old ID ?
>> 4. can I format/reset the namenode to enable the datanode to reconnect ?
>>
>> thanks!
>> -
>> Boaz Yarom
>> CS System Team
>> 03-640-8961 / 7637
>>
>>
Re: datanode has no storageID
Posted by bo...@post.tau.ac.il.
hi,
thanks for the reply,
I stopped all the processes,
moved the hadoop-data-node folders from all,
moved hadoop-name-node folder on the namenode machine
and formatted the namenode.
still some machines, the same as before has no storageID in
current/VERSION file.
these machine are'nt shown in the web-GUI.
and this is the last line from the datanode log file:
2012-08-30 10:05:29,430 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration =
DatanodeRegistration(datanode-machine-name.cs.tau.ac.il:50010,
storageID=, infoPort=50075, ipcPort=50020)
any ideas ?
--
Boaz Yarom
CS System Team
03-640-8961 / 7637
Quoting Arpit Gupta <ar...@hortonworks.com>:
>> 2. how/where does the namenode stores the datanodes's storageids ?
>
> When the datanode connects with the namenode for the first time it
> will register with the namenode and during the registration the of
> the datanode the storage id for the datanode is generated.
>
>>
>> 4. can I format/reset the namenode to enable the datanode to reconnect ?
>
> If you do not care about the existing data you can certainly do
> that. Make sure you stop all processes, format the namenode, clean
> up the data directories on all your datanodes (dfs.data.dir prop in
> hdfs-site.xml) and then start the namenode and the datanodes.
>
> --
> Arpit Gupta
> Hortonworks Inc.
> http://hortonworks.com/
>
> On Aug 28, 2012, at 9:13 AM, boazya@post.tau.ac.il wrote:
>
>> Hi,
>> hope it's not a newby question...
>> I installed several versions of hadoop for testing,
>> (0.20.203, 0.21.0, and 1.0.3)
>> on various machines.
>> now I am using 1.0.3 on all the machines,
>> I face a problem that in some of the machhines, the datanode gets no
>> storageID from the namenode.
>> where it works, the datanode has the following lines in the log file:
>> (and current/VERSION has a storageID=< some ID > )
>> ---------------------------------------------------------------
>> 2012-08-28 19:04:31,415 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration =
>> DatanodeRegistration(datanode-works.cs.tau.ac.il:50010,
>> storageID=DS-996163017-<machines-ip>-50010-1342683478942,
>> infoPort=50075, ipcPort=50020)
>> 2012-08-28 19:04:31,418 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: Starting
>> asynchronous block report scan
>> 2012-08-28 19:04:31,418 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>> DatanodeRegistration(<machines-ip>:50010,
>> storageID=DS-996163017-<machines-ip>-50010-1342683478942,
>> infoPort=50075, ipcPort=50020)In DataNode.run, data =
>> FSDataset{dirpath='/var/cache/hdfs/hadoop-data-node/current'}
>> 2012-08-28 19:04:31,419 INFO org.apache.hadoop.ipc.Server: IPC
>> Server Responder: starting
>> -------------------------------------------------------------------------------
>>
>> where it doesn't work I have only the first line and it hangs.
>> (and current/VERSION has a 'storageID=' empty value line )
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> 2012-08-28 18:42:01,297 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration =
>> DatanodeRegistration(machinename.cs.tau.ac.il:50010, storageID=,
>> infoPort=50075, ipcPort=50020)
>> 2012-08-28 18:42:01,287 INFO org.apache.hadoop.ipc.Server: Starting
>> SocketReader
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> 1. any Ideas?
>> 2. how/where does the namenode stores the datanodes's storageids ?
>> 3. how can I get a new storageid for a datanode or it's old ID ?
>> 4. can I format/reset the namenode to enable the datanode to reconnect ?
>>
>> thanks!
>> -
>> Boaz Yarom
>> CS System Team
>> 03-640-8961 / 7637
>>
>>
Re: datanode has no storageID
Posted by Arpit Gupta <ar...@hortonworks.com>.
> 2. how/where does the namenode stores the datanodes's storageids ?
When the datanode connects with the namenode for the first time it will register with the namenode and during the registration the of the datanode the storage id for the datanode is generated.
>
> 4. can I format/reset the namenode to enable the datanode to reconnect ?
If you do not care about the existing data you can certainly do that. Make sure you stop all processes, format the namenode, clean up the data directories on all your datanodes (dfs.data.dir prop in hdfs-site.xml) and then start the namenode and the datanodes.
--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/
On Aug 28, 2012, at 9:13 AM, boazya@post.tau.ac.il wrote:
> Hi,
> hope it's not a newby question...
> I installed several versions of hadoop for testing,
> (0.20.203, 0.21.0, and 1.0.3)
> on various machines.
> now I am using 1.0.3 on all the machines,
> I face a problem that in some of the machhines, the datanode gets no
> storageID from the namenode.
> where it works, the datanode has the following lines in the log file:
> (and current/VERSION has a storageID=< some ID > )
> ---------------------------------------------------------------
> 2012-08-28 19:04:31,415 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(datanode-works.cs.tau.ac.il:50010, storageID=DS-996163017-<machines-ip>-50010-1342683478942, infoPort=50075, ipcPort=50020)
> 2012-08-28 19:04:31,418 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting asynchronous block report scan
> 2012-08-28 19:04:31,418 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(<machines-ip>:50010, storageID=DS-996163017-<machines-ip>-50010-1342683478942, infoPort=50075, ipcPort=50020)In DataNode.run, data = FSDataset{dirpath='/var/cache/hdfs/hadoop-data-node/current'}
> 2012-08-28 19:04:31,419 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> -------------------------------------------------------------------------------
>
> where it doesn't work I have only the first line and it hangs.
> (and current/VERSION has a 'storageID=' empty value line )
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 2012-08-28 18:42:01,297 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(machinename.cs.tau.ac.il:50010, storageID=, infoPort=50075, ipcPort=50020)
> 2012-08-28 18:42:01,287 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 1. any Ideas?
> 2. how/where does the namenode stores the datanodes's storageids ?
> 3. how can I get a new storageid for a datanode or it's old ID ?
> 4. can I format/reset the namenode to enable the datanode to reconnect ?
>
> thanks!
> -
> Boaz Yarom
> CS System Team
> 03-640-8961 / 7637
>
>
Re: datanode has no storageID
Posted by Arpit Gupta <ar...@hortonworks.com>.
> 2. how/where does the namenode stores the datanodes's storageids ?
When the datanode connects with the namenode for the first time it will register with the namenode and during the registration the of the datanode the storage id for the datanode is generated.
>
> 4. can I format/reset the namenode to enable the datanode to reconnect ?
If you do not care about the existing data you can certainly do that. Make sure you stop all processes, format the namenode, clean up the data directories on all your datanodes (dfs.data.dir prop in hdfs-site.xml) and then start the namenode and the datanodes.
--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/
On Aug 28, 2012, at 9:13 AM, boazya@post.tau.ac.il wrote:
> Hi,
> hope it's not a newby question...
> I installed several versions of hadoop for testing,
> (0.20.203, 0.21.0, and 1.0.3)
> on various machines.
> now I am using 1.0.3 on all the machines,
> I face a problem that in some of the machhines, the datanode gets no
> storageID from the namenode.
> where it works, the datanode has the following lines in the log file:
> (and current/VERSION has a storageID=< some ID > )
> ---------------------------------------------------------------
> 2012-08-28 19:04:31,415 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(datanode-works.cs.tau.ac.il:50010, storageID=DS-996163017-<machines-ip>-50010-1342683478942, infoPort=50075, ipcPort=50020)
> 2012-08-28 19:04:31,418 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting asynchronous block report scan
> 2012-08-28 19:04:31,418 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(<machines-ip>:50010, storageID=DS-996163017-<machines-ip>-50010-1342683478942, infoPort=50075, ipcPort=50020)In DataNode.run, data = FSDataset{dirpath='/var/cache/hdfs/hadoop-data-node/current'}
> 2012-08-28 19:04:31,419 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> -------------------------------------------------------------------------------
>
> where it doesn't work I have only the first line and it hangs.
> (and current/VERSION has a 'storageID=' empty value line )
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 2012-08-28 18:42:01,297 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(machinename.cs.tau.ac.il:50010, storageID=, infoPort=50075, ipcPort=50020)
> 2012-08-28 18:42:01,287 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 1. any Ideas?
> 2. how/where does the namenode stores the datanodes's storageids ?
> 3. how can I get a new storageid for a datanode or it's old ID ?
> 4. can I format/reset the namenode to enable the datanode to reconnect ?
>
> thanks!
> -
> Boaz Yarom
> CS System Team
> 03-640-8961 / 7637
>
>
Re: datanode has no storageID
Posted by Arpit Gupta <ar...@hortonworks.com>.
> 2. how/where does the namenode stores the datanodes's storageids ?
When the datanode connects with the namenode for the first time it will register with the namenode and during the registration the of the datanode the storage id for the datanode is generated.
>
> 4. can I format/reset the namenode to enable the datanode to reconnect ?
If you do not care about the existing data you can certainly do that. Make sure you stop all processes, format the namenode, clean up the data directories on all your datanodes (dfs.data.dir prop in hdfs-site.xml) and then start the namenode and the datanodes.
--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/
On Aug 28, 2012, at 9:13 AM, boazya@post.tau.ac.il wrote:
> Hi,
> hope it's not a newby question...
> I installed several versions of hadoop for testing,
> (0.20.203, 0.21.0, and 1.0.3)
> on various machines.
> now I am using 1.0.3 on all the machines,
> I face a problem that in some of the machhines, the datanode gets no
> storageID from the namenode.
> where it works, the datanode has the following lines in the log file:
> (and current/VERSION has a storageID=< some ID > )
> ---------------------------------------------------------------
> 2012-08-28 19:04:31,415 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(datanode-works.cs.tau.ac.il:50010, storageID=DS-996163017-<machines-ip>-50010-1342683478942, infoPort=50075, ipcPort=50020)
> 2012-08-28 19:04:31,418 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting asynchronous block report scan
> 2012-08-28 19:04:31,418 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(<machines-ip>:50010, storageID=DS-996163017-<machines-ip>-50010-1342683478942, infoPort=50075, ipcPort=50020)In DataNode.run, data = FSDataset{dirpath='/var/cache/hdfs/hadoop-data-node/current'}
> 2012-08-28 19:04:31,419 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> -------------------------------------------------------------------------------
>
> where it doesn't work I have only the first line and it hangs.
> (and current/VERSION has a 'storageID=' empty value line )
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 2012-08-28 18:42:01,297 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(machinename.cs.tau.ac.il:50010, storageID=, infoPort=50075, ipcPort=50020)
> 2012-08-28 18:42:01,287 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 1. any Ideas?
> 2. how/where does the namenode stores the datanodes's storageids ?
> 3. how can I get a new storageid for a datanode or it's old ID ?
> 4. can I format/reset the namenode to enable the datanode to reconnect ?
>
> thanks!
> -
> Boaz Yarom
> CS System Team
> 03-640-8961 / 7637
>
>
Re: datanode has no storageID
Posted by Arpit Gupta <ar...@hortonworks.com>.
> 2. how/where does the namenode stores the datanodes's storageids ?
When the datanode connects with the namenode for the first time it will register with the namenode and during the registration the of the datanode the storage id for the datanode is generated.
>
> 4. can I format/reset the namenode to enable the datanode to reconnect ?
If you do not care about the existing data you can certainly do that. Make sure you stop all processes, format the namenode, clean up the data directories on all your datanodes (dfs.data.dir prop in hdfs-site.xml) and then start the namenode and the datanodes.
--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/
On Aug 28, 2012, at 9:13 AM, boazya@post.tau.ac.il wrote:
> Hi,
> hope it's not a newby question...
> I installed several versions of hadoop for testing,
> (0.20.203, 0.21.0, and 1.0.3)
> on various machines.
> now I am using 1.0.3 on all the machines,
> I face a problem that in some of the machhines, the datanode gets no
> storageID from the namenode.
> where it works, the datanode has the following lines in the log file:
> (and current/VERSION has a storageID=< some ID > )
> ---------------------------------------------------------------
> 2012-08-28 19:04:31,415 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(datanode-works.cs.tau.ac.il:50010, storageID=DS-996163017-<machines-ip>-50010-1342683478942, infoPort=50075, ipcPort=50020)
> 2012-08-28 19:04:31,418 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting asynchronous block report scan
> 2012-08-28 19:04:31,418 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(<machines-ip>:50010, storageID=DS-996163017-<machines-ip>-50010-1342683478942, infoPort=50075, ipcPort=50020)In DataNode.run, data = FSDataset{dirpath='/var/cache/hdfs/hadoop-data-node/current'}
> 2012-08-28 19:04:31,419 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> -------------------------------------------------------------------------------
>
> where it doesn't work I have only the first line and it hangs.
> (and current/VERSION has a 'storageID=' empty value line )
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 2012-08-28 18:42:01,297 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(machinename.cs.tau.ac.il:50010, storageID=, infoPort=50075, ipcPort=50020)
> 2012-08-28 18:42:01,287 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 1. any Ideas?
> 2. how/where does the namenode stores the datanodes's storageids ?
> 3. how can I get a new storageid for a datanode or it's old ID ?
> 4. can I format/reset the namenode to enable the datanode to reconnect ?
>
> thanks!
> -
> Boaz Yarom
> CS System Team
> 03-640-8961 / 7637
>
>