You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by schubert zhang <zs...@gmail.com> on 2009/03/25 10:12:18 UTC
HDFS unbalance issue. (HBase over HDFS)
Hi all,
I am using hbase-0.19.1 and hadoop-0.19.
My cluster have 5+1 nodes, and there are about 512 regions in HBase (256MB
per region).
But I found the blocks in HDFS is very unbalanced. Following is the status
from HDFS web GUI.
(Node: I don't know if this mailing list can display html!)
HDFS blocks:
node1 509036 blocks
node2 157937 blocks
node3 15783 blocks
node4 15117 blocks
node5 20158 blocks
But my HBase regions are very balanced.
node1 88 regions
node2 108 regions
node3 111 regions
node4 102 regions
node5 105 regions
NodeLast
ContactAdmin StateConfigured
Capacity (GB)Used
(GB)Non DFS
Used (GB)Remaining
(GB)Used
(%)Used
(%)Remaining
(%)Blocksnd1-rack0-cloud<http://nd1-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>
0In Service822.8578.6743.28200.8670.3324.41509036nd2-rack0-cloud<http://nd2-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>
0In Service822.8190.0242.96589.8223.0971.68157937nd3-rack0-cloud<http://nd3-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>
0In Service822.851.9542.61728.246.3188.5115783nd4-rack0-cloud<http://nd4-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>
6In Service822.846.1942.84733.775.6189.1815117nd5-rack0-cloud<http://nd5-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>
1In Service1215.6152.3762.911100.324.3190.5220158
But my HBase regions are very balanced.
AddressStart CodeLoadnd1-rack0-cloud:60020 <http://nd1-rack0-cloud:60030/>
1237967027050requests=383, regions=88, usedHeap=978, maxHeap=1991
nd2-rack0-cloud:60020 <http://nd2-rack0-cloud:60030/>1237788871362requests=422,
regions=108, usedHeap=1433,
maxHeap=1991nd3-rack0-cloud:60020<http://nd3-rack0-cloud:60030/>
1237788881667requests=962, regions=111, usedHeap=1534, maxHeap=1991
nd4-rack0-cloud:60020 <http://nd4-rack0-cloud:60030/>1237788859541requests=369,
regions=102, usedHeap=1059,
maxHeap=1991nd5-rack0-cloud:60020<http://nd5-rack0-cloud:60030/>
1237788899331requests=384, regions=105, usedHeap=1535,
maxHeap=1983Total:servers:
5 requests=2520, regions=514
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by stack <st...@duboce.net>.
Thank you for figuring it (The issue is HADOOP-5124 for others interested).
St.Ack
On Wed, Apr 1, 2009 at 10:26 AM, zsongbo <zs...@gmail.com> wrote:
> Hi Stack,
>
> I have just send a email to your email address with the attached fix (since
> I cannot attach big file here).
> Please check it.
> Someone have reported this issue in hadoop, and Hadoop's Hairong has made
> patch on 0.21.0.
>
> Regards,
> Schubert
>
> On Wed, Apr 1, 2009 at 3:44 PM, stack <st...@duboce.net> wrote:
>
> > What was the fix?
> > Thanks,
> > St.Ack
> >
> > On Wed, Apr 1, 2009 at 6:54 AM, zsongbo <zs...@gmail.com> wrote:
> >
> > > Thanks stack. (i am schubert)
> > > Yes, I have found the fix in 0.20.
> > > And I just made a temporary fix based on branch-0.19 and it work fine.
> > >
> > > On Mon, Mar 30, 2009 at 7:16 PM, stack <st...@duboce.net> wrote:
> > >
> > > > Thanks for doing the digging Schubert. I agree, its an ugly issue.
> > > > Another
> > > > gentleman reported he'd tripped over the same thing in private mail.
> > I'd
> > > > suggest you file an issue against HADOOP HDFS and add the below (I"ve
> > > > opened
> > > > HBASE-1296 so we can track it in our project).
> > > >
> > > > Good stuff,
> > > > St.Ack
> > > >
> > > > On Fri, Mar 27, 2009 at 9:50 AM, schubert zhang <zs...@gmail.com>
> > > wrote:
> > > >
> > > > > Sorry "But I found the namenode is fair to process the invalidating
> > for
> > > > > each
> > > > > datanode."
> > > > > should be:
> > > > >
> > > > > "But I found the namenode is unfair to process the invalidating for
> > > each
> > > > > datanode."
> > > > >
> > > > >
> > > > > On Fri, Mar 27, 2009 at 3:49 PM, schubert zhang <zsongbo@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Thanks Samuel,
> > > > > > Your information is very correct.
> > > > > > I have also read code about garbage collection of invalidating
> > > blocks.
> > > > > >
> > > > > > But I found the namenode is fair to process the invalidating for
> > each
> > > > > > datanode.
> > > > > > In my cluster, there are 5 datanode. The storage IDs are:
> > > > > >
> > > > > > node1: DS- 978762906-10.24.1.12-50010-1237686434530
> > > > > > node2: DS- 489086185-10.24.1.14-50010-1237686416330
> > > > > > node3: DS-1170985665-10.24.1.16-50010-1237686426395
> > > > > > node4: DS-1024388083-10.24.1.18-50010-1237686404482
> > > > > > node5: DS-2136798339-10.24.1.20-50010-1237686444430
> > > > > > I know the storage ID is generated
> > > > > > by
> > > > org.apache.hadoop.hdfs.server.datanode.DataNode.setNewStorageID(...).
> > > > > >
> > > > > > In org.apache.hadoop.hdfs.server.namenode.FSNamesystem
> > > > > >
> > > > > > // Keeps a Collection for every named machine containing
> > > > > > // blocks that have recently been invalidated and are thought
> to
> > > live
> > > > > > // on the machine in question.
> > > > > > // Mapping: StorageID -> ArrayList<Block>
> > > > > > //
> > > > > > private Map<String, Collection<Block>> recentInvalidateSets =
> > > > > > new TreeMap<String, Collection<Block>>();
> > > > > >
> > > > > > In
> > > >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.ReplicationMonitor
> > > > > > This thread run in interval: replicationRecheckInterval=3000
> > > > > milliseconds.
> > > > > >
> > > > > > Into computeDatanodeWork()
> > > > > > nodesToProcess = 2.
> > > > > >
> > > > > > then into computeInvalidateWork(nodesToProcess)
> > > > > > the for cycle will only exec 2 cycles.
> > > > > >
> > > > > > for each cycle, go into invalidateWorkForOneNode()
> > > > > > it will always get the first node to invalidate blocks on this
> > node.
> > > > > > String firstNodeId =
> > > recentInvalidateSets.keySet().iterator().next();
> > > > > >
> > > > > > TreeMap is a stored map, so, the ketSet is:
> > > > > > [1024388083-10.24.1.18-50010-1237686404482,
> > > > > > 1170985665-10.24.1.16-50010-1237686426395,
> > > > > > 2136798339-10.24.1.20-50010-1237686444430,
> > > > > > 489086185-10.24.1.14-50010-1237686416330,
> > > > > > 978762906-10.24.1.12-50010-1237686434530]
> > > > > >
> > > > > > So, the sequence of node list in recentInvalidateSets is:
> > > > > > [node4, node3, node5, node2, node1]
> > > > > >
> > > > > > So, every time in invalidateWorkForOneNode(), it will always
> > process
> > > > > node4
> > > > > > then node3, then node2 and then node1.
> > > > > >
> > > > > > My application is a HBase write-heavy application.
> > > > > > So, there is many blocks need invalidate in each datanode. So
> when
> > > each
> > > > > > 3000 milliseconds, at most, there is only two datanode is
> > processed.
> > > > > Since
> > > > > > the node1 is the last one in the TreeMap, it have no change to be
> > > > garbage
> > > > > > collected.
> > > > > >
> > > > > > I think HDFS namenode should fix this issue.
> > > > > >
> > > > > > Schubert
> > > > > >
> > > > > > On Thu, Mar 26, 2009 at 2:57 PM, Samuel Guo <gu...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > >> After a file is deleted, HDFS does not immediately reclaim the
> > > > available
> > > > > >> physical storage. It does so only lazily during garbage
> > collection.
> > > > When
> > > > > a
> > > > > >> file is deleted by the application, the master remove the file's
> > > > > metadata
> > > > > >> from *FSNamesystem* and logs the deletion immediately. And the
> > > file's
> > > > > >> deleted blocks information will be collected in each
> > > > > DataNodeDescriptor's
> > > > > >> *invalidateBlocks* set in Namenode. During the heartbeats
> between
> > NN
> > > > and
> > > > > >> DN,
> > > > > >> NN will scan the specified DN's DataNodeDescriptor's
> > > invalidateBlocks
> > > > > set,
> > > > > >> find the blocks to be deleted in DN and send a *DNA_INVALIDATE*
> > > > > >> BlockCommand
> > > > > >> to DN. And the *BlockScanner* thread running on DN will scan,
> find
> > > and
> > > > > >> delete these blocks after DN receives the *DNA_INVALIDATE*
> > > > BlockCommand.
> > > > > >>
> > > > > >> You can search *DNA_INVALIDATE* in DataNode.java and
> NameNode.java
> > > > > files,
> > > > > >> and find the logic of the garbage collection. Hope it will be
> > > helpful.
> > > > > >>
> > > > > >> On Thu, Mar 26, 2009 at 11:07 AM, schubert zhang <
> > zsongbo@gmail.com
> > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Tanks Andrew and Billy.
> > > > > >> > I think the subject of this mail thread is not appropriate, it
> > may
> > > > not
> > > > > >> be a
> > > > > >> > balance issue.
> > > > > >> > The problem seems the block deleting scheduler in HDFS.
> > > > > >> >
> > > > > >> > Last night(timezone:+8), I slow down my application, and this
> > > > morning,
> > > > > I
> > > > > >> > found almost all garbage blocks are deleted.
> > > > > >> > Here is the current blocks number of each datanode:
> > > > > >> > node1: 10651
> > > > > >> > node2: 10477
> > > > > >> > node3: 12185
> > > > > >> > node4: 11607
> > > > > >> > node5: 14000
> > > > > >> >
> > > > > >> > It seems fine.
> > > > > >> > But I want to study the code of HDFS and make clear the policy
> > of
> > > > > >> deleting
> > > > > >> > blocks on datanodes. If anyone in the hadoop community can
> give
> > > me
> > > > > some
> > > > > >> > advices?
> > > > > >> >
> > > > > >> > Schubert
> > > > > >> >
> > > > > >> > On Thu, Mar 26, 2009 at 7:55 AM, Andrew Purtell <
> > > > apurtell@apache.org>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> >
> > > > > >> > >
> > > > > >> > > > From: schubert zhang <zs...@gmail.com>
> > > > > >> > > > From another point of view, I think HBase cannot control
> to
> > > > > >> > > > delete blocks on which node, it would just delete files,
> and
> > > > > >> > > > HDFS delete blocks where the blocks locating.
> > > > > >> > >
> > > > > >> > > Yes, that is exactly correct.
> > > > > >> > >
> > > > > >> > > Best regards,
> > > > > >> > >
> > > > > >> > > - Andy
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by zsongbo <zs...@gmail.com>.
Hi Stack,
I have just send a email to your email address with the attached fix (since
I cannot attach big file here).
Please check it.
Someone have reported this issue in hadoop, and Hadoop's Hairong has made
patch on 0.21.0.
Regards,
Schubert
On Wed, Apr 1, 2009 at 3:44 PM, stack <st...@duboce.net> wrote:
> What was the fix?
> Thanks,
> St.Ack
>
> On Wed, Apr 1, 2009 at 6:54 AM, zsongbo <zs...@gmail.com> wrote:
>
> > Thanks stack. (i am schubert)
> > Yes, I have found the fix in 0.20.
> > And I just made a temporary fix based on branch-0.19 and it work fine.
> >
> > On Mon, Mar 30, 2009 at 7:16 PM, stack <st...@duboce.net> wrote:
> >
> > > Thanks for doing the digging Schubert. I agree, its an ugly issue.
> > > Another
> > > gentleman reported he'd tripped over the same thing in private mail.
> I'd
> > > suggest you file an issue against HADOOP HDFS and add the below (I"ve
> > > opened
> > > HBASE-1296 so we can track it in our project).
> > >
> > > Good stuff,
> > > St.Ack
> > >
> > > On Fri, Mar 27, 2009 at 9:50 AM, schubert zhang <zs...@gmail.com>
> > wrote:
> > >
> > > > Sorry "But I found the namenode is fair to process the invalidating
> for
> > > > each
> > > > datanode."
> > > > should be:
> > > >
> > > > "But I found the namenode is unfair to process the invalidating for
> > each
> > > > datanode."
> > > >
> > > >
> > > > On Fri, Mar 27, 2009 at 3:49 PM, schubert zhang <zs...@gmail.com>
> > > wrote:
> > > >
> > > > > Thanks Samuel,
> > > > > Your information is very correct.
> > > > > I have also read code about garbage collection of invalidating
> > blocks.
> > > > >
> > > > > But I found the namenode is fair to process the invalidating for
> each
> > > > > datanode.
> > > > > In my cluster, there are 5 datanode. The storage IDs are:
> > > > >
> > > > > node1: DS- 978762906-10.24.1.12-50010-1237686434530
> > > > > node2: DS- 489086185-10.24.1.14-50010-1237686416330
> > > > > node3: DS-1170985665-10.24.1.16-50010-1237686426395
> > > > > node4: DS-1024388083-10.24.1.18-50010-1237686404482
> > > > > node5: DS-2136798339-10.24.1.20-50010-1237686444430
> > > > > I know the storage ID is generated
> > > > > by
> > > org.apache.hadoop.hdfs.server.datanode.DataNode.setNewStorageID(...).
> > > > >
> > > > > In org.apache.hadoop.hdfs.server.namenode.FSNamesystem
> > > > >
> > > > > // Keeps a Collection for every named machine containing
> > > > > // blocks that have recently been invalidated and are thought to
> > live
> > > > > // on the machine in question.
> > > > > // Mapping: StorageID -> ArrayList<Block>
> > > > > //
> > > > > private Map<String, Collection<Block>> recentInvalidateSets =
> > > > > new TreeMap<String, Collection<Block>>();
> > > > >
> > > > > In
> > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.ReplicationMonitor
> > > > > This thread run in interval: replicationRecheckInterval=3000
> > > > milliseconds.
> > > > >
> > > > > Into computeDatanodeWork()
> > > > > nodesToProcess = 2.
> > > > >
> > > > > then into computeInvalidateWork(nodesToProcess)
> > > > > the for cycle will only exec 2 cycles.
> > > > >
> > > > > for each cycle, go into invalidateWorkForOneNode()
> > > > > it will always get the first node to invalidate blocks on this
> node.
> > > > > String firstNodeId =
> > recentInvalidateSets.keySet().iterator().next();
> > > > >
> > > > > TreeMap is a stored map, so, the ketSet is:
> > > > > [1024388083-10.24.1.18-50010-1237686404482,
> > > > > 1170985665-10.24.1.16-50010-1237686426395,
> > > > > 2136798339-10.24.1.20-50010-1237686444430,
> > > > > 489086185-10.24.1.14-50010-1237686416330,
> > > > > 978762906-10.24.1.12-50010-1237686434530]
> > > > >
> > > > > So, the sequence of node list in recentInvalidateSets is:
> > > > > [node4, node3, node5, node2, node1]
> > > > >
> > > > > So, every time in invalidateWorkForOneNode(), it will always
> process
> > > > node4
> > > > > then node3, then node2 and then node1.
> > > > >
> > > > > My application is a HBase write-heavy application.
> > > > > So, there is many blocks need invalidate in each datanode. So when
> > each
> > > > > 3000 milliseconds, at most, there is only two datanode is
> processed.
> > > > Since
> > > > > the node1 is the last one in the TreeMap, it have no change to be
> > > garbage
> > > > > collected.
> > > > >
> > > > > I think HDFS namenode should fix this issue.
> > > > >
> > > > > Schubert
> > > > >
> > > > > On Thu, Mar 26, 2009 at 2:57 PM, Samuel Guo <gu...@gmail.com>
> > > wrote:
> > > > >
> > > > >> After a file is deleted, HDFS does not immediately reclaim the
> > > available
> > > > >> physical storage. It does so only lazily during garbage
> collection.
> > > When
> > > > a
> > > > >> file is deleted by the application, the master remove the file's
> > > > metadata
> > > > >> from *FSNamesystem* and logs the deletion immediately. And the
> > file's
> > > > >> deleted blocks information will be collected in each
> > > > DataNodeDescriptor's
> > > > >> *invalidateBlocks* set in Namenode. During the heartbeats between
> NN
> > > and
> > > > >> DN,
> > > > >> NN will scan the specified DN's DataNodeDescriptor's
> > invalidateBlocks
> > > > set,
> > > > >> find the blocks to be deleted in DN and send a *DNA_INVALIDATE*
> > > > >> BlockCommand
> > > > >> to DN. And the *BlockScanner* thread running on DN will scan, find
> > and
> > > > >> delete these blocks after DN receives the *DNA_INVALIDATE*
> > > BlockCommand.
> > > > >>
> > > > >> You can search *DNA_INVALIDATE* in DataNode.java and NameNode.java
> > > > files,
> > > > >> and find the logic of the garbage collection. Hope it will be
> > helpful.
> > > > >>
> > > > >> On Thu, Mar 26, 2009 at 11:07 AM, schubert zhang <
> zsongbo@gmail.com
> > >
> > > > >> wrote:
> > > > >>
> > > > >> > Tanks Andrew and Billy.
> > > > >> > I think the subject of this mail thread is not appropriate, it
> may
> > > not
> > > > >> be a
> > > > >> > balance issue.
> > > > >> > The problem seems the block deleting scheduler in HDFS.
> > > > >> >
> > > > >> > Last night(timezone:+8), I slow down my application, and this
> > > morning,
> > > > I
> > > > >> > found almost all garbage blocks are deleted.
> > > > >> > Here is the current blocks number of each datanode:
> > > > >> > node1: 10651
> > > > >> > node2: 10477
> > > > >> > node3: 12185
> > > > >> > node4: 11607
> > > > >> > node5: 14000
> > > > >> >
> > > > >> > It seems fine.
> > > > >> > But I want to study the code of HDFS and make clear the policy
> of
> > > > >> deleting
> > > > >> > blocks on datanodes. If anyone in the hadoop community can give
> > me
> > > > some
> > > > >> > advices?
> > > > >> >
> > > > >> > Schubert
> > > > >> >
> > > > >> > On Thu, Mar 26, 2009 at 7:55 AM, Andrew Purtell <
> > > apurtell@apache.org>
> > > > >> > wrote:
> > > > >> >
> > > > >> >
> > > > >> > >
> > > > >> > > > From: schubert zhang <zs...@gmail.com>
> > > > >> > > > From another point of view, I think HBase cannot control to
> > > > >> > > > delete blocks on which node, it would just delete files, and
> > > > >> > > > HDFS delete blocks where the blocks locating.
> > > > >> > >
> > > > >> > > Yes, that is exactly correct.
> > > > >> > >
> > > > >> > > Best regards,
> > > > >> > >
> > > > >> > > - Andy
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by stack <st...@duboce.net>.
What was the fix?
Thanks,
St.Ack
On Wed, Apr 1, 2009 at 6:54 AM, zsongbo <zs...@gmail.com> wrote:
> Thanks stack. (i am schubert)
> Yes, I have found the fix in 0.20.
> And I just made a temporary fix based on branch-0.19 and it work fine.
>
> On Mon, Mar 30, 2009 at 7:16 PM, stack <st...@duboce.net> wrote:
>
> > Thanks for doing the digging Schubert. I agree, its an ugly issue.
> > Another
> > gentleman reported he'd tripped over the same thing in private mail. I'd
> > suggest you file an issue against HADOOP HDFS and add the below (I"ve
> > opened
> > HBASE-1296 so we can track it in our project).
> >
> > Good stuff,
> > St.Ack
> >
> > On Fri, Mar 27, 2009 at 9:50 AM, schubert zhang <zs...@gmail.com>
> wrote:
> >
> > > Sorry "But I found the namenode is fair to process the invalidating for
> > > each
> > > datanode."
> > > should be:
> > >
> > > "But I found the namenode is unfair to process the invalidating for
> each
> > > datanode."
> > >
> > >
> > > On Fri, Mar 27, 2009 at 3:49 PM, schubert zhang <zs...@gmail.com>
> > wrote:
> > >
> > > > Thanks Samuel,
> > > > Your information is very correct.
> > > > I have also read code about garbage collection of invalidating
> blocks.
> > > >
> > > > But I found the namenode is fair to process the invalidating for each
> > > > datanode.
> > > > In my cluster, there are 5 datanode. The storage IDs are:
> > > >
> > > > node1: DS- 978762906-10.24.1.12-50010-1237686434530
> > > > node2: DS- 489086185-10.24.1.14-50010-1237686416330
> > > > node3: DS-1170985665-10.24.1.16-50010-1237686426395
> > > > node4: DS-1024388083-10.24.1.18-50010-1237686404482
> > > > node5: DS-2136798339-10.24.1.20-50010-1237686444430
> > > > I know the storage ID is generated
> > > > by
> > org.apache.hadoop.hdfs.server.datanode.DataNode.setNewStorageID(...).
> > > >
> > > > In org.apache.hadoop.hdfs.server.namenode.FSNamesystem
> > > >
> > > > // Keeps a Collection for every named machine containing
> > > > // blocks that have recently been invalidated and are thought to
> live
> > > > // on the machine in question.
> > > > // Mapping: StorageID -> ArrayList<Block>
> > > > //
> > > > private Map<String, Collection<Block>> recentInvalidateSets =
> > > > new TreeMap<String, Collection<Block>>();
> > > >
> > > > In
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.ReplicationMonitor
> > > > This thread run in interval: replicationRecheckInterval=3000
> > > milliseconds.
> > > >
> > > > Into computeDatanodeWork()
> > > > nodesToProcess = 2.
> > > >
> > > > then into computeInvalidateWork(nodesToProcess)
> > > > the for cycle will only exec 2 cycles.
> > > >
> > > > for each cycle, go into invalidateWorkForOneNode()
> > > > it will always get the first node to invalidate blocks on this node.
> > > > String firstNodeId =
> recentInvalidateSets.keySet().iterator().next();
> > > >
> > > > TreeMap is a stored map, so, the ketSet is:
> > > > [1024388083-10.24.1.18-50010-1237686404482,
> > > > 1170985665-10.24.1.16-50010-1237686426395,
> > > > 2136798339-10.24.1.20-50010-1237686444430,
> > > > 489086185-10.24.1.14-50010-1237686416330,
> > > > 978762906-10.24.1.12-50010-1237686434530]
> > > >
> > > > So, the sequence of node list in recentInvalidateSets is:
> > > > [node4, node3, node5, node2, node1]
> > > >
> > > > So, every time in invalidateWorkForOneNode(), it will always process
> > > node4
> > > > then node3, then node2 and then node1.
> > > >
> > > > My application is a HBase write-heavy application.
> > > > So, there is many blocks need invalidate in each datanode. So when
> each
> > > > 3000 milliseconds, at most, there is only two datanode is processed.
> > > Since
> > > > the node1 is the last one in the TreeMap, it have no change to be
> > garbage
> > > > collected.
> > > >
> > > > I think HDFS namenode should fix this issue.
> > > >
> > > > Schubert
> > > >
> > > > On Thu, Mar 26, 2009 at 2:57 PM, Samuel Guo <gu...@gmail.com>
> > wrote:
> > > >
> > > >> After a file is deleted, HDFS does not immediately reclaim the
> > available
> > > >> physical storage. It does so only lazily during garbage collection.
> > When
> > > a
> > > >> file is deleted by the application, the master remove the file's
> > > metadata
> > > >> from *FSNamesystem* and logs the deletion immediately. And the
> file's
> > > >> deleted blocks information will be collected in each
> > > DataNodeDescriptor's
> > > >> *invalidateBlocks* set in Namenode. During the heartbeats between NN
> > and
> > > >> DN,
> > > >> NN will scan the specified DN's DataNodeDescriptor's
> invalidateBlocks
> > > set,
> > > >> find the blocks to be deleted in DN and send a *DNA_INVALIDATE*
> > > >> BlockCommand
> > > >> to DN. And the *BlockScanner* thread running on DN will scan, find
> and
> > > >> delete these blocks after DN receives the *DNA_INVALIDATE*
> > BlockCommand.
> > > >>
> > > >> You can search *DNA_INVALIDATE* in DataNode.java and NameNode.java
> > > files,
> > > >> and find the logic of the garbage collection. Hope it will be
> helpful.
> > > >>
> > > >> On Thu, Mar 26, 2009 at 11:07 AM, schubert zhang <zsongbo@gmail.com
> >
> > > >> wrote:
> > > >>
> > > >> > Tanks Andrew and Billy.
> > > >> > I think the subject of this mail thread is not appropriate, it may
> > not
> > > >> be a
> > > >> > balance issue.
> > > >> > The problem seems the block deleting scheduler in HDFS.
> > > >> >
> > > >> > Last night(timezone:+8), I slow down my application, and this
> > morning,
> > > I
> > > >> > found almost all garbage blocks are deleted.
> > > >> > Here is the current blocks number of each datanode:
> > > >> > node1: 10651
> > > >> > node2: 10477
> > > >> > node3: 12185
> > > >> > node4: 11607
> > > >> > node5: 14000
> > > >> >
> > > >> > It seems fine.
> > > >> > But I want to study the code of HDFS and make clear the policy of
> > > >> deleting
> > > >> > blocks on datanodes. If anyone in the hadoop community can give
> me
> > > some
> > > >> > advices?
> > > >> >
> > > >> > Schubert
> > > >> >
> > > >> > On Thu, Mar 26, 2009 at 7:55 AM, Andrew Purtell <
> > apurtell@apache.org>
> > > >> > wrote:
> > > >> >
> > > >> >
> > > >> > >
> > > >> > > > From: schubert zhang <zs...@gmail.com>
> > > >> > > > From another point of view, I think HBase cannot control to
> > > >> > > > delete blocks on which node, it would just delete files, and
> > > >> > > > HDFS delete blocks where the blocks locating.
> > > >> > >
> > > >> > > Yes, that is exactly correct.
> > > >> > >
> > > >> > > Best regards,
> > > >> > >
> > > >> > > - Andy
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by zsongbo <zs...@gmail.com>.
Thanks stack. (i am schubert)
Yes, I have found the fix in 0.20.
And I just made a temporary fix based on branch-0.19 and it work fine.
On Mon, Mar 30, 2009 at 7:16 PM, stack <st...@duboce.net> wrote:
> Thanks for doing the digging Schubert. I agree, its an ugly issue.
> Another
> gentleman reported he'd tripped over the same thing in private mail. I'd
> suggest you file an issue against HADOOP HDFS and add the below (I"ve
> opened
> HBASE-1296 so we can track it in our project).
>
> Good stuff,
> St.Ack
>
> On Fri, Mar 27, 2009 at 9:50 AM, schubert zhang <zs...@gmail.com> wrote:
>
> > Sorry "But I found the namenode is fair to process the invalidating for
> > each
> > datanode."
> > should be:
> >
> > "But I found the namenode is unfair to process the invalidating for each
> > datanode."
> >
> >
> > On Fri, Mar 27, 2009 at 3:49 PM, schubert zhang <zs...@gmail.com>
> wrote:
> >
> > > Thanks Samuel,
> > > Your information is very correct.
> > > I have also read code about garbage collection of invalidating blocks.
> > >
> > > But I found the namenode is fair to process the invalidating for each
> > > datanode.
> > > In my cluster, there are 5 datanode. The storage IDs are:
> > >
> > > node1: DS- 978762906-10.24.1.12-50010-1237686434530
> > > node2: DS- 489086185-10.24.1.14-50010-1237686416330
> > > node3: DS-1170985665-10.24.1.16-50010-1237686426395
> > > node4: DS-1024388083-10.24.1.18-50010-1237686404482
> > > node5: DS-2136798339-10.24.1.20-50010-1237686444430
> > > I know the storage ID is generated
> > > by
> org.apache.hadoop.hdfs.server.datanode.DataNode.setNewStorageID(...).
> > >
> > > In org.apache.hadoop.hdfs.server.namenode.FSNamesystem
> > >
> > > // Keeps a Collection for every named machine containing
> > > // blocks that have recently been invalidated and are thought to live
> > > // on the machine in question.
> > > // Mapping: StorageID -> ArrayList<Block>
> > > //
> > > private Map<String, Collection<Block>> recentInvalidateSets =
> > > new TreeMap<String, Collection<Block>>();
> > >
> > > In
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.ReplicationMonitor
> > > This thread run in interval: replicationRecheckInterval=3000
> > milliseconds.
> > >
> > > Into computeDatanodeWork()
> > > nodesToProcess = 2.
> > >
> > > then into computeInvalidateWork(nodesToProcess)
> > > the for cycle will only exec 2 cycles.
> > >
> > > for each cycle, go into invalidateWorkForOneNode()
> > > it will always get the first node to invalidate blocks on this node.
> > > String firstNodeId = recentInvalidateSets.keySet().iterator().next();
> > >
> > > TreeMap is a stored map, so, the ketSet is:
> > > [1024388083-10.24.1.18-50010-1237686404482,
> > > 1170985665-10.24.1.16-50010-1237686426395,
> > > 2136798339-10.24.1.20-50010-1237686444430,
> > > 489086185-10.24.1.14-50010-1237686416330,
> > > 978762906-10.24.1.12-50010-1237686434530]
> > >
> > > So, the sequence of node list in recentInvalidateSets is:
> > > [node4, node3, node5, node2, node1]
> > >
> > > So, every time in invalidateWorkForOneNode(), it will always process
> > node4
> > > then node3, then node2 and then node1.
> > >
> > > My application is a HBase write-heavy application.
> > > So, there is many blocks need invalidate in each datanode. So when each
> > > 3000 milliseconds, at most, there is only two datanode is processed.
> > Since
> > > the node1 is the last one in the TreeMap, it have no change to be
> garbage
> > > collected.
> > >
> > > I think HDFS namenode should fix this issue.
> > >
> > > Schubert
> > >
> > > On Thu, Mar 26, 2009 at 2:57 PM, Samuel Guo <gu...@gmail.com>
> wrote:
> > >
> > >> After a file is deleted, HDFS does not immediately reclaim the
> available
> > >> physical storage. It does so only lazily during garbage collection.
> When
> > a
> > >> file is deleted by the application, the master remove the file's
> > metadata
> > >> from *FSNamesystem* and logs the deletion immediately. And the file's
> > >> deleted blocks information will be collected in each
> > DataNodeDescriptor's
> > >> *invalidateBlocks* set in Namenode. During the heartbeats between NN
> and
> > >> DN,
> > >> NN will scan the specified DN's DataNodeDescriptor's invalidateBlocks
> > set,
> > >> find the blocks to be deleted in DN and send a *DNA_INVALIDATE*
> > >> BlockCommand
> > >> to DN. And the *BlockScanner* thread running on DN will scan, find and
> > >> delete these blocks after DN receives the *DNA_INVALIDATE*
> BlockCommand.
> > >>
> > >> You can search *DNA_INVALIDATE* in DataNode.java and NameNode.java
> > files,
> > >> and find the logic of the garbage collection. Hope it will be helpful.
> > >>
> > >> On Thu, Mar 26, 2009 at 11:07 AM, schubert zhang <zs...@gmail.com>
> > >> wrote:
> > >>
> > >> > Tanks Andrew and Billy.
> > >> > I think the subject of this mail thread is not appropriate, it may
> not
> > >> be a
> > >> > balance issue.
> > >> > The problem seems the block deleting scheduler in HDFS.
> > >> >
> > >> > Last night(timezone:+8), I slow down my application, and this
> morning,
> > I
> > >> > found almost all garbage blocks are deleted.
> > >> > Here is the current blocks number of each datanode:
> > >> > node1: 10651
> > >> > node2: 10477
> > >> > node3: 12185
> > >> > node4: 11607
> > >> > node5: 14000
> > >> >
> > >> > It seems fine.
> > >> > But I want to study the code of HDFS and make clear the policy of
> > >> deleting
> > >> > blocks on datanodes. If anyone in the hadoop community can give me
> > some
> > >> > advices?
> > >> >
> > >> > Schubert
> > >> >
> > >> > On Thu, Mar 26, 2009 at 7:55 AM, Andrew Purtell <
> apurtell@apache.org>
> > >> > wrote:
> > >> >
> > >> >
> > >> > >
> > >> > > > From: schubert zhang <zs...@gmail.com>
> > >> > > > From another point of view, I think HBase cannot control to
> > >> > > > delete blocks on which node, it would just delete files, and
> > >> > > > HDFS delete blocks where the blocks locating.
> > >> > >
> > >> > > Yes, that is exactly correct.
> > >> > >
> > >> > > Best regards,
> > >> > >
> > >> > > - Andy
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by stack <st...@duboce.net>.
Thanks for doing the digging Schubert. I agree, its an ugly issue. Another
gentleman reported he'd tripped over the same thing in private mail. I'd
suggest you file an issue against HADOOP HDFS and add the below (I"ve opened
HBASE-1296 so we can track it in our project).
Good stuff,
St.Ack
On Fri, Mar 27, 2009 at 9:50 AM, schubert zhang <zs...@gmail.com> wrote:
> Sorry "But I found the namenode is fair to process the invalidating for
> each
> datanode."
> should be:
>
> "But I found the namenode is unfair to process the invalidating for each
> datanode."
>
>
> On Fri, Mar 27, 2009 at 3:49 PM, schubert zhang <zs...@gmail.com> wrote:
>
> > Thanks Samuel,
> > Your information is very correct.
> > I have also read code about garbage collection of invalidating blocks.
> >
> > But I found the namenode is fair to process the invalidating for each
> > datanode.
> > In my cluster, there are 5 datanode. The storage IDs are:
> >
> > node1: DS- 978762906-10.24.1.12-50010-1237686434530
> > node2: DS- 489086185-10.24.1.14-50010-1237686416330
> > node3: DS-1170985665-10.24.1.16-50010-1237686426395
> > node4: DS-1024388083-10.24.1.18-50010-1237686404482
> > node5: DS-2136798339-10.24.1.20-50010-1237686444430
> > I know the storage ID is generated
> > by org.apache.hadoop.hdfs.server.datanode.DataNode.setNewStorageID(...).
> >
> > In org.apache.hadoop.hdfs.server.namenode.FSNamesystem
> >
> > // Keeps a Collection for every named machine containing
> > // blocks that have recently been invalidated and are thought to live
> > // on the machine in question.
> > // Mapping: StorageID -> ArrayList<Block>
> > //
> > private Map<String, Collection<Block>> recentInvalidateSets =
> > new TreeMap<String, Collection<Block>>();
> >
> > In org.apache.hadoop.hdfs.server.namenode.FSNamesystem.ReplicationMonitor
> > This thread run in interval: replicationRecheckInterval=3000
> milliseconds.
> >
> > Into computeDatanodeWork()
> > nodesToProcess = 2.
> >
> > then into computeInvalidateWork(nodesToProcess)
> > the for cycle will only exec 2 cycles.
> >
> > for each cycle, go into invalidateWorkForOneNode()
> > it will always get the first node to invalidate blocks on this node.
> > String firstNodeId = recentInvalidateSets.keySet().iterator().next();
> >
> > TreeMap is a stored map, so, the ketSet is:
> > [1024388083-10.24.1.18-50010-1237686404482,
> > 1170985665-10.24.1.16-50010-1237686426395,
> > 2136798339-10.24.1.20-50010-1237686444430,
> > 489086185-10.24.1.14-50010-1237686416330,
> > 978762906-10.24.1.12-50010-1237686434530]
> >
> > So, the sequence of node list in recentInvalidateSets is:
> > [node4, node3, node5, node2, node1]
> >
> > So, every time in invalidateWorkForOneNode(), it will always process
> node4
> > then node3, then node2 and then node1.
> >
> > My application is a HBase write-heavy application.
> > So, there is many blocks need invalidate in each datanode. So when each
> > 3000 milliseconds, at most, there is only two datanode is processed.
> Since
> > the node1 is the last one in the TreeMap, it have no change to be garbage
> > collected.
> >
> > I think HDFS namenode should fix this issue.
> >
> > Schubert
> >
> > On Thu, Mar 26, 2009 at 2:57 PM, Samuel Guo <gu...@gmail.com> wrote:
> >
> >> After a file is deleted, HDFS does not immediately reclaim the available
> >> physical storage. It does so only lazily during garbage collection. When
> a
> >> file is deleted by the application, the master remove the file's
> metadata
> >> from *FSNamesystem* and logs the deletion immediately. And the file's
> >> deleted blocks information will be collected in each
> DataNodeDescriptor's
> >> *invalidateBlocks* set in Namenode. During the heartbeats between NN and
> >> DN,
> >> NN will scan the specified DN's DataNodeDescriptor's invalidateBlocks
> set,
> >> find the blocks to be deleted in DN and send a *DNA_INVALIDATE*
> >> BlockCommand
> >> to DN. And the *BlockScanner* thread running on DN will scan, find and
> >> delete these blocks after DN receives the *DNA_INVALIDATE* BlockCommand.
> >>
> >> You can search *DNA_INVALIDATE* in DataNode.java and NameNode.java
> files,
> >> and find the logic of the garbage collection. Hope it will be helpful.
> >>
> >> On Thu, Mar 26, 2009 at 11:07 AM, schubert zhang <zs...@gmail.com>
> >> wrote:
> >>
> >> > Tanks Andrew and Billy.
> >> > I think the subject of this mail thread is not appropriate, it may not
> >> be a
> >> > balance issue.
> >> > The problem seems the block deleting scheduler in HDFS.
> >> >
> >> > Last night(timezone:+8), I slow down my application, and this morning,
> I
> >> > found almost all garbage blocks are deleted.
> >> > Here is the current blocks number of each datanode:
> >> > node1: 10651
> >> > node2: 10477
> >> > node3: 12185
> >> > node4: 11607
> >> > node5: 14000
> >> >
> >> > It seems fine.
> >> > But I want to study the code of HDFS and make clear the policy of
> >> deleting
> >> > blocks on datanodes. If anyone in the hadoop community can give me
> some
> >> > advices?
> >> >
> >> > Schubert
> >> >
> >> > On Thu, Mar 26, 2009 at 7:55 AM, Andrew Purtell <ap...@apache.org>
> >> > wrote:
> >> >
> >> >
> >> > >
> >> > > > From: schubert zhang <zs...@gmail.com>
> >> > > > From another point of view, I think HBase cannot control to
> >> > > > delete blocks on which node, it would just delete files, and
> >> > > > HDFS delete blocks where the blocks locating.
> >> > >
> >> > > Yes, that is exactly correct.
> >> > >
> >> > > Best regards,
> >> > >
> >> > > - Andy
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >>
> >
> >
>
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by schubert zhang <zs...@gmail.com>.
Sorry "But I found the namenode is fair to process the invalidating for each
datanode."
should be:
"But I found the namenode is unfair to process the invalidating for each
datanode."
On Fri, Mar 27, 2009 at 3:49 PM, schubert zhang <zs...@gmail.com> wrote:
> Thanks Samuel,
> Your information is very correct.
> I have also read code about garbage collection of invalidating blocks.
>
> But I found the namenode is fair to process the invalidating for each
> datanode.
> In my cluster, there are 5 datanode. The storage IDs are:
>
> node1: DS- 978762906-10.24.1.12-50010-1237686434530
> node2: DS- 489086185-10.24.1.14-50010-1237686416330
> node3: DS-1170985665-10.24.1.16-50010-1237686426395
> node4: DS-1024388083-10.24.1.18-50010-1237686404482
> node5: DS-2136798339-10.24.1.20-50010-1237686444430
> I know the storage ID is generated
> by org.apache.hadoop.hdfs.server.datanode.DataNode.setNewStorageID(...).
>
> In org.apache.hadoop.hdfs.server.namenode.FSNamesystem
>
> // Keeps a Collection for every named machine containing
> // blocks that have recently been invalidated and are thought to live
> // on the machine in question.
> // Mapping: StorageID -> ArrayList<Block>
> //
> private Map<String, Collection<Block>> recentInvalidateSets =
> new TreeMap<String, Collection<Block>>();
>
> In org.apache.hadoop.hdfs.server.namenode.FSNamesystem.ReplicationMonitor
> This thread run in interval: replicationRecheckInterval=3000 milliseconds.
>
> Into computeDatanodeWork()
> nodesToProcess = 2.
>
> then into computeInvalidateWork(nodesToProcess)
> the for cycle will only exec 2 cycles.
>
> for each cycle, go into invalidateWorkForOneNode()
> it will always get the first node to invalidate blocks on this node.
> String firstNodeId = recentInvalidateSets.keySet().iterator().next();
>
> TreeMap is a stored map, so, the ketSet is:
> [1024388083-10.24.1.18-50010-1237686404482,
> 1170985665-10.24.1.16-50010-1237686426395,
> 2136798339-10.24.1.20-50010-1237686444430,
> 489086185-10.24.1.14-50010-1237686416330,
> 978762906-10.24.1.12-50010-1237686434530]
>
> So, the sequence of node list in recentInvalidateSets is:
> [node4, node3, node5, node2, node1]
>
> So, every time in invalidateWorkForOneNode(), it will always process node4
> then node3, then node2 and then node1.
>
> My application is a HBase write-heavy application.
> So, there is many blocks need invalidate in each datanode. So when each
> 3000 milliseconds, at most, there is only two datanode is processed. Since
> the node1 is the last one in the TreeMap, it have no change to be garbage
> collected.
>
> I think HDFS namenode should fix this issue.
>
> Schubert
>
> On Thu, Mar 26, 2009 at 2:57 PM, Samuel Guo <gu...@gmail.com> wrote:
>
>> After a file is deleted, HDFS does not immediately reclaim the available
>> physical storage. It does so only lazily during garbage collection. When a
>> file is deleted by the application, the master remove the file's metadata
>> from *FSNamesystem* and logs the deletion immediately. And the file's
>> deleted blocks information will be collected in each DataNodeDescriptor's
>> *invalidateBlocks* set in Namenode. During the heartbeats between NN and
>> DN,
>> NN will scan the specified DN's DataNodeDescriptor's invalidateBlocks set,
>> find the blocks to be deleted in DN and send a *DNA_INVALIDATE*
>> BlockCommand
>> to DN. And the *BlockScanner* thread running on DN will scan, find and
>> delete these blocks after DN receives the *DNA_INVALIDATE* BlockCommand.
>>
>> You can search *DNA_INVALIDATE* in DataNode.java and NameNode.java files,
>> and find the logic of the garbage collection. Hope it will be helpful.
>>
>> On Thu, Mar 26, 2009 at 11:07 AM, schubert zhang <zs...@gmail.com>
>> wrote:
>>
>> > Tanks Andrew and Billy.
>> > I think the subject of this mail thread is not appropriate, it may not
>> be a
>> > balance issue.
>> > The problem seems the block deleting scheduler in HDFS.
>> >
>> > Last night(timezone:+8), I slow down my application, and this morning, I
>> > found almost all garbage blocks are deleted.
>> > Here is the current blocks number of each datanode:
>> > node1: 10651
>> > node2: 10477
>> > node3: 12185
>> > node4: 11607
>> > node5: 14000
>> >
>> > It seems fine.
>> > But I want to study the code of HDFS and make clear the policy of
>> deleting
>> > blocks on datanodes. If anyone in the hadoop community can give me some
>> > advices?
>> >
>> > Schubert
>> >
>> > On Thu, Mar 26, 2009 at 7:55 AM, Andrew Purtell <ap...@apache.org>
>> > wrote:
>> >
>> >
>> > >
>> > > > From: schubert zhang <zs...@gmail.com>
>> > > > From another point of view, I think HBase cannot control to
>> > > > delete blocks on which node, it would just delete files, and
>> > > > HDFS delete blocks where the blocks locating.
>> > >
>> > > Yes, that is exactly correct.
>> > >
>> > > Best regards,
>> > >
>> > > - Andy
>> > >
>> > >
>> > >
>> > >
>> > >
>> >
>>
>
>
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by schubert zhang <zs...@gmail.com>.
Sorry "But I found the namenode is fair to process the invalidating for each
datanode."
should be:
"But I found the namenode is unfair to process the invalidating for each
datanode."
On Fri, Mar 27, 2009 at 3:49 PM, schubert zhang <zs...@gmail.com> wrote:
> Thanks Samuel,
> Your information is very correct.
> I have also read code about garbage collection of invalidating blocks.
>
> But I found the namenode is fair to process the invalidating for each
> datanode.
> In my cluster, there are 5 datanode. The storage IDs are:
>
> node1: DS- 978762906-10.24.1.12-50010-1237686434530
> node2: DS- 489086185-10.24.1.14-50010-1237686416330
> node3: DS-1170985665-10.24.1.16-50010-1237686426395
> node4: DS-1024388083-10.24.1.18-50010-1237686404482
> node5: DS-2136798339-10.24.1.20-50010-1237686444430
> I know the storage ID is generated
> by org.apache.hadoop.hdfs.server.datanode.DataNode.setNewStorageID(...).
>
> In org.apache.hadoop.hdfs.server.namenode.FSNamesystem
>
> // Keeps a Collection for every named machine containing
> // blocks that have recently been invalidated and are thought to live
> // on the machine in question.
> // Mapping: StorageID -> ArrayList<Block>
> //
> private Map<String, Collection<Block>> recentInvalidateSets =
> new TreeMap<String, Collection<Block>>();
>
> In org.apache.hadoop.hdfs.server.namenode.FSNamesystem.ReplicationMonitor
> This thread run in interval: replicationRecheckInterval=3000 milliseconds.
>
> Into computeDatanodeWork()
> nodesToProcess = 2.
>
> then into computeInvalidateWork(nodesToProcess)
> the for cycle will only exec 2 cycles.
>
> for each cycle, go into invalidateWorkForOneNode()
> it will always get the first node to invalidate blocks on this node.
> String firstNodeId = recentInvalidateSets.keySet().iterator().next();
>
> TreeMap is a stored map, so, the ketSet is:
> [1024388083-10.24.1.18-50010-1237686404482,
> 1170985665-10.24.1.16-50010-1237686426395,
> 2136798339-10.24.1.20-50010-1237686444430,
> 489086185-10.24.1.14-50010-1237686416330,
> 978762906-10.24.1.12-50010-1237686434530]
>
> So, the sequence of node list in recentInvalidateSets is:
> [node4, node3, node5, node2, node1]
>
> So, every time in invalidateWorkForOneNode(), it will always process node4
> then node3, then node2 and then node1.
>
> My application is a HBase write-heavy application.
> So, there is many blocks need invalidate in each datanode. So when each
> 3000 milliseconds, at most, there is only two datanode is processed. Since
> the node1 is the last one in the TreeMap, it have no change to be garbage
> collected.
>
> I think HDFS namenode should fix this issue.
>
> Schubert
>
> On Thu, Mar 26, 2009 at 2:57 PM, Samuel Guo <gu...@gmail.com> wrote:
>
>> After a file is deleted, HDFS does not immediately reclaim the available
>> physical storage. It does so only lazily during garbage collection. When a
>> file is deleted by the application, the master remove the file's metadata
>> from *FSNamesystem* and logs the deletion immediately. And the file's
>> deleted blocks information will be collected in each DataNodeDescriptor's
>> *invalidateBlocks* set in Namenode. During the heartbeats between NN and
>> DN,
>> NN will scan the specified DN's DataNodeDescriptor's invalidateBlocks set,
>> find the blocks to be deleted in DN and send a *DNA_INVALIDATE*
>> BlockCommand
>> to DN. And the *BlockScanner* thread running on DN will scan, find and
>> delete these blocks after DN receives the *DNA_INVALIDATE* BlockCommand.
>>
>> You can search *DNA_INVALIDATE* in DataNode.java and NameNode.java files,
>> and find the logic of the garbage collection. Hope it will be helpful.
>>
>> On Thu, Mar 26, 2009 at 11:07 AM, schubert zhang <zs...@gmail.com>
>> wrote:
>>
>> > Tanks Andrew and Billy.
>> > I think the subject of this mail thread is not appropriate, it may not
>> be a
>> > balance issue.
>> > The problem seems the block deleting scheduler in HDFS.
>> >
>> > Last night(timezone:+8), I slow down my application, and this morning, I
>> > found almost all garbage blocks are deleted.
>> > Here is the current blocks number of each datanode:
>> > node1: 10651
>> > node2: 10477
>> > node3: 12185
>> > node4: 11607
>> > node5: 14000
>> >
>> > It seems fine.
>> > But I want to study the code of HDFS and make clear the policy of
>> deleting
>> > blocks on datanodes. If anyone in the hadoop community can give me some
>> > advices?
>> >
>> > Schubert
>> >
>> > On Thu, Mar 26, 2009 at 7:55 AM, Andrew Purtell <ap...@apache.org>
>> > wrote:
>> >
>> >
>> > >
>> > > > From: schubert zhang <zs...@gmail.com>
>> > > > From another point of view, I think HBase cannot control to
>> > > > delete blocks on which node, it would just delete files, and
>> > > > HDFS delete blocks where the blocks locating.
>> > >
>> > > Yes, that is exactly correct.
>> > >
>> > > Best regards,
>> > >
>> > > - Andy
>> > >
>> > >
>> > >
>> > >
>> > >
>> >
>>
>
>
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by schubert zhang <zs...@gmail.com>.
Thanks Samuel,
Your information is very correct.
I have also read code about garbage collection of invalidating blocks.
But I found the namenode is fair to process the invalidating for each
datanode.
In my cluster, there are 5 datanode. The storage IDs are:
node1: DS- 978762906-10.24.1.12-50010-1237686434530
node2: DS- 489086185-10.24.1.14-50010-1237686416330
node3: DS-1170985665-10.24.1.16-50010-1237686426395
node4: DS-1024388083-10.24.1.18-50010-1237686404482
node5: DS-2136798339-10.24.1.20-50010-1237686444430
I know the storage ID is generated
by org.apache.hadoop.hdfs.server.datanode.DataNode.setNewStorageID(...).
In org.apache.hadoop.hdfs.server.namenode.FSNamesystem
// Keeps a Collection for every named machine containing
// blocks that have recently been invalidated and are thought to live
// on the machine in question.
// Mapping: StorageID -> ArrayList<Block>
//
private Map<String, Collection<Block>> recentInvalidateSets =
new TreeMap<String, Collection<Block>>();
In org.apache.hadoop.hdfs.server.namenode.FSNamesystem.ReplicationMonitor
This thread run in interval: replicationRecheckInterval=3000 milliseconds.
Into computeDatanodeWork()
nodesToProcess = 2.
then into computeInvalidateWork(nodesToProcess)
the for cycle will only exec 2 cycles.
for each cycle, go into invalidateWorkForOneNode()
it will always get the first node to invalidate blocks on this node.
String firstNodeId = recentInvalidateSets.keySet().iterator().next();
TreeMap is a stored map, so, the ketSet is:
[1024388083-10.24.1.18-50010-1237686404482,
1170985665-10.24.1.16-50010-1237686426395,
2136798339-10.24.1.20-50010-1237686444430,
489086185-10.24.1.14-50010-1237686416330,
978762906-10.24.1.12-50010-1237686434530]
So, the sequence of node list in recentInvalidateSets is:
[node4, node3, node5, node2, node1]
So, every time in invalidateWorkForOneNode(), it will always process node4
then node3, then node2 and then node1.
My application is a HBase write-heavy application.
So, there is many blocks need invalidate in each datanode. So when each 3000
milliseconds, at most, there is only two datanode is processed. Since the
node1 is the last one in the TreeMap, it have no change to be garbage
collected.
I think HDFS namenode should fix this issue.
Schubert
On Thu, Mar 26, 2009 at 2:57 PM, Samuel Guo <gu...@gmail.com> wrote:
> After a file is deleted, HDFS does not immediately reclaim the available
> physical storage. It does so only lazily during garbage collection. When a
> file is deleted by the application, the master remove the file's metadata
> from *FSNamesystem* and logs the deletion immediately. And the file's
> deleted blocks information will be collected in each DataNodeDescriptor's
> *invalidateBlocks* set in Namenode. During the heartbeats between NN and
> DN,
> NN will scan the specified DN's DataNodeDescriptor's invalidateBlocks set,
> find the blocks to be deleted in DN and send a *DNA_INVALIDATE*
> BlockCommand
> to DN. And the *BlockScanner* thread running on DN will scan, find and
> delete these blocks after DN receives the *DNA_INVALIDATE* BlockCommand.
>
> You can search *DNA_INVALIDATE* in DataNode.java and NameNode.java files,
> and find the logic of the garbage collection. Hope it will be helpful.
>
> On Thu, Mar 26, 2009 at 11:07 AM, schubert zhang <zs...@gmail.com>
> wrote:
>
> > Tanks Andrew and Billy.
> > I think the subject of this mail thread is not appropriate, it may not be
> a
> > balance issue.
> > The problem seems the block deleting scheduler in HDFS.
> >
> > Last night(timezone:+8), I slow down my application, and this morning, I
> > found almost all garbage blocks are deleted.
> > Here is the current blocks number of each datanode:
> > node1: 10651
> > node2: 10477
> > node3: 12185
> > node4: 11607
> > node5: 14000
> >
> > It seems fine.
> > But I want to study the code of HDFS and make clear the policy of
> deleting
> > blocks on datanodes. If anyone in the hadoop community can give me some
> > advices?
> >
> > Schubert
> >
> > On Thu, Mar 26, 2009 at 7:55 AM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> >
> > >
> > > > From: schubert zhang <zs...@gmail.com>
> > > > From another point of view, I think HBase cannot control to
> > > > delete blocks on which node, it would just delete files, and
> > > > HDFS delete blocks where the blocks locating.
> > >
> > > Yes, that is exactly correct.
> > >
> > > Best regards,
> > >
> > > - Andy
> > >
> > >
> > >
> > >
> > >
> >
>
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by schubert zhang <zs...@gmail.com>.
Thanks Samuel,
Your information is very correct.
I have also read code about garbage collection of invalidating blocks.
But I found the namenode is fair to process the invalidating for each
datanode.
In my cluster, there are 5 datanode. The storage IDs are:
node1: DS- 978762906-10.24.1.12-50010-1237686434530
node2: DS- 489086185-10.24.1.14-50010-1237686416330
node3: DS-1170985665-10.24.1.16-50010-1237686426395
node4: DS-1024388083-10.24.1.18-50010-1237686404482
node5: DS-2136798339-10.24.1.20-50010-1237686444430
I know the storage ID is generated
by org.apache.hadoop.hdfs.server.datanode.DataNode.setNewStorageID(...).
In org.apache.hadoop.hdfs.server.namenode.FSNamesystem
// Keeps a Collection for every named machine containing
// blocks that have recently been invalidated and are thought to live
// on the machine in question.
// Mapping: StorageID -> ArrayList<Block>
//
private Map<String, Collection<Block>> recentInvalidateSets =
new TreeMap<String, Collection<Block>>();
In org.apache.hadoop.hdfs.server.namenode.FSNamesystem.ReplicationMonitor
This thread run in interval: replicationRecheckInterval=3000 milliseconds.
Into computeDatanodeWork()
nodesToProcess = 2.
then into computeInvalidateWork(nodesToProcess)
the for cycle will only exec 2 cycles.
for each cycle, go into invalidateWorkForOneNode()
it will always get the first node to invalidate blocks on this node.
String firstNodeId = recentInvalidateSets.keySet().iterator().next();
TreeMap is a stored map, so, the ketSet is:
[1024388083-10.24.1.18-50010-1237686404482,
1170985665-10.24.1.16-50010-1237686426395,
2136798339-10.24.1.20-50010-1237686444430,
489086185-10.24.1.14-50010-1237686416330,
978762906-10.24.1.12-50010-1237686434530]
So, the sequence of node list in recentInvalidateSets is:
[node4, node3, node5, node2, node1]
So, every time in invalidateWorkForOneNode(), it will always process node4
then node3, then node2 and then node1.
My application is a HBase write-heavy application.
So, there is many blocks need invalidate in each datanode. So when each 3000
milliseconds, at most, there is only two datanode is processed. Since the
node1 is the last one in the TreeMap, it have no change to be garbage
collected.
I think HDFS namenode should fix this issue.
Schubert
On Thu, Mar 26, 2009 at 2:57 PM, Samuel Guo <gu...@gmail.com> wrote:
> After a file is deleted, HDFS does not immediately reclaim the available
> physical storage. It does so only lazily during garbage collection. When a
> file is deleted by the application, the master remove the file's metadata
> from *FSNamesystem* and logs the deletion immediately. And the file's
> deleted blocks information will be collected in each DataNodeDescriptor's
> *invalidateBlocks* set in Namenode. During the heartbeats between NN and
> DN,
> NN will scan the specified DN's DataNodeDescriptor's invalidateBlocks set,
> find the blocks to be deleted in DN and send a *DNA_INVALIDATE*
> BlockCommand
> to DN. And the *BlockScanner* thread running on DN will scan, find and
> delete these blocks after DN receives the *DNA_INVALIDATE* BlockCommand.
>
> You can search *DNA_INVALIDATE* in DataNode.java and NameNode.java files,
> and find the logic of the garbage collection. Hope it will be helpful.
>
> On Thu, Mar 26, 2009 at 11:07 AM, schubert zhang <zs...@gmail.com>
> wrote:
>
> > Tanks Andrew and Billy.
> > I think the subject of this mail thread is not appropriate, it may not be
> a
> > balance issue.
> > The problem seems the block deleting scheduler in HDFS.
> >
> > Last night(timezone:+8), I slow down my application, and this morning, I
> > found almost all garbage blocks are deleted.
> > Here is the current blocks number of each datanode:
> > node1: 10651
> > node2: 10477
> > node3: 12185
> > node4: 11607
> > node5: 14000
> >
> > It seems fine.
> > But I want to study the code of HDFS and make clear the policy of
> deleting
> > blocks on datanodes. If anyone in the hadoop community can give me some
> > advices?
> >
> > Schubert
> >
> > On Thu, Mar 26, 2009 at 7:55 AM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> >
> > >
> > > > From: schubert zhang <zs...@gmail.com>
> > > > From another point of view, I think HBase cannot control to
> > > > delete blocks on which node, it would just delete files, and
> > > > HDFS delete blocks where the blocks locating.
> > >
> > > Yes, that is exactly correct.
> > >
> > > Best regards,
> > >
> > > - Andy
> > >
> > >
> > >
> > >
> > >
> >
>
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by Samuel Guo <gu...@gmail.com>.
After a file is deleted, HDFS does not immediately reclaim the available
physical storage. It does so only lazily during garbage collection. When a
file is deleted by the application, the master remove the file's metadata
from *FSNamesystem* and logs the deletion immediately. And the file's
deleted blocks information will be collected in each DataNodeDescriptor's
*invalidateBlocks* set in Namenode. During the heartbeats between NN and DN,
NN will scan the specified DN's DataNodeDescriptor's invalidateBlocks set,
find the blocks to be deleted in DN and send a *DNA_INVALIDATE* BlockCommand
to DN. And the *BlockScanner* thread running on DN will scan, find and
delete these blocks after DN receives the *DNA_INVALIDATE* BlockCommand.
You can search *DNA_INVALIDATE* in DataNode.java and NameNode.java files,
and find the logic of the garbage collection. Hope it will be helpful.
On Thu, Mar 26, 2009 at 11:07 AM, schubert zhang <zs...@gmail.com> wrote:
> Tanks Andrew and Billy.
> I think the subject of this mail thread is not appropriate, it may not be a
> balance issue.
> The problem seems the block deleting scheduler in HDFS.
>
> Last night(timezone:+8), I slow down my application, and this morning, I
> found almost all garbage blocks are deleted.
> Here is the current blocks number of each datanode:
> node1: 10651
> node2: 10477
> node3: 12185
> node4: 11607
> node5: 14000
>
> It seems fine.
> But I want to study the code of HDFS and make clear the policy of deleting
> blocks on datanodes. If anyone in the hadoop community can give me some
> advices?
>
> Schubert
>
> On Thu, Mar 26, 2009 at 7:55 AM, Andrew Purtell <ap...@apache.org>
> wrote:
>
>
> >
> > > From: schubert zhang <zs...@gmail.com>
> > > From another point of view, I think HBase cannot control to
> > > delete blocks on which node, it would just delete files, and
> > > HDFS delete blocks where the blocks locating.
> >
> > Yes, that is exactly correct.
> >
> > Best regards,
> >
> > - Andy
> >
> >
> >
> >
> >
>
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by schubert zhang <zs...@gmail.com>.
Tanks Andrew and Billy.
I think the subject of this mail thread is not appropriate, it may not be a
balance issue.
The problem seems the block deleting scheduler in HDFS.
Last night(timezone:+8), I slow down my application, and this morning, I
found almost all garbage blocks are deleted.
Here is the current blocks number of each datanode:
node1: 10651
node2: 10477
node3: 12185
node4: 11607
node5: 14000
It seems fine.
But I want to study the code of HDFS and make clear the policy of deleting
blocks on datanodes. If anyone in the hadoop community can give me some
advices?
Schubert
On Thu, Mar 26, 2009 at 7:55 AM, Andrew Purtell <ap...@apache.org> wrote:
>
> > From: schubert zhang <zs...@gmail.com>
> > From another point of view, I think HBase cannot control to
> > delete blocks on which node, it would just delete files, and
> > HDFS delete blocks where the blocks locating.
>
> Yes, that is exactly correct.
>
> Best regards,
>
> - Andy
>
>
>
>
>
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by Billy Pearson <sa...@pearsonwholesale.com>.
If you load your data on HDFS from node1 then it will always get more blocks
as hadoop saves one copy to local datanode then copies to others to meet
replication setting.
Same thing should be happening on hbase where ever the region is open and a
compaction happens the data should be written local datanode first then
copied.
As for the datanode always using cpu when ever there is compactions going on
if your replication is set to 3 then there is 3 datanodes effected by each
compaction happening.
Billy
"schubert zhang" <zs...@gmail.com> wrote in
message news:fa03480d0903250534u2ba56c2bm94c67d2e92a0e4e4@mail.gmail.com...
> Then, I stop my application (the application write to and read from
> HBase).
> After one hour, when I come back to see the status of HDFS, some blocks
> are
> deleted. Following is current status.
>
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 2956
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 2962
>
> node1: 464518
> node2: 42495
> node3: 7505
> node4: 7205
> node5: 7636
>
> On each node, the datanode process is busy (top).
>
> I want to know the reason of these phenomenons. Thanks.
>
> Schubert
>
> On Wed, Mar 25, 2009 at 6:37 PM, schubert zhang
> <zs...@gmail.com> wrote:
>
>> From another point of view, I think HBase cannot control to delete blocks
>> on which node, it would just delete files, and HDFS delete blocks where
>> the
>> blocks locating.
>>
>> Schubert
>>
>> On Wed, Mar 25, 2009 at 6:28 PM, schubert zhang
>> <zs...@gmail.com> wrote:
>>
>>> Thanks Ryan. Balancer may take a long time.
>>>
>>> The number of block are too different. But maybe it is caused by HBase
>>> not
>>> deleting garbage blocks on regionserver1 and regionserver2 and maybe
>>> others.
>>>
>>> We grep the logs of hadoop and find there is no any "deleting block" in
>>> node1 and node2.
>>>
>>> Following is the grep (grep -c "ask 10.24.1.1?:50010 to delete") result
>>> of
>>> hasoop logs:
>>>
>>> namenode:
>>>
>>> -----grep -c "ask 10.24.1.12:50010 to delete"-----node1
>>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to
>>> delete"
>>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
>>> 4754
>>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to
>>> delete"
>>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
>>> 1062
>>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to
>>> delete"
>>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>>> 0
>>>
>>> -----grep -c "ask 10.24.1.14:50010 to delete"-----node2
>>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to
>>> delete"
>>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>>> 1494
>>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to
>>> delete"
>>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
>>> 3305
>>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to
>>> delete"
>>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
>>> 3385
>>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to
>>> delete"
>>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>>> 1494
>>>
>>> -----grep -c "ask 10.24.1.16:50010 to delete"-----node3
>>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to
>>> delete"
>>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
>>> 8022
>>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to
>>> delete"
>>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
>>> 8238
>>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to
>>> delete"
>>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>>> 4302
>>>
>>> -----grep -c "ask 10.24.1.18:50010 to delete"-----node4
>>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to
>>> delete"
>>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
>>> 8591
>>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to
>>> delete"
>>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
>>> 9111
>>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to
>>> delete"
>>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>>> 5038
>>>
>>> -----grep -c "ask 10.24.1.20:50010 to delete"-----node5
>>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to
>>> delete"
>>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
>>> 3794
>>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to
>>> delete"
>>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
>>> 3946
>>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to
>>> delete"
>>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>>> 2989
>>>
>>> So, I think it may caused by HBase.
>>> I just grep the log of the zero "delete block" node. and find:
>>> [schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
>>> hadoop-schubert-datanode-nd1-rack0-cloud.log.2009-03-24
>>> 104739
>>> [schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
>>> hadoop-schubert-datanode-nd1-rack0-cloud.log.2009-03-23
>>> 465927
>>> [schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
>>> hadoop-schubert-datanode-nd1-rack0-cloud.log
>>> 0
>>>
>>>
>>>
>>>
>>> On Wed, Mar 25, 2009 at 5:14 PM, Ryan Rawson
>>> <ry...@gmail.com> wrote:
>>>
>>>> Try
>>>> hadoop/bin/start-balancer.sh
>>>>
>>>> HDFS doesnt auto-balance. Balancing in HDFS requires moving data
>>>> around,
>>>> whereas balancing in HBase just means opening a file on a different
>>>> machine.
>>>>
>>>> On Wed, Mar 25, 2009 at 2:12 AM, schubert zhang
>>>> <zs...@gmail.com>
>>>> wrote:
>>>>
>>>> > Hi all,
>>>> > I am using hbase-0.19.1 and hadoop-0.19.
>>>> > My cluster have 5+1 nodes, and there are about 512 regions in HBase
>>>> (256MB
>>>> > per region).
>>>> >
>>>> > But I found the blocks in HDFS is very unbalanced. Following is the
>>>> status
>>>> > from HDFS web GUI.
>>>> >
>>>> > (Node: I don't know if this mailing list can display html!)
>>>> >
>>>> > HDFS blocks:
>>>> > node1 509036 blocks
>>>> > node2 157937 blocks
>>>> > node3 15783 blocks
>>>> > node4 15117 blocks
>>>> > node5 20158 blocks
>>>> >
>>>> > But my HBase regions are very balanced.
>>>> > node1 88 regions
>>>> > node2 108 regions
>>>> > node3 111 regions
>>>> > node4 102 regions
>>>> > node5 105 regions
>>>> >
>>>> >
>>>> >
>>>> > NodeLast
>>>> > ContactAdmin StateConfigured
>>>> > Capacity (GB)Used
>>>> > (GB)Non DFS
>>>> > Used (GB)Remaining
>>>> > (GB)Used
>>>> > (%)Used
>>>> > (%)Remaining
>>>> > (%)Blocksnd1-rack0-cloud<
>>>> >
>>>> http://nd1-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>>>> > >
>>>> > 0In Service822.8578.6743.28200.8670.3324.41509036nd2-rack0-cloud<
>>>> >
>>>> http://nd2-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>>>> > >
>>>> > 0In Service822.8190.0242.96589.8223.0971.68157937nd3-rack0-cloud<
>>>> >
>>>> http://nd3-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>>>> > >
>>>> > 0In Service822.851.9542.61728.246.3188.5115783nd4-rack0-cloud<
>>>> >
>>>> http://nd4-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>>>> > >
>>>> > 6In Service822.846.1942.84733.775.6189.1815117nd5-rack0-cloud<
>>>> >
>>>> http://nd5-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>>>> > >
>>>> > 1In Service1215.6152.3762.911100.324.3190.5220158
>>>> >
>>>> >
>>>> > But my HBase regions are very balanced.
>>>> >
>>>> > AddressStart CodeLoadnd1-rack0-cloud:60020 <
>>>> http://nd1-rack0-cloud:60030/>
>>>> > 1237967027050requests=383, regions=88, usedHeap=978, maxHeap=1991
>>>> > nd2-rack0-cloud:60020 <http://nd2-rack0-cloud:60030/
>>>> > >1237788871362requests=422,
>>>> > regions=108, usedHeap=1433,
>>>> > maxHeap=1991nd3-rack0-cloud:60020<http://nd3-rack0-cloud:60030/>
>>>> > 1237788881667requests=962, regions=111, usedHeap=1534, maxHeap=1991
>>>> > nd4-rack0-cloud:60020 <http://nd4-rack0-cloud:60030/
>>>> > >1237788859541requests=369,
>>>> > regions=102, usedHeap=1059,
>>>> > maxHeap=1991nd5-rack0-cloud:60020<http://nd5-rack0-cloud:60030/>
>>>> > 1237788899331requests=384, regions=105, usedHeap=1535,
>>>> > maxHeap=1983Total:servers:
>>>> > 5 requests=2520, regions=514
>>>> >
>>>>
>>>
>>>
>>
>
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by schubert zhang <zs...@gmail.com>.
Then, I stop my application (the application write to and read from HBase).
After one hour, when I come back to see the status of HDFS, some blocks are
deleted. Following is current status.
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
2956
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
2962
node1: 464518
node2: 42495
node3: 7505
node4: 7205
node5: 7636
On each node, the datanode process is busy (top).
I want to know the reason of these phenomenons. Thanks.
Schubert
On Wed, Mar 25, 2009 at 6:37 PM, schubert zhang <zs...@gmail.com> wrote:
> From another point of view, I think HBase cannot control to delete blocks
> on which node, it would just delete files, and HDFS delete blocks where the
> blocks locating.
>
> Schubert
>
> On Wed, Mar 25, 2009 at 6:28 PM, schubert zhang <zs...@gmail.com> wrote:
>
>> Thanks Ryan. Balancer may take a long time.
>>
>> The number of block are too different. But maybe it is caused by HBase not
>> deleting garbage blocks on regionserver1 and regionserver2 and maybe others.
>>
>> We grep the logs of hadoop and find there is no any "deleting block" in
>> node1 and node2.
>>
>> Following is the grep (grep -c "ask 10.24.1.1?:50010 to delete") result of
>> hasoop logs:
>>
>> namenode:
>>
>> -----grep -c "ask 10.24.1.12:50010 to delete"-----node1
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
>> 4754
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
>> 1062
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>> 0
>>
>> -----grep -c "ask 10.24.1.14:50010 to delete"-----node2
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>> 1494
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
>> 3305
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
>> 3385
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>> 1494
>>
>> -----grep -c "ask 10.24.1.16:50010 to delete"-----node3
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
>> 8022
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
>> 8238
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>> 4302
>>
>> -----grep -c "ask 10.24.1.18:50010 to delete"-----node4
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
>> 8591
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
>> 9111
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>> 5038
>>
>> -----grep -c "ask 10.24.1.20:50010 to delete"-----node5
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
>> 3794
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
>> 3946
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>> 2989
>>
>> So, I think it may caused by HBase.
>> I just grep the log of the zero "delete block" node. and find:
>> [schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
>> hadoop-schubert-datanode-nd1-rack0-cloud.log.2009-03-24
>> 104739
>> [schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
>> hadoop-schubert-datanode-nd1-rack0-cloud.log.2009-03-23
>> 465927
>> [schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
>> hadoop-schubert-datanode-nd1-rack0-cloud.log
>> 0
>>
>>
>>
>>
>> On Wed, Mar 25, 2009 at 5:14 PM, Ryan Rawson <ry...@gmail.com> wrote:
>>
>>> Try
>>> hadoop/bin/start-balancer.sh
>>>
>>> HDFS doesnt auto-balance. Balancing in HDFS requires moving data around,
>>> whereas balancing in HBase just means opening a file on a different
>>> machine.
>>>
>>> On Wed, Mar 25, 2009 at 2:12 AM, schubert zhang <zs...@gmail.com>
>>> wrote:
>>>
>>> > Hi all,
>>> > I am using hbase-0.19.1 and hadoop-0.19.
>>> > My cluster have 5+1 nodes, and there are about 512 regions in HBase
>>> (256MB
>>> > per region).
>>> >
>>> > But I found the blocks in HDFS is very unbalanced. Following is the
>>> status
>>> > from HDFS web GUI.
>>> >
>>> > (Node: I don't know if this mailing list can display html!)
>>> >
>>> > HDFS blocks:
>>> > node1 509036 blocks
>>> > node2 157937 blocks
>>> > node3 15783 blocks
>>> > node4 15117 blocks
>>> > node5 20158 blocks
>>> >
>>> > But my HBase regions are very balanced.
>>> > node1 88 regions
>>> > node2 108 regions
>>> > node3 111 regions
>>> > node4 102 regions
>>> > node5 105 regions
>>> >
>>> >
>>> >
>>> > NodeLast
>>> > ContactAdmin StateConfigured
>>> > Capacity (GB)Used
>>> > (GB)Non DFS
>>> > Used (GB)Remaining
>>> > (GB)Used
>>> > (%)Used
>>> > (%)Remaining
>>> > (%)Blocksnd1-rack0-cloud<
>>> >
>>> http://nd1-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>>> > >
>>> > 0In Service822.8578.6743.28200.8670.3324.41509036nd2-rack0-cloud<
>>> >
>>> http://nd2-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>>> > >
>>> > 0In Service822.8190.0242.96589.8223.0971.68157937nd3-rack0-cloud<
>>> >
>>> http://nd3-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>>> > >
>>> > 0In Service822.851.9542.61728.246.3188.5115783nd4-rack0-cloud<
>>> >
>>> http://nd4-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>>> > >
>>> > 6In Service822.846.1942.84733.775.6189.1815117nd5-rack0-cloud<
>>> >
>>> http://nd5-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>>> > >
>>> > 1In Service1215.6152.3762.911100.324.3190.5220158
>>> >
>>> >
>>> > But my HBase regions are very balanced.
>>> >
>>> > AddressStart CodeLoadnd1-rack0-cloud:60020 <
>>> http://nd1-rack0-cloud:60030/>
>>> > 1237967027050requests=383, regions=88, usedHeap=978, maxHeap=1991
>>> > nd2-rack0-cloud:60020 <http://nd2-rack0-cloud:60030/
>>> > >1237788871362requests=422,
>>> > regions=108, usedHeap=1433,
>>> > maxHeap=1991nd3-rack0-cloud:60020<http://nd3-rack0-cloud:60030/>
>>> > 1237788881667requests=962, regions=111, usedHeap=1534, maxHeap=1991
>>> > nd4-rack0-cloud:60020 <http://nd4-rack0-cloud:60030/
>>> > >1237788859541requests=369,
>>> > regions=102, usedHeap=1059,
>>> > maxHeap=1991nd5-rack0-cloud:60020<http://nd5-rack0-cloud:60030/>
>>> > 1237788899331requests=384, regions=105, usedHeap=1535,
>>> > maxHeap=1983Total:servers:
>>> > 5 requests=2520, regions=514
>>> >
>>>
>>
>>
>
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by schubert zhang <zs...@gmail.com>.
Then, I stop my application (the application write to and read from HBase).
After one hour, when I come back to see the status of HDFS, some blocks are
deleted. Following is current status.
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
2956
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
2962
node1: 464518
node2: 42495
node3: 7505
node4: 7205
node5: 7636
On each node, the datanode process is busy (top).
I want to know the reason of these phenomenons. Thanks.
Schubert
On Wed, Mar 25, 2009 at 6:37 PM, schubert zhang <zs...@gmail.com> wrote:
> From another point of view, I think HBase cannot control to delete blocks
> on which node, it would just delete files, and HDFS delete blocks where the
> blocks locating.
>
> Schubert
>
> On Wed, Mar 25, 2009 at 6:28 PM, schubert zhang <zs...@gmail.com> wrote:
>
>> Thanks Ryan. Balancer may take a long time.
>>
>> The number of block are too different. But maybe it is caused by HBase not
>> deleting garbage blocks on regionserver1 and regionserver2 and maybe others.
>>
>> We grep the logs of hadoop and find there is no any "deleting block" in
>> node1 and node2.
>>
>> Following is the grep (grep -c "ask 10.24.1.1?:50010 to delete") result of
>> hasoop logs:
>>
>> namenode:
>>
>> -----grep -c "ask 10.24.1.12:50010 to delete"-----node1
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
>> 4754
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
>> 1062
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>> 0
>>
>> -----grep -c "ask 10.24.1.14:50010 to delete"-----node2
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>> 1494
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
>> 3305
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
>> 3385
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>> 1494
>>
>> -----grep -c "ask 10.24.1.16:50010 to delete"-----node3
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
>> 8022
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
>> 8238
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>> 4302
>>
>> -----grep -c "ask 10.24.1.18:50010 to delete"-----node4
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
>> 8591
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
>> 9111
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>> 5038
>>
>> -----grep -c "ask 10.24.1.20:50010 to delete"-----node5
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
>> 3794
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
>> 3946
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>> 2989
>>
>> So, I think it may caused by HBase.
>> I just grep the log of the zero "delete block" node. and find:
>> [schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
>> hadoop-schubert-datanode-nd1-rack0-cloud.log.2009-03-24
>> 104739
>> [schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
>> hadoop-schubert-datanode-nd1-rack0-cloud.log.2009-03-23
>> 465927
>> [schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
>> hadoop-schubert-datanode-nd1-rack0-cloud.log
>> 0
>>
>>
>>
>>
>> On Wed, Mar 25, 2009 at 5:14 PM, Ryan Rawson <ry...@gmail.com> wrote:
>>
>>> Try
>>> hadoop/bin/start-balancer.sh
>>>
>>> HDFS doesnt auto-balance. Balancing in HDFS requires moving data around,
>>> whereas balancing in HBase just means opening a file on a different
>>> machine.
>>>
>>> On Wed, Mar 25, 2009 at 2:12 AM, schubert zhang <zs...@gmail.com>
>>> wrote:
>>>
>>> > Hi all,
>>> > I am using hbase-0.19.1 and hadoop-0.19.
>>> > My cluster have 5+1 nodes, and there are about 512 regions in HBase
>>> (256MB
>>> > per region).
>>> >
>>> > But I found the blocks in HDFS is very unbalanced. Following is the
>>> status
>>> > from HDFS web GUI.
>>> >
>>> > (Node: I don't know if this mailing list can display html!)
>>> >
>>> > HDFS blocks:
>>> > node1 509036 blocks
>>> > node2 157937 blocks
>>> > node3 15783 blocks
>>> > node4 15117 blocks
>>> > node5 20158 blocks
>>> >
>>> > But my HBase regions are very balanced.
>>> > node1 88 regions
>>> > node2 108 regions
>>> > node3 111 regions
>>> > node4 102 regions
>>> > node5 105 regions
>>> >
>>> >
>>> >
>>> > NodeLast
>>> > ContactAdmin StateConfigured
>>> > Capacity (GB)Used
>>> > (GB)Non DFS
>>> > Used (GB)Remaining
>>> > (GB)Used
>>> > (%)Used
>>> > (%)Remaining
>>> > (%)Blocksnd1-rack0-cloud<
>>> >
>>> http://nd1-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>>> > >
>>> > 0In Service822.8578.6743.28200.8670.3324.41509036nd2-rack0-cloud<
>>> >
>>> http://nd2-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>>> > >
>>> > 0In Service822.8190.0242.96589.8223.0971.68157937nd3-rack0-cloud<
>>> >
>>> http://nd3-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>>> > >
>>> > 0In Service822.851.9542.61728.246.3188.5115783nd4-rack0-cloud<
>>> >
>>> http://nd4-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>>> > >
>>> > 6In Service822.846.1942.84733.775.6189.1815117nd5-rack0-cloud<
>>> >
>>> http://nd5-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>>> > >
>>> > 1In Service1215.6152.3762.911100.324.3190.5220158
>>> >
>>> >
>>> > But my HBase regions are very balanced.
>>> >
>>> > AddressStart CodeLoadnd1-rack0-cloud:60020 <
>>> http://nd1-rack0-cloud:60030/>
>>> > 1237967027050requests=383, regions=88, usedHeap=978, maxHeap=1991
>>> > nd2-rack0-cloud:60020 <http://nd2-rack0-cloud:60030/
>>> > >1237788871362requests=422,
>>> > regions=108, usedHeap=1433,
>>> > maxHeap=1991nd3-rack0-cloud:60020<http://nd3-rack0-cloud:60030/>
>>> > 1237788881667requests=962, regions=111, usedHeap=1534, maxHeap=1991
>>> > nd4-rack0-cloud:60020 <http://nd4-rack0-cloud:60030/
>>> > >1237788859541requests=369,
>>> > regions=102, usedHeap=1059,
>>> > maxHeap=1991nd5-rack0-cloud:60020<http://nd5-rack0-cloud:60030/>
>>> > 1237788899331requests=384, regions=105, usedHeap=1535,
>>> > maxHeap=1983Total:servers:
>>> > 5 requests=2520, regions=514
>>> >
>>>
>>
>>
>
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by schubert zhang <zs...@gmail.com>.
Tanks Andrew and Billy.
I think the subject of this mail thread is not appropriate, it may not be a
balance issue.
The problem seems the block deleting scheduler in HDFS.
Last night(timezone:+8), I slow down my application, and this morning, I
found almost all garbage blocks are deleted.
Here is the current blocks number of each datanode:
node1: 10651
node2: 10477
node3: 12185
node4: 11607
node5: 14000
It seems fine.
But I want to study the code of HDFS and make clear the policy of deleting
blocks on datanodes. If anyone in the hadoop community can give me some
advices?
Schubert
On Thu, Mar 26, 2009 at 7:55 AM, Andrew Purtell <ap...@apache.org> wrote:
>
> > From: schubert zhang <zs...@gmail.com>
> > From another point of view, I think HBase cannot control to
> > delete blocks on which node, it would just delete files, and
> > HDFS delete blocks where the blocks locating.
>
> Yes, that is exactly correct.
>
> Best regards,
>
> - Andy
>
>
>
>
>
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by Andrew Purtell <ap...@apache.org>.
> From: schubert zhang <zs...@gmail.com>
> From another point of view, I think HBase cannot control to
> delete blocks on which node, it would just delete files, and
> HDFS delete blocks where the blocks locating.
Yes, that is exactly correct.
Best regards,
- Andy
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by Andrew Purtell <ap...@apache.org>.
> From: schubert zhang <zs...@gmail.com>
> From another point of view, I think HBase cannot control to
> delete blocks on which node, it would just delete files, and
> HDFS delete blocks where the blocks locating.
Yes, that is exactly correct.
Best regards,
- Andy
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by schubert zhang <zs...@gmail.com>.
>From another point of view, I think HBase cannot control to delete blocks on
which node, it would just delete files, and HDFS delete blocks where the
blocks locating.
Schubert
On Wed, Mar 25, 2009 at 6:28 PM, schubert zhang <zs...@gmail.com> wrote:
> Thanks Ryan. Balancer may take a long time.
>
> The number of block are too different. But maybe it is caused by HBase not
> deleting garbage blocks on regionserver1 and regionserver2 and maybe others.
>
> We grep the logs of hadoop and find there is no any "deleting block" in
> node1 and node2.
>
> Following is the grep (grep -c "ask 10.24.1.1?:50010 to delete") result of
> hasoop logs:
>
> namenode:
>
> -----grep -c "ask 10.24.1.12:50010 to delete"-----node1
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
> 4754
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
> 1062
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 0
>
> -----grep -c "ask 10.24.1.14:50010 to delete"-----node2
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 1494
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
> 3305
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
> 3385
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 1494
>
> -----grep -c "ask 10.24.1.16:50010 to delete"-----node3
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
> 8022
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
> 8238
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 4302
>
> -----grep -c "ask 10.24.1.18:50010 to delete"-----node4
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
> 8591
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
> 9111
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 5038
>
> -----grep -c "ask 10.24.1.20:50010 to delete"-----node5
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
> 3794
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
> 3946
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 2989
>
> So, I think it may caused by HBase.
> I just grep the log of the zero "delete block" node. and find:
> [schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
> hadoop-schubert-datanode-nd1-rack0-cloud.log.2009-03-24
> 104739
> [schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
> hadoop-schubert-datanode-nd1-rack0-cloud.log.2009-03-23
> 465927
> [schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
> hadoop-schubert-datanode-nd1-rack0-cloud.log
> 0
>
>
>
>
> On Wed, Mar 25, 2009 at 5:14 PM, Ryan Rawson <ry...@gmail.com> wrote:
>
>> Try
>> hadoop/bin/start-balancer.sh
>>
>> HDFS doesnt auto-balance. Balancing in HDFS requires moving data around,
>> whereas balancing in HBase just means opening a file on a different
>> machine.
>>
>> On Wed, Mar 25, 2009 at 2:12 AM, schubert zhang <zs...@gmail.com>
>> wrote:
>>
>> > Hi all,
>> > I am using hbase-0.19.1 and hadoop-0.19.
>> > My cluster have 5+1 nodes, and there are about 512 regions in HBase
>> (256MB
>> > per region).
>> >
>> > But I found the blocks in HDFS is very unbalanced. Following is the
>> status
>> > from HDFS web GUI.
>> >
>> > (Node: I don't know if this mailing list can display html!)
>> >
>> > HDFS blocks:
>> > node1 509036 blocks
>> > node2 157937 blocks
>> > node3 15783 blocks
>> > node4 15117 blocks
>> > node5 20158 blocks
>> >
>> > But my HBase regions are very balanced.
>> > node1 88 regions
>> > node2 108 regions
>> > node3 111 regions
>> > node4 102 regions
>> > node5 105 regions
>> >
>> >
>> >
>> > NodeLast
>> > ContactAdmin StateConfigured
>> > Capacity (GB)Used
>> > (GB)Non DFS
>> > Used (GB)Remaining
>> > (GB)Used
>> > (%)Used
>> > (%)Remaining
>> > (%)Blocksnd1-rack0-cloud<
>> >
>> http://nd1-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>> > >
>> > 0In Service822.8578.6743.28200.8670.3324.41509036nd2-rack0-cloud<
>> >
>> http://nd2-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>> > >
>> > 0In Service822.8190.0242.96589.8223.0971.68157937nd3-rack0-cloud<
>> >
>> http://nd3-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>> > >
>> > 0In Service822.851.9542.61728.246.3188.5115783nd4-rack0-cloud<
>> >
>> http://nd4-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>> > >
>> > 6In Service822.846.1942.84733.775.6189.1815117nd5-rack0-cloud<
>> >
>> http://nd5-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>> > >
>> > 1In Service1215.6152.3762.911100.324.3190.5220158
>> >
>> >
>> > But my HBase regions are very balanced.
>> >
>> > AddressStart CodeLoadnd1-rack0-cloud:60020 <
>> http://nd1-rack0-cloud:60030/>
>> > 1237967027050requests=383, regions=88, usedHeap=978, maxHeap=1991
>> > nd2-rack0-cloud:60020 <http://nd2-rack0-cloud:60030/
>> > >1237788871362requests=422,
>> > regions=108, usedHeap=1433,
>> > maxHeap=1991nd3-rack0-cloud:60020<http://nd3-rack0-cloud:60030/>
>> > 1237788881667requests=962, regions=111, usedHeap=1534, maxHeap=1991
>> > nd4-rack0-cloud:60020 <http://nd4-rack0-cloud:60030/
>> > >1237788859541requests=369,
>> > regions=102, usedHeap=1059,
>> > maxHeap=1991nd5-rack0-cloud:60020<http://nd5-rack0-cloud:60030/>
>> > 1237788899331requests=384, regions=105, usedHeap=1535,
>> > maxHeap=1983Total:servers:
>> > 5 requests=2520, regions=514
>> >
>>
>
>
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by schubert zhang <zs...@gmail.com>.
>From another point of view, I think HBase cannot control to delete blocks on
which node, it would just delete files, and HDFS delete blocks where the
blocks locating.
Schubert
On Wed, Mar 25, 2009 at 6:28 PM, schubert zhang <zs...@gmail.com> wrote:
> Thanks Ryan. Balancer may take a long time.
>
> The number of block are too different. But maybe it is caused by HBase not
> deleting garbage blocks on regionserver1 and regionserver2 and maybe others.
>
> We grep the logs of hadoop and find there is no any "deleting block" in
> node1 and node2.
>
> Following is the grep (grep -c "ask 10.24.1.1?:50010 to delete") result of
> hasoop logs:
>
> namenode:
>
> -----grep -c "ask 10.24.1.12:50010 to delete"-----node1
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
> 4754
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
> 1062
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 0
>
> -----grep -c "ask 10.24.1.14:50010 to delete"-----node2
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 1494
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
> 3305
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
> 3385
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 1494
>
> -----grep -c "ask 10.24.1.16:50010 to delete"-----node3
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
> 8022
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
> 8238
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 4302
>
> -----grep -c "ask 10.24.1.18:50010 to delete"-----node4
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
> 8591
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
> 9111
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 5038
>
> -----grep -c "ask 10.24.1.20:50010 to delete"-----node5
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
> 3794
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
> 3946
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 2989
>
> So, I think it may caused by HBase.
> I just grep the log of the zero "delete block" node. and find:
> [schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
> hadoop-schubert-datanode-nd1-rack0-cloud.log.2009-03-24
> 104739
> [schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
> hadoop-schubert-datanode-nd1-rack0-cloud.log.2009-03-23
> 465927
> [schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
> hadoop-schubert-datanode-nd1-rack0-cloud.log
> 0
>
>
>
>
> On Wed, Mar 25, 2009 at 5:14 PM, Ryan Rawson <ry...@gmail.com> wrote:
>
>> Try
>> hadoop/bin/start-balancer.sh
>>
>> HDFS doesnt auto-balance. Balancing in HDFS requires moving data around,
>> whereas balancing in HBase just means opening a file on a different
>> machine.
>>
>> On Wed, Mar 25, 2009 at 2:12 AM, schubert zhang <zs...@gmail.com>
>> wrote:
>>
>> > Hi all,
>> > I am using hbase-0.19.1 and hadoop-0.19.
>> > My cluster have 5+1 nodes, and there are about 512 regions in HBase
>> (256MB
>> > per region).
>> >
>> > But I found the blocks in HDFS is very unbalanced. Following is the
>> status
>> > from HDFS web GUI.
>> >
>> > (Node: I don't know if this mailing list can display html!)
>> >
>> > HDFS blocks:
>> > node1 509036 blocks
>> > node2 157937 blocks
>> > node3 15783 blocks
>> > node4 15117 blocks
>> > node5 20158 blocks
>> >
>> > But my HBase regions are very balanced.
>> > node1 88 regions
>> > node2 108 regions
>> > node3 111 regions
>> > node4 102 regions
>> > node5 105 regions
>> >
>> >
>> >
>> > NodeLast
>> > ContactAdmin StateConfigured
>> > Capacity (GB)Used
>> > (GB)Non DFS
>> > Used (GB)Remaining
>> > (GB)Used
>> > (%)Used
>> > (%)Remaining
>> > (%)Blocksnd1-rack0-cloud<
>> >
>> http://nd1-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>> > >
>> > 0In Service822.8578.6743.28200.8670.3324.41509036nd2-rack0-cloud<
>> >
>> http://nd2-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>> > >
>> > 0In Service822.8190.0242.96589.8223.0971.68157937nd3-rack0-cloud<
>> >
>> http://nd3-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>> > >
>> > 0In Service822.851.9542.61728.246.3188.5115783nd4-rack0-cloud<
>> >
>> http://nd4-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>> > >
>> > 6In Service822.846.1942.84733.775.6189.1815117nd5-rack0-cloud<
>> >
>> http://nd5-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>> > >
>> > 1In Service1215.6152.3762.911100.324.3190.5220158
>> >
>> >
>> > But my HBase regions are very balanced.
>> >
>> > AddressStart CodeLoadnd1-rack0-cloud:60020 <
>> http://nd1-rack0-cloud:60030/>
>> > 1237967027050requests=383, regions=88, usedHeap=978, maxHeap=1991
>> > nd2-rack0-cloud:60020 <http://nd2-rack0-cloud:60030/
>> > >1237788871362requests=422,
>> > regions=108, usedHeap=1433,
>> > maxHeap=1991nd3-rack0-cloud:60020<http://nd3-rack0-cloud:60030/>
>> > 1237788881667requests=962, regions=111, usedHeap=1534, maxHeap=1991
>> > nd4-rack0-cloud:60020 <http://nd4-rack0-cloud:60030/
>> > >1237788859541requests=369,
>> > regions=102, usedHeap=1059,
>> > maxHeap=1991nd5-rack0-cloud:60020<http://nd5-rack0-cloud:60030/>
>> > 1237788899331requests=384, regions=105, usedHeap=1535,
>> > maxHeap=1983Total:servers:
>> > 5 requests=2520, regions=514
>> >
>>
>
>
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by schubert zhang <zs...@gmail.com>.
Thanks Ryan. Balancer may take a long time.
The number of block are too different. But maybe it is caused by HBase not
deleting garbage blocks on regionserver1 and regionserver2 and maybe others.
We grep the logs of hadoop and find there is no any "deleting block" in
node1 and node2.
Following is the grep (grep -c "ask 10.24.1.1?:50010 to delete") result of
hasoop logs:
namenode:
-----grep -c "ask 10.24.1.12:50010 to delete"-----node1
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
4754
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
1062
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
0
-----grep -c "ask 10.24.1.14:50010 to delete"-----node2
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
1494
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
3305
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
3385
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
1494
-----grep -c "ask 10.24.1.16:50010 to delete"-----node3
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
8022
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
8238
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
4302
-----grep -c "ask 10.24.1.18:50010 to delete"-----node4
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
8591
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
9111
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
5038
-----grep -c "ask 10.24.1.20:50010 to delete"-----node5
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
3794
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
3946
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
2989
So, I think it may caused by HBase.
I just grep the log of the zero "delete block" node. and find:
[schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
hadoop-schubert-datanode-nd1-rack0-cloud.log.2009-03-24
104739
[schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
hadoop-schubert-datanode-nd1-rack0-cloud.log.2009-03-23
465927
[schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
hadoop-schubert-datanode-nd1-rack0-cloud.log
0
On Wed, Mar 25, 2009 at 5:14 PM, Ryan Rawson <ry...@gmail.com> wrote:
> Try
> hadoop/bin/start-balancer.sh
>
> HDFS doesnt auto-balance. Balancing in HDFS requires moving data around,
> whereas balancing in HBase just means opening a file on a different
> machine.
>
> On Wed, Mar 25, 2009 at 2:12 AM, schubert zhang <zs...@gmail.com> wrote:
>
> > Hi all,
> > I am using hbase-0.19.1 and hadoop-0.19.
> > My cluster have 5+1 nodes, and there are about 512 regions in HBase
> (256MB
> > per region).
> >
> > But I found the blocks in HDFS is very unbalanced. Following is the
> status
> > from HDFS web GUI.
> >
> > (Node: I don't know if this mailing list can display html!)
> >
> > HDFS blocks:
> > node1 509036 blocks
> > node2 157937 blocks
> > node3 15783 blocks
> > node4 15117 blocks
> > node5 20158 blocks
> >
> > But my HBase regions are very balanced.
> > node1 88 regions
> > node2 108 regions
> > node3 111 regions
> > node4 102 regions
> > node5 105 regions
> >
> >
> >
> > NodeLast
> > ContactAdmin StateConfigured
> > Capacity (GB)Used
> > (GB)Non DFS
> > Used (GB)Remaining
> > (GB)Used
> > (%)Used
> > (%)Remaining
> > (%)Blocksnd1-rack0-cloud<
> >
> http://nd1-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> > >
> > 0In Service822.8578.6743.28200.8670.3324.41509036nd2-rack0-cloud<
> >
> http://nd2-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> > >
> > 0In Service822.8190.0242.96589.8223.0971.68157937nd3-rack0-cloud<
> >
> http://nd3-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> > >
> > 0In Service822.851.9542.61728.246.3188.5115783nd4-rack0-cloud<
> >
> http://nd4-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> > >
> > 6In Service822.846.1942.84733.775.6189.1815117nd5-rack0-cloud<
> >
> http://nd5-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> > >
> > 1In Service1215.6152.3762.911100.324.3190.5220158
> >
> >
> > But my HBase regions are very balanced.
> >
> > AddressStart CodeLoadnd1-rack0-cloud:60020 <
> http://nd1-rack0-cloud:60030/>
> > 1237967027050requests=383, regions=88, usedHeap=978, maxHeap=1991
> > nd2-rack0-cloud:60020 <http://nd2-rack0-cloud:60030/
> > >1237788871362requests=422,
> > regions=108, usedHeap=1433,
> > maxHeap=1991nd3-rack0-cloud:60020<http://nd3-rack0-cloud:60030/>
> > 1237788881667requests=962, regions=111, usedHeap=1534, maxHeap=1991
> > nd4-rack0-cloud:60020 <http://nd4-rack0-cloud:60030/
> > >1237788859541requests=369,
> > regions=102, usedHeap=1059,
> > maxHeap=1991nd5-rack0-cloud:60020<http://nd5-rack0-cloud:60030/>
> > 1237788899331requests=384, regions=105, usedHeap=1535,
> > maxHeap=1983Total:servers:
> > 5 requests=2520, regions=514
> >
>
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by schubert zhang <zs...@gmail.com>.
Thanks Ryan. Balancer may take a long time.
The number of block are too different. But maybe it is caused by HBase not
deleting garbage blocks on regionserver1 and regionserver2 and maybe others.
We grep the logs of hadoop and find there is no any "deleting block" in
node1 and node2.
Following is the grep (grep -c "ask 10.24.1.1?:50010 to delete") result of
hasoop logs:
namenode:
-----grep -c "ask 10.24.1.12:50010 to delete"-----node1
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
4754
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
1062
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
0
-----grep -c "ask 10.24.1.14:50010 to delete"-----node2
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
1494
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
3305
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
3385
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
1494
-----grep -c "ask 10.24.1.16:50010 to delete"-----node3
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
8022
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
8238
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
4302
-----grep -c "ask 10.24.1.18:50010 to delete"-----node4
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
8591
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
9111
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
5038
-----grep -c "ask 10.24.1.20:50010 to delete"-----node5
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
3794
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
3946
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
2989
So, I think it may caused by HBase.
I just grep the log of the zero "delete block" node. and find:
[schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
hadoop-schubert-datanode-nd1-rack0-cloud.log.2009-03-24
104739
[schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
hadoop-schubert-datanode-nd1-rack0-cloud.log.2009-03-23
465927
[schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
hadoop-schubert-datanode-nd1-rack0-cloud.log
0
On Wed, Mar 25, 2009 at 5:14 PM, Ryan Rawson <ry...@gmail.com> wrote:
> Try
> hadoop/bin/start-balancer.sh
>
> HDFS doesnt auto-balance. Balancing in HDFS requires moving data around,
> whereas balancing in HBase just means opening a file on a different
> machine.
>
> On Wed, Mar 25, 2009 at 2:12 AM, schubert zhang <zs...@gmail.com> wrote:
>
> > Hi all,
> > I am using hbase-0.19.1 and hadoop-0.19.
> > My cluster have 5+1 nodes, and there are about 512 regions in HBase
> (256MB
> > per region).
> >
> > But I found the blocks in HDFS is very unbalanced. Following is the
> status
> > from HDFS web GUI.
> >
> > (Node: I don't know if this mailing list can display html!)
> >
> > HDFS blocks:
> > node1 509036 blocks
> > node2 157937 blocks
> > node3 15783 blocks
> > node4 15117 blocks
> > node5 20158 blocks
> >
> > But my HBase regions are very balanced.
> > node1 88 regions
> > node2 108 regions
> > node3 111 regions
> > node4 102 regions
> > node5 105 regions
> >
> >
> >
> > NodeLast
> > ContactAdmin StateConfigured
> > Capacity (GB)Used
> > (GB)Non DFS
> > Used (GB)Remaining
> > (GB)Used
> > (%)Used
> > (%)Remaining
> > (%)Blocksnd1-rack0-cloud<
> >
> http://nd1-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> > >
> > 0In Service822.8578.6743.28200.8670.3324.41509036nd2-rack0-cloud<
> >
> http://nd2-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> > >
> > 0In Service822.8190.0242.96589.8223.0971.68157937nd3-rack0-cloud<
> >
> http://nd3-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> > >
> > 0In Service822.851.9542.61728.246.3188.5115783nd4-rack0-cloud<
> >
> http://nd4-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> > >
> > 6In Service822.846.1942.84733.775.6189.1815117nd5-rack0-cloud<
> >
> http://nd5-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> > >
> > 1In Service1215.6152.3762.911100.324.3190.5220158
> >
> >
> > But my HBase regions are very balanced.
> >
> > AddressStart CodeLoadnd1-rack0-cloud:60020 <
> http://nd1-rack0-cloud:60030/>
> > 1237967027050requests=383, regions=88, usedHeap=978, maxHeap=1991
> > nd2-rack0-cloud:60020 <http://nd2-rack0-cloud:60030/
> > >1237788871362requests=422,
> > regions=108, usedHeap=1433,
> > maxHeap=1991nd3-rack0-cloud:60020<http://nd3-rack0-cloud:60030/>
> > 1237788881667requests=962, regions=111, usedHeap=1534, maxHeap=1991
> > nd4-rack0-cloud:60020 <http://nd4-rack0-cloud:60030/
> > >1237788859541requests=369,
> > regions=102, usedHeap=1059,
> > maxHeap=1991nd5-rack0-cloud:60020<http://nd5-rack0-cloud:60030/>
> > 1237788899331requests=384, regions=105, usedHeap=1535,
> > maxHeap=1983Total:servers:
> > 5 requests=2520, regions=514
> >
>
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by Ryan Rawson <ry...@gmail.com>.
Try
hadoop/bin/start-balancer.sh
HDFS doesnt auto-balance. Balancing in HDFS requires moving data around,
whereas balancing in HBase just means opening a file on a different machine.
On Wed, Mar 25, 2009 at 2:12 AM, schubert zhang <zs...@gmail.com> wrote:
> Hi all,
> I am using hbase-0.19.1 and hadoop-0.19.
> My cluster have 5+1 nodes, and there are about 512 regions in HBase (256MB
> per region).
>
> But I found the blocks in HDFS is very unbalanced. Following is the status
> from HDFS web GUI.
>
> (Node: I don't know if this mailing list can display html!)
>
> HDFS blocks:
> node1 509036 blocks
> node2 157937 blocks
> node3 15783 blocks
> node4 15117 blocks
> node5 20158 blocks
>
> But my HBase regions are very balanced.
> node1 88 regions
> node2 108 regions
> node3 111 regions
> node4 102 regions
> node5 105 regions
>
>
>
> NodeLast
> ContactAdmin StateConfigured
> Capacity (GB)Used
> (GB)Non DFS
> Used (GB)Remaining
> (GB)Used
> (%)Used
> (%)Remaining
> (%)Blocksnd1-rack0-cloud<
> http://nd1-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >
> 0In Service822.8578.6743.28200.8670.3324.41509036nd2-rack0-cloud<
> http://nd2-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >
> 0In Service822.8190.0242.96589.8223.0971.68157937nd3-rack0-cloud<
> http://nd3-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >
> 0In Service822.851.9542.61728.246.3188.5115783nd4-rack0-cloud<
> http://nd4-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >
> 6In Service822.846.1942.84733.775.6189.1815117nd5-rack0-cloud<
> http://nd5-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >
> 1In Service1215.6152.3762.911100.324.3190.5220158
>
>
> But my HBase regions are very balanced.
>
> AddressStart CodeLoadnd1-rack0-cloud:60020 <http://nd1-rack0-cloud:60030/>
> 1237967027050requests=383, regions=88, usedHeap=978, maxHeap=1991
> nd2-rack0-cloud:60020 <http://nd2-rack0-cloud:60030/
> >1237788871362requests=422,
> regions=108, usedHeap=1433,
> maxHeap=1991nd3-rack0-cloud:60020<http://nd3-rack0-cloud:60030/>
> 1237788881667requests=962, regions=111, usedHeap=1534, maxHeap=1991
> nd4-rack0-cloud:60020 <http://nd4-rack0-cloud:60030/
> >1237788859541requests=369,
> regions=102, usedHeap=1059,
> maxHeap=1991nd5-rack0-cloud:60020<http://nd5-rack0-cloud:60030/>
> 1237788899331requests=384, regions=105, usedHeap=1535,
> maxHeap=1983Total:servers:
> 5 requests=2520, regions=514
>
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by Ryan Rawson <ry...@gmail.com>.
Try
hadoop/bin/start-balancer.sh
HDFS doesnt auto-balance. Balancing in HDFS requires moving data around,
whereas balancing in HBase just means opening a file on a different machine.
On Wed, Mar 25, 2009 at 2:12 AM, schubert zhang <zs...@gmail.com> wrote:
> Hi all,
> I am using hbase-0.19.1 and hadoop-0.19.
> My cluster have 5+1 nodes, and there are about 512 regions in HBase (256MB
> per region).
>
> But I found the blocks in HDFS is very unbalanced. Following is the status
> from HDFS web GUI.
>
> (Node: I don't know if this mailing list can display html!)
>
> HDFS blocks:
> node1 509036 blocks
> node2 157937 blocks
> node3 15783 blocks
> node4 15117 blocks
> node5 20158 blocks
>
> But my HBase regions are very balanced.
> node1 88 regions
> node2 108 regions
> node3 111 regions
> node4 102 regions
> node5 105 regions
>
>
>
> NodeLast
> ContactAdmin StateConfigured
> Capacity (GB)Used
> (GB)Non DFS
> Used (GB)Remaining
> (GB)Used
> (%)Used
> (%)Remaining
> (%)Blocksnd1-rack0-cloud<
> http://nd1-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >
> 0In Service822.8578.6743.28200.8670.3324.41509036nd2-rack0-cloud<
> http://nd2-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >
> 0In Service822.8190.0242.96589.8223.0971.68157937nd3-rack0-cloud<
> http://nd3-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >
> 0In Service822.851.9542.61728.246.3188.5115783nd4-rack0-cloud<
> http://nd4-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >
> 6In Service822.846.1942.84733.775.6189.1815117nd5-rack0-cloud<
> http://nd5-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >
> 1In Service1215.6152.3762.911100.324.3190.5220158
>
>
> But my HBase regions are very balanced.
>
> AddressStart CodeLoadnd1-rack0-cloud:60020 <http://nd1-rack0-cloud:60030/>
> 1237967027050requests=383, regions=88, usedHeap=978, maxHeap=1991
> nd2-rack0-cloud:60020 <http://nd2-rack0-cloud:60030/
> >1237788871362requests=422,
> regions=108, usedHeap=1433,
> maxHeap=1991nd3-rack0-cloud:60020<http://nd3-rack0-cloud:60030/>
> 1237788881667requests=962, regions=111, usedHeap=1534, maxHeap=1991
> nd4-rack0-cloud:60020 <http://nd4-rack0-cloud:60030/
> >1237788859541requests=369,
> regions=102, usedHeap=1059,
> maxHeap=1991nd5-rack0-cloud:60020<http://nd5-rack0-cloud:60030/>
> 1237788899331requests=384, regions=105, usedHeap=1535,
> maxHeap=1983Total:servers:
> 5 requests=2520, regions=514
>
Re: HDFS unbalance issue. (HBase over HDFS)
Posted by Andrew Purtell <ap...@apache.org>.
Hello Schubert,
HDFS does not automatically balance block placements. You should
run the balancer utility for that. However, please be advised
that the HBase DFSClient caches block locations so running the
block balancer while HBase is running is not advised -- It can
lead to spurious, but serious, errors. Shut down HBase first,
run the DFS balancer utility, then restart.
Best regards,
- Andy
> From: schubert zhang <zs...@gmail.com>
> Subject: HDFS unbalance issue. (HBase over HDFS)
> To: core-user@hadoop.apache.org, hbase-user@hadoop.apache.org
> Date: Wednesday, March 25, 2009, 2:12 AM
>
> Hi all,
> I am using hbase-0.19.1 and hadoop-0.19.
> My cluster have 5+1 nodes, and there are about 512 regions
> in HBase (256MB per region).
>
> But I found the blocks in HDFS is very unbalanced.
> Following is the status from HDFS web GUI.
>
> HDFS blocks:
> node1 509036 blocks
> node2 157937 blocks
> node3 15783 blocks
> node4 15117 blocks
> node5 20158 blocks
>
> But my HBase regions are very balanced.
> node1 88 regions
> node2 108 regions
> node3 111 regions
> node4 102 regions
> node5 105 regions