You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Surendra Singh Lilhore (JIRA)" <ji...@apache.org> on 2018/09/04 08:18:09 UTC
[jira] [Comment Edited] (HDFS-13768) Adding replicas to volume map
makes DataNode start slowly
[ https://issues.apache.org/jira/browse/HDFS-13768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16602728#comment-16602728 ]
Surendra Singh Lilhore edited comment on HDFS-13768 at 9/4/18 8:18 AM:
-----------------------------------------------------------------------
Thanks [~RANith] for patch.
Some review comment
1. You can start you first task in BlockPoolSlice#getVolumeMap() and then BlockPoolSlice#addToReplicasMap() can submit the sub task when item is directory.
{code:java}
if (!success) {
// add finalized replicas
addToReplicasMap(volumeMap, finalizedDir, lazyWriteReplicaMap, true);
// add rbw replicas
addToReplicasMap(volumeMap, rbwDir, lazyWriteReplicaMap, false);
}
{code}
This can be changed to
{code:java}
if (!success) {
// add finalized replicas
AddReplicaProcessor task = new AddReplicaProcessor(volumeMap,
finalizedDir, lazyWriteReplicaMap, true);
forkJoinPool.invoke(task);
// add rbw replicas
task = new AddReplicaProcessor(volumeMap, rbwDir, lazyWriteReplicaMap,
false);
forkJoinPool.invoke(task);
}
{code}
2. You used {{forkJoinPool.invoke()}} in BlockPoolSlice#addToReplicasMap(). It will wait till the task is finished. You need to use here {{task.fork()}}.
{code:java}
+ forkJoinPool.invoke(new AddReplicaProcessor(volumeMap, file,
+ lazyWriteReplicaMap, isFinalized));
{code}
instead of this use
{code:java}
+ AddReplicaProcessor task = new AddReplicaProcessor(volumeMap, file,
+ lazyWriteReplicaMap, isFinalized);
+ task.fork();
{code}
3. line 78, you shutdown the {{forkJoinPool}}. This may be by mistake. Pls check once.
{code:java}
+ forkJoinPool.shutdown();
{code}
4. What if the exception occur's in {{AddReplicaProcessor}} ?, you need to report it back to the next level.
was (Author: surendrasingh):
Thanks [~RANith] for patch.
Some review comment
1. You can start you first task in BlockPoolSlice#getVolumeMap() and then BlockPoolSlice#addToReplicasMap() can submit the sub task when item is directory.
{code}
if (!success) {
// add finalized replicas
addToReplicasMap(volumeMap, finalizedDir, lazyWriteReplicaMap, true);
// add rbw replicas
addToReplicasMap(volumeMap, rbwDir, lazyWriteReplicaMap, false);
}
{code}
This can be changed to
{code}
if (!success) {
// add finalized replicas
AddReplicaProcessor task = new AddReplicaProcessor(volumeMap,
finalizedDir, lazyWriteReplicaMap, true);
forkJoinPool.invoke(task);
// add rbw replicas
task = new AddReplicaProcessor(volumeMap, rbwDir, lazyWriteReplicaMap,
false);
forkJoinPool.invoke(task);
}
{code}
2. You used {{forkJoinPool.invoke()}} in BlockPoolSlice#addToReplicasMap(). It will wait till the task is finished. You need to use here {{task.fork()}}.
{code}
+ forkJoinPool.invoke(new AddReplicaProcessor(volumeMap, file,
+ lazyWriteReplicaMap, isFinalized));
{code}
instead of this use
{code}
+ AddReplicaProcessor task = new AddReplicaProcessor(volumeMap, file,
+ lazyWriteReplicaMap, isFinalized);
+ task.fork();
{code}
2. line 78, you shutdown the {{forkJoinPool}}. This may be by mistake. Pls check once.
{code}
+ forkJoinPool.shutdown();
{code}
3. What if the exception occur's in {{AddReplicaProcessor}} ?, you need to report it back to the next level.
> Adding replicas to volume map makes DataNode start slowly
> -----------------------------------------------------------
>
> Key: HDFS-13768
> URL: https://issues.apache.org/jira/browse/HDFS-13768
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 3.1.0
> Reporter: Yiqun Lin
> Assignee: Ranith Sardar
> Priority: Major
> Attachments: HDFS-13768.patch
>
>
> We find DN starting so slowly when rolling upgrade our cluster. When we restart DNs, the DNs start so slowly and not register to NN immediately. And this cause a lots of following error:
> {noformat}
> DataXceiver error processing WRITE_BLOCK operation src: /xx.xx.xx.xx:64360 dst: /xx.xx.xx.xx:50010
> java.io.IOException: Not ready to serve the block pool, BP-1508644862-xx.xx.xx.xx-1493781183457.
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.checkAndWaitForBP(DataXceiver.java:1290)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.checkAccess(DataXceiver.java:1298)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:630)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:169)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:106)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:246)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Looking into the logic of DN startup, it will do the initial block pool operation before the registration. And during initializing block pool operation, we found the adding replicas to volume map is the most expensive operation. Related log:
> {noformat}
> 2018-07-26 10:46:23,771 INFO [Thread-105] org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time to add replicas to map for block pool BP-1508644862-xx.xx.xx.xx-1493781183457 on volume /home/hard_disk/1/dfs/dn/current: 242722ms
> 2018-07-26 10:46:26,231 INFO [Thread-109] org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time to add replicas to map for block pool BP-1508644862-xx.xx.xx.xx-1493781183457 on volume /home/hard_disk/5/dfs/dn/current: 245182ms
> 2018-07-26 10:46:32,146 INFO [Thread-112] org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time to add replicas to map for block pool BP-1508644862-xx.xx.xx.xx-1493781183457 on volume /home/hard_disk/8/dfs/dn/current: 251097ms
> 2018-07-26 10:47:08,283 INFO [Thread-106] org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time to add replicas to map for block pool BP-1508644862-xx.xx.xx.xx-1493781183457 on volume /home/hard_disk/2/dfs/dn/current: 287235ms
> {noformat}
> Currently DN uses independent thread to scan and add replica for each volume, but we still need to wait the slowest thread to finish its work. So the main problem here is that we could make the thread to run faster.
> The jstack we get when DN blocking in the adding replica:
> {noformat}
> "Thread-113" #419 daemon prio=5 os_prio=0 tid=0x00007f40879ff000 nid=0x145da runnable [0x00007f4043a38000]
> java.lang.Thread.State: RUNNABLE
> at java.io.UnixFileSystem.list(Native Method)
> at java.io.File.list(File.java:1122)
> at java.io.File.listFiles(File.java:1207)
> at org.apache.hadoop.fs.FileUtil.listFiles(FileUtil.java:1165)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.addToReplicasMap(BlockPoolSlice.java:445)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.addToReplicasMap(BlockPoolSlice.java:448)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.addToReplicasMap(BlockPoolSlice.java:448)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.getVolumeMap(BlockPoolSlice.java:342)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getVolumeMap(FsVolumeImpl.java:864)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList$1.run(FsVolumeList.java:191)
> {noformat}
> One improvement maybe we can use ForkJoinPool to do this recursive task, rather than a sync way. This will be a great improvement because it can greatly speed up recovery process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org