You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Hairong Kuang (JIRA)" <ji...@apache.org> on 2010/10/14 22:22:33 UTC
[jira] Created: (HDFS-1458) Improve checkpoint performance by
avoiding uncessary image downloads
Improve checkpoint performance by avoiding uncessary image downloads
--------------------------------------------------------------------
Key: HDFS-1458
URL: https://issues.apache.org/jira/browse/HDFS-1458
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Affects Versions: 0.22.0
Reporter: Hairong Kuang
Assignee: Hairong Kuang
Fix For: 0.22.0
If secondary namenode could verify that the image it has on its disk is the same as the one in the primary NameNode, it could skip downloading the image from the primary NN, thus completely eliminating the image download overhead.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1458) Improve checkpoint performance by
avoiding unnecessary image downloads
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925982#action_12925982 ]
Hairong Kuang commented on HDFS-1458:
-------------------------------------
Cool! I did not realize that BackupNode/CheckpointNode already does this, I will take a look. FB will deploy AvatarNode so does not need this either. :-) But I guess the community is still using Secondary NN. There might be a need of this optimization.
> Improve checkpoint performance by avoiding unnecessary image downloads
> ----------------------------------------------------------------------
>
> Key: HDFS-1458
> URL: https://issues.apache.org/jira/browse/HDFS-1458
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.22.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: checkpoint-checkfsimageissame.patch
>
>
> If secondary namenode could verify that the image it has on its disk is the same as the one in the primary NameNode, it could skip downloading the image from the primary NN, thus completely eliminating the image download overhead.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1458) Improve checkpoint performance by
avoiding unnecessary image downloads
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-1458:
--------------------------------
Summary: Improve checkpoint performance by avoiding unnecessary image downloads (was: Improve checkpoint performance by avoiding uncessary image downloads)
> Improve checkpoint performance by avoiding unnecessary image downloads
> ----------------------------------------------------------------------
>
> Key: HDFS-1458
> URL: https://issues.apache.org/jira/browse/HDFS-1458
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.22.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: checkpoint-checkfsimageissame.patch
>
>
> If secondary namenode could verify that the image it has on its disk is the same as the one in the primary NameNode, it could skip downloading the image from the primary NN, thus completely eliminating the image download overhead.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1458) Improve checkpoint performance by
avoiding uncessary image downloads
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925697#action_12925697 ]
Hairong Kuang commented on HDFS-1458:
-------------------------------------
When discussing this with Dhruba, I realized that the secondary does not need to load the image if the primary and the secondary have the same image. This will give yet another boost on checkpoint performance.
> Improve checkpoint performance by avoiding uncessary image downloads
> --------------------------------------------------------------------
>
> Key: HDFS-1458
> URL: https://issues.apache.org/jira/browse/HDFS-1458
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.22.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: checkpoint-checkfsimageissame.patch
>
>
> If secondary namenode could verify that the image it has on its disk is the same as the one in the primary NameNode, it could skip downloading the image from the primary NN, thus completely eliminating the image download overhead.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1458) Improve checkpoint performance by
avoiding uncessary image downloads
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922777#action_12922777 ]
Hairong Kuang commented on HDFS-1458:
-------------------------------------
Lu, thanks for the patch. It looks good. Two comments:
1. Can we avoid computing checksum on the fly? I linked this issue to HDFS-903. It would be nice that it gets computed when the image is saved to disk.
2. Can checksum be returned with rollEditLog?
> Improve checkpoint performance by avoiding uncessary image downloads
> --------------------------------------------------------------------
>
> Key: HDFS-1458
> URL: https://issues.apache.org/jira/browse/HDFS-1458
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.22.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: checkpoint-checkfsimageissame.patch
>
>
> If secondary namenode could verify that the image it has on its disk is the same as the one in the primary NameNode, it could skip downloading the image from the primary NN, thus completely eliminating the image download overhead.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1458) Improve checkpoint performance by
avoiding unnecessary image downloads
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-1458:
--------------------------------
Attachment: trunkNoDownloadImage.patch
This patch does the proposed optimization. If the image checksum at seondary NameNode is the same as the one at primary Namenode, the secondary neither download the image from the primary nor load the image from disk.
> Improve checkpoint performance by avoiding unnecessary image downloads
> ----------------------------------------------------------------------
>
> Key: HDFS-1458
> URL: https://issues.apache.org/jira/browse/HDFS-1458
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.22.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: checkpoint-checkfsimageissame.patch, trunkNoDownloadImage.patch
>
>
> If secondary namenode could verify that the image it has on its disk is the same as the one in the primary NameNode, it could skip downloading the image from the primary NN, thus completely eliminating the image download overhead.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1458) Improve checkpoint performance by
avoiding unnecessary image downloads
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925965#action_12925965 ]
Konstantin Shvachko commented on HDFS-1458:
-------------------------------------------
Just curious,
- why do we bother improving a deprecated SNN for 0.22, if we have a direct replacement CheckpointNode?
- or is it intended for FB-0.20?
- This is already done in BackupNode / CheckpointNode by verifying checkpoint time and the versions.
You can add the verification of the checksum and its done.
What do I miss?
> Improve checkpoint performance by avoiding unnecessary image downloads
> ----------------------------------------------------------------------
>
> Key: HDFS-1458
> URL: https://issues.apache.org/jira/browse/HDFS-1458
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.22.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: checkpoint-checkfsimageissame.patch
>
>
> If secondary namenode could verify that the image it has on its disk is the same as the one in the primary NameNode, it could skip downloading the image from the primary NN, thus completely eliminating the image download overhead.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1458) Improve checkpoint performance by
avoiding uncessary image downloads
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922790#action_12922790 ]
dhruba borthakur commented on HDFS-1458:
----------------------------------------
If we are recomputing the checksum everytime at the primary NN, then it this patch does not reduce load on the primary namenode, isn't it?
> Improve checkpoint performance by avoiding uncessary image downloads
> --------------------------------------------------------------------
>
> Key: HDFS-1458
> URL: https://issues.apache.org/jira/browse/HDFS-1458
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.22.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: checkpoint-checkfsimageissame.patch
>
>
> If secondary namenode could verify that the image it has on its disk is the same as the one in the primary NameNode, it could skip downloading the image from the primary NN, thus completely eliminating the image download overhead.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1458) Improve checkpoint performance by
avoiding unnecessary image downloads
Posted by "Lu Yilei (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927292#action_12927292 ]
Lu Yilei commented on HDFS-1458:
--------------------------------
To Hairong.
Thanks your good idea.
1. We can generate a UUID when SNN save fsimage. And we can add new fields for the UUID. If the UUID of the NameNode is equal the UUID of the SNN, we can't download the fsimage of NameNode.
2. Can checksum be returned with rollEditLog? Yes.
> Improve checkpoint performance by avoiding unnecessary image downloads
> ----------------------------------------------------------------------
>
> Key: HDFS-1458
> URL: https://issues.apache.org/jira/browse/HDFS-1458
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.22.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: checkpoint-checkfsimageissame.patch
>
>
> If secondary namenode could verify that the image it has on its disk is the same as the one in the primary NameNode, it could skip downloading the image from the primary NN, thus completely eliminating the image download overhead.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1458) Improve checkpoint performance by
avoiding uncessary image downloads
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922799#action_12922799 ]
Hairong Kuang commented on HDFS-1458:
-------------------------------------
It will reduce the network traffic.
Besides using checksum, another option is to use checkpoint time. But it is not as reliable as checksum.
> Improve checkpoint performance by avoiding uncessary image downloads
> --------------------------------------------------------------------
>
> Key: HDFS-1458
> URL: https://issues.apache.org/jira/browse/HDFS-1458
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.22.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: checkpoint-checkfsimageissame.patch
>
>
> If secondary namenode could verify that the image it has on its disk is the same as the one in the primary NameNode, it could skip downloading the image from the primary NN, thus completely eliminating the image download overhead.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1458) Improve checkpoint performance by
avoiding uncessary image downloads
Posted by "Lu Yilei (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lu Yilei updated HDFS-1458:
---------------------------
Attachment: checkpoint-checkfsimageissame.patch
Check NameNode fsimage and SecondaryNameNode when download CheckpointFiles by RPC, if they are the same, SecondaryNameNode will not download the fsimage from the NameNode.
> Improve checkpoint performance by avoiding uncessary image downloads
> --------------------------------------------------------------------
>
> Key: HDFS-1458
> URL: https://issues.apache.org/jira/browse/HDFS-1458
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.22.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: checkpoint-checkfsimageissame.patch
>
>
> If secondary namenode could verify that the image it has on its disk is the same as the one in the primary NameNode, it could skip downloading the image from the primary NN, thus completely eliminating the image download overhead.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.