You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2009/12/10 01:50:18 UTC
[jira] Created: (HDFS-822) Appends to already-finalized blocks can
rename across volumes
Appends to already-finalized blocks can rename across volumes
-------------------------------------------------------------
Key: HDFS-822
URL: https://issues.apache.org/jira/browse/HDFS-822
Project: Hadoop HDFS
Issue Type: Improvement
Components: data-node
Affects Versions: 0.21.0, 0.22.0
Reporter: Todd Lipcon
This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-822) Appends to already-finalized blocks
can rename across volumes
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800973#action_12800973 ]
Todd Lipcon commented on HDFS-822:
----------------------------------
bq. So basically this patch delays the decision to check disk space as opposed to Todd's proposal.
Only problem with this is that it now ignores the dfs.du.reserved setting, since that's only set at allocation-time. Maybe that's OK, since there are plenty of other ways to overrun the reserved space - just wanted to bring it up.
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch, HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-822) Appends to already-finalized blocks can
rename across volumes
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-822:
-------------------------------
Attachment: HDFS-800.patch
This new patch simply does what I proposed and it sets up a pipeline without checking the available space.
If the disk does not have space when actual writes happen, the write will fail and the out-of-space datanode will be taken out of the pipeline. So basically this patch delays the decision to check disk space as opposed to Todd's proposal.
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-822) Appends to already-finalized blocks
can rename across volumes
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800837#action_12800837 ]
Hairong Kuang commented on HDFS-822:
------------------------------------
It seems to me that moving a block to be appended to a different volume when the current volume has space is a bug, which I'd like to fix in this jira. However, moving a block to be appended to a different volume when the current volume does not have enough space for the estimated new bytes is an optimization, which probably we should work on it in a different jira.
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-822) Appends to already-finalized blocks
can rename across volumes
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800877#action_12800877 ]
Todd Lipcon commented on HDFS-822:
----------------------------------
Sounds reasonable, so long as we're confident that holding the FSDataset lock for 10-20 seconds isn't a bug (eg appending to a 250MB block will take 10 seconds at 25M/sec cross-volume copy rate, could very well be slower if either disk is stressed)
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-822) Appends to already-finalized blocks
can rename across volumes
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800952#action_12800952 ]
dhruba borthakur commented on HDFS-822:
---------------------------------------
> But anyway my proposal is to disallow the across volume rename for now. Does this sound OK to everybody?
sounds ok to me.
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-822) Appends to already-finalized blocks
can rename across volumes
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800947#action_12800947 ]
Hairong Kuang commented on HDFS-822:
------------------------------------
> if the datanode dies in the middle of a rename, we can have a partial file in the destination as well as the complete file in the source, isn't it?
Current code handles this. It removes the partial file in the destination if rename fails.
Sorry that I did not make my proposal clear. What I am thinking to do in this jira is to disallow across volume rename. So appending to a block simply moves the block to the rbw directory in the same volume. This is definitely an improvement to 0.21 & trunk because 0.21 & trunk may move the block to a different volume even if the current volume has space. This is a critical bug that should be fixed. As to whether we should rename across volume when the current volume is full, lets do it in a different jira.
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-822) Appends to already-finalized blocks can
rename across volumes
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-822:
-------------------------------
Attachment: HDFS-822.patch
Todd's comment makes sense. This patch incorporates his comment and adds a unit test.
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch, HDFS-822.patch, HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-822) Appends to already-finalized blocks can
rename across volumes
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-822:
-------------------------------
Priority: Blocker (was: Major)
Fix Version/s: 0.22.0
0.21.0
Assignee: Hairong Kuang
This is a valid point. Let's get it fixed in 0.21.
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-822) Appends to already-finalized blocks
can rename across volumes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802647#action_12802647 ]
Hudson commented on HDFS-822:
-----------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #173 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/173/])
. Appends to already-finalized blocks can rename across volumes. Contributed by Hairong Kuang.
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch, HDFS-822.patch, HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-822) Appends to already-finalized blocks
can rename across volumes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804902#action_12804902 ]
Hudson commented on HDFS-822:
-----------------------------
Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #205 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/205/])
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch, HDFS-822.patch, HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-822) Appends to already-finalized blocks
can rename across volumes
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800917#action_12800917 ]
Todd Lipcon commented on HDFS-822:
----------------------------------
bq. Moreover, rename across two mount points is not atomic, ie. if the datanode dies in the middle of a rename, we can have a partial file in the destination as well as the complete file in the source, isn't it?
Yes, that's true. But wouldn't that just present as a corrupt replica when the block scanner catches up to it, or a client tries to read it?
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-822) Appends to already-finalized blocks
can rename across volumes
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789979#action_12789979 ]
Todd Lipcon commented on HDFS-822:
----------------------------------
I think in that case it *should* do a cross-volume block rename. However, it should avoid holding the FSDataset lock while doing so
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-822) Appends to already-finalized blocks
can rename across volumes
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800950#action_12800950 ]
Hairong Kuang commented on HDFS-822:
------------------------------------
> Current code handles this. It removes the partial file in the destination if rename fails.
I checked the code again. Unfortunately the code does not handle this.
But anyway my proposal is to disallow the across volume rename for now. Does this sound OK to everybody?
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-822) Appends to already-finalized blocks
can rename across volumes
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800915#action_12800915 ]
dhruba borthakur commented on HDFS-822:
---------------------------------------
If there is space in the same volume, then the rename occur within the same volume. In this case, holding the FSDataset lock across the rename is ok.
If there is no space, then the rename is actually a datacopy. In this case, we should not hold the FSDataset lock across the rename. Moreover, rename across two mount points is not atomic, ie. if the datanode dies in the middle of a rename, we can have a partial file in the destination as well as the complete file in the source, isn't it?
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-822) Appends to already-finalized blocks can
rename across volumes
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-822:
-------------------------------
Status: Patch Available (was: Open)
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch, HDFS-822.patch, HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-822) Appends to already-finalized blocks can
rename across volumes
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-822:
-------------------------------
Attachment: HDFS-822.patch
This patch makes sure that the replica to be appended is moved to the same volume if the volume has enough space available.
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-822) Appends to already-finalized blocks
can rename across volumes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802835#action_12802835 ]
Hudson commented on HDFS-822:
-----------------------------
Integrated in Hadoop-Hdfs-trunk #206 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/206/])
. Appends to already-finalized blocks can rename across volumes. Contributed by Hairong Kuang.
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch, HDFS-822.patch, HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-822) Appends to already-finalized blocks
can rename across volumes
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802576#action_12802576 ]
Hadoop QA commented on HDFS-822:
--------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12430802/HDFS-822.patch
against trunk revision 899747.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/196/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/196/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/196/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/196/console
This message is automatically generated.
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch, HDFS-822.patch, HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-822) Appends to already-finalized blocks can
rename across volumes
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-822:
-------------------------------
Attachment: (was: HDFS-800.patch)
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-822) Appends to already-finalized blocks
can rename across volumes
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799966#action_12799966 ]
Hairong Kuang commented on HDFS-822:
------------------------------------
I am looking at this issue. If datanode releases the lock while copying replicas, I am not sure how can we ensure data integrity. In append, there are two potential file copies, one is copy on write (the unlink operation) and other is cross-volumn copy as Todd pointed out.
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-822) Appends to already-finalized blocks
can rename across volumes
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789964#action_12789964 ]
dhruba borthakur commented on HDFS-822:
---------------------------------------
This looks like a bug that might be good to fix in 0.21 itself.
On a related note, suppose the last partial block of a file has three replicas on three different volumes on three different datanodes. And let's suppose that these volumes (disks) have no more free space on them whereas there is plenty of free space on other datanodes and disks. Now, if an application re-opens the file to append to it, the append will fail because there isn't any free space on those devices (even though the cluster has tons of free space). Is it possible to avoid this inconsistent behaviour?
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-822) Appends to already-finalized blocks
can rename across volumes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804887#action_12804887 ]
Hudson commented on HDFS-822:
-----------------------------
Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #101 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/101/])
. Appends to already-finalized blocks can rename across volumes. Contributed by Hairong Kuang.
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch, HDFS-822.patch, HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-822) Appends to already-finalized blocks
can rename across volumes
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788428#action_12788428 ]
Todd Lipcon commented on HDFS-822:
----------------------------------
I haven't tried to actually produce this bug yet, but it may be reasonably important, since this is also a synchronized method, and a full cross-volume block copy could take several seconds at least, during which the DN can't do a whole lot.
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-822) Appends to already-finalized blocks
can rename across volumes
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802498#action_12802498 ]
Todd Lipcon commented on HDFS-822:
----------------------------------
Looks good to me. +1
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch, HDFS-822.patch, HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-822) Appends to already-finalized blocks
can rename across volumes
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800951#action_12800951 ]
Todd Lipcon commented on HDFS-822:
----------------------------------
In that case, shouldn't the current patch throw an exception when the current volume's full, rather than calling getNextVolume?
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-822) Appends to already-finalized blocks
can rename across volumes
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800409#action_12800409 ]
Todd Lipcon commented on HDFS-822:
----------------------------------
Patch looks like an improvement over the current code, but is there still a concern about holding the FSDataset in the case that this volume is full?
Given that we don't have an intra-datanode balancer, it seems reasonably common that some disks are entirely full while others have free space (eg on an old cluster where 500G disks are replaced piecemeal with TB)
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-822) Appends to already-finalized blocks can
rename across volumes
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-822:
-------------------------------
Attachment: HDFS-822.patch
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch, HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-822) Appends to already-finalized blocks can
rename across volumes
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-822:
-------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I've just committed this.
> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
> Key: HDFS-822
> URL: https://issues.apache.org/jira/browse/HDFS-822
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0, 0.22.0
>
> Attachments: HDFS-822.patch, HDFS-822.patch, HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block is already finalized, it needs to move it into the RBW directory so it can go back into a "being written" state. This is done using volumes.getNextVolume without preference to the volume that the block currently exists on. It seems to me that this could cause a lot of slow cross-volume copies on applications that periodically append/close/append/close a file. Instead, getNextVolume could provide an alternate form that gives preference to a particular volume, so the rename stays on the same disk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.