You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Hairong Kuang (JIRA)" <ji...@apache.org> on 2009/08/12 20:33:14 UTC
[jira] Created: (HDFS-543) Break FSDatasetInterface#writeToBlock()
into writeToTemporary, writeToRBW, and append
Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append
-------------------------------------------------------------------------------------
Key: HDFS-543
URL: https://issues.apache.org/jira/browse/HDFS-543
Project: Hadoop HDFS
Issue Type: Sub-task
Components: data-node
Affects Versions: Append Branch
Reporter: Hairong Kuang
Assignee: Hairong Kuang
Fix For: Append Branch
FSDatasetInterface#writeToBlock() currently allows to create/recover a temporary replica, create/recover a RBW replica, or append to a replica. The implementation of the method in FSDataset is very complicated and error prone. I'd like to break this method into 3:
1. writeToTemporary allows to create a Temporary replica or recover from a packet error for a Tempoary replica;
2. writeToRBW allows to create a RBW replica or recover from a packet error for a RBW replica;
3. append allows to append to an existing Finalized replica.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-543) Break
FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and
append
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748270#action_12748270 ]
dhruba borthakur commented on HDFS-543:
---------------------------------------
+1 looks good.
For writeToTemporary() and FSDataSet.append(), this patch does not invoke stopWriters() to terminate any existing writers. Will it be wrong to invoke stopWriters() in all these three methods?
> Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append
> -------------------------------------------------------------------------------------
>
> Key: HDFS-543
> URL: https://issues.apache.org/jira/browse/HDFS-543
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: Append Branch
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: Append Branch
>
> Attachments: writeToReplica.patch, writeToReplica1.patch, writeToReplica2.patch, writeToReplica3.patch
>
>
> FSDatasetInterface#writeToBlock() currently allows to create/recover a temporary replica, create/recover a RBW replica, or append to a replica. The implementation of the method in FSDataset is very complicated and error prone. I'd like to break this method into 3:
> 1. writeToTemporary allows to create a Temporary replica or recover from a packet error for a Tempoary replica;
> 2. writeToRBW allows to create a RBW replica or recover from a packet error for a RBW replica;
> 3. append allows to append to an existing Finalized replica.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-543) Break
FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and
append
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748170#action_12748170 ]
Suresh Srinivas commented on HDFS-543:
--------------------------------------
+1 for the patch
> Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append
> -------------------------------------------------------------------------------------
>
> Key: HDFS-543
> URL: https://issues.apache.org/jira/browse/HDFS-543
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: Append Branch
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: Append Branch
>
> Attachments: writeToReplica.patch, writeToReplica1.patch, writeToReplica2.patch, writeToReplica3.patch
>
>
> FSDatasetInterface#writeToBlock() currently allows to create/recover a temporary replica, create/recover a RBW replica, or append to a replica. The implementation of the method in FSDataset is very complicated and error prone. I'd like to break this method into 3:
> 1. writeToTemporary allows to create a Temporary replica or recover from a packet error for a Tempoary replica;
> 2. writeToRBW allows to create a RBW replica or recover from a packet error for a RBW replica;
> 3. append allows to append to an existing Finalized replica.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-543) Break FSDatasetInterface#writeToBlock()
into writeToTemporary, writeToRBW, and append
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-543:
-------------------------------
Attachment: writeToReplica.patch
This patch
1. replaces writeToBlock by append, writeToTemporary, and writeToRbw to FSDatasetInterface and implements these 3 APIs in both FSDataset and SimulatedFSDataset.
2. Changes ReplicaInPipeline to make sure that each replica has only one writer.
3. When DataNode starts, replicas under tmp are removed.
4. Starting from this patch, any replicas created by a client are initially placed under rbw sbudir and are loaded as RWR (ReplicaWaitingToBeRecovered) at DataNode startup time. RWR replicas are sent as part of block reports. Finalized & RWR are not distinguished in a block report for now.
> Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append
> -------------------------------------------------------------------------------------
>
> Key: HDFS-543
> URL: https://issues.apache.org/jira/browse/HDFS-543
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: Append Branch
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: Append Branch
>
> Attachments: writeToReplica.patch
>
>
> FSDatasetInterface#writeToBlock() currently allows to create/recover a temporary replica, create/recover a RBW replica, or append to a replica. The implementation of the method in FSDataset is very complicated and error prone. I'd like to break this method into 3:
> 1. writeToTemporary allows to create a Temporary replica or recover from a packet error for a Tempoary replica;
> 2. writeToRBW allows to create a RBW replica or recover from a packet error for a RBW replica;
> 3. append allows to append to an existing Finalized replica.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-543) Break FSDatasetInterface#writeToBlock()
into writeToTemporary, writeToRBW, and append
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-543:
-------------------------------
Attachment: writeToReplica3.patch
This patch additionally adds a BlockNotFoundException to handle the error when a replica does not exists for append or a replica with the right state does not exist.
> Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append
> -------------------------------------------------------------------------------------
>
> Key: HDFS-543
> URL: https://issues.apache.org/jira/browse/HDFS-543
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: Append Branch
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: Append Branch
>
> Attachments: writeToReplica.patch, writeToReplica1.patch, writeToReplica2.patch, writeToReplica3.patch
>
>
> FSDatasetInterface#writeToBlock() currently allows to create/recover a temporary replica, create/recover a RBW replica, or append to a replica. The implementation of the method in FSDataset is very complicated and error prone. I'd like to break this method into 3:
> 1. writeToTemporary allows to create a Temporary replica or recover from a packet error for a Tempoary replica;
> 2. writeToRBW allows to create a RBW replica or recover from a packet error for a RBW replica;
> 3. append allows to append to an existing Finalized replica.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-543) Break
FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and
append
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748076#action_12748076 ]
Konstantin Boudnik commented on HDFS-543:
-----------------------------------------
+1 on the new test implementation.
> Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append
> -------------------------------------------------------------------------------------
>
> Key: HDFS-543
> URL: https://issues.apache.org/jira/browse/HDFS-543
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: Append Branch
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: Append Branch
>
> Attachments: writeToReplica.patch, writeToReplica1.patch, writeToReplica2.patch
>
>
> FSDatasetInterface#writeToBlock() currently allows to create/recover a temporary replica, create/recover a RBW replica, or append to a replica. The implementation of the method in FSDataset is very complicated and error prone. I'd like to break this method into 3:
> 1. writeToTemporary allows to create a Temporary replica or recover from a packet error for a Tempoary replica;
> 2. writeToRBW allows to create a RBW replica or recover from a packet error for a RBW replica;
> 3. append allows to append to an existing Finalized replica.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-543) Break
FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and
append
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748007#action_12748007 ]
Konstantin Boudnik commented on HDFS-543:
-----------------------------------------
+1 on the test framework version: new tests are suppose to be JUnit4 compliant.
> Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append
> -------------------------------------------------------------------------------------
>
> Key: HDFS-543
> URL: https://issues.apache.org/jira/browse/HDFS-543
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: Append Branch
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: Append Branch
>
> Attachments: writeToReplica.patch, writeToReplica1.patch
>
>
> FSDatasetInterface#writeToBlock() currently allows to create/recover a temporary replica, create/recover a RBW replica, or append to a replica. The implementation of the method in FSDataset is very complicated and error prone. I'd like to break this method into 3:
> 1. writeToTemporary allows to create a Temporary replica or recover from a packet error for a Tempoary replica;
> 2. writeToRBW allows to create a RBW replica or recover from a packet error for a RBW replica;
> 3. append allows to append to an existing Finalized replica.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HDFS-543) Break FSDatasetInterface#writeToBlock()
into writeToTemporary, writeToRBW, and append
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang resolved HDFS-543.
--------------------------------
Resolution: Fixed
Release Note: I've already committed this.
Hadoop Flags: [Reviewed]
> Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append
> -------------------------------------------------------------------------------------
>
> Key: HDFS-543
> URL: https://issues.apache.org/jira/browse/HDFS-543
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: Append Branch
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: Append Branch
>
> Attachments: writeToReplica.patch, writeToReplica1.patch, writeToReplica2.patch, writeToReplica3.patch
>
>
> FSDatasetInterface#writeToBlock() currently allows to create/recover a temporary replica, create/recover a RBW replica, or append to a replica. The implementation of the method in FSDataset is very complicated and error prone. I'd like to break this method into 3:
> 1. writeToTemporary allows to create a Temporary replica or recover from a packet error for a Tempoary replica;
> 2. writeToRBW allows to create a RBW replica or recover from a packet error for a RBW replica;
> 3. append allows to append to an existing Finalized replica.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-543) Break
FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and
append
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748437#action_12748437 ]
Hairong Kuang commented on HDFS-543:
------------------------------------
Thanks Suresh, Dhruba, and Cos for spending time reviewing this patch.
> For writeToTemporary() and FSDataSet.append(), this patch does not invoke stopWriters() to terminate any existing writers.
writeToTemporary() is to create a new temporary replica, so there is no writer to stop. Append() is for appending to a finalized replica, so the replica has no writer either.
> Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append
> -------------------------------------------------------------------------------------
>
> Key: HDFS-543
> URL: https://issues.apache.org/jira/browse/HDFS-543
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: Append Branch
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: Append Branch
>
> Attachments: writeToReplica.patch, writeToReplica1.patch, writeToReplica2.patch, writeToReplica3.patch
>
>
> FSDatasetInterface#writeToBlock() currently allows to create/recover a temporary replica, create/recover a RBW replica, or append to a replica. The implementation of the method in FSDataset is very complicated and error prone. I'd like to break this method into 3:
> 1. writeToTemporary allows to create a Temporary replica or recover from a packet error for a Tempoary replica;
> 2. writeToRBW allows to create a RBW replica or recover from a packet error for a RBW replica;
> 3. append allows to append to an existing Finalized replica.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-543) Break
FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and
append
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747808#action_12747808 ]
Suresh Srinivas commented on HDFS-543:
--------------------------------------
Comments:
# nit: Some of the lines exceed 80 chars
# TestSimulatedFSDataset.java - unrelated to this patch there are two warnings of unused variables {{blockReport2}} and {{bytesAdded}}
# SimulatedFSDataset.java - unrelated to this patch private constructor is unused
# TestWriteToReplica.java is missing. Not sure if that is intentional. Also this test was not based on new junit.
> Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append
> -------------------------------------------------------------------------------------
>
> Key: HDFS-543
> URL: https://issues.apache.org/jira/browse/HDFS-543
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: Append Branch
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: Append Branch
>
> Attachments: writeToReplica.patch, writeToReplica1.patch
>
>
> FSDatasetInterface#writeToBlock() currently allows to create/recover a temporary replica, create/recover a RBW replica, or append to a replica. The implementation of the method in FSDataset is very complicated and error prone. I'd like to break this method into 3:
> 1. writeToTemporary allows to create a Temporary replica or recover from a packet error for a Tempoary replica;
> 2. writeToRBW allows to create a RBW replica or recover from a packet error for a RBW replica;
> 3. append allows to append to an existing Finalized replica.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-543) Break FSDatasetInterface#writeToBlock()
into writeToTemporary, writeToRBW, and append
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-543:
-------------------------------
Release Note: (was: I've already committed this.)
I've already committed this.
> Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append
> -------------------------------------------------------------------------------------
>
> Key: HDFS-543
> URL: https://issues.apache.org/jira/browse/HDFS-543
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: Append Branch
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: Append Branch
>
> Attachments: writeToReplica.patch, writeToReplica1.patch, writeToReplica2.patch, writeToReplica3.patch
>
>
> FSDatasetInterface#writeToBlock() currently allows to create/recover a temporary replica, create/recover a RBW replica, or append to a replica. The implementation of the method in FSDataset is very complicated and error prone. I'd like to break this method into 3:
> 1. writeToTemporary allows to create a Temporary replica or recover from a packet error for a Tempoary replica;
> 2. writeToRBW allows to create a RBW replica or recover from a packet error for a RBW replica;
> 3. append allows to append to an existing Finalized replica.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-543) Break
FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and
append
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747702#action_12747702 ]
Hairong Kuang commented on HDFS-543:
------------------------------------
One more change that the new patch makes is that it removes "isRecovery" parameter from FSDatasetInterface#writeToTemporary() based on the fact that Temporary replica does not support packet recovery.
> Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append
> -------------------------------------------------------------------------------------
>
> Key: HDFS-543
> URL: https://issues.apache.org/jira/browse/HDFS-543
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: Append Branch
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: Append Branch
>
> Attachments: writeToReplica.patch, writeToReplica1.patch
>
>
> FSDatasetInterface#writeToBlock() currently allows to create/recover a temporary replica, create/recover a RBW replica, or append to a replica. The implementation of the method in FSDataset is very complicated and error prone. I'd like to break this method into 3:
> 1. writeToTemporary allows to create a Temporary replica or recover from a packet error for a Tempoary replica;
> 2. writeToRBW allows to create a RBW replica or recover from a packet error for a RBW replica;
> 3. append allows to append to an existing Finalized replica.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-543) Break FSDatasetInterface#writeToBlock()
into writeToTemporary, writeToRBW, and append
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-543:
-------------------------------
Attachment: writeToReplica2.patch
This patch incorporates comments 1 and 4. I also changed TestDataNodeRestart, which was committed in HDFS-544, to use junit4.
For comment 2, the constructor is used in Reflection so can not be removed. For comment 3, I took a quick look, it seems not trivial to remove those variables. Not sure if this is a programming bug or simply bad style. I will leave them there for now.
> Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append
> -------------------------------------------------------------------------------------
>
> Key: HDFS-543
> URL: https://issues.apache.org/jira/browse/HDFS-543
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: Append Branch
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: Append Branch
>
> Attachments: writeToReplica.patch, writeToReplica1.patch, writeToReplica2.patch
>
>
> FSDatasetInterface#writeToBlock() currently allows to create/recover a temporary replica, create/recover a RBW replica, or append to a replica. The implementation of the method in FSDataset is very complicated and error prone. I'd like to break this method into 3:
> 1. writeToTemporary allows to create a Temporary replica or recover from a packet error for a Tempoary replica;
> 2. writeToRBW allows to create a RBW replica or recover from a packet error for a RBW replica;
> 3. append allows to append to an existing Finalized replica.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-543) Break
FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and
append
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748060#action_12748060 ]
Hairong Kuang commented on HDFS-543:
------------------------------------
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 15 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
> Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append
> -------------------------------------------------------------------------------------
>
> Key: HDFS-543
> URL: https://issues.apache.org/jira/browse/HDFS-543
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: Append Branch
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: Append Branch
>
> Attachments: writeToReplica.patch, writeToReplica1.patch, writeToReplica2.patch
>
>
> FSDatasetInterface#writeToBlock() currently allows to create/recover a temporary replica, create/recover a RBW replica, or append to a replica. The implementation of the method in FSDataset is very complicated and error prone. I'd like to break this method into 3:
> 1. writeToTemporary allows to create a Temporary replica or recover from a packet error for a Tempoary replica;
> 2. writeToRBW allows to create a RBW replica or recover from a packet error for a RBW replica;
> 3. append allows to append to an existing Finalized replica.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-543) Break FSDatasetInterface#writeToBlock()
into writeToTemporary, writeToRBW, and append
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-543:
-------------------------------
Attachment: writeToReplica1.patch
> 1. BlockReceiver.java - when clientName is empty, the block being written to is considered temporary. Should it be more explicit that this? Can a real client set its name to empty also?
This is what we do in the trunk. A real client's name is guaranteed to be not empty.
> 2. BlockReceiver.java - for client append the current code checks for finalized and isRecovery to be true. Is there a need to use isRecovery? Also what happens in the case finalized is true and isRecovery false?
This is also what the trunk does now. In the append case, isRecovery must be true. If isFinalized is true and isRecovery is false, BlockReciever will throw an exception BlockAlreadyExists.
For comments 5, 6, and 7, I simplified the updateBlock logic by calling stopWriter. This part will be rewritten anyway in the future.
The new patch incorporates all remaining comments.
> Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append
> -------------------------------------------------------------------------------------
>
> Key: HDFS-543
> URL: https://issues.apache.org/jira/browse/HDFS-543
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: Append Branch
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: Append Branch
>
> Attachments: writeToReplica.patch, writeToReplica1.patch
>
>
> FSDatasetInterface#writeToBlock() currently allows to create/recover a temporary replica, create/recover a RBW replica, or append to a replica. The implementation of the method in FSDataset is very complicated and error prone. I'd like to break this method into 3:
> 1. writeToTemporary allows to create a Temporary replica or recover from a packet error for a Tempoary replica;
> 2. writeToRBW allows to create a RBW replica or recover from a packet error for a RBW replica;
> 3. append allows to append to an existing Finalized replica.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-543) Break
FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and
append
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747551#action_12747551 ]
Suresh Srinivas commented on HDFS-543:
--------------------------------------
# BlockReceiver.java - when {{clientName}} is empty, the block being written to is considered temporary. Should it be more explicit that this? Can a real client set its name to empty also?
# BlockReceiver.java - for client append the current code checks for {{finalized}} and {{isRecovery}} to be true. Is there a need to use {{isRecovery}}? Also what happens in the case {{finalized}} is true and {{isRecovery}} false?
# ReplicaBeingWritten.java has an unused import {{List}}
# {{ReplicaInPipeline.stopAndWaitWriter}} - prefer method name to be {{stopWriter}}
# FSDataset.updateBlock - we could use a utility class to interrupt thread that could be used for {{ReplicaInPipeline.stopAndWaitWriter}}. If not at least we should have similar checks as in {{stopAndWaitWriter}} (such as if it is same as current thread, if the thread is alive).
# FSDataset.tryUpdateBlock - should the check for {{thread.isAlive()}} moved to {{ReplicaInPipeline.getWriter()}}?
# nit: {{FSDataset.tryUpdateBlock}} - no need for else where write is set to null
# FSDataset.append - when renaming metadata file succeeds and renaming block file fails, should we rename the metadata file back to old file?
# FSDataset.append - creating RBW file and metadata file from the existing finalized block, should this functionality to move to ReplicaInfo as a method? It could be done by having a method {{static ReplicaBeingWritten FinalizedReplica.createRbwFile(ReplicaInfo info))}}?
# For all new debug logs, add check for isDebugEnabled.
I will review the tests in the next version of the patch
> Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append
> -------------------------------------------------------------------------------------
>
> Key: HDFS-543
> URL: https://issues.apache.org/jira/browse/HDFS-543
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: Append Branch
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: Append Branch
>
> Attachments: writeToReplica.patch
>
>
> FSDatasetInterface#writeToBlock() currently allows to create/recover a temporary replica, create/recover a RBW replica, or append to a replica. The implementation of the method in FSDataset is very complicated and error prone. I'd like to break this method into 3:
> 1. writeToTemporary allows to create a Temporary replica or recover from a packet error for a Tempoary replica;
> 2. writeToRBW allows to create a RBW replica or recover from a packet error for a RBW replica;
> 3. append allows to append to an existing Finalized replica.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.