You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hairong Kuang (JIRA)" <ji...@apache.org> on 2007/09/17 21:01:47 UTC
[jira] Created: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Restructure data node code so that block sending/receiving is seperated from data transfer header handling
----------------------------------------------------------------------------------------------------------
Key: HADOOP-1908
URL: https://issues.apache.org/jira/browse/HADOOP-1908
Project: Hadoop
Issue Type: Improvement
Components: dfs
Affects Versions: 0.14.1
Reporter: Hairong Kuang
Assignee: Hairong Kuang
Fix For: 0.15.0
This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528950 ]
Raghu Angadi commented on HADOOP-1908:
--------------------------------------
> Another change in protocol to note is that previous datanode used to send empty 'DATA_CHUNK' (ie. with len == 0), now it sends just '(int)0' to
Actually, when a DFSClient writes a block it still sends an empty DATA_CHUNK, but when a Datanode writes a block, it sends '0'.
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch, datanodeRestructure.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528949 ]
Raghu Angadi commented on HADOOP-1908:
--------------------------------------
+1. Couple of observations :
# As you noted in the conversation we had, this patch results in an exception printed to Datanode log on the mirrors for most of the blocks received.
# Another change in protocol to note is that previous datanode used to send empty 'DATA_CHUNK' (ie. with len == 0), now it sends just '(int)0' to properly end a stream.
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch, datanodeRestructure.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529316 ]
Hadoop QA commented on HADOOP-1908:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12366313/datanodeRestructure.patch
against trunk revision r577917.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests -1. The patch failed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/802/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/802/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/802/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/802/console
This message is automatically generated.
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch, datanodeRestructure.patch, datanodeRestructure.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528841 ]
Raghu Angadi commented on HADOOP-1908:
--------------------------------------
bq. I think that a datanode should report an error if the next datanode in the pipeline reported an error. This allows the client to detect if it failed to create the desired number of replicas.
Even if this is the behavior we want, which I don't think we should do, it belongs in a different Jira. As you noted, it needs more changes just returning an error.
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528893 ]
Hairong Kuang commented on HADOOP-1908:
---------------------------------------
I agree with Dhruba that a datanode should report an error if the next datanode in the pipeline reported an error as what I tried to do in my first patch. I also agree with Raghu that this should be done in a separate jira.
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1908:
----------------------------------
Status: Patch Available (was: Open)
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch, datanodeRestructure.patch, datanodeRestructure.patch, datanodeRestructure1.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1908:
----------------------------------
Status: Patch Available (was: Open)
Thanks a lot, Raghu!
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch, datanodeRestructure.patch, datanodeRestructure.patch, datanodeRestructure1.patch, datanodeRestructure2.patch, datanodeRestructure3.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528927 ]
Raghu Angadi commented on HADOOP-1908:
--------------------------------------
Sure thats fine. I personally not a big fan of including any code 'just to be sure'. Another example of extra checks :
{code}
if(out != null)
IOUtils.closeStream(out);
{code}
We added IOUtils.closeStream() exactly to capture this common case that checks if stream null and also ignores IOException from out.close(). I will review the rest of it now.
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch, datanodeRestructure.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530272 ]
Raghu Angadi commented on HADOOP-1908:
--------------------------------------
+1. Code review.
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch, datanodeRestructure.patch, datanodeRestructure.patch, datanodeRestructure1.patch, datanodeRestructure2.patch, datanodeRestructure3.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1908:
----------------------------------
Attachment: datanodeRestructure1.patch
I am not able to reproduce the error reported in Hudson build. I am uploading a patch with very minor changes to trigger the build again.
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch, datanodeRestructure.patch, datanodeRestructure.patch, datanodeRestructure1.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1908:
----------------------------------
Status: Patch Available (was: Open)
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch, datanodeRestructure.patch, datanodeRestructure.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530809 ]
Hudson commented on HADOOP-1908:
--------------------------------
Integrated in Hadoop-Nightly #252 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/252/])
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch, datanodeRestructure.patch, datanodeRestructure.patch, datanodeRestructure1.patch, datanodeRestructure2.patch, datanodeRestructure3.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529599 ]
Hadoop QA commented on HADOOP-1908:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12366388/datanodeRestructure1.patch
against trunk revision r578249.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/806/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/806/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/806/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/806/console
This message is automatically generated.
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch, datanodeRestructure.patch, datanodeRestructure.patch, datanodeRestructure1.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1908:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Hairong!
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch, datanodeRestructure.patch, datanodeRestructure.patch, datanodeRestructure1.patch, datanodeRestructure2.patch, datanodeRestructure3.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528836 ]
dhruba borthakur commented on HADOOP-1908:
------------------------------------------
I think that a datanode should report an error if the next datanode in the pipeline reported an error. This allows the client to detect if it failed to create the desired number of replicas.
The recovery from this error is simple. The client abandons the entire block, fetches a new blockid and block locations from the namenode and restarts flushing the current block.
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1908:
----------------------------------
Attachment: datanodeRestructure2.patch
This patch touches the code that effects the core functionality of data nodes. Many thanks to Raghu for his time on reviewing my patch line by line to make sure that the patch works correctly . This new patch incorporates all his comments.
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch, datanodeRestructure.patch, datanodeRestructure.patch, datanodeRestructure1.patch, datanodeRestructure2.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1908:
----------------------------------
Status: Open (was: Patch Available)
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch, datanodeRestructure.patch, datanodeRestructure.patch, datanodeRestructure1.patch, datanodeRestructure2.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1908:
----------------------------------
Attachment: datanodeRestructure.patch
Thanks Raghu for his detailed review and comments. This patch incorporates all of his comments except for comment 2.
I keep all those code that closes streams. The reason is that those streams are not a socket's raw streams. Those are buffered data streams that buit on top of a socket's raw streams. I think it is a good practice that make sure those are closed and therefore related resources are released.
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch, datanodeRestructure.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1908:
----------------------------------
Attachment: datanode.patch
In addition to code restructure, this patch introduces the following changes:
1. Add a Throttler interface to allow data transfer throttling. A Throttler parameter is added to both block send & receiving interface. If the throttler parameter is null, throttling is disallowed.
2. Minor changes to data transfer protocols:
* OP_READ_BLOCK: return chunk offset only when start offset is not 0;
return OP_STATUS_ERROR if received an invalid block id or an invalid length
* OP_WRITE_BLOCK: return OP_STATUS_ERROR if recived an invalid bytesPerChecksum or invalid block id
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528580 ]
Raghu Angadi commented on HADOOP-1908:
--------------------------------------
The overall structure looks good. A few comments and a few corner cases :
# Optional startOffset for OP_READ_BLOCK. Is there any use of making it optional? (See the first comment).
# Extra closeStream() calls are not necessary (set the above comment).
# BlockSender needs a close() method that closes any files that were open. If there is an exception around DataNode.java:1339, this could leak file descriptors. Also this close() method can used inside BlockSender's 'finally' blocks as well.
# Same for BlockReceiver
# in writeBlock(), failure to read status from mirror results in this datanode reporting error to source. This is not expected. It should return success as long as local writeBlock() is successful.
# In BlockSender constructor, the initiall 'XXX' comment and the following line should be removed since they are not relevant.
# The following code prints a success log which could be followed by a failure log : {code}
blockSender.sendBlock(out, null);
//REVIEW we are printing this even though it could be an error.
LOG.info("Transmitted block " + b + " to " + curTarget);
DataInputStream in = new DataInputStream(new BufferedInputStream(
sock.getInputStream(), BUFFER_SIZE));
short opStatus = in.readShort();
if(opStatus != FSConstants.OP_STATUS_SUCCESS ) {
LOG.info("Block " +b + " transfer failed.");
}
{code} You could just use the warning in following catch {}.
# Another log related : Inside writeBlock() we try to write status back to remote side even if there were read errors from the remote side, which will again fail resulting in a spurious error message.
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528123 ]
Raghu Angadi commented on HADOOP-1908:
--------------------------------------
I will look at the patch soon. Regd returning chunk offset only if offset is not 0 sound fine but this would be the only optional field in the protocol. Having such an optional field would make generic header parsing etc (like the changes proposed here) harder to make. Why not just return the start offset for every OP_READ_BLOCK?. My opinion.
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529265 ]
Raghu Angadi commented on HADOOP-1908:
--------------------------------------
+1. Thanks Hairong.
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch, datanodeRestructure.patch, datanodeRestructure.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528504 ]
Raghu Angadi commented on HADOOP-1908:
--------------------------------------
I am still looking at the patch.
There are extra 'closeStream()' calls added as part of this patch. Previously we intentionally did not close the streams.
These are not required since when we close the socket, all the associated streams are closed by definition. Could you remove the extra closeStream() calls?
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1908:
----------------------------------
Attachment: datanodeRestructure.patch
This patch includes all of Raghu's new comments:
1. Remove additional check for null streams. It also makes BlockSender & BlockReceiver to be "Closable", so we can use IOUtils.closeStream to close block senders & receivers.
2. Add code to receive the response for a block write request so it eliminates the additional exception thrown by a data node when sends write response back. But the code ignores the response for now and leaves HADOOP-1927 to correctly handle the response.
3. DFSClient sends a "0" to mark the end of a block. I also marked the protocol change in the comment of DATA_TRANSFER_VERSION increment.
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch, datanodeRestructure.patch, datanodeRestructure.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1908:
----------------------------------
Attachment: datanodeRestructure3.patch
The previous patch included a fix to HADOOP-1940. But since Chris created a jira and submitted a patch to it, the new patch removes the fix to HADOOP-1940.
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch, datanodeRestructure.patch, datanodeRestructure.patch, datanodeRestructure1.patch, datanodeRestructure2.patch, datanodeRestructure3.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529679 ]
Raghu Angadi commented on HADOOP-1908:
--------------------------------------
I checked {{writeBlock()}} to see if the first build failures could be explained. I noticed the the following (not related to the build failure) :
# BlockReceiver : It invokes {{data.finalizeBlock()}} before closing the block files. This will fail on Windows
# BlockReciever.receiveChunk() : {code}
//
// If stream-copy fails, continue
// writing to disk. We shouldn't
// interrupt client write.
//
mirrorOut = null;
{code} The above does not work the the way comments intends because mirrorOut is just an arg and has no effect on mirrorOut up its stack (i.e. in BlockReceiver.receiveBlock() and writeBlock().) This worked ebefore because all of these were in single function.
# Minor : DFSClient.endBlock() : Couple of checks in '{{while (bytesLeft > 0) {..}}' are not necessary because they were required when the loop was 'while (bytesLeft >= 0){.. }}'.
# In BlockReceiver.receiveBlock() : the check '{{if (lastLen > 0 && lastLen != bytesPerChecksum) {..}}' was meant to be done for every chunk received. Now this is checked only for the last-but-one chunk.
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch, datanodeRestructure.patch, datanodeRestructure.patch, datanodeRestructure1.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1908:
----------------------------------
Status: Open (was: Patch Available)
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch, datanodeRestructure.patch, datanodeRestructure.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1908) Restructure data node code so that
block sending/receiving is seperated from data transfer header handling
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530327 ]
Hadoop QA commented on HADOOP-1908:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12366540/datanodeRestructure3.patch
against trunk revision r579353.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/828/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/828/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/828/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/828/console
This message is automatically generated.
> Restructure data node code so that block sending/receiving is seperated from data transfer header handling
> ----------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1908
> URL: https://issues.apache.org/jira/browse/HADOOP-1908
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanode.patch, datanodeRestructure.patch, datanodeRestructure.patch, datanodeRestructure1.patch, datanodeRestructure2.patch, datanodeRestructure3.patch
>
>
> This jira is intended for code sharing. I'd like to have a BlockSender which is resposible for reading a block from a disk and writing it to an output stream, and a BlockReceiver which receives a block from an input stream, writes it to a disk, and possibly writes it a mirror ouput stream to support pipeline writes. Block sender and receiver code is independent of any block IO protocols so it could be shared by block transfer, block write, block read, and future block replacement introduced by data node rebalancing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.