You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hairong Kuang (JIRA)" <ji...@apache.org> on 2007/09/18 00:20:43 UTC

[jira] Created: (HADOOP-1912) Datanode should support block replacement

Datanode should support block replacement
-----------------------------------------

                 Key: HADOOP-1912
                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
             Project: Hadoop
          Issue Type: New Feature
          Components: dfs
    Affects Versions: 0.14.1
            Reporter: Hairong Kuang
            Assignee: Hairong Kuang
             Fix For: 0.15.0


This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.

In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1912) Datanode should support block replacement

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-1912:
----------------------------------

    Status: Open  (was: Patch Available)

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.16.0
>
>         Attachments: replace.patch, replace1.patch, replace2.patch, replace3.patch, replace4.patch, replace5.patch, replace6.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1912) Datanode should support block replacement

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-1912:
----------------------------------

    Attachment: replace2.patch

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>         Attachments: replace.patch, replace1.patch, replace2.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1912) Datanode should support block replacement

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12539275 ] 

Hadoop QA commented on HADOOP-1912:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12368396/replace6.patch
against trunk revision r590273.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests +1.  The patch passed core unit tests.

    contrib tests +1.  The patch passed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1046/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1046/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1046/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1046/console

This message is automatically generated.

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.16.0
>
>         Attachments: replace.patch, replace1.patch, replace2.patch, replace3.patch, replace4.patch, replace5.patch, replace6.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1912) Datanode should support block replacement

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-1912:
----------------------------------

    Fix Version/s: 0.16.0
           Status: Patch Available  (was: Open)

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.16.0
>
>         Attachments: replace.patch, replace1.patch, replace2.patch, replace3.patch, replace4.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1912) Datanode should support block replacement

Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536856 ] 

Raghu Angadi commented on HADOOP-1912:
--------------------------------------

> 3. For comment 3, your change is not exactly the same as the logic in the current code. If you'd merge two checks into one, I will change the code to be [...]

I see now. I misread the code. No need to merge the conditions. thanks.

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>         Attachments: replace.patch, replace1.patch, replace2.patch, replace3.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1912) Datanode should support block replacement

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-1912:
----------------------------------

    Attachment: replace1.patch

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.15.0
>
>         Attachments: replace.patch, replace1.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1912) Datanode should support block replacement

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-1912:
----------------------------------

    Attachment:     (was: replace2.patch)

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>         Attachments: replace.patch, replace1.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1912) Datanode should support block replacement

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-1912:
----------------------------------

    Attachment: replace3.patch

This patch makes changes to the implementation of I/O throttling as suggested by Raghu. I also added a junit test for testing Throttling.

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>         Attachments: replace.patch, replace1.patch, replace2.patch, replace3.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1912) Datanode should support block replacement

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-1912:
----------------------------------

    Attachment: replace6.patch

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.16.0
>
>         Attachments: replace.patch, replace1.patch, replace2.patch, replace3.patch, replace4.patch, replace5.patch, replace6.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1912) Datanode should support block replacement

Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Raghu Angadi updated HADOOP-1912:
---------------------------------

    Comment: was deleted

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>         Attachments: replace.patch, replace1.patch, replace2.patch, replace3.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1912) Datanode should support block replacement

Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536854 ] 

Raghu Angadi commented on HADOOP-1912:
--------------------------------------

regd 3: 

{code} (priSet.contains(delNodeHint) || ( addedNode != null && !priSet.contains(addedNode)) {code}
looks same as 
{code} (addedNode != null || priSet.contains(delNodeHint)) {code}

Except the latter has one less look up. What am I missing :) ? In what case do these result in different results?

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>         Attachments: replace.patch, replace1.patch, replace2.patch, replace3.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1912) Datanode should support block replacement

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12539669 ] 

Hudson commented on HADOOP-1912:
--------------------------------

Integrated in Hadoop-Nightly #290 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/290/])

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.16.0
>
>         Attachments: replace.patch, replace1.patch, replace2.patch, replace3.patch, replace4.patch, replace5.patch, replace6.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1912) Datanode should support block replacement

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12537656 ] 

Hadoop QA commented on HADOOP-1912:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12368262/replace5.patch
against trunk revision r588083.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs -1.  The patch appears to introduce 1 new Findbugs warnings.

    core tests +1.  The patch passed core unit tests.

    contrib tests +1.  The patch passed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/981/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/981/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/981/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/981/console

This message is automatically generated.

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.16.0
>
>         Attachments: replace.patch, replace1.patch, replace2.patch, replace3.patch, replace4.patch, replace5.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1912) Datanode should support block replacement

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-1912:
----------------------------------

    Attachment: replace.patch

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.15.0
>
>         Attachments: replace.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1912) Datanode should support block replacement

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536802 ] 

Hairong Kuang commented on HADOOP-1912:
---------------------------------------

Thanks Raghu! 

1. Throttling test is in TestBlockReplacement.

2. Regd throttler, yes I totally agree with you. I will make ThrottlerBase to be package private and it would be nice if a throttler can be shared by mutlple threads. Let's see how this could be done.

3. For comment 3, your change is not exactly the same as the logic in the current code. If you'd merge two checks into one, I will change the code to be if(priSet.contains(delNodeHint) || ( addedNode != null && !priSet.contains(addedNode)) { cur = delNodeHint;}

4. For addBlock, the current code may have null locations in machineSet which cause serialization error, so I use an ArrayList first and then convert it into an array when constructing the result.


> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>         Attachments: replace.patch, replace1.patch, replace2.patch, replace3.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1912) Datanode should support block replacement

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12537819 ] 

Hadoop QA commented on HADOOP-1912:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12368396/replace6.patch
against trunk revision r588341.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests -1.  The patch failed core unit tests.

    contrib tests +1.  The patch passed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/998/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/998/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/998/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/998/console

This message is automatically generated.

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.16.0
>
>         Attachments: replace.patch, replace1.patch, replace2.patch, replace3.patch, replace4.patch, replace5.patch, replace6.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1912) Datanode should support block replacement

Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12537158 ] 

Raghu Angadi commented on HADOOP-1912:
--------------------------------------

There is a warning in eclipse for bandwidthPerSec member of Throttler since it not read anywhere.


> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.16.0
>
>         Attachments: replace.patch, replace1.patch, replace2.patch, replace3.patch, replace4.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1912) Datanode should support block replacement

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-1912:
----------------------------------

    Attachment: replace2.patch

The old patch was outdated. So here is a new one.

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>         Attachments: replace.patch, replace1.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1912) Datanode should support block replacement

Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-1912:
-------------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I just committed this. Thanks Hairong!

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.16.0
>
>         Attachments: replace.patch, replace1.patch, replace2.patch, replace3.patch, replace4.patch, replace5.patch, replace6.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1912) Datanode should support block replacement

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-1912:
----------------------------------

    Attachment: replace4.patch

Thanks Raghu for providing an implementation of a Throttler that can be shared by mutiple threads. This new patch incorporates his algorithm and a couple of minor changes.

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>         Attachments: replace.patch, replace1.patch, replace2.patch, replace3.patch, replace4.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1912) Datanode should support block replacement

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-1912:
----------------------------------

    Status: Patch Available  (was: Open)

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.16.0
>
>         Attachments: replace.patch, replace1.patch, replace2.patch, replace3.patch, replace4.patch, replace5.patch, replace6.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1912) Datanode should support block replacement

Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536911 ] 

Raghu Angadi commented on HADOOP-1912:
--------------------------------------

+1 code review.

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>         Attachments: replace.patch, replace1.patch, replace2.patch, replace3.patch, replace4.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1912) Datanode should support block replacement

Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536388 ] 

Raghu Angadi commented on HADOOP-1912:
--------------------------------------

Pretty much looks fine.

# I could not find throttling test. 
# Regd throttler : each connection is individually throttled. I think ideally we should use one throttler that is used by all connections. This will make sure we use up allowed b/w when ever possible. In the current scheme, transfer rate betwen A & B can not use extra b/w if another connection between B & C cannot use its quota (because C has many connections). Also when throttler is shared, small  blocks can not escape below the the radar. 
## Please make ThrottlerBase package private so that it can be used by HADOOP-2012
# minor : in FSNamesystem.java : {code}
        ///
        if( priSet.contains(delNodeHint)) {
          cur = delNodeHint;
        } else if(addedNode != null && !priSet.contains(addedNode)){
          cur = delNodeHint;
         }
/// Can be replaced by
       if (   addedNode != null || priSet.contains(delNodeHint) ) {
          cur = delNodeHint;
      }
{code}
# minor : it increases allocation in addBlock() in FSNameSystem.java. Is the current implementation more correct?

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>         Attachments: replace.patch, replace1.patch, replace2.patch, replace3.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1912) Datanode should support block replacement

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-1912:
----------------------------------

    Attachment: replace5.patch

This patch removes field bandwidthPerSec in Throttler.

> Datanode should support block replacement
> -----------------------------------------
>
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.16.0
>
>         Attachments: replace.patch, replace1.patch, replace2.patch, replace3.patch, replace4.patch, replace5.patch
>
>
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides to move a block B from Source S to Destination D. It also chooses a proxy source PS, which contains a replica of B, to speed up block copy.  The block placement is carried in the following steps:
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B> <source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>". 
> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived RPC informing the namenode that a block B is received and please delete a replica of B from source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each data node limits maximum number of concurrent data transfers (including both sending and receiving) for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle( int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller has sent or received since the last throttle is called. The method calculates the caller's I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data transfers is changed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.