[jira] [Commented] (HDFS-17868) introduce BlockPlacementPolicyCrossDC for multi datacenter stretched hdfs cluster

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-17868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18048716#comment-18048716 ] 

ASF GitHub Bot commented on HDFS-17868:
---------------------------------------

eubnara opened a new pull request, #8159:
URL: https://github.com/apache/hadoop/pull/8159

   
   
   <!--
     Thanks for sending a pull request!
       1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
       2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
   -->
   
   ### Description of PR
   
   I wrote description on https://issues.apache.org/jira/browse/HDFS-17868.
   
   
   ### How was this patch tested?
   
   Manual test on the private hdfs cluster.
   
   
   ### For code changes:
   
   - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
   - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
   - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
   
   




> introduce BlockPlacementPolicyCrossDC for multi datacenter stretched hdfs cluster
> ---------------------------------------------------------------------------------
>
>                 Key: HDFS-17868
>                 URL: https://issues.apache.org/jira/browse/HDFS-17868
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: block placement
>            Reporter: YUBI LEE
>            Priority: Major
>
> I got ideas from https://dan.naver.com/25/sessions/692, https://www.youtube.com/watch?v=1h4k_Dbt0t8, I implemented "BlockPlacementPolicyCrossDC" policy. Thanks to [~acedia28].
> It would be better if [~acedia28] shares the better version of the block placement policy.
> It introduces some configurations:
> (default value written in parenthesis)
> {code}
> dfs.block.replicator.cross.dc.async.enabled (false)
> dfs.block.replicator.cross.dc.preferred.datacenter
> dfs.block.replicator.cross.dc.bandwidth.limit.mb (5120)
> dfs.block.replicator.cross.dc.bandwidth.refill.period.sec (1)
> dfs.block.replicator.cross.dc.sync.paths
> dfs.block.replicator.cross.dc.limited.sync.paths
> {code}
> According to ideas from the session I mentioned above, this policy introduces 3 ways to write hdfs block.
> - sync write: the original hdfs way
> - limited sync write: using bucket4j, sync write < threshold, async write > threshold.
> - async write: return datanode candidates only which locate the same datacenter to hdfs client, under replicated blocks will replicated later in asynchronous way.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org