You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "Dinesh Chitlangia (Jira)" <ji...@apache.org> on 2024/03/28 14:50:00 UTC

[jira] [Resolved] (HDFS-17216) When distcp handle the small files, the bandwidth parameter will be invalid, resulting in serious overspeed behavior

     [ https://issues.apache.org/jira/browse/HDFS-17216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dinesh Chitlangia resolved HDFS-17216.
--------------------------------------
    Fix Version/s: 3.5.0
         Assignee: xiaojunxiang
       Resolution: Fixed

Thanks [~bigdata_zoodev]  for the contribution and [~hiwangzhihui] for the review.

> When distcp handle the small files, the bandwidth parameter will be invalid, resulting in serious overspeed behavior
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-17216
>                 URL: https://issues.apache.org/jira/browse/HDFS-17216
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: distcp
>    Affects Versions: 3.3.4
>            Reporter: xiaojunxiang
>            Assignee: xiaojunxiang
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 3.5.0
>
>         Attachments: DiscpAnalyze.jpg
>
>
> When distcp copies small files (file size slightly smaller than the bandwidth), the throbber only starts to throb after 1 second, and the throttled is specific to a single file. so the throbber becomes invalid, causing distcp to fill the cluster bandwidth and crush production traffic, which is a terrible thing. 
> Also, it takes time for files to set up the IO pipeline, so you shouldn't test with very small files, which will slow the transfer, especially as bandwidth kicks in, which will amplify the impact of small files on the rate



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org