You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Iyappan Srinivasan (JIRA)" <ji...@apache.org> on 2009/04/17 10:20:15 UTC
[jira] Updated: (HADOOP-5693) The distcp command is working slower
in trunk when compared to hadoop-0.18.3
[ https://issues.apache.org/jira/browse/HADOOP-5693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Iyappan Srinivasan updated HADOOP-5693:
---------------------------------------
Description:
1st scenario :
Source cluster (3 slaves) is hadoop-0.18.3 and destination cluster is trunk (3 slaves).
a) The source has 3 files in a directory equaling 0.4 GB
b) distcp is used to copy these contents to a destination directory.
(bin/hadoop --config ~/cluster_dir distcp -update -delete hftp://<IPFQDN>:<port>//dir1/file1 hdfs://<IPFQDN:PORT>//copied/)
c) It takes 34 seconds
2nd scenario :
Source cluster is trunk. and destination cluster (3 slaves) is hadoop-0.18.3
a) The source has 3 files in a directory equaling 0.4 GB
b) distcp is used to copy these contents to a destination directory.
c) It takes 19 seconds
>From this we can conclude, In the latest trunk, distcp has considerabley slowed down ( around 78%).
was:
1st scenario :
Source cluster (3 slaves) is hadoop-0.18.3-2486615 and destination cluster is hadoop-0.20.2678219V3 (3 slaves).
a) The source has 3 files in a directory equaling 0.4 GB
b) distcp is used to copy these contents to a destination directory.
(bin/hadoop --config ~/cluster_dir distcp -update -delete hftp://<IPFQDN>:<port>//dir1/file1 hdfs://<IPFQDN:PORT>//copied/)
c) It takes 36 seconds
2nd scenario :
Source cluster is hadoop-0.20.2678219V3. and destination cluster (3 slaves) is hadoop-0.18.3-2486615.
a) The source has 3 files in a directory equaling 0.4 GB
b) distcp is used to copy these contents to a destination directory.
c) It takes 20 seconds
>From this we can conclude, In the latest 0.20.2678219V3, distcp has considerabley slowed down ( around 90%).
Summary: The distcp command is working slower in trunk when compared to hadoop-0.18.3 (was: The distcp command is working slower in hadoop 0.20.2678219V3 when compared to hadoop-0.18.3-2486615)
> The distcp command is working slower in trunk when compared to hadoop-0.18.3
> ----------------------------------------------------------------------------
>
> Key: HADOOP-5693
> URL: https://issues.apache.org/jira/browse/HADOOP-5693
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.20.0
> Environment: cluster is brought up with hod in both 0.18.3 and 0.20 versions
> Reporter: Iyappan Srinivasan
>
> 1st scenario :
> Source cluster (3 slaves) is hadoop-0.18.3 and destination cluster is trunk (3 slaves).
> a) The source has 3 files in a directory equaling 0.4 GB
> b) distcp is used to copy these contents to a destination directory.
> (bin/hadoop --config ~/cluster_dir distcp -update -delete hftp://<IPFQDN>:<port>//dir1/file1 hdfs://<IPFQDN:PORT>//copied/)
> c) It takes 34 seconds
> 2nd scenario :
> Source cluster is trunk. and destination cluster (3 slaves) is hadoop-0.18.3
> a) The source has 3 files in a directory equaling 0.4 GB
> b) distcp is used to copy these contents to a destination directory.
> c) It takes 19 seconds
> From this we can conclude, In the latest trunk, distcp has considerabley slowed down ( around 78%).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.