You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Eremikhin Alexey <a....@corp.badoo.com> on 2016/01/27 13:24:19 UTC
WebHDFS listing takes 0.7ms per item
Hi,
I've met a performance issue after migration from Hadoop 1.2.1 to 2.7.1
Listing of directories in WebHDFS started taking significantly more time
than it took before.
I've performed some experiments and concluded that it takes 0.7 per item
in directory.
That means listing of directory with 10000 objects takes 7 seconds.
But that was checked on Hadoop 1.2.1 it takes about 700ms (on the same
sever).
Java version does not affect test results (7/8)
Does anyone know about such a behaviour?
There are details on tests and configuration.
*
alexxz@hadoop1.mlan:~> hadoop version*
Hadoop 2.7.1
Subversion Unknown -r Unknown
Compiled by root on 2015-09-11T13:58Z
Compiled with protoc 2.5.0
From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a
This command was run using
/local/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar
*alexxz@hadoop1.mlan:~> java -version*
java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
*Experiments*
alexxz@hadoop1.mlan:~> hadoop fs -rm -r /tmp/test
16/01/27 11:43:02 INFO fs.TrashPolicyDefault: Namenode trash
configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted /tmp/test
alexxz@hadoop1.mlan:~> hadoop fs -mkdir /tmp/test
alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..10}
alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
per request:' | head -1
Time per request: 16.132 [ms] (mean)
alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..100}
alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
per request:' | head -1
Time per request: 88.119 [ms] (mean)
alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..400}
alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
per request:' | head -1
Time per request: 302.568 [ms] (mean)
alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..1000}
alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
per request:' | head -1
Time per request: 746.773 [ms] (mean)
alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..2000}
alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
per request:' | head -1
Time per request: 1530.869 [ms] (mean)
alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..4000}
alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
per request:' | head -1
Time per request: 2891.558 [ms] (mean)
alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..10000}
alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
per request:' | head -1
Time per request: 7625.777 [ms] (mean)
Re: WebHDFS listing takes 0.7ms per item
Posted by Eremikhin Alexey <a....@corp.badoo.com>.
OK. I've found what was the issue.
It comes from https://issues.apache.org/jira/browse/HDFS-6565
And slows down HDFS listing in 37 times.
I've created a reverse patch to
https://issues.apache.org/jira/browse/HDFS-6565 .
How can I raise that question for mainline developers?
*2.7.1 without patch
*alexxz@hadoop1.mlan:~> time curl
'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | wc
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left Speed
100 2469k 0 2469k 0 0 335k 0 --:--:-- 0:00:07 --:--:--
338k
10002 10002 2528929
real 0m7.397s
user 0m0.084s
sys 0m0.028s
*2.7.1 with patch
*alexxz@hadoop1.mlan:~> time curl
'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | wc
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left Speed
100 2469k 0 2469k 0 0 14.5M 0 --:--:-- --:--:-- --:--:--
15.0M
10002 10002 2528929
real 0m0.180s
user 0m0.100s
sys 0m0.012s
*
*
On 27/01/16 15:24, Eremikhin Alexey wrote:
> Hi,
> I've met a performance issue after migration from Hadoop 1.2.1 to 2.7.1
> Listing of directories in WebHDFS started taking significantly more
> time than it took before.
> I've performed some experiments and concluded that it takes 0.7 per
> item in directory.
> That means listing of directory with 10000 objects takes 7 seconds.
> But that was checked on Hadoop 1.2.1 it takes about 700ms (on the same
> sever).
> Java version does not affect test results (7/8)
>
> Does anyone know about such a behaviour?
>
> There are details on tests and configuration.
> *
> alexxz@hadoop1.mlan:~> hadoop version*
> Hadoop 2.7.1
> Subversion Unknown -r Unknown
> Compiled by root on 2015-09-11T13:58Z
> Compiled with protoc 2.5.0
> From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a
> This command was run using
> /local/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar
>
> *alexxz@hadoop1.mlan:~> java -version*
> java version "1.8.0_60"
> Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
> Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
>
> *Experiments*
> alexxz@hadoop1.mlan:~> hadoop fs -rm -r /tmp/test
> 16/01/27 11:43:02 INFO fs.TrashPolicyDefault: Namenode trash
> configuration: Deletion interval = 0 minutes, Emptier interval = 0
> minutes.
> Deleted /tmp/test
> alexxz@hadoop1.mlan:~> hadoop fs -mkdir /tmp/test
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..10}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 16.132 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..100}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 88.119 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..400}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 302.568 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..1000}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 746.773 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..2000}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 1530.869 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..4000}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 2891.558 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..10000}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 7625.777 [ms] (mean)
>
Re: WebHDFS listing takes 0.7ms per item
Posted by Eremikhin Alexey <a....@corp.badoo.com>.
OK. I've found what was the issue.
It comes from https://issues.apache.org/jira/browse/HDFS-6565
And slows down HDFS listing in 37 times.
I've created a reverse patch to
https://issues.apache.org/jira/browse/HDFS-6565 .
How can I raise that question for mainline developers?
*2.7.1 without patch
*alexxz@hadoop1.mlan:~> time curl
'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | wc
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left Speed
100 2469k 0 2469k 0 0 335k 0 --:--:-- 0:00:07 --:--:--
338k
10002 10002 2528929
real 0m7.397s
user 0m0.084s
sys 0m0.028s
*2.7.1 with patch
*alexxz@hadoop1.mlan:~> time curl
'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | wc
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left Speed
100 2469k 0 2469k 0 0 14.5M 0 --:--:-- --:--:-- --:--:--
15.0M
10002 10002 2528929
real 0m0.180s
user 0m0.100s
sys 0m0.012s
*
*
On 27/01/16 15:24, Eremikhin Alexey wrote:
> Hi,
> I've met a performance issue after migration from Hadoop 1.2.1 to 2.7.1
> Listing of directories in WebHDFS started taking significantly more
> time than it took before.
> I've performed some experiments and concluded that it takes 0.7 per
> item in directory.
> That means listing of directory with 10000 objects takes 7 seconds.
> But that was checked on Hadoop 1.2.1 it takes about 700ms (on the same
> sever).
> Java version does not affect test results (7/8)
>
> Does anyone know about such a behaviour?
>
> There are details on tests and configuration.
> *
> alexxz@hadoop1.mlan:~> hadoop version*
> Hadoop 2.7.1
> Subversion Unknown -r Unknown
> Compiled by root on 2015-09-11T13:58Z
> Compiled with protoc 2.5.0
> From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a
> This command was run using
> /local/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar
>
> *alexxz@hadoop1.mlan:~> java -version*
> java version "1.8.0_60"
> Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
> Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
>
> *Experiments*
> alexxz@hadoop1.mlan:~> hadoop fs -rm -r /tmp/test
> 16/01/27 11:43:02 INFO fs.TrashPolicyDefault: Namenode trash
> configuration: Deletion interval = 0 minutes, Emptier interval = 0
> minutes.
> Deleted /tmp/test
> alexxz@hadoop1.mlan:~> hadoop fs -mkdir /tmp/test
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..10}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 16.132 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..100}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 88.119 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..400}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 302.568 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..1000}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 746.773 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..2000}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 1530.869 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..4000}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 2891.558 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..10000}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 7625.777 [ms] (mean)
>
Re: WebHDFS listing takes 0.7ms per item
Posted by Eremikhin Alexey <a....@corp.badoo.com>.
OK. I've found what was the issue.
It comes from https://issues.apache.org/jira/browse/HDFS-6565
And slows down HDFS listing in 37 times.
I've created a reverse patch to
https://issues.apache.org/jira/browse/HDFS-6565 .
How can I raise that question for mainline developers?
*2.7.1 without patch
*alexxz@hadoop1.mlan:~> time curl
'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | wc
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left Speed
100 2469k 0 2469k 0 0 335k 0 --:--:-- 0:00:07 --:--:--
338k
10002 10002 2528929
real 0m7.397s
user 0m0.084s
sys 0m0.028s
*2.7.1 with patch
*alexxz@hadoop1.mlan:~> time curl
'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | wc
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left Speed
100 2469k 0 2469k 0 0 14.5M 0 --:--:-- --:--:-- --:--:--
15.0M
10002 10002 2528929
real 0m0.180s
user 0m0.100s
sys 0m0.012s
*
*
On 27/01/16 15:24, Eremikhin Alexey wrote:
> Hi,
> I've met a performance issue after migration from Hadoop 1.2.1 to 2.7.1
> Listing of directories in WebHDFS started taking significantly more
> time than it took before.
> I've performed some experiments and concluded that it takes 0.7 per
> item in directory.
> That means listing of directory with 10000 objects takes 7 seconds.
> But that was checked on Hadoop 1.2.1 it takes about 700ms (on the same
> sever).
> Java version does not affect test results (7/8)
>
> Does anyone know about such a behaviour?
>
> There are details on tests and configuration.
> *
> alexxz@hadoop1.mlan:~> hadoop version*
> Hadoop 2.7.1
> Subversion Unknown -r Unknown
> Compiled by root on 2015-09-11T13:58Z
> Compiled with protoc 2.5.0
> From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a
> This command was run using
> /local/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar
>
> *alexxz@hadoop1.mlan:~> java -version*
> java version "1.8.0_60"
> Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
> Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
>
> *Experiments*
> alexxz@hadoop1.mlan:~> hadoop fs -rm -r /tmp/test
> 16/01/27 11:43:02 INFO fs.TrashPolicyDefault: Namenode trash
> configuration: Deletion interval = 0 minutes, Emptier interval = 0
> minutes.
> Deleted /tmp/test
> alexxz@hadoop1.mlan:~> hadoop fs -mkdir /tmp/test
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..10}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 16.132 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..100}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 88.119 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..400}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 302.568 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..1000}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 746.773 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..2000}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 1530.869 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..4000}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 2891.558 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..10000}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 7625.777 [ms] (mean)
>
Re: WebHDFS listing takes 0.7ms per item
Posted by Eremikhin Alexey <a....@corp.badoo.com>.
OK. I've found what was the issue.
It comes from https://issues.apache.org/jira/browse/HDFS-6565
And slows down HDFS listing in 37 times.
I've created a reverse patch to
https://issues.apache.org/jira/browse/HDFS-6565 .
How can I raise that question for mainline developers?
*2.7.1 without patch
*alexxz@hadoop1.mlan:~> time curl
'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | wc
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left Speed
100 2469k 0 2469k 0 0 335k 0 --:--:-- 0:00:07 --:--:--
338k
10002 10002 2528929
real 0m7.397s
user 0m0.084s
sys 0m0.028s
*2.7.1 with patch
*alexxz@hadoop1.mlan:~> time curl
'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | wc
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left Speed
100 2469k 0 2469k 0 0 14.5M 0 --:--:-- --:--:-- --:--:--
15.0M
10002 10002 2528929
real 0m0.180s
user 0m0.100s
sys 0m0.012s
*
*
On 27/01/16 15:24, Eremikhin Alexey wrote:
> Hi,
> I've met a performance issue after migration from Hadoop 1.2.1 to 2.7.1
> Listing of directories in WebHDFS started taking significantly more
> time than it took before.
> I've performed some experiments and concluded that it takes 0.7 per
> item in directory.
> That means listing of directory with 10000 objects takes 7 seconds.
> But that was checked on Hadoop 1.2.1 it takes about 700ms (on the same
> sever).
> Java version does not affect test results (7/8)
>
> Does anyone know about such a behaviour?
>
> There are details on tests and configuration.
> *
> alexxz@hadoop1.mlan:~> hadoop version*
> Hadoop 2.7.1
> Subversion Unknown -r Unknown
> Compiled by root on 2015-09-11T13:58Z
> Compiled with protoc 2.5.0
> From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a
> This command was run using
> /local/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar
>
> *alexxz@hadoop1.mlan:~> java -version*
> java version "1.8.0_60"
> Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
> Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
>
> *Experiments*
> alexxz@hadoop1.mlan:~> hadoop fs -rm -r /tmp/test
> 16/01/27 11:43:02 INFO fs.TrashPolicyDefault: Namenode trash
> configuration: Deletion interval = 0 minutes, Emptier interval = 0
> minutes.
> Deleted /tmp/test
> alexxz@hadoop1.mlan:~> hadoop fs -mkdir /tmp/test
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..10}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 16.132 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..100}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 88.119 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..400}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 302.568 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..1000}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 746.773 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..2000}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 1530.869 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..4000}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 2891.558 [ms] (mean)
> alexxz@hadoop1.mlan:~> hadoop fs -touchz /tmp/test/{1..10000}
> alexxz@hadoop1.mlan:~> /usr/sbin/ab2 -n 40 -c 1
> 'http://hadoop1:50070/webhdfs/v1/tmp/test?op=LISTSTATUS' | grep 'Time
> per request:' | head -1
> Time per request: 7625.777 [ms] (mean)
>