You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by Apache Wiki <wi...@apache.org> on 2008/01/08 09:21:49 UTC
[Lucene-hadoop Wiki] Trivial Update of "DataProcessingBenchmarks" by udanax
Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification.
The following page has been changed by udanax:
http://wiki.apache.org/lucene-hadoop/DataProcessingBenchmarks
New page:
== Data Processing Benchmarks ==
SQL > select ipaddress, count(*) from access_log group by ipaddress order by count(*) desc limit 0,100;
[[BR]]''σ ,,count. ipaddress,, (τ ,,count,, (γ ,,count(ipaddress). ipaddress,, (access_log)))''
||<bgcolor="#E5E5E5">||<bgcolor="#E5E5E5">!MySql ||<bgcolor="#E5E5E5">Hadoop||
||<bgcolor="#E5E5E5">Index ||index ||non-index||
||<bgcolor="#E5E5E5">Machine ||1 ||1000||
||<bgcolor="#E5E5E5">Rows ||3,700,000 ||49,324,734||
||<bgcolor="#E5E5E5">Results ||100 ||100||
||<bgcolor="#E5E5E5">Time ||3.715 sec ||112.03 sec||
{{{
bash# ./bin/hadoop jar ./log_examples.jar loganalysis -m 1000 -r 1000 udanax/logfiles udanax/rank 100
* Connecting Time Count Map/Reduce Job start.
----------------------------------------------------------------------
08/01/08 16:13:46 INFO mapred.FileInputFormat: Total input paths to process : 18
08/01/08 16:13:47 INFO mapred.JobClient: Running job: job_200801081529_0005
08/01/08 16:13:48 INFO mapred.JobClient: map 0% reduce 0%
08/01/08 16:13:54 INFO mapred.JobClient: map 1% reduce 0%
08/01/08 16:13:56 INFO mapred.JobClient: map 2% reduce 0%
08/01/08 16:14:01 INFO mapred.JobClient: map 6% reduce 0%
08/01/08 16:14:03 INFO mapred.JobClient: map 8% reduce 0%
08/01/08 16:14:05 INFO mapred.JobClient: map 10% reduce 0%
08/01/08 16:14:07 INFO mapred.JobClient: map 23% reduce 0%
08/01/08 16:14:08 INFO mapred.JobClient: map 24% reduce 0%
08/01/08 16:14:11 INFO mapred.JobClient: map 28% reduce 0%
08/01/08 16:14:13 INFO mapred.JobClient: map 29% reduce 0%
08/01/08 16:14:15 INFO mapred.JobClient: map 31% reduce 0%
08/01/08 16:14:16 INFO mapred.JobClient: map 33% reduce 0%
08/01/08 16:14:17 INFO mapred.JobClient: map 39% reduce 1%
08/01/08 16:14:18 INFO mapred.JobClient: map 40% reduce 1%
08/01/08 16:14:21 INFO mapred.JobClient: map 43% reduce 1%
08/01/08 16:14:23 INFO mapred.JobClient: map 44% reduce 2%
08/01/08 16:14:24 INFO mapred.JobClient: map 45% reduce 2%
08/01/08 16:14:25 INFO mapred.JobClient: map 46% reduce 2%
08/01/08 16:14:26 INFO mapred.JobClient: map 48% reduce 3%
08/01/08 16:14:27 INFO mapred.JobClient: map 52% reduce 4%
08/01/08 16:14:30 INFO mapred.JobClient: map 53% reduce 4%
08/01/08 16:14:31 INFO mapred.JobClient: map 54% reduce 5%
08/01/08 16:14:33 INFO mapred.JobClient: map 55% reduce 5%
08/01/08 16:14:34 INFO mapred.JobClient: map 56% reduce 5%
08/01/08 16:14:35 INFO mapred.JobClient: map 57% reduce 5%
08/01/08 16:14:36 INFO mapred.JobClient: map 61% reduce 7%
08/01/08 16:14:37 INFO mapred.JobClient: map 63% reduce 7%
08/01/08 16:14:41 INFO mapred.JobClient: map 64% reduce 7%
08/01/08 16:14:43 INFO mapred.JobClient: map 65% reduce 7%
08/01/08 16:14:45 INFO mapred.JobClient: map 67% reduce 8%
08/01/08 16:14:47 INFO mapred.JobClient: map 72% reduce 8%
08/01/08 16:14:48 INFO mapred.JobClient: map 73% reduce 9%
08/01/08 16:14:52 INFO mapred.JobClient: map 74% reduce 9%
08/01/08 16:14:55 INFO mapred.JobClient: map 76% reduce 10%
08/01/08 16:14:57 INFO mapred.JobClient: map 80% reduce 11%
08/01/08 16:14:59 INFO mapred.JobClient: map 81% reduce 11%
08/01/08 16:15:01 INFO mapred.JobClient: map 82% reduce 11%
08/01/08 16:15:02 INFO mapred.JobClient: map 83% reduce 11%
08/01/08 16:15:05 INFO mapred.JobClient: map 86% reduce 12%
08/01/08 16:15:06 INFO mapred.JobClient: map 87% reduce 12%
08/01/08 16:15:07 INFO mapred.JobClient: map 88% reduce 13%
08/01/08 16:15:09 INFO mapred.JobClient: map 89% reduce 13%
08/01/08 16:15:10 INFO mapred.JobClient: map 89% reduce 14%
08/01/08 16:15:11 INFO mapred.JobClient: map 90% reduce 14%
08/01/08 16:15:12 INFO mapred.JobClient: map 91% reduce 14%
08/01/08 16:15:14 INFO mapred.JobClient: map 92% reduce 14%
08/01/08 16:15:15 INFO mapred.JobClient: map 94% reduce 15%
08/01/08 16:15:16 INFO mapred.JobClient: map 95% reduce 16%
08/01/08 16:15:17 INFO mapred.JobClient: map 96% reduce 16%
08/01/08 16:15:18 INFO mapred.JobClient: map 97% reduce 16%
08/01/08 16:15:20 INFO mapred.JobClient: map 98% reduce 16%
08/01/08 16:15:22 INFO mapred.JobClient: map 98% reduce 17%
08/01/08 16:15:24 INFO mapred.JobClient: map 100% reduce 17%
08/01/08 16:15:26 INFO mapred.JobClient: map 100% reduce 18%
08/01/08 16:15:27 INFO mapred.JobClient: map 100% reduce 31%
08/01/08 16:15:28 INFO mapred.JobClient: map 100% reduce 36%
08/01/08 16:15:29 INFO mapred.JobClient: map 100% reduce 45%
08/01/08 16:15:30 INFO mapred.JobClient: map 100% reduce 74%
08/01/08 16:15:31 INFO mapred.JobClient: map 100% reduce 81%
08/01/08 16:15:32 INFO mapred.JobClient: map 100% reduce 84%
08/01/08 16:15:33 INFO mapred.JobClient: map 100% reduce 92%
08/01/08 16:15:34 INFO mapred.JobClient: map 100% reduce 100%
08/01/08 16:15:35 INFO mapred.JobClient: Job complete: job_200801081529_0005
08/01/08 16:15:35 INFO mapred.JobClient: Counters: 11
08/01/08 16:15:35 INFO mapred.JobClient: Job Counters
08/01/08 16:15:35 INFO mapred.JobClient: Launched map tasks=1355
08/01/08 16:15:35 INFO mapred.JobClient: Launched reduce tasks=1457
08/01/08 16:15:35 INFO mapred.JobClient: Map-Reduce Framework
08/01/08 16:15:35 INFO mapred.JobClient: Map input records=49324734
08/01/08 16:15:35 INFO mapred.JobClient: Map output records=49324721
08/01/08 16:15:35 INFO mapred.JobClient: Map input bytes=8551673779
08/01/08 16:15:35 INFO mapred.JobClient: Map output bytes=790763358
08/01/08 16:15:35 INFO mapred.JobClient: Combine input records=49324734
08/01/08 16:15:35 INFO mapred.JobClient: Combine output records=705771
08/01/08 16:15:35 INFO mapred.JobClient: Reduce input groups=201330
08/01/08 16:15:35 INFO mapred.JobClient: Reduce input records=705771
08/01/08 16:15:35 INFO mapred.JobClient: Reduce output records=201330
* Sort by Connection Time Count Map/Reduce Job start.
----------------------------------------------------------------------
08/01/08 16:15:35 INFO mapred.FileInputFormat: Total input paths to process : 100
08/01/08 16:15:36 INFO mapred.JobClient: Running job: job_200801081529_0006
08/01/08 16:15:37 INFO mapred.JobClient: map 0% reduce 0%
08/01/08 16:15:40 INFO mapred.JobClient: map 10% reduce 0%
08/01/08 16:15:41 INFO mapred.JobClient: map 42% reduce 0%
08/01/08 16:15:42 INFO mapred.JobClient: map 91% reduce 0%
08/01/08 16:15:43 INFO mapred.JobClient: map 100% reduce 0%
08/01/08 16:17:54 INFO mapred.JobClient: map 100% reduce 6%
08/01/08 16:20:54 INFO mapred.JobClient: map 100% reduce 17%
08/01/08 16:22:41 INFO mapred.JobClient: map 100% reduce 29%
08/01/08 16:25:52 INFO mapred.JobClient: map 100% reduce 37%
08/01/08 16:27:44 INFO mapred.JobClient: map 100% reduce 51%
08/01/08 16:28:12 INFO mapred.JobClient: map 100% reduce 69%
08/01/08 16:30:35 INFO mapred.JobClient: map 100% reduce 82%
08/01/08 16:32:25 INFO mapred.JobClient: map 100% reduce 99%
08/01/08 16:33:54 INFO mapred.JobClient: map 100% reduce 100%
08/01/08 16:33:55 INFO mapred.JobClient: Job complete: job_200801081529_0006
08/01/08 16:33:11 INFO mapred.JobClient: Counters: 11
08/01/08 16:33:55 INFO mapred.JobClient: Job Counters
08/01/08 16:33:55 INFO mapred.JobClient: Launched map tasks=1080
08/01/08 16:33:55 INFO mapred.JobClient: Launched reduce tasks=1
08/01/08 16:33:55 INFO mapred.JobClient: Map-Reduce Framework
08/01/08 16:33:55 INFO mapred.JobClient: Map input records=201330
08/01/08 16:33:55 INFO mapred.JobClient: Map output records=201330
08/01/08 16:33:55 INFO mapred.JobClient: Map input bytes=5080608
08/01/08 16:33:55 INFO mapred.JobClient: Map output bytes=5108994
08/01/08 16:33:55 INFO mapred.JobClient: Combine input records=201330
08/01/08 16:33:55 INFO mapred.JobClient: Combine output records=8406
08/01/08 16:33:55 INFO mapred.JobClient: Reduce input groups=19270
08/01/08 16:33:55 INFO mapred.JobClient: Reduce input records=84069
08/01/08 16:33:55 INFO mapred.JobClient: Reduce output records=200
------------------------------------
* Top 100 connector list :
+--------------+-------------------+
| Count | Ip Address |
+--------------+-------------------+
| 374932 | 121.165.51.179 |
| 357615 | 121.150.85.42 |
| 304878 | 211.204.83.50 |
| 274461 | 10.8.107.219 |
| 264475 | 222.238.215.220 |
| 246650 | 123.254.226.176 |
| 242124 | 218.50.17.33 |
| 229223 | 116.34.227.130 |
| 222771 | 61.98.219.133 |
| 196677 | 116.122.252.19 |
| 186095 | 124.46.159.54 |
| 181853 | 211.172.61.217 |
| 178051 | 123.214.135.227 |
| 177545 | 222.118.198.139 |
| 175318 | 121.190.177.28 |
| 174289 | 211.107.182.103 |
| 166152 | 123.212.206.195 |
| 165495 | 218.50.28.205 |
| 164378 | 59.3.99.227 |
| 160612 | 121.144.99.83 |
| 158529 | 219.252.85.66 |
| 153793 | 121.138.193.64 |
| 151995 | 121.145.170.142 |
| 146165 | 211.245.129.176 |
| 145766 | 124.50.192.5 |
| 141955 | 124.49.112.28 |
| 139306 | 74.6.22.134 |
| 138591 | 121.146.230.146 |
| 136510 | 222.104.236.39 |
| 135713 | 222.119.64.36 |
| 135395 | 211.214.79.219 |
| 131614 | 124.50.39.137 |
| 130607 | 211.203.169.195 |
| 129805 | 121.184.49.152 |
| 128691 | 125.138.220.135 |
| 127131 | 222.109.169.162 |
| 126593 | 58.235.62.31 |
| 126572 | 211.209.220.182 |
| 124235 | 121.166.8.143 |
| 120835 | 210.94.77.21 |
| 119125 | 121.132.10.153 |
| 118614 | 222.237.43.156 |
| 116033 | 222.232.155.39 |
| 114882 | 124.60.159.9 |
| 114030 | 218.238.122.191 |
| 112950 | 121.146.203.67 |
| 110689 | 222.232.72.138 |
| 109602 | 59.27.78.202 |
| 107644 | 121.171.103.242 |
| 107455 | 221.150.183.226 |
| 107152 | 218.55.200.198 |
| 105625 | 122.36.134.52 |
| 105108 | 58.142.103.228 |
| 103540 | 121.189.210.44 |
| 103379 | 125.191.119.191 |
| 103302 | 121.135.12.61 |
| 103068 | 59.150.103.15 |
| 102877 | 211.33.116.85 |
| 102724 | 65.55.213.39 |
| 102605 | 121.131.89.5 |
| 102203 | 221.150.198.209 |
| 102059 | 125.190.91.14 |
| 101727 | 125.187.35.180 |
| 101624 | 58.228.81.115 |
| 100364 | 203.247.80.42 |
| 98559 | 121.188.233.2 |
| 97796 | 124.51.131.138 |
| 95466 | 222.239.59.19 |
| 94386 | 122.128.194.104 |
| 89315 | 61.252.130.101 |
| 89205 | 116.37.203.91 |
| 87911 | 125.187.54.146 |
| 87851 | 125.142.135.237 |
| 86262 | 121.165.66.27 |
| 85723 | 58.234.15.202 |
| 85086 | 121.165.73.254 |
| 84824 | 121.53.95.93 |
| 84762 | 211.36.65.234 |
| 84409 | 58.121.165.252 |
| 84329 | 168.131.153.175 |
| 82710 | 121.162.28.131 |
| 82041 | 121.139.151.137 |
| 81887 | 124.111.242.97 |
| 81578 | 125.178.79.218 |
| 80827 | 124.62.208.102 |
| 79866 | 211.175.253.135 |
| 79226 | 202.30.106.20 |
| 78535 | 58.122.138.79 |
| 77824 | 221.161.127.18 |
| 77141 | 211.59.140.121 |
| 76623 | 122.35.247.26 |
| 73814 | 210.183.41.161 |
| 73328 | 125.182.29.4 |
| 73220 | 218.236.193.29 |
| 72808 | 61.101.164.172 |
| 72413 | 58.120.250.151 |
| 72154 | 211.210.164.215 |
| 72083 | 122.44.149.231 |
| 71646 | 124.49.150.145 |
| 70915 | 211.48.70.247 |
+--------------+-------------------+
Processing time : 112.03 sec
}}}