You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pig.apache.org by Apache Wiki <wi...@apache.org> on 2009/02/24 20:18:34 UTC
[Pig Wiki] Update of "PigMix" by AlanGates
Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Pig Wiki" for change notification.
The following page has been changed by AlanGates:
http://wiki.apache.org/pig/PigMix
------------------------------------------------------------------------------
== Runs ==
- == November 19, 2008 ==
- PigMix was run on a 25 machine hadoop cluster. The cluster was running hadoop version 0.18.1. The tests were run against two
+ The following table includes runs done of the pig mix. All of these runs have been done on a cluster with 26 slaves plus one machine acting as the name node and job tracker. The cluster was running
+ hadoop version 0.18.1. (TODO: Need to get specific hardware info on those machines).
+
+ The tests were run against two
versions of pig: top of trunk, and top of types branch both as of Nov 21 2008.
- So far only the latency tests have been run, not the scalability tests. Equivalent map reduce programs have not yet been run.
-
The tests were run three times for each version and the results averaged.
- || Query || Top of Trunk || Top of Types Branch ||
- || L1 explode || 261 || 283 ||
- || L2 fr join || 1665 || 253 ||
- || L3 join || 1912 || 320 ||
- || L4 distinct agg || 254 || 193 ||
- || L5 anti-join || 1535 || 281 ||
- || L6 large group by key || 294 || 226 ||
- || L7 nested split || 243 || 204 ||
- || L8 group all || 462 || 194 ||
- || L9 order by 1 field || 5294 || 867 ||
- || L10 order by multiple fields || 1403 || 565 ||
- || L11 distinct + union || 316 || 255 ||
- || L12 multi-store || fails || 781 ||
+ tot = top of trunk
+ totb = top of types branch
+
+ || Version || Map Reduce Java Code || tot 11/21/08 || totb 11/21/08 || totb 1/20/09 ||
+ || Date Run || 11/22/08 || 11/21/08 || 11/21/08 || 1/20/09 ||
+ || L1 explode || 116 || 261 || 283 || 218 ||
+ || L2 fr join || 41 || 1665 || 253 || 168 ||
+ || L3 join || 97 || 1912 || 320 || 258 ||
+ || L4 distinct agg || 68 || 254 || 193 || 110 ||
+ || L5 anti-join || 90 || 1535 || 281 || 209 ||
+ || L6 large group by key || 61 || 294 || 226 || 126 ||
+ || L7 nested split || 72 || 243 || 204 || 107 ||
+ || L8 group all || 56 || 462 || 194 || 104 ||
+ || L9 order by 1 field || 286 || 5294 || 867 || 851 ||
+ || L10 order by multiple fields || 634 || 1403 || 565 || 469 ||
+ || L11 distinct + union || 120 || 316 || 255 || 164 ||
+ || L12 multi-store || 150 || fails || 781 || 499 ||
+ || Total time || 1791 || 13638 || 4420 || 3284 ||
+ || Compared to hadoop || 1.0 || 7.6 || 2.5 || 1.8 ||
+
+ The totb run of 1/20/09 includes the change to make BufferedPositionedInputStream use a buffer instead of relying on hadoop to buffer.
== Features Tested ==
Based on a sample of user queries, PigMix includes tests for the following features.