You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by "Dasha Boudnik (JIRA)" <ji...@apache.org> on 2014/02/20 03:45:20 UTC
[jira] [Updated] (BIGTOP-1213) BigTop map defining
TestHadoopExamples smokes ~ orderering is nondeterministic of tests.
Should be orderd.
[ https://issues.apache.org/jira/browse/BIGTOP-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dasha Boudnik updated BIGTOP-1213:
----------------------------------
Attachment: BIGTOP-1213.patch
All the suggested changes are included in this patch. Namely:
- TeraGen should now always run before Terasort. I changed Map to LinkedHashMap, as the latter guarantees to iterate in the order in which the entries were put into the map
- parameters can be set for pi; I kept the default at 2 maps, 1000 samples just to avoid changing the code more than I needed to
- sleep is no longer included, as it's indeed not in newer versions for YARN
Hope that covers everything!
> BigTop map defining TestHadoopExamples smokes ~ orderering is nondeterministic of tests. Should be orderd.
> -----------------------------------------------------------------------------------------------------------
>
> Key: BIGTOP-1213
> URL: https://issues.apache.org/jira/browse/BIGTOP-1213
> Project: Bigtop
> Issue Type: Bug
> Reporter: jay vyas
> Assignee: Dasha Boudnik
> Attachments: BIGTOP-1213.patch
>
>
> According to the JDK contract for a Map, the order of iteration is not gauranteed.
> But in TestHadoopExamples.groovy, we define a map and iterate through it.
> The tests should run in the defined order,
> 1) both for determinism, as well
> 2) so that TeraGen always runs before TeraSort
> {noformat}
> static Map examples =
> [
> pi :'5 10',
> wordcount :"$EXAMPLES/text $EXAMPLES_OUT/wordcount",
> teragen :"${terasort_rows} teragen${terasortid}",
> terasort :"teragen${terasortid} terasort${terasortid}",
> teravalidate :"terasort${terasortid} tervalidate${terasortid}",
> ...
> ];
> {noformat}
> examples.each { k, v -> res[k] = [k.toString(), v.toString()] as Object[]; }
> {noformat}
> While implementing this JIRA, we can also add in some other minor improvements :
> - Parameterization of calculate pi so that it can run fast on VMS
> - Remove "sleep" , as its not in newer versions of YARN ( i think, need to confirm)
> TL;DR, Im proposing a fix to add concreted deterministic JDK-independant ordering to the Map so tests always run in same order + a few minor upgrades to this class while were at it.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)