You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Zhang, Liyun" <li...@intel.com> on 2014/12/26 09:47:37 UTC

some question about streaming_local.conf

Hi all:
  I have 2 questiones about pig/test/e2e/pig/tests/streaming_local.conf:
1.
$cfg = {
        'driver' => 'Pig',
        'nummachines' => 5,

        'groups' => [
                                {
                                # This group is for local mode testing
                                'name' => 'StreamingLocal',
                'sortBenchmark' => 1,
                'sortResults' => 1,
                'floatpostprocess' => 1,
                'delimiter' => '   ',
                'tests' => [
                                                {
                                        #Section 1.1: perl script, no parameters
                        'num' => 1,
                        'execonly' => 'local',  // this line
                        'pig' => q#

all e2e test cases are only executed in local mode now. Can these e2e tests run in other mode, like "mapreduce","tez","spark"?
when i replace 'execonly'=>'local' with 'execonly'=>'spark', all cases pass when POStream is implemented in spark mode.
I think we can remove 'execonly'=>'local' and can test these e2e tests in other modes.

2. when using ship with streaming, it will send streaming binary and supporting files, if any, from the client node to the compute nodes..I found we use "perl ./libexec/GroupBy.pl" in StreamingLocal_3.pig, this path is a relative path to current executed path. can we use "perl GroupBy.pl" because i think the file "./libexec/GroupBy.pl" has been shipped to compute nodes.
StreamingLocal_3.pig
/test/e2e/pigMD `perl ./libexec/GroupBy.pl '\t' 0` ship('./libexec/GroupBy.pl');
A = load './data/singlefile/studenttab10k';
B = group A by $0;
C = foreach B generate flatten(A);
D = stream C through CMD;
store D into './testout/root-1419582821-streaming_local.conf/StreamingLocal_3.out';



Best regards
Zhang,Liyun