about how to implement ship in other mode like "spark"

["Zhang, Liyun" <li...@intel.com>]

Hi all,
  I want to ask a question about "ship" in pig:
    Ship with streaming, it will send streaming binary and supporting files, if any, from the client node to the compute nodes.
  I found that the implementation of ship in Mapreduce mode is:

/home/zly/prj/oss/pig/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java
line 721:
setupDistributedCache(pigContext, conf, pigContext.getProperties(),
                    "pig.streaming.ship.files", true);

this function gets all "pig.streaming.ship.files" from the properties, then copy the ship files to hadoop using fs.copyFromLocalFile, at the same time, symlink feature is turned on by using DistributedCache.createSymlink(conf). For example, if ship file "/tmp/teststreaming.pl" is copyed from local to hadoop, the hadoop file will be hdfs://xxxx:8020/tmp/tempxxxx/tmp-xxx#teststreaming.pl. /tmp/hadoop-root/mapred/local/1419842279890/tmp-1268857767 is a cache for hdfs://xxxx:8020/tmp/tempxxxx/tmp-xxx#teststreaming.pl . teststreaming.pl will be generated as a link to  /tmp/hadoop-root/mapred/local/1419842279890/tmp-1268857767 in the current execution path.  If i want to implement ship in other mode like spark, the only thing i need to do is copying the shiped files from the shiped path to current execution path?



Best regards
Zhang,Liyun

Re: about how to implement ship in other mode like "spark"

[Xuefu Zhang <xz...@cloudera.com>]

You might want to consider SparkContext.addFile() for distributing the file
at the client and SparkFiles.get() for retrieving the file at the execution
node.

--Xuefu

On Fri, Jan 2, 2015 at 7:15 PM, Zhang, Liyun <li...@intel.com> wrote:

> Hi all,
>   I want to ask a question about "ship" in pig:
>     Ship with streaming, it will send streaming binary and supporting
> files, if any, from the client node to the compute nodes.
>   I found that the implementation of ship in Mapreduce mode is:
>
>
> /home/zly/prj/oss/pig/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java
> line 721:
> setupDistributedCache(pigContext, conf, pigContext.getProperties(),
>                     "pig.streaming.ship.files", true);
>
> this function gets all "pig.streaming.ship.files" from the properties,
> then copy the ship files to hadoop using fs.copyFromLocalFile, at the same
> time, symlink feature is turned on by using
> DistributedCache.createSymlink(conf). For example, if ship file "/tmp/
> teststreaming.pl" is copyed from local to hadoop, the hadoop file will be
> hdfs://xxxx:8020/tmp/tempxxxx/tmp-xxx#teststreaming.pl.
> /tmp/hadoop-root/mapred/local/1419842279890/tmp-1268857767 is a cache for
> hdfs://xxxx:8020/tmp/tempxxxx/tmp-xxx#teststreaming.pl . teststreaming.pl
> will be generated as a link to
> /tmp/hadoop-root/mapred/local/1419842279890/tmp-1268857767 in the current
> execution path.  If i want to implement ship in other mode like spark, the
> only thing i need to do is copying the shiped files from the shiped path to
> current execution path?
>
>
>
> Best regards
> Zhang,Liyun
>
>