[jira] [Commented] (PIG-4059) Pig on Spark

["Rohini Palaniswamy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16027590#comment-16027590 ] 

Rohini Palaniswamy commented on PIG-4059:
-----------------------------------------

+1 to commit for https://reviews.apache.org/r/57317/diff/9.  

[~daijy]/[~xuefuz],
    Could one of you +1 as well for the final patch to be merged?

> Pig on Spark
> ------------
>
>                 Key: PIG-4059
>                 URL: https://issues.apache.org/jira/browse/PIG-4059
>             Project: Pig
>          Issue Type: New Feature
>          Components: spark
>            Reporter: Rohini Palaniswamy
>            Assignee: Praveen Rachabattuni
>              Labels: spork
>             Fix For: spark-branch
>
>         Attachments: Pig-on-Spark-Design-Doc.pdf, Pig-on-Spark-Scope.pdf
>
>
> Setting up your development environment:
> 0. download spark release package(currently pig on spark only support spark 1.6).
> 1. Check out Pig Spark branch.
> 2. Build Pig by running "ant jar" and "ant -Dhadoopversion=23 jar" for hadoop-2.x versions
> 3. Configure these environmental variables:
>     export HADOOP_USER_CLASSPATH_FIRST="true"
> Now we support “local” and "yarn-client" mode, you can export system variable “SPARK_MASTER” like:
>     export SPARK_MASTER=local or export SPARK_MASTER="yarn-client"
> 4. In local mode: ./pig -x spark_local xxx.pig
>     In yarn-client mode: 
>     export SPARK_HOME=xx; 
>     export SPARK_JAR=hdfs://example.com:8020/xxxx (the hdfs location where you upload the spark-assembly*.jar)
>     ./pig -x spark xxx.pig



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)