[jira] [Updated] (PIG-4946) Remove redudant code of bin/pig in spark mode after PIG-4903 check in

["liyunzhang_intel (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

liyunzhang_intel updated PIG-4946:
----------------------------------
    Attachment: PIG-4946.patch

[~kexianda]:
After PIG-4903 check in, the process of starting a pig script changes(before we add all jars under $PIG_HOME/lib and $PIG_HOME/lib/spark/* to the classpath and export them to SPARK_YARN_DIST_FILES and SPARK_DIST_CLASSPATH) now we dynamically load libs(PIG-4893)

I found that some redudant code in PIG-4903's check in. So checkout latest code and patch -p1<PIG-4946.patch to verify this jira.

  Can you verify the process as following both in spark local and spark yarn-client mode?

*export SPARK_MASTER="yarn-client”*
1. not export SPARK_HOME  to run:
{code}
     error message like "Error: SPARK_HOME is not set!" is thrown out
{code}
2. export SPARK_HOME but not export SPARK_JAR to run:
{code}
    error message like "Error: SPARK_JAR is not set, SPARK_JAR stands for the hdfs location of spark-assembly*.jar. This allows YARN to cache spark-assembly*.jar on nodes so that it     doesn't need to be distributed each time an application runs."
{code}
3. export SPARK_HOME and SPARK_JAR  to run:
{code}
export SPARK_SOURCE=/home/zly/prj/oss/spark_source/
export SPARK_JAR=hdfs://zly1.sh.intel.com:8020/user/root/spark-assembly-1.6.0-hadoop2.6.0.jar
./pig -x spark xx.pig
{code}

*export SPARK_MASTER="local"
1. we need not export SPARK_HOME, SPARK_JAR in local mode, so even we don't export these two variables, the script should pass.
     

> Remove redudant code of bin/pig in spark mode after PIG-4903 check in
> ---------------------------------------------------------------------
>
>                 Key: PIG-4946
>                 URL: https://issues.apache.org/jira/browse/PIG-4946
>             Project: Pig
>          Issue Type: Bug
>            Reporter: liyunzhang_intel
>            Assignee: liyunzhang_intel
>         Attachments: PIG-4946.patch
>
>
> After PIG-4903 checkin, some redudant code of bin/pig in spark branch is generated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)