[jira] [Updated] (OOZIE-2923) Improve Spark options parsing

["Andras Piros (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OOZIE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andras Piros updated OOZIE-2923:
--------------------------------
    Attachment: OOZIE-2923.001.patch

> Improve Spark options parsing
> -----------------------------
>
>                 Key: OOZIE-2923
>                 URL: https://issues.apache.org/jira/browse/OOZIE-2923
>             Project: Oozie
>          Issue Type: Improvement
>          Components: action
>    Affects Versions: 4.3.0
>            Reporter: Andras Piros
>            Assignee: Andras Piros
>         Attachments: OOZIE-2923.001.patch
>
>
> There are two issues w/ Spark action's argument parsing within {{SparkMain}}
> h5. Driver and executor extra classpaths: equals sign used
> When the user specifies {{\-\-conf spark.executor.extraClassPath=XYZ}} or {{\-\-conf spark.driver.extraClassPath=ABC}}, the option {{\-\-conf}} will be added to {{sparkArgs}}. Then when the code tries to evaluate {{spark.executor.extraClassPath=XYZ}}, it uses special logic and set {{addToSparkArgs = false}}. As a result there will be a extra {{\-\-conf}} in the {{sparkArgs}} eventually.
> For example: {{\-\-conf spark.executor.extraClassPath=XYZ \-\-conf otherProperty=ABC}} will become {{\-\-conf \-\-conf otherProperty=ABC}}, which will cause spark job submit failure later.
> We might need to remove one prior {{\-\-conf}} in {{sparkArgs}} if the current evaluated {{opt}} is {{EXECUTOR_CLASSPATH}} or {{DRIVER_CLASSPATH}}.
> h5. User provided files and archives: equals sign used
> For the following workflow XML snippet:
> {code:xml}
> <spark-opts>--files=${nameNode}/home/share/hive-site.xml --num-executors 4 --executor-memory 7g --driver-memory 7g</spark-opts>
> {code}
> the {{\-\-files=$\{nameNode\}/home/share/hive-site.xml}} {{opt}} will be placed into {{sparkArgs}} in previous Oozie version without any modification, because we don't have special handling for {{\-\-files}} {{opt}}.
> If the user specifies {{\-\-files=$\{nameNode\}/home/share/hive-site.xml --num-executor 4}}, then {{SparkMain}} code treats {{\-\-num-executor}} as a file path / name. That caused the issue as I described in my previous comment. We might need to change the handling logic for {{FILES_OPTION}} and {{ARCHIVES_OPTION}} to be the same to {{DRIVER_CLASSPATH_OPTION}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)