[jira] [Commented] (OOZIE-2391) spark-opts value in workflow.xml is not parsed properly

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OOZIE-2391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230282#comment-15230282 ] 

Hadoop QA commented on OOZIE-2391:
----------------------------------

Testing JIRA OOZIE-2391

Cleaning local git workspace

----------------------------

{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.    {color:green}+1{color} the patch does not introduce any @author tags
.    {color:green}+1{color} the patch does not introduce any tabs
.    {color:green}+1{color} the patch does not introduce any trailing spaces
.    {color:green}+1{color} the patch does not introduce any line longer than 132
.    {color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.    {color:green}+1{color} the patch does not seem to introduce new RAT warnings
{color:green}+1 JAVADOC{color}
.    {color:green}+1{color} the patch does not seem to introduce new Javadoc warnings
{color:green}+1 COMPILE{color}
.    {color:green}+1{color} HEAD compiles
.    {color:green}+1{color} patch compiles
.    {color:green}+1{color} the patch does not seem to introduce new javac warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.    {color:green}+1{color} the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
.    {color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.    Tests run: 1775
{color:green}+1 DISTRO{color}
.    {color:green}+1{color} distro tarball builds with the patch 

----------------------------
{color:green}*+1 Overall result, good!, no -1s*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/2818/

> spark-opts value in workflow.xml is not parsed properly
> -------------------------------------------------------
>
>                 Key: OOZIE-2391
>                 URL: https://issues.apache.org/jira/browse/OOZIE-2391
>             Project: Oozie
>          Issue Type: Bug
>    Affects Versions: trunk
>            Reporter: Praveen Bathala
>            Assignee: Nathan Skone
>              Labels: newbie
>         Attachments: OOZIE-2391-001.patch, OOZIE-2391-002.patch
>
>
> When providing spark-opts in workflow.xml for oozie, the "--conf" property values is not parsed properly.
> Example spark opts
> {code:xml}
> <spark-opts>--conf spark.executor.extraJavaOptions=-Dconfig.url=hdfs:///my.conf -Dmy.env=${env}</spark-opts>
> {code}
> In this case the value for property "spark.executor.extraJavaOptions" is "-Dconfig.url=hdfs:///my.conf -Dmy.env=${env}", but in the oozie code here https://github.com/apache/oozie/blob/38e33811bc3f9d3b1a1a024dcab6af760734b696/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L110 the string is just split by space, which make the "-Dmy.env=${env}" a separate argument to spark and spark fails job as that argument is not valid.
> This doesn't work with enclosing the whole value in quotes too.
> Error from stderr:
> Error: Unrecognized option '-Dmy.env=development"'.
> Run with --help for usage help or --verbose for debug output
> Intercepting System.exit(1)
> Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], exit code [1]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)