[jira] [Commented] (OOZIE-3122) Oozie spark action unable to handle some options in spark-defaults.conf

["Peter Cseh (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OOZIE-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16248984#comment-16248984 ] 

Peter Cseh commented on OOZIE-3122:
-----------------------------------

Most of these issues are handled by OOZIE-2984. This was resolved soon after 4.3.0 was released, but there is no release with the fix in it yet. 

> Oozie spark action unable to handle some options in spark-defaults.conf
> -----------------------------------------------------------------------
>
>                 Key: OOZIE-3122
>                 URL: https://issues.apache.org/jira/browse/OOZIE-3122
>             Project: Oozie
>          Issue Type: Bug
>          Components: action
>    Affects Versions: 4.3.0
>         Environment: Operating on EMR cluster (5.4.0) which includes Oozie 4.3.0
>            Reporter: Simon Whitelaw
>              Labels: usability
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> When using the setting oozie.service.SparkConfigurationService.spark.configurations *=etc/spark/conf to specify a spark-defaults.conf file for spark to use, a few options are not handled properly by the oozie spark action and cause the job to fail. These include the following:
> * spark.driver.extraClassPath - causes a blank --conf tag to be sent to the spark submit and the spark submit will fail
> * spark.executor.extraClassPath - causes a blank --conf tag to be sent to the spark submit and the spark submit will fail
> * spark.executor.extraJavaOptions with multiple options set (e.g. spark.executor.extraJavaOptions  -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CMSClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p') - causes all options except for the first to be passed as singletons rather than extraJavaOptions
> * spark.driver.extraJavaOptions with multiple option set - causes all options except for the first to be passed as singletons rather than extraJavaOptions
> * Within the spark.driver.extraJavaOptions and spark.executor.extraJavaOptions, the option -XX:OnOutOfMemoryError='kill -9 %p' is not parsed correctly even from the spark-opts tag and requires additional double quotations within the single quote marks: '"kill -9 %p"'
> These issues were noticed running the oozie spark action on an amazon EMR cluster. Workaround involves sending the options as part of the spark-opts tag, but is not ideal because in EMR, the spark-defaults.conf file is autogenerated for the specifications of the cluster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)