[jira] [Commented] (SYSTEMML-745) Spark context not initialized error under yarn-cluster for small data

["Glenn Weidner (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SYSTEMML-745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15312694#comment-15312694 ] 

Glenn Weidner commented on SYSTEMML-745:
----------------------------------------

Note the DML script runs successfully (e.g., 'Hello Apache SystemML' visible in log for simple hello world example).  Since creating spark context is an expensive operation, it is done only when required.  However, the Spark Application Master apparently checks if it's been created and if not, fails the job:

16/06/02 07:48:48 INFO DMLScript: SystemML Statistics:
Total execution time:		0.528 sec.
Number of executed MR Jobs:	0.

16/06/02 07:48:48 INFO DMLScript: END DML run 06/02/2016 07:48:48
16/06/02 07:48:48 INFO ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
16/06/02 07:48:57 ERROR ApplicationMaster: SparkContext did not initialize after waiting for 100000 ms. Please check earlier log output for errors. Failing the application.


> Spark context not initialized error under yarn-cluster for small data
> ---------------------------------------------------------------------
>
>                 Key: SYSTEMML-745
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-745
>             Project: SystemML
>          Issue Type: Bug
>          Components: Runtime
>            Reporter: Glenn Weidner
>
> Sample command to reproduce issue:
> ./bin/spark-submit --master yarn-cluster --class org.apache.sysml.api.DMLScript ../systemml/lib/systemml-0.10.0-incubating.jar -s "print('Hello Apache SystemML')
> Work-around is to add '-exec spark':
> ./bin/spark-submit --master yarn-cluster --class org.apache.sysml.api.DMLScript ../systemml/lib/systemml-0.10.0-incubating.jar -s "print('Hello Apache SystemML') -exec spark



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)