Unable to execute sparkr jobs through Chronos

[Rodrick Brown <ro...@orchard-app.com>]

We use Chronos extensively for running our batch based spark jobs and recently
started using sparkr however I'm seeing an issue trying to get sparkr jobs to
run successfully when launched through Chronos basically always failing with a
generic error below

  

Launching java with spark-submit command /opt/spark-1.6.1/bin/spark-submit
--jars /data/orchard/etc/config/load-tradedata-accumulo-
prod.jar,/data/orchard/jars/dataloader-library-
626d9b10bef28a163ad61d8a2e87bb71d455ea34-assembled.jar --driver-memory "6000M"
sparkr-shell /tmp/RtmpzJUGsJ/backend_port11c5c2f3ca40b

Error: Master must start with yarn, spark, mesos, or local

Run with --help for usage help or --verbose for debug output Error in
sparkR.init(master = sparkr_master, appName = job_name, sparkJars =
sparkr_jars, :

JVM is not ready after 10 seconds

Calls: orchard_sparkR_init -> sparkR.init

In addition: Warning message:

In processSparkJars(sparkJars) :

 sparkJars as a comma-separated string is deprecated, use character vector
instead

Execution halted  

  

I believe it may have something to do with the sparkr-shell that's forked from
R

If I run this job outside of Chronos directly from the CLI everything runs
fine it connects to my mesos cluster and succeeds.

  

I'm not sure what could be causing this issue.

  

I've tried using R CMD BATCH myscript.R and Rscript \--vanilla myscript.R both
suffer from the same issue.

  

Here is the job definition

  

{  
   "name": "PROD_applepie_loan_detail_R",  
   "command": "env && exec /data/orchard/bin/orchard_job_executor.py
-j PROD_applepie_loan_detail_R",  
   "shell": true,  
   "epsilon": "PT15M",  
   "executor": "",  
   "executorFlags": "",  
   "retries": 3,  
   "owner": "ops@orchardplatform.com",  
   "ownerName": "",  
   "runAsUser": "orchard",  
   "description": "",  
   "async": false,  
   "successCount": 0,  
   "errorCount": 0,  
   "lastSuccess": "",  
   "lastError": "",  
   "cpus": 0.1,  
   "disk": 10,  
   "mem": 6000,  
   "disabled": false,  
   "softError": false,  
   "dataProcessingJobType": false,  
   "errorsSinceLastSuccess": 0,  
   "uris": [ "file:///data/orchard/R/sparkr_env.R",
"file:///data/orchard/R/applepie_loan_detail.R"],  
   "environmentVariables": [  
     {  
       "name": "SPARKR_MASTER",  
       "value": "mesos://leader.mesos:5050"  
     }  
   ],  
   "arguments": [],  
   "highPriority": false,  
   "runAsUser": "fedora",  
   "constraints": [[ "rack", "EQUALS", "spark" ]],  
   "schedule": "R/2016-06-09T02:00:00.000Z/P1W",  
   "scheduleTimeZone": "UTC"  
 }

  

  

\--

**Rodrick Brown** / Systems Engineer 

+1 917 445 6839 /
[rodrick@orchardplatform.com](mailto:charlie@orchardplatform.com)

**Orchard Platform** 

101 5th Avenue, 4th Floor, New York, NY 10003

[http://www.orchardplatform.com](http://www.orchardplatform.com/)

[Orchard Blog](http://www.orchardplatform.com/blog/) | [Marketplace Lending
Meetup](http://www.meetup.com/Peer-to-Peer-Lending-P2P/)


-- 
*NOTICE TO RECIPIENTS*: This communication is confidential and intended for 
the use of the addressee only. If you are not an intended recipient of this 
communication, please delete it immediately and notify the sender by return 
email. Unauthorized reading, dissemination, distribution or copying of this 
communication is prohibited. This communication does not constitute an 
offer to sell or a solicitation of an indication of interest to purchase 
any loan, security or any other financial product or instrument, nor is it 
an offer to sell or a solicitation of an indication of interest to purchase 
any products or services to any persons who are prohibited from receiving 
such information under applicable law. The contents of this communication 
may not be accurate or complete and are subject to change without notice. 
As such, Orchard App, Inc. (including its subsidiaries and affiliates, 
"Orchard") makes no representation regarding the accuracy or completeness 
of the information contained herein. The intended recipient is advised to 
consult its own professional advisors, including those specializing in 
legal, tax and accounting matters. Orchard does not provide legal, tax or 
accounting advice.

Re: Unable to execute sparkr jobs through Chronos

[Sun Rui <su...@163.com>]

I saw in the job definition an Env Var: SPARKR_MASTER. What is that for? I don’t think SparkR uses it.
> On Jun 17, 2016, at 10:08, Sun Rui <su...@163.com> wrote:
> 
> It seems that spark master URL is not correct. What is it?
>> On Jun 16, 2016, at 18:57, Rodrick Brown <rodrick@orchard-app.com <ma...@orchard-app.com>> wrote:
>> 
>> Master must start with yarn, spark, mesos, or local
> 

Re: Unable to execute sparkr jobs through Chronos

[Sun Rui <su...@163.com>]

It seems that spark master URL is not correct. What is it?
> On Jun 16, 2016, at 18:57, Rodrick Brown <ro...@orchard-app.com> wrote:
> 
> Master must start with yarn, spark, mesos, or local