You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Rodrick Brown <ro...@orchard-app.com> on 2016/06/16 10:57:07 UTC
Unable to execute sparkr jobs with Chronos
We use Chronos extensively for running our batch based spark jobs and recently
started using sparkr however I'm seeing an issue trying to get sparkr jobs to
run successfully when launched through Chronos basically always failing with a
generic error below
Launching java with spark-submit command /opt/spark-1.6.1/bin/spark-submit
--jars /data/orchard/etc/config/load-tradedata-accumulo-
prod.jar,/data/orchard/jars/dataloader-library-
626d9b10bef28a163ad61d8a2e87bb71d455ea34-assembled.jar --driver-memory "6000M"
sparkr-shell /tmp/RtmpzJUGsJ/backend_port11c5c2f3ca40b
Error: Master must start with yarn, spark, mesos, or local
Run with --help for usage help or --verbose for debug output Error in
sparkR.init(master = sparkr_master, appName = job_name, sparkJars =
sparkr_jars, :
JVM is not ready after 10 seconds
Calls: orchard_sparkR_init -> sparkR.init
In addition: Warning message:
In processSparkJars(sparkJars) :
sparkJars as a comma-separated string is deprecated, use character vector
instead
Execution halted
I believe it may have something to do with the sparkr-shell that's forked from
R
If I run this job outside of Chronos directly from the CLI everything runs
fine it connects to my mesos cluster and succeeds.
I'm not sure what could be causing this issue.
I've tried using R CMD BATCH myscript.R and Rscript \--vanilla myscript.R both
suffer from the same issue.
Here is the job definition
{
"name": "PROD_applepie_loan_detail_R",
"command": "env && exec /data/orchard/bin/orchard_job_executor.py
-j PROD_applepie_loan_detail_R",
"shell": true,
"epsilon": "PT15M",
"executor": "",
"executorFlags": "",
"retries": 3,
"owner": "ops@orchardplatform.com",
"ownerName": "",
"runAsUser": "orchard",
"description": "",
"async": false,
"successCount": 0,
"errorCount": 0,
"lastSuccess": "",
"lastError": "",
"cpus": 0.1,
"disk": 10,
"mem": 6000,
"disabled": false,
"softError": false,
"dataProcessingJobType": false,
"errorsSinceLastSuccess": 0,
"uris": [ "file:///data/orchard/R/sparkr_env.R",
"file:///data/orchard/R/applepie_loan_detail.R"],
"environmentVariables": [
{
"name": "SPARKR_MASTER",
"value": "mesos://leader.mesos:5050"
}
],
"arguments": [],
"highPriority": false,
"runAsUser": "fedora",
"constraints": [[ "rack", "EQUALS", "spark" ]],
"schedule": "R/2016-06-09T02:00:00.000Z/P1W",
"scheduleTimeZone": "UTC"
}
\--
**Rodrick Brown** / Systems Engineer
+1 917 445 6839 /
[rodrick@orchardplatform.com](mailto:charlie@orchardplatform.com)
**Orchard Platform**
101 5th Avenue, 4th Floor, New York, NY 10003
[http://www.orchardplatform.com](http://www.orchardplatform.com/)
[Orchard Blog](http://www.orchardplatform.com/blog/) | [Marketplace Lending
Meetup](http://www.meetup.com/Peer-to-Peer-Lending-P2P/)
--
*NOTICE TO RECIPIENTS*: This communication is confidential and intended for
the use of the addressee only. If you are not an intended recipient of this
communication, please delete it immediately and notify the sender by return
email. Unauthorized reading, dissemination, distribution or copying of this
communication is prohibited. This communication does not constitute an
offer to sell or a solicitation of an indication of interest to purchase
any loan, security or any other financial product or instrument, nor is it
an offer to sell or a solicitation of an indication of interest to purchase
any products or services to any persons who are prohibited from receiving
such information under applicable law. The contents of this communication
may not be accurate or complete and are subject to change without notice.
As such, Orchard App, Inc. (including its subsidiaries and affiliates,
"Orchard") makes no representation regarding the accuracy or completeness
of the information contained herein. The intended recipient is advised to
consult its own professional advisors, including those specializing in
legal, tax and accounting matters. Orchard does not provide legal, tax or
accounting advice.
Re: Unable to execute sparkr jobs with Chronos
Posted by haosdent <ha...@gmail.com>.
Hi, @Brown. The error looks like you need to pass the flag `--master
mesos://leader.mesos:5050` to spark-submit.
On Thu, Jun 16, 2016 at 6:57 PM, Rodrick Brown <ro...@orchard-app.com>
wrote:
> We use Chronos extensively for running our batch based spark jobs and
> recently started using sparkr however I'm seeing an issue trying to get
> sparkr jobs to run successfully when launched through Chronos basically
> always failing with a generic error below
>
> Launching java with spark-submit command /opt/spark-1.6.1/bin/spark-submit
> --jars
> /data/orchard/etc/config/load-tradedata-accumulo-prod.jar,/data/orchard/jars/dataloader-library-626d9b10bef28a163ad61d8a2e87bb71d455ea34-assembled.jar
> --driver-memory "6000M" sparkr-shell
> /tmp/RtmpzJUGsJ/backend_port11c5c2f3ca40b
> Error: Master must start with yarn, spark, mesos, or local
> Run with --help for usage help or --verbose for debug output Error in
> sparkR.init(master = sparkr_master, appName = job_name, sparkJars =
> sparkr_jars, :
> JVM is not ready after 10 seconds
> Calls: orchard_sparkR_init -> sparkR.init
> In addition: Warning message:
> In processSparkJars(sparkJars) :
> sparkJars as a comma-separated string is deprecated, use character vector
> instead
> Execution halted
>
> I believe it may have something to do with the sparkr-shell that's forked
> from R
> If I run this job outside of Chronos directly from the CLI everything runs
> fine it connects to my mesos cluster and succeeds.
>
> I'm not sure what could be causing this issue.
>
> I've tried using R CMD BATCH myscript.R and Rscript --vanilla myscript.R
> both suffer from the same issue.
>
> Here is the job definition
>
> {
> "name": "PROD_applepie_loan_detail_R",
> "command": "env && exec /data/orchard/bin/orchard_job_executor.py -j
> PROD_applepie_loan_detail_R",
> "shell": true,
> "epsilon": "PT15M",
> "executor": "",
> "executorFlags": "",
> "retries": 3,
> "owner": "ops@orchardplatform.com",
> "ownerName": "",
> "runAsUser": "orchard",
> "description": "",
> "async": false,
> "successCount": 0,
> "errorCount": 0,
> "lastSuccess": "",
> "lastError": "",
> "cpus": 0.1,
> "disk": 10,
> "mem": 6000,
> "disabled": false,
> "softError": false,
> "dataProcessingJobType": false,
> "errorsSinceLastSuccess": 0,
> "uris": [ "file:///data/orchard/R/sparkr_env.R",
> "file:///data/orchard/R/applepie_loan_detail.R"],
> "environmentVariables": [
> {
> "name": "SPARKR_MASTER",
> "value": "mesos://leader.mesos:5050"
> }
> ],
> "arguments": [],
> "highPriority": false,
> "runAsUser": "fedora",
> "constraints": [[ "rack", "EQUALS", "spark" ]],
> "schedule": "R/2016-06-09T02:00:00.000Z/P1W",
> "scheduleTimeZone": "UTC"
> }
>
> --
>
> *Rodrick Brown* / Systems Engineer
>
> +1 917 445 6839 / rodrick@orchardplatform.com
> <ch...@orchardplatform.com>
>
> *Orchard Platform*
>
> 101 5th Avenue, 4th Floor, New York, NY 10003
>
> http://www.orchardplatform.com
>
> Orchard Blog <http://www.orchardplatform.com/blog/> | Marketplace Lending
> Meetup <http://www.meetup.com/Peer-to-Peer-Lending-P2P/>
>
> *NOTICE TO RECIPIENTS*: This communication is confidential and intended
> for the use of the addressee only. If you are not an intended recipient of
> this communication, please delete it immediately and notify the sender by
> return email. Unauthorized reading, dissemination, distribution or copying
> of this communication is prohibited. This communication does not constitute
> an offer to sell or a solicitation of an indication of interest to purchase
> any loan, security or any other financial product or instrument, nor is it
> an offer to sell or a solicitation of an indication of interest to purchase
> any products or services to any persons who are prohibited from receiving
> such information under applicable law. The contents of this communication
> may not be accurate or complete and are subject to change without notice.
> As such, Orchard App, Inc. (including its subsidiaries and affiliates,
> "Orchard") makes no representation regarding the accuracy or completeness
> of the information contained herein. The intended recipient is advised to
> consult its own professional advisors, including those specializing in
> legal, tax and accounting matters. Orchard does not provide legal, tax or
> accounting advice.
>
--
Best Regards,
Haosdent Huang