Re: Launching master/slave jobs in Auora

["Erb, Stephan" <St...@blue-yonder.com>]

Hi Chris,


most of the time you have a couple of subsystems that you need to tie together to build a functioning system or service (database, a couple of Aurora services and cron jobs, links to your log aggregator, links to your monitoring infrastructure, a link to your package repository, etc). We embed those into some kind of dashboard in front of our Aurora cluster. A few details can be found here [1].


Working with the plain Aurora client will probably work just as well and might even be the most straightforward approach when you start from scratch.


Having a separate job for master and worker does not  add much overall complexity for us. We opted for a simple naming convention to differentiate between the two. Given the flexibility of Aurora's job configuration you can even define both types in the same .aurora file. ​


[1] http://www.blue-yonder.com/blog-e/2015/07/18/scalable-data-science-apache-aurora-at-blue-yonder/​


Let me know if you have any more questions,

Stephan


________________________________
From: Chris Bannister <c....@gmail.com>
Sent: Tuesday, December 29, 2015 2:31 PM
To: user@aurora.apache.org
Subject: Re: Launching master/slave jobs in Auora

Thanks both for the responses, very useful. The only real issue, which isnt much of a problem, is that the jobs are created and killed separately. They will then be displayed separately in the UI. I think this could be solved by having something akin to Kubernetes Labels, a way to group multiple jobs some way.

Could you comment or expand on how you found launching multiple jobs? Is this done just with the Aurora client, or with some other tools?

Thanks

On Tue, 22 Dec 2015 at 15:57 Zameer Manji <zm...@apache.org>> wrote:
+1 to the two jobs idea.

Not only do you get the benefits that Stephan mentioned, but Aurora already assumes that a job is a pool of like tasks. This assumption is used right now for the maintenance tooling (where it try to keep 95% of the instances up at a given time), but it can be used elsewhere in the future. By going the two jobs route you avoid the case where Aurora reschedules shard 0 and disrupts all of the work done.

It is possible to make shard 0 the coordinator/driver task and I have seen production systems which do this. However, I advise against it for the reasons mentioned. If you do find creating 2 jobs is more difficult to orchestrate, could you please detail why?

On Tue, Dec 22, 2015 at 9:38 AM, Erb, Stephan <St...@blue-yonder.com>> wrote:

Hi Chris,


we are spawning an internal batch processing framework on Aurora, consisting of a single master and multiple workers.


We opted for the 2 jobs idea. Main advantage I see with this approach that you actually keep separate things separate without having to teach all external systems (service discovery, load balancer, your monitoring solution, etc) that the first instance is different.


Best Regards,

Stephan

​



________________________________
From: Chris Bannister <c....@gmail.com>>
Sent: Tuesday, December 22, 2015 2:47 PM
To: user@aurora.apache.org<ma...@aurora.apache.org>
Subject: Launching master/slave jobs in Auora

Hi, I'm doing some work to get Apache Spark running in Aurora and it seems to work reasonably well without many changes to Spark, the only issue I'm running into is launching it over many instances.

Spark runs in a driver executor model, where the driver coordinates works on the executors. The problem I have is that I want to launch the executors and drivers independently, ie I want to have 10 executors and 1 driver. I can accomplish this by having 2 jobs, a driver and an executor job, but launching this seems a bit complicated to orchestrate. Another option would be to declare the job with 2 tasks, have the driver run on shard 0 and executors on the rest.

Has anyone had any experience with running similar systems in Aurora? I imagine Heron must have to do something similar, launching the topology master and workers.

Chris



--
Zameer Manji