You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Tom Arnfeld <to...@duedil.com> on 2014/07/09 22:06:18 UTC
Python Celery on Mesos
Hey everyone,
Posted this on IRC earlier but thought i'd bring it up here. Has anyone given any thought to building an extension to Celery (a popular python framework for building task based apps http://www.celeryproject.org/) to allow it to behave as a Mesos framework and run tasks on Mesos? I'm not very familiar with the internals of Celery so it might be more complex to achieve than it looks from the surface, but something we'd be interested in working with.
Tom.
--
Tom Arnfeld
Developer // DueDil
tom@duedil.com
(+44) 7525940046
25 Christopher Street, London, EC2A 2BS
Company Number: 06999618
What is DueDil? | Product features | Try it for free
Re: Python Celery on Mesos
Posted by Tom Arnfeld <to...@duedil.com>.
Yeah that is possible, though I was thinking more making a mesos framework
that behaves as a broker to fully utilise the point of mesos (fine grained
task allocation).
Not sure I fully understand the landscape of how Celery components for
together enough to know whether that's possible or not, but it's an
interesting experiment.
Tom.
On 10 Jul 2014, at 00:14, Hunter Owens <ow...@gmail.com> wrote:
You could run Celery through Marathon, you'd just also need to have a
broker (Reddis, RabbitMQ) running concurrently on mesos (or elsewhere,
technically) . Though it may be useful to framework the ability to package
it all together.
-Hunter
On Wed, Jul 9, 2014 at 4:06 PM, Tom Arnfeld <to...@duedil.com> wrote:
> Hey everyone,
>
> Posted this on IRC earlier but thought i'd bring it up here. Has anyone
> given any thought to building an extension to Celery (a popular python
> framework for building task based apps http://www.celeryproject.org/) to
> allow it to behave as a Mesos framework and run tasks on Mesos? I'm not
> very familiar with the internals of Celery so it might be more complex to
> achieve than it looks from the surface, but something we'd be interested in
> working with.
>
> Tom.
>
> --
> *Tom Arnfeld*
> Developer // DueDil
>
> tom@duedil.com
> (+44) 7525940046
>
>
>
> 25 Christopher Street, London, EC2A 2BS
> Company Number: 06999618
>
> What is DueDil? <https://www.duedil.com/about> | Product features
> <https://www.duedil.com/features?ref=ft> | Try it for free
> <https://www.duedil.com/>
>
>
Re: Python Celery on Mesos
Posted by Hunter Owens <ow...@gmail.com>.
You could run Celery through Marathon, you'd just also need to have a
broker (Reddis, RabbitMQ) running concurrently on mesos (or elsewhere,
technically) . Though it may be useful to framework the ability to package
it all together.
-Hunter
On Wed, Jul 9, 2014 at 4:06 PM, Tom Arnfeld <to...@duedil.com> wrote:
> Hey everyone,
>
> Posted this on IRC earlier but thought i'd bring it up here. Has anyone
> given any thought to building an extension to Celery (a popular python
> framework for building task based apps http://www.celeryproject.org/) to
> allow it to behave as a Mesos framework and run tasks on Mesos? I'm not
> very familiar with the internals of Celery so it might be more complex to
> achieve than it looks from the surface, but something we'd be interested in
> working with.
>
> Tom.
>
> --
> *Tom Arnfeld*
> Developer // DueDil
>
> tom@duedil.com
> (+44) 7525940046
>
>
>
> 25 Christopher Street, London, EC2A 2BS
> Company Number: 06999618
>
> What is DueDil? <https://www.duedil.com/about> | Product features
> <https://www.duedil.com/features?ref=ft> | Try it for free
> <https://www.duedil.com/>
>
>