You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Stefano Baghino <st...@teralytics.ch> on 2017/08/03 06:47:38 UTC
Mesos with Python: mesos.native is missing
Hello everyone!
I'm trying to use Apache Mesos to run jobs automated with Apache Airflow
(workflow management).
In order to run Airflow on top of Marathon (which we need to have Airflow
automatically restart in case of failure), I created a container
(Dockerfile here
<https://gist.github.com/stefanobaghino/50b2919eef6acb108d3647ccf570d937#file-dockerfile>)
and then provided the necessary environment variables to let it know to: 1)
use Mesos and 2) where to find the Mesos master. The problem arises when I
launch the container: apparently, `mesos.native` is missing (stacktrace here
<https://gist.github.com/stefanobaghino/50b2919eef6acb108d3647ccf570d937#file-stderr>).
I tried installing `mesos` or `mesos.native` via pip in the container but
both appear to not be there (`mesos` is actually there when you search, but
there is no actual package to install as it's just a placeholder).
Is my diagnosis correct? How can I install the required Mesos packages?
Both the container definition and the error stacktrace are available in
this gist
<https://gist.github.com/stefanobaghino/50b2919eef6acb108d3647ccf570d937#file-stderr>.
Thanks!
--
Stefano Baghino | TERALYTICS
*software engineer*
Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
phone: +41 43 508 24 57
email: stefano.baghino@teralytics.ch
www.teralytics.net
Company registration number: CH-020.3.037.709-7 | Trade register Canton
Zurich
Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
de Vries
This e-mail message contains confidential information which is for the sole
attention and use of the intended recipient. Please notify us at once if
you think that it may not be intended for you and delete it immediately.
Re: Mesos with Python: mesos.native is missing
Posted by Stefano Baghino <st...@teralytics.ch>.
As a follow up to my message this morning, I've been able to to finally
make the whole thing work: Airflow is dockerized and deployed on Marathon
and uses Mesos as an executor. To solve the problem, I copied a small
portion of the official Mesos image to install Mesos on the container. I
then copied the installed Python bindings to a directory where those could
be found at runtime. I have updated the original gist and here
<https://gist.github.com/stefanobaghino/50b2919eef6acb108d3647ccf570d937/revisions>
you can see what I changed.
On Thu, Aug 3, 2017 at 12:13 PM, Hitec Adam Cecile <ad...@hitec.lu>
wrote:
> Hello,
>
> There's a problem with the build system not guessing the WHL filename
> correctly. Look at the JIRA bug, I reported it with a fix.
>
> Regards, Adam
>
>
> Le 3 août 2017 08:47:38 GMT+02:00, Stefano Baghino <
> stefano.baghino@teralytics.ch> a écrit :
>>
>> Hello everyone!
>>
>> I'm trying to use Apache Mesos to run jobs automated with Apache Airflow
>> (workflow management).
>>
>> In order to run Airflow on top of Marathon (which we need to have Airflow
>> automatically restart in case of failure), I created a container
>> (Dockerfile here
>> <https://gist.github.com/stefanobaghino/50b2919eef6acb108d3647ccf570d937#file-dockerfile>)
>> and then provided the necessary environment variables to let it know to: 1)
>> use Mesos and 2) where to find the Mesos master. The problem arises when I
>> launch the container: apparently, `mesos.native` is missing (stacktrace
>> here
>> <https://gist.github.com/stefanobaghino/50b2919eef6acb108d3647ccf570d937#file-stderr>).
>> I tried installing `mesos` or `mesos.native` via pip in the container but
>> both appear to not be there (`mesos` is actually there when you search, but
>> there is no actual package to install as it's just a placeholder).
>>
>> Is my diagnosis correct? How can I install the required Mesos packages?
>> Both the container definition and the error stacktrace are available in
>> this gist
>> <https://gist.github.com/stefanobaghino/50b2919eef6acb108d3647ccf570d937#file-stderr>.
>> Thanks!
>>
>> --
>> Stefano Baghino | TERALYTICS
>> *software engineer*
>>
>> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
>> phone: +41 43 508 24 57
>> email: stefano.baghino@teralytics.ch
>> www.teralytics.net
>>
>> Company registration number: CH-020.3.037.709-7 | Trade register Canton
>> Zurich
>> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz,
>> Yann de Vries
>>
>> This e-mail message contains confidential information which is for the
>> sole attention and use of the intended recipient. Please notify us at once
>> if you think that it may not be intended for you and delete it immediately.
>>
>
> --
> Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma
> brièveté.
>
--
Stefano Baghino | TERALYTICS
*software engineer*
Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
phone: +41 43 508 24 57
email: stefano.baghino@teralytics.ch
www.teralytics.net
Company registration number: CH-020.3.037.709-7 | Trade register Canton
Zurich
Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
de Vries
This e-mail message contains confidential information which is for the sole
attention and use of the intended recipient. Please notify us at once if
you think that it may not be intended for you and delete it immediately.
Re: Mesos with Python: mesos.native is missing
Posted by Hitec Adam Cecile <ad...@hitec.lu>.
Hello,
There's a problem with the build system not guessing the WHL filename correctly. Look at the JIRA bug, I reported it with a fix.
Regards, Adam
Le 3 août 2017 08:47:38 GMT+02:00, Stefano Baghino <st...@teralytics.ch> a écrit :
>Hello everyone!
>
>I'm trying to use Apache Mesos to run jobs automated with Apache
>Airflow
>(workflow management).
>
>In order to run Airflow on top of Marathon (which we need to have
>Airflow
>automatically restart in case of failure), I created a container
>(Dockerfile here
><https://gist.github.com/stefanobaghino/50b2919eef6acb108d3647ccf570d937#file-dockerfile>)
>and then provided the necessary environment variables to let it know
>to: 1)
>use Mesos and 2) where to find the Mesos master. The problem arises
>when I
>launch the container: apparently, `mesos.native` is missing (stacktrace
>here
><https://gist.github.com/stefanobaghino/50b2919eef6acb108d3647ccf570d937#file-stderr>).
>I tried installing `mesos` or `mesos.native` via pip in the container
>but
>both appear to not be there (`mesos` is actually there when you search,
>but
>there is no actual package to install as it's just a placeholder).
>
>Is my diagnosis correct? How can I install the required Mesos packages?
>Both the container definition and the error stacktrace are available in
>this gist
><https://gist.github.com/stefanobaghino/50b2919eef6acb108d3647ccf570d937#file-stderr>.
>Thanks!
>
>--
>Stefano Baghino | TERALYTICS
>*software engineer*
>
>Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
>phone: +41 43 508 24 57
>email: stefano.baghino@teralytics.ch
>www.teralytics.net
>
>Company registration number: CH-020.3.037.709-7 | Trade register Canton
>Zurich
>Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz,
>Yann
>de Vries
>
>This e-mail message contains confidential information which is for the
>sole
>attention and use of the intended recipient. Please notify us at once
>if
>you think that it may not be intended for you and delete it
>immediately.
--
Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma brièveté.