You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Ash Berlin-Taylor <as...@apache.org> on 2019/01/25 11:46:55 UTC
[DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Is anyone using the Mesos Executor? I think we should deprecate and remove it.
The reason I say that is I have a feeling it's been broken since 1.9.0 and no-one has noticed:
class MesosExecutor(BaseExecutor, LoginMixin):
def start(self):
# ...
self.log.info(
'MesosFramework master : %s, name : %s, cpu : %s, mem : %s, checkpoint : %s',
master, framework.name,
str(task_cpu), str(task_memory), str(framework.checkpoint)
)
That mixin should have been LoggingMixin, not LoginMixin. But given this bug has been in since https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29 which was in the 1.9.0 release I propose we simply remove this executor from master.
Thoughts?
Ash
Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
https://jira.apache.org/jira/browse/AIRFLOW-4313
:-)
Op za 13 apr. 2019 om 12:28 schreef Felix Uellendall <
felix.uellendall@gmx.de>:
> +1 (non-binding)
>
> Am 12/04/2019 um 23:42 schrieb Tao Feng:
> > +1 on removing mesos executor.
> >
> > On Fri, Apr 12, 2019 at 2:25 PM Daniel Imberman <
> daniel.imberman@gmail.com>
> > wrote:
> >
> >> Since we are now doing version 2.0 spring cleaning can we revisit this?
> >>
> >> On Wed, Feb 27, 2019 at 8:38 AM Maxime Beauchemin <
> >> maximebeauchemin@gmail.com> wrote:
> >>
> >>> +1 as it relies on pickles
> >>>
> >>> On Wed, Feb 27, 2019 at 4:51 AM Ash Berlin-Taylor <as...@apache.org>
> >> wrote:
> >>>> I say lets remove it in 2.0.
> >>>>
> >>>>> On 27 Feb 2019, at 11:50, Driesprong, Fokko <fo...@driesprong.frl>
> >>>> wrote:
> >>>>> Are we going to remove the MesosExecutor? I think it is a safe bet
> >>> since
> >>>> in
> >>>>> the Survey of Ash (thanks again for that!), it also shows that nobody
> >>> is
> >>>>> really using it.
> >>>>>
> >>>>> https://jira.apache.org/jira/browse/AIRFLOW-3969
> >>>>>
> >>>>> Cheers, Fokko
> >>>>>
> >>>>> Op di 12 feb. 2019 om 10:28 schreef Ash Berlin-Taylor <
> >> ash@apache.org
> >>>> :
> >>>>>> I’m glad yarn wasn’t the only option - it would have meant I’d have
> >>>> never
> >>>>>> been in a position to use Airflow! (Many of our workflows don’t
> >> touch
> >>>>>> EMR/Hadoop, and running Celery is much more of a known element to a
> >>>> python
> >>>>>> developer than configuring Yarn)
> >>>>>>
> >>>>>>> On 11 Feb 2019, at 17:14, Maxime Beauchemin <
> >>>> maximebeauchemin@gmail.com>
> >>>>>> wrote:
> >>>>>>> From memory, I think MesosExecutor depends on pickling to get DAG
> >>>>>>> definitions to workers, which we should also deprecate.
> >>>>>>>
> >>>>>>> About CeleryExecutor, we never had the intention to make it the
> >>>>>> recommended
> >>>>>>> option for production early on. The intent back in 2014 was to
> >> write
> >>> a
> >>>>>>> YarnExecutor quickly (that was before the rise of k8s), but
> >> realized
> >>>> that
> >>>>>>> container support wasn't ready in Yarn at the time, and had hard
> >>>>>>> restrictions on what we could install on Hadoop/Yarn data nodes, so
> >>> we
> >>>>>>> moved forward with Celery.
> >>>>>>>
> >>>>>>> I think it's practical to have LocalExecutor around though for unit
> >>>> test,
> >>>>>>> local use, and for POCs.
> >>>>>>>
> >>>>>>> Max
> >>>>>>>
> >>>>>>> On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot <
> >>>>>>> o.girardot@lateral-thoughts.com> wrote:
> >>>>>>>
> >>>>>>>> Hi, we tried to use it for a new project 1year ago, but it was too
> >>>>>> broken
> >>>>>>>> to be usable, and we didn't have the time to make it work.
> >>>>>>>> Of course we are still hoping to use it one day, but that doesn't
> >>> mean
> >>>>>> it
> >>>>>>>> should not move to contrib, it was already clear from the
> >>>> documentation
> >>>>>>>> that it was not a first class citizen.
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>>
> >>>>>>>> Olivier.
> >>>>>>>>
> >>>>>>>> Le dim. 27 janv. 2019 à 12:12, Ash Berlin-Taylor <as...@apache.org>
> >> a
> >>>>>> écrit
> >>>>>>>> :
> >>>>>>>>
> >>>>>>>>> That PR pre-dates 1.9.0 being released.
> >>>>>>>>>
> >>>>>>>>>> On 27 Jan 2019, at 06:57, airflowuser <
> >> airflowuser@protonmail.com
> >>>>>>>> .INVALID>
> >>>>>>>>> wrote:
> >>>>>>>>>> The Mesos executor is being used (at least by some people) and
> >>> there
> >>>>>> is
> >>>>>>>>> an open PR to extend it:
> >>>>>>>>>> https://github.com/apache/airflow/pull/3739
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Sent with ProtonMail Secure Email.
> >>>>>>>>>>
> >>>>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> >>>>>>>>>> On Friday, January 25, 2019 7:30 PM, Kaxil Naik <
> >>>> kaxilnaik@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>> What about Dask Executor as well.
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman <
> >>>>>>>> daniel.imberman@gmail.com
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> @sid basically my response lol
> >>>>>>>>>>>> I haven't seen any emails from any users and if it's been
> >> broken
> >>>> for
> >>>>>>>>> two
> >>>>>>>>>>>> versions then there doesn't seem to be enough interest to
> >>> maintain
> >>>>>>>> it.
> >>>>>>>>>>>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39132@gmail.com
> >>> wrote:
> >>>>>>>>>>>>> LOL :_0
> >>>>>>>>>>>>> -s
> >>>>>>>>>>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin
> >>> bdbruin@gmail.com
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> I agree. Maybe we should make a choice on what we consider
> >> 1st
> >>>>>>>> class
> >>>>>>>>>>>>>> executors and move others to contrib (I'm looking at you
> >>> celery)
> >>>>>>>>>>>>>> Sent from my iPhone
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor ash@apache.org
> >>>>>> wrote:
> >>>>>>>>>>>>>>> Is anyone using the Mesos Executor? I think we should
> >>> deprecate
> >>>>>>>> and
> >>>>>>>>>>>>>>> remove it.
> >>>>>>>>>>>>>>> The reason I say that is I have a feeling it's been broken
> >>>> since
> >>>>>>>>>>>>>>> 1.9.0
> >>>>>>>>>>>>>> and no-one has noticed:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> class MesosExecutor(BaseExecutor, LoginMixin):
> >>>>>>>>>>>>>>> def start(self):
> >>>>>>>>>>>>>>> # ...
> >>>>>>>>>>>>>>> self.log.info(
> >>>>>>>>>>>>>>> 'MesosFramework master : %s, name : %s, cpu : %s, mem :
> >>>>>>>>>>>>>>> %s,
> >>>>>>>>>>>>>> checkpoint : %s',
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> master, framework.name,
> >>>>>>>>>>>>>>> str(task_cpu), str(task_memory),
> >>>>>>>> str(framework.checkpoint)
> >>>>>>>>>>>>>>> )
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> That mixin should have been LoggingMixin, not LoginMixin.
> >> But
> >>>>>>>> given
> >>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>> bug has been in since
> >>>>>>>>>>>>
> >>
> https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29
> >>>>>>>>>>>>>> which was in the 1.9.0 release I propose we simply remove
> >> this
> >>>>>>>>> executor
> >>>>>>>>>>>>>> from master.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thoughts?
> >>>>>>>>>>>>>>> Ash
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
>
Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Posted by Felix Uellendall <fe...@gmx.de>.
+1 (non-binding)
Am 12/04/2019 um 23:42 schrieb Tao Feng:
> +1 on removing mesos executor.
>
> On Fri, Apr 12, 2019 at 2:25 PM Daniel Imberman <da...@gmail.com>
> wrote:
>
>> Since we are now doing version 2.0 spring cleaning can we revisit this?
>>
>> On Wed, Feb 27, 2019 at 8:38 AM Maxime Beauchemin <
>> maximebeauchemin@gmail.com> wrote:
>>
>>> +1 as it relies on pickles
>>>
>>> On Wed, Feb 27, 2019 at 4:51 AM Ash Berlin-Taylor <as...@apache.org>
>> wrote:
>>>> I say lets remove it in 2.0.
>>>>
>>>>> On 27 Feb 2019, at 11:50, Driesprong, Fokko <fo...@driesprong.frl>
>>>> wrote:
>>>>> Are we going to remove the MesosExecutor? I think it is a safe bet
>>> since
>>>> in
>>>>> the Survey of Ash (thanks again for that!), it also shows that nobody
>>> is
>>>>> really using it.
>>>>>
>>>>> https://jira.apache.org/jira/browse/AIRFLOW-3969
>>>>>
>>>>> Cheers, Fokko
>>>>>
>>>>> Op di 12 feb. 2019 om 10:28 schreef Ash Berlin-Taylor <
>> ash@apache.org
>>>> :
>>>>>> I’m glad yarn wasn’t the only option - it would have meant I’d have
>>>> never
>>>>>> been in a position to use Airflow! (Many of our workflows don’t
>> touch
>>>>>> EMR/Hadoop, and running Celery is much more of a known element to a
>>>> python
>>>>>> developer than configuring Yarn)
>>>>>>
>>>>>>> On 11 Feb 2019, at 17:14, Maxime Beauchemin <
>>>> maximebeauchemin@gmail.com>
>>>>>> wrote:
>>>>>>> From memory, I think MesosExecutor depends on pickling to get DAG
>>>>>>> definitions to workers, which we should also deprecate.
>>>>>>>
>>>>>>> About CeleryExecutor, we never had the intention to make it the
>>>>>> recommended
>>>>>>> option for production early on. The intent back in 2014 was to
>> write
>>> a
>>>>>>> YarnExecutor quickly (that was before the rise of k8s), but
>> realized
>>>> that
>>>>>>> container support wasn't ready in Yarn at the time, and had hard
>>>>>>> restrictions on what we could install on Hadoop/Yarn data nodes, so
>>> we
>>>>>>> moved forward with Celery.
>>>>>>>
>>>>>>> I think it's practical to have LocalExecutor around though for unit
>>>> test,
>>>>>>> local use, and for POCs.
>>>>>>>
>>>>>>> Max
>>>>>>>
>>>>>>> On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot <
>>>>>>> o.girardot@lateral-thoughts.com> wrote:
>>>>>>>
>>>>>>>> Hi, we tried to use it for a new project 1year ago, but it was too
>>>>>> broken
>>>>>>>> to be usable, and we didn't have the time to make it work.
>>>>>>>> Of course we are still hoping to use it one day, but that doesn't
>>> mean
>>>>>> it
>>>>>>>> should not move to contrib, it was already clear from the
>>>> documentation
>>>>>>>> that it was not a first class citizen.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Olivier.
>>>>>>>>
>>>>>>>> Le dim. 27 janv. 2019 à 12:12, Ash Berlin-Taylor <as...@apache.org>
>> a
>>>>>> écrit
>>>>>>>> :
>>>>>>>>
>>>>>>>>> That PR pre-dates 1.9.0 being released.
>>>>>>>>>
>>>>>>>>>> On 27 Jan 2019, at 06:57, airflowuser <
>> airflowuser@protonmail.com
>>>>>>>> .INVALID>
>>>>>>>>> wrote:
>>>>>>>>>> The Mesos executor is being used (at least by some people) and
>>> there
>>>>>> is
>>>>>>>>> an open PR to extend it:
>>>>>>>>>> https://github.com/apache/airflow/pull/3739
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Sent with ProtonMail Secure Email.
>>>>>>>>>>
>>>>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>>>> On Friday, January 25, 2019 7:30 PM, Kaxil Naik <
>>>> kaxilnaik@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>> What about Dask Executor as well.
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman <
>>>>>>>> daniel.imberman@gmail.com
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> @sid basically my response lol
>>>>>>>>>>>> I haven't seen any emails from any users and if it's been
>> broken
>>>> for
>>>>>>>>> two
>>>>>>>>>>>> versions then there doesn't seem to be enough interest to
>>> maintain
>>>>>>>> it.
>>>>>>>>>>>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39132@gmail.com
>>> wrote:
>>>>>>>>>>>>> LOL :_0
>>>>>>>>>>>>> -s
>>>>>>>>>>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin
>>> bdbruin@gmail.com
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I agree. Maybe we should make a choice on what we consider
>> 1st
>>>>>>>> class
>>>>>>>>>>>>>> executors and move others to contrib (I'm looking at you
>>> celery)
>>>>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor ash@apache.org
>>>>>> wrote:
>>>>>>>>>>>>>>> Is anyone using the Mesos Executor? I think we should
>>> deprecate
>>>>>>>> and
>>>>>>>>>>>>>>> remove it.
>>>>>>>>>>>>>>> The reason I say that is I have a feeling it's been broken
>>>> since
>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>> and no-one has noticed:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> class MesosExecutor(BaseExecutor, LoginMixin):
>>>>>>>>>>>>>>> def start(self):
>>>>>>>>>>>>>>> # ...
>>>>>>>>>>>>>>> self.log.info(
>>>>>>>>>>>>>>> 'MesosFramework master : %s, name : %s, cpu : %s, mem :
>>>>>>>>>>>>>>> %s,
>>>>>>>>>>>>>> checkpoint : %s',
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> master, framework.name,
>>>>>>>>>>>>>>> str(task_cpu), str(task_memory),
>>>>>>>> str(framework.checkpoint)
>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That mixin should have been LoggingMixin, not LoginMixin.
>> But
>>>>>>>> given
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> bug has been in since
>>>>>>>>>>>>
>> https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29
>>>>>>>>>>>>>> which was in the 1.9.0 release I propose we simply remove
>> this
>>>>>>>>> executor
>>>>>>>>>>>>>> from master.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>> Ash
>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Posted by Tao Feng <fe...@gmail.com>.
+1 on removing mesos executor.
On Fri, Apr 12, 2019 at 2:25 PM Daniel Imberman <da...@gmail.com>
wrote:
> Since we are now doing version 2.0 spring cleaning can we revisit this?
>
> On Wed, Feb 27, 2019 at 8:38 AM Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > +1 as it relies on pickles
> >
> > On Wed, Feb 27, 2019 at 4:51 AM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> >
> > > I say lets remove it in 2.0.
> > >
> > > > On 27 Feb 2019, at 11:50, Driesprong, Fokko <fo...@driesprong.frl>
> > > wrote:
> > > >
> > > > Are we going to remove the MesosExecutor? I think it is a safe bet
> > since
> > > in
> > > > the Survey of Ash (thanks again for that!), it also shows that nobody
> > is
> > > > really using it.
> > > >
> > > > https://jira.apache.org/jira/browse/AIRFLOW-3969
> > > >
> > > > Cheers, Fokko
> > > >
> > > > Op di 12 feb. 2019 om 10:28 schreef Ash Berlin-Taylor <
> ash@apache.org
> > >:
> > > >
> > > >> I’m glad yarn wasn’t the only option - it would have meant I’d have
> > > never
> > > >> been in a position to use Airflow! (Many of our workflows don’t
> touch
> > > >> EMR/Hadoop, and running Celery is much more of a known element to a
> > > python
> > > >> developer than configuring Yarn)
> > > >>
> > > >>> On 11 Feb 2019, at 17:14, Maxime Beauchemin <
> > > maximebeauchemin@gmail.com>
> > > >> wrote:
> > > >>>
> > > >>> From memory, I think MesosExecutor depends on pickling to get DAG
> > > >>> definitions to workers, which we should also deprecate.
> > > >>>
> > > >>> About CeleryExecutor, we never had the intention to make it the
> > > >> recommended
> > > >>> option for production early on. The intent back in 2014 was to
> write
> > a
> > > >>> YarnExecutor quickly (that was before the rise of k8s), but
> realized
> > > that
> > > >>> container support wasn't ready in Yarn at the time, and had hard
> > > >>> restrictions on what we could install on Hadoop/Yarn data nodes, so
> > we
> > > >>> moved forward with Celery.
> > > >>>
> > > >>> I think it's practical to have LocalExecutor around though for unit
> > > test,
> > > >>> local use, and for POCs.
> > > >>>
> > > >>> Max
> > > >>>
> > > >>> On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot <
> > > >>> o.girardot@lateral-thoughts.com> wrote:
> > > >>>
> > > >>>> Hi, we tried to use it for a new project 1year ago, but it was too
> > > >> broken
> > > >>>> to be usable, and we didn't have the time to make it work.
> > > >>>> Of course we are still hoping to use it one day, but that doesn't
> > mean
> > > >> it
> > > >>>> should not move to contrib, it was already clear from the
> > > documentation
> > > >>>> that it was not a first class citizen.
> > > >>>>
> > > >>>> Regards,
> > > >>>>
> > > >>>> Olivier.
> > > >>>>
> > > >>>> Le dim. 27 janv. 2019 à 12:12, Ash Berlin-Taylor <as...@apache.org>
> a
> > > >> écrit
> > > >>>> :
> > > >>>>
> > > >>>>> That PR pre-dates 1.9.0 being released.
> > > >>>>>
> > > >>>>>> On 27 Jan 2019, at 06:57, airflowuser <
> airflowuser@protonmail.com
> > > >>>> .INVALID>
> > > >>>>> wrote:
> > > >>>>>>
> > > >>>>>> The Mesos executor is being used (at least by some people) and
> > there
> > > >> is
> > > >>>>> an open PR to extend it:
> > > >>>>>> https://github.com/apache/airflow/pull/3739
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Sent with ProtonMail Secure Email.
> > > >>>>>>
> > > >>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > >>>>>> On Friday, January 25, 2019 7:30 PM, Kaxil Naik <
> > > kaxilnaik@gmail.com>
> > > >>>>> wrote:
> > > >>>>>>
> > > >>>>>>> What about Dask Executor as well.
> > > >>>>>>>
> > > >>>>>>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman <
> > > >>>> daniel.imberman@gmail.com
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> @sid basically my response lol
> > > >>>>>>>> I haven't seen any emails from any users and if it's been
> broken
> > > for
> > > >>>>> two
> > > >>>>>>>> versions then there doesn't seem to be enough interest to
> > maintain
> > > >>>> it.
> > > >>>>>>>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39132@gmail.com
> > wrote:
> > > >>>>>>>>>
> > > >>>>>>>>> LOL :_0
> > > >>>>>>>>> -s
> > > >>>>>>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin
> > bdbruin@gmail.com
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> I agree. Maybe we should make a choice on what we consider
> 1st
> > > >>>> class
> > > >>>>>>>>>> executors and move others to contrib (I'm looking at you
> > celery)
> > > >>>>>>>>>> Sent from my iPhone
> > > >>>>>>>>>>
> > > >>>>>>>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor ash@apache.org
> > > >> wrote:
> > > >>>>>>>>>>> Is anyone using the Mesos Executor? I think we should
> > deprecate
> > > >>>> and
> > > >>>>>>>>>>> remove it.
> > > >>>>>>>>>>> The reason I say that is I have a feeling it's been broken
> > > since
> > > >>>>>>>>>>> 1.9.0
> > > >>>>>>>>>
> > > >>>>>>>>>> and no-one has noticed:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> class MesosExecutor(BaseExecutor, LoginMixin):
> > > >>>>>>>>>>> def start(self):
> > > >>>>>>>>>>> # ...
> > > >>>>>>>>>>> self.log.info(
> > > >>>>>>>>>>> 'MesosFramework master : %s, name : %s, cpu : %s, mem :
> > > >>>>>>>>>>> %s,
> > > >>>>>>>>>
> > > >>>>>>>>>> checkpoint : %s',
> > > >>>>>>>>>>
> > > >>>>>>>>>>> master, framework.name,
> > > >>>>>>>>>>> str(task_cpu), str(task_memory),
> > > >>>> str(framework.checkpoint)
> > > >>>>>>>>>>> )
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> That mixin should have been LoggingMixin, not LoginMixin.
> But
> > > >>>> given
> > > >>>>>>>>>>> this
> > > >>>>>>>>>>> bug has been in since
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>
> > > >>>>
> > > >>
> > >
> >
> https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29
> > > >>>>>>>>
> > > >>>>>>>>>> which was in the 1.9.0 release I propose we simply remove
> this
> > > >>>>> executor
> > > >>>>>>>>>> from master.
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Thoughts?
> > > >>>>>>>>>>> Ash
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>
> > > >>
> > >
> > >
> >
>
Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Posted by Daniel Imberman <da...@gmail.com>.
Since we are now doing version 2.0 spring cleaning can we revisit this?
On Wed, Feb 27, 2019 at 8:38 AM Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:
> +1 as it relies on pickles
>
> On Wed, Feb 27, 2019 at 4:51 AM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > I say lets remove it in 2.0.
> >
> > > On 27 Feb 2019, at 11:50, Driesprong, Fokko <fo...@driesprong.frl>
> > wrote:
> > >
> > > Are we going to remove the MesosExecutor? I think it is a safe bet
> since
> > in
> > > the Survey of Ash (thanks again for that!), it also shows that nobody
> is
> > > really using it.
> > >
> > > https://jira.apache.org/jira/browse/AIRFLOW-3969
> > >
> > > Cheers, Fokko
> > >
> > > Op di 12 feb. 2019 om 10:28 schreef Ash Berlin-Taylor <ash@apache.org
> >:
> > >
> > >> I’m glad yarn wasn’t the only option - it would have meant I’d have
> > never
> > >> been in a position to use Airflow! (Many of our workflows don’t touch
> > >> EMR/Hadoop, and running Celery is much more of a known element to a
> > python
> > >> developer than configuring Yarn)
> > >>
> > >>> On 11 Feb 2019, at 17:14, Maxime Beauchemin <
> > maximebeauchemin@gmail.com>
> > >> wrote:
> > >>>
> > >>> From memory, I think MesosExecutor depends on pickling to get DAG
> > >>> definitions to workers, which we should also deprecate.
> > >>>
> > >>> About CeleryExecutor, we never had the intention to make it the
> > >> recommended
> > >>> option for production early on. The intent back in 2014 was to write
> a
> > >>> YarnExecutor quickly (that was before the rise of k8s), but realized
> > that
> > >>> container support wasn't ready in Yarn at the time, and had hard
> > >>> restrictions on what we could install on Hadoop/Yarn data nodes, so
> we
> > >>> moved forward with Celery.
> > >>>
> > >>> I think it's practical to have LocalExecutor around though for unit
> > test,
> > >>> local use, and for POCs.
> > >>>
> > >>> Max
> > >>>
> > >>> On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot <
> > >>> o.girardot@lateral-thoughts.com> wrote:
> > >>>
> > >>>> Hi, we tried to use it for a new project 1year ago, but it was too
> > >> broken
> > >>>> to be usable, and we didn't have the time to make it work.
> > >>>> Of course we are still hoping to use it one day, but that doesn't
> mean
> > >> it
> > >>>> should not move to contrib, it was already clear from the
> > documentation
> > >>>> that it was not a first class citizen.
> > >>>>
> > >>>> Regards,
> > >>>>
> > >>>> Olivier.
> > >>>>
> > >>>> Le dim. 27 janv. 2019 à 12:12, Ash Berlin-Taylor <as...@apache.org> a
> > >> écrit
> > >>>> :
> > >>>>
> > >>>>> That PR pre-dates 1.9.0 being released.
> > >>>>>
> > >>>>>> On 27 Jan 2019, at 06:57, airflowuser <airflowuser@protonmail.com
> > >>>> .INVALID>
> > >>>>> wrote:
> > >>>>>>
> > >>>>>> The Mesos executor is being used (at least by some people) and
> there
> > >> is
> > >>>>> an open PR to extend it:
> > >>>>>> https://github.com/apache/airflow/pull/3739
> > >>>>>>
> > >>>>>>
> > >>>>>> Sent with ProtonMail Secure Email.
> > >>>>>>
> > >>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > >>>>>> On Friday, January 25, 2019 7:30 PM, Kaxil Naik <
> > kaxilnaik@gmail.com>
> > >>>>> wrote:
> > >>>>>>
> > >>>>>>> What about Dask Executor as well.
> > >>>>>>>
> > >>>>>>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman <
> > >>>> daniel.imberman@gmail.com
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> @sid basically my response lol
> > >>>>>>>> I haven't seen any emails from any users and if it's been broken
> > for
> > >>>>> two
> > >>>>>>>> versions then there doesn't seem to be enough interest to
> maintain
> > >>>> it.
> > >>>>>>>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39132@gmail.com
> wrote:
> > >>>>>>>>>
> > >>>>>>>>> LOL :_0
> > >>>>>>>>> -s
> > >>>>>>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin
> bdbruin@gmail.com
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> I agree. Maybe we should make a choice on what we consider 1st
> > >>>> class
> > >>>>>>>>>> executors and move others to contrib (I'm looking at you
> celery)
> > >>>>>>>>>> Sent from my iPhone
> > >>>>>>>>>>
> > >>>>>>>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor ash@apache.org
> > >> wrote:
> > >>>>>>>>>>> Is anyone using the Mesos Executor? I think we should
> deprecate
> > >>>> and
> > >>>>>>>>>>> remove it.
> > >>>>>>>>>>> The reason I say that is I have a feeling it's been broken
> > since
> > >>>>>>>>>>> 1.9.0
> > >>>>>>>>>
> > >>>>>>>>>> and no-one has noticed:
> > >>>>>>>>>>
> > >>>>>>>>>>> class MesosExecutor(BaseExecutor, LoginMixin):
> > >>>>>>>>>>> def start(self):
> > >>>>>>>>>>> # ...
> > >>>>>>>>>>> self.log.info(
> > >>>>>>>>>>> 'MesosFramework master : %s, name : %s, cpu : %s, mem :
> > >>>>>>>>>>> %s,
> > >>>>>>>>>
> > >>>>>>>>>> checkpoint : %s',
> > >>>>>>>>>>
> > >>>>>>>>>>> master, framework.name,
> > >>>>>>>>>>> str(task_cpu), str(task_memory),
> > >>>> str(framework.checkpoint)
> > >>>>>>>>>>> )
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> That mixin should have been LoggingMixin, not LoginMixin. But
> > >>>> given
> > >>>>>>>>>>> this
> > >>>>>>>>>>> bug has been in since
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29
> > >>>>>>>>
> > >>>>>>>>>> which was in the 1.9.0 release I propose we simply remove this
> > >>>>> executor
> > >>>>>>>>>> from master.
> > >>>>>>>>>>
> > >>>>>>>>>>> Thoughts?
> > >>>>>>>>>>> Ash
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>
> > >>
> >
> >
>
Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Posted by Maxime Beauchemin <ma...@gmail.com>.
+1 as it relies on pickles
On Wed, Feb 27, 2019 at 4:51 AM Ash Berlin-Taylor <as...@apache.org> wrote:
> I say lets remove it in 2.0.
>
> > On 27 Feb 2019, at 11:50, Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
> >
> > Are we going to remove the MesosExecutor? I think it is a safe bet since
> in
> > the Survey of Ash (thanks again for that!), it also shows that nobody is
> > really using it.
> >
> > https://jira.apache.org/jira/browse/AIRFLOW-3969
> >
> > Cheers, Fokko
> >
> > Op di 12 feb. 2019 om 10:28 schreef Ash Berlin-Taylor <as...@apache.org>:
> >
> >> I’m glad yarn wasn’t the only option - it would have meant I’d have
> never
> >> been in a position to use Airflow! (Many of our workflows don’t touch
> >> EMR/Hadoop, and running Celery is much more of a known element to a
> python
> >> developer than configuring Yarn)
> >>
> >>> On 11 Feb 2019, at 17:14, Maxime Beauchemin <
> maximebeauchemin@gmail.com>
> >> wrote:
> >>>
> >>> From memory, I think MesosExecutor depends on pickling to get DAG
> >>> definitions to workers, which we should also deprecate.
> >>>
> >>> About CeleryExecutor, we never had the intention to make it the
> >> recommended
> >>> option for production early on. The intent back in 2014 was to write a
> >>> YarnExecutor quickly (that was before the rise of k8s), but realized
> that
> >>> container support wasn't ready in Yarn at the time, and had hard
> >>> restrictions on what we could install on Hadoop/Yarn data nodes, so we
> >>> moved forward with Celery.
> >>>
> >>> I think it's practical to have LocalExecutor around though for unit
> test,
> >>> local use, and for POCs.
> >>>
> >>> Max
> >>>
> >>> On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot <
> >>> o.girardot@lateral-thoughts.com> wrote:
> >>>
> >>>> Hi, we tried to use it for a new project 1year ago, but it was too
> >> broken
> >>>> to be usable, and we didn't have the time to make it work.
> >>>> Of course we are still hoping to use it one day, but that doesn't mean
> >> it
> >>>> should not move to contrib, it was already clear from the
> documentation
> >>>> that it was not a first class citizen.
> >>>>
> >>>> Regards,
> >>>>
> >>>> Olivier.
> >>>>
> >>>> Le dim. 27 janv. 2019 à 12:12, Ash Berlin-Taylor <as...@apache.org> a
> >> écrit
> >>>> :
> >>>>
> >>>>> That PR pre-dates 1.9.0 being released.
> >>>>>
> >>>>>> On 27 Jan 2019, at 06:57, airflowuser <airflowuser@protonmail.com
> >>>> .INVALID>
> >>>>> wrote:
> >>>>>>
> >>>>>> The Mesos executor is being used (at least by some people) and there
> >> is
> >>>>> an open PR to extend it:
> >>>>>> https://github.com/apache/airflow/pull/3739
> >>>>>>
> >>>>>>
> >>>>>> Sent with ProtonMail Secure Email.
> >>>>>>
> >>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> >>>>>> On Friday, January 25, 2019 7:30 PM, Kaxil Naik <
> kaxilnaik@gmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>>> What about Dask Executor as well.
> >>>>>>>
> >>>>>>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman <
> >>>> daniel.imberman@gmail.com
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> @sid basically my response lol
> >>>>>>>> I haven't seen any emails from any users and if it's been broken
> for
> >>>>> two
> >>>>>>>> versions then there doesn't seem to be enough interest to maintain
> >>>> it.
> >>>>>>>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39132@gmail.com wrote:
> >>>>>>>>>
> >>>>>>>>> LOL :_0
> >>>>>>>>> -s
> >>>>>>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin bdbruin@gmail.com
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> I agree. Maybe we should make a choice on what we consider 1st
> >>>> class
> >>>>>>>>>> executors and move others to contrib (I'm looking at you celery)
> >>>>>>>>>> Sent from my iPhone
> >>>>>>>>>>
> >>>>>>>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor ash@apache.org
> >> wrote:
> >>>>>>>>>>> Is anyone using the Mesos Executor? I think we should deprecate
> >>>> and
> >>>>>>>>>>> remove it.
> >>>>>>>>>>> The reason I say that is I have a feeling it's been broken
> since
> >>>>>>>>>>> 1.9.0
> >>>>>>>>>
> >>>>>>>>>> and no-one has noticed:
> >>>>>>>>>>
> >>>>>>>>>>> class MesosExecutor(BaseExecutor, LoginMixin):
> >>>>>>>>>>> def start(self):
> >>>>>>>>>>> # ...
> >>>>>>>>>>> self.log.info(
> >>>>>>>>>>> 'MesosFramework master : %s, name : %s, cpu : %s, mem :
> >>>>>>>>>>> %s,
> >>>>>>>>>
> >>>>>>>>>> checkpoint : %s',
> >>>>>>>>>>
> >>>>>>>>>>> master, framework.name,
> >>>>>>>>>>> str(task_cpu), str(task_memory),
> >>>> str(framework.checkpoint)
> >>>>>>>>>>> )
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> That mixin should have been LoggingMixin, not LoginMixin. But
> >>>> given
> >>>>>>>>>>> this
> >>>>>>>>>>> bug has been in since
> >>>>>>>>
> >>>>>>>>
> >>>>>
> >>>>
> >>
> https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29
> >>>>>>>>
> >>>>>>>>>> which was in the 1.9.0 release I propose we simply remove this
> >>>>> executor
> >>>>>>>>>> from master.
> >>>>>>>>>>
> >>>>>>>>>>> Thoughts?
> >>>>>>>>>>> Ash
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>
> >>
>
>
Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Posted by Ash Berlin-Taylor <as...@apache.org>.
I say lets remove it in 2.0.
> On 27 Feb 2019, at 11:50, Driesprong, Fokko <fo...@driesprong.frl> wrote:
>
> Are we going to remove the MesosExecutor? I think it is a safe bet since in
> the Survey of Ash (thanks again for that!), it also shows that nobody is
> really using it.
>
> https://jira.apache.org/jira/browse/AIRFLOW-3969
>
> Cheers, Fokko
>
> Op di 12 feb. 2019 om 10:28 schreef Ash Berlin-Taylor <as...@apache.org>:
>
>> I’m glad yarn wasn’t the only option - it would have meant I’d have never
>> been in a position to use Airflow! (Many of our workflows don’t touch
>> EMR/Hadoop, and running Celery is much more of a known element to a python
>> developer than configuring Yarn)
>>
>>> On 11 Feb 2019, at 17:14, Maxime Beauchemin <ma...@gmail.com>
>> wrote:
>>>
>>> From memory, I think MesosExecutor depends on pickling to get DAG
>>> definitions to workers, which we should also deprecate.
>>>
>>> About CeleryExecutor, we never had the intention to make it the
>> recommended
>>> option for production early on. The intent back in 2014 was to write a
>>> YarnExecutor quickly (that was before the rise of k8s), but realized that
>>> container support wasn't ready in Yarn at the time, and had hard
>>> restrictions on what we could install on Hadoop/Yarn data nodes, so we
>>> moved forward with Celery.
>>>
>>> I think it's practical to have LocalExecutor around though for unit test,
>>> local use, and for POCs.
>>>
>>> Max
>>>
>>> On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot <
>>> o.girardot@lateral-thoughts.com> wrote:
>>>
>>>> Hi, we tried to use it for a new project 1year ago, but it was too
>> broken
>>>> to be usable, and we didn't have the time to make it work.
>>>> Of course we are still hoping to use it one day, but that doesn't mean
>> it
>>>> should not move to contrib, it was already clear from the documentation
>>>> that it was not a first class citizen.
>>>>
>>>> Regards,
>>>>
>>>> Olivier.
>>>>
>>>> Le dim. 27 janv. 2019 à 12:12, Ash Berlin-Taylor <as...@apache.org> a
>> écrit
>>>> :
>>>>
>>>>> That PR pre-dates 1.9.0 being released.
>>>>>
>>>>>> On 27 Jan 2019, at 06:57, airflowuser <airflowuser@protonmail.com
>>>> .INVALID>
>>>>> wrote:
>>>>>>
>>>>>> The Mesos executor is being used (at least by some people) and there
>> is
>>>>> an open PR to extend it:
>>>>>> https://github.com/apache/airflow/pull/3739
>>>>>>
>>>>>>
>>>>>> Sent with ProtonMail Secure Email.
>>>>>>
>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>> On Friday, January 25, 2019 7:30 PM, Kaxil Naik <ka...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>>> What about Dask Executor as well.
>>>>>>>
>>>>>>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman <
>>>> daniel.imberman@gmail.com
>>>>>>> wrote:
>>>>>>>
>>>>>>>> @sid basically my response lol
>>>>>>>> I haven't seen any emails from any users and if it's been broken for
>>>>> two
>>>>>>>> versions then there doesn't seem to be enough interest to maintain
>>>> it.
>>>>>>>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39132@gmail.com wrote:
>>>>>>>>>
>>>>>>>>> LOL :_0
>>>>>>>>> -s
>>>>>>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin bdbruin@gmail.com
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I agree. Maybe we should make a choice on what we consider 1st
>>>> class
>>>>>>>>>> executors and move others to contrib (I'm looking at you celery)
>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>
>>>>>>>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor ash@apache.org
>> wrote:
>>>>>>>>>>> Is anyone using the Mesos Executor? I think we should deprecate
>>>> and
>>>>>>>>>>> remove it.
>>>>>>>>>>> The reason I say that is I have a feeling it's been broken since
>>>>>>>>>>> 1.9.0
>>>>>>>>>
>>>>>>>>>> and no-one has noticed:
>>>>>>>>>>
>>>>>>>>>>> class MesosExecutor(BaseExecutor, LoginMixin):
>>>>>>>>>>> def start(self):
>>>>>>>>>>> # ...
>>>>>>>>>>> self.log.info(
>>>>>>>>>>> 'MesosFramework master : %s, name : %s, cpu : %s, mem :
>>>>>>>>>>> %s,
>>>>>>>>>
>>>>>>>>>> checkpoint : %s',
>>>>>>>>>>
>>>>>>>>>>> master, framework.name,
>>>>>>>>>>> str(task_cpu), str(task_memory),
>>>> str(framework.checkpoint)
>>>>>>>>>>> )
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> That mixin should have been LoggingMixin, not LoginMixin. But
>>>> given
>>>>>>>>>>> this
>>>>>>>>>>> bug has been in since
>>>>>>>>
>>>>>>>>
>>>>>
>>>>
>> https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29
>>>>>>>>
>>>>>>>>>> which was in the 1.9.0 release I propose we simply remove this
>>>>> executor
>>>>>>>>>> from master.
>>>>>>>>>>
>>>>>>>>>>> Thoughts?
>>>>>>>>>>> Ash
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>
>>
Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Are we going to remove the MesosExecutor? I think it is a safe bet since in
the Survey of Ash (thanks again for that!), it also shows that nobody is
really using it.
https://jira.apache.org/jira/browse/AIRFLOW-3969
Cheers, Fokko
Op di 12 feb. 2019 om 10:28 schreef Ash Berlin-Taylor <as...@apache.org>:
> I’m glad yarn wasn’t the only option - it would have meant I’d have never
> been in a position to use Airflow! (Many of our workflows don’t touch
> EMR/Hadoop, and running Celery is much more of a known element to a python
> developer than configuring Yarn)
>
> > On 11 Feb 2019, at 17:14, Maxime Beauchemin <ma...@gmail.com>
> wrote:
> >
> > From memory, I think MesosExecutor depends on pickling to get DAG
> > definitions to workers, which we should also deprecate.
> >
> > About CeleryExecutor, we never had the intention to make it the
> recommended
> > option for production early on. The intent back in 2014 was to write a
> > YarnExecutor quickly (that was before the rise of k8s), but realized that
> > container support wasn't ready in Yarn at the time, and had hard
> > restrictions on what we could install on Hadoop/Yarn data nodes, so we
> > moved forward with Celery.
> >
> > I think it's practical to have LocalExecutor around though for unit test,
> > local use, and for POCs.
> >
> > Max
> >
> > On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot <
> > o.girardot@lateral-thoughts.com> wrote:
> >
> >> Hi, we tried to use it for a new project 1year ago, but it was too
> broken
> >> to be usable, and we didn't have the time to make it work.
> >> Of course we are still hoping to use it one day, but that doesn't mean
> it
> >> should not move to contrib, it was already clear from the documentation
> >> that it was not a first class citizen.
> >>
> >> Regards,
> >>
> >> Olivier.
> >>
> >> Le dim. 27 janv. 2019 à 12:12, Ash Berlin-Taylor <as...@apache.org> a
> écrit
> >> :
> >>
> >>> That PR pre-dates 1.9.0 being released.
> >>>
> >>>> On 27 Jan 2019, at 06:57, airflowuser <airflowuser@protonmail.com
> >> .INVALID>
> >>> wrote:
> >>>>
> >>>> The Mesos executor is being used (at least by some people) and there
> is
> >>> an open PR to extend it:
> >>>> https://github.com/apache/airflow/pull/3739
> >>>>
> >>>>
> >>>> Sent with ProtonMail Secure Email.
> >>>>
> >>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> >>>> On Friday, January 25, 2019 7:30 PM, Kaxil Naik <ka...@gmail.com>
> >>> wrote:
> >>>>
> >>>>> What about Dask Executor as well.
> >>>>>
> >>>>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman <
> >> daniel.imberman@gmail.com
> >>>>> wrote:
> >>>>>
> >>>>>> @sid basically my response lol
> >>>>>> I haven't seen any emails from any users and if it's been broken for
> >>> two
> >>>>>> versions then there doesn't seem to be enough interest to maintain
> >> it.
> >>>>>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39132@gmail.com wrote:
> >>>>>>>
> >>>>>>> LOL :_0
> >>>>>>> -s
> >>>>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin bdbruin@gmail.com
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> I agree. Maybe we should make a choice on what we consider 1st
> >> class
> >>>>>>>> executors and move others to contrib (I'm looking at you celery)
> >>>>>>>> Sent from my iPhone
> >>>>>>>>
> >>>>>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor ash@apache.org
> wrote:
> >>>>>>>>> Is anyone using the Mesos Executor? I think we should deprecate
> >> and
> >>>>>>>>> remove it.
> >>>>>>>>> The reason I say that is I have a feeling it's been broken since
> >>>>>>>>> 1.9.0
> >>>>>>>
> >>>>>>>> and no-one has noticed:
> >>>>>>>>
> >>>>>>>>> class MesosExecutor(BaseExecutor, LoginMixin):
> >>>>>>>>> def start(self):
> >>>>>>>>> # ...
> >>>>>>>>> self.log.info(
> >>>>>>>>> 'MesosFramework master : %s, name : %s, cpu : %s, mem :
> >>>>>>>>> %s,
> >>>>>>>
> >>>>>>>> checkpoint : %s',
> >>>>>>>>
> >>>>>>>>> master, framework.name,
> >>>>>>>>> str(task_cpu), str(task_memory),
> >> str(framework.checkpoint)
> >>>>>>>>> )
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> That mixin should have been LoggingMixin, not LoginMixin. But
> >> given
> >>>>>>>>> this
> >>>>>>>>> bug has been in since
> >>>>>>
> >>>>>>
> >>>
> >>
> https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29
> >>>>>>
> >>>>>>>> which was in the 1.9.0 release I propose we simply remove this
> >>> executor
> >>>>>>>> from master.
> >>>>>>>>
> >>>>>>>>> Thoughts?
> >>>>>>>>> Ash
> >>>>
> >>>>
> >>>
> >>>
> >>
>
>
Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Posted by Ash Berlin-Taylor <as...@apache.org>.
I’m glad yarn wasn’t the only option - it would have meant I’d have never been in a position to use Airflow! (Many of our workflows don’t touch EMR/Hadoop, and running Celery is much more of a known element to a python developer than configuring Yarn)
> On 11 Feb 2019, at 17:14, Maxime Beauchemin <ma...@gmail.com> wrote:
>
> From memory, I think MesosExecutor depends on pickling to get DAG
> definitions to workers, which we should also deprecate.
>
> About CeleryExecutor, we never had the intention to make it the recommended
> option for production early on. The intent back in 2014 was to write a
> YarnExecutor quickly (that was before the rise of k8s), but realized that
> container support wasn't ready in Yarn at the time, and had hard
> restrictions on what we could install on Hadoop/Yarn data nodes, so we
> moved forward with Celery.
>
> I think it's practical to have LocalExecutor around though for unit test,
> local use, and for POCs.
>
> Max
>
> On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot <
> o.girardot@lateral-thoughts.com> wrote:
>
>> Hi, we tried to use it for a new project 1year ago, but it was too broken
>> to be usable, and we didn't have the time to make it work.
>> Of course we are still hoping to use it one day, but that doesn't mean it
>> should not move to contrib, it was already clear from the documentation
>> that it was not a first class citizen.
>>
>> Regards,
>>
>> Olivier.
>>
>> Le dim. 27 janv. 2019 à 12:12, Ash Berlin-Taylor <as...@apache.org> a écrit
>> :
>>
>>> That PR pre-dates 1.9.0 being released.
>>>
>>>> On 27 Jan 2019, at 06:57, airflowuser <airflowuser@protonmail.com
>> .INVALID>
>>> wrote:
>>>>
>>>> The Mesos executor is being used (at least by some people) and there is
>>> an open PR to extend it:
>>>> https://github.com/apache/airflow/pull/3739
>>>>
>>>>
>>>> Sent with ProtonMail Secure Email.
>>>>
>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>> On Friday, January 25, 2019 7:30 PM, Kaxil Naik <ka...@gmail.com>
>>> wrote:
>>>>
>>>>> What about Dask Executor as well.
>>>>>
>>>>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman <
>> daniel.imberman@gmail.com
>>>>> wrote:
>>>>>
>>>>>> @sid basically my response lol
>>>>>> I haven't seen any emails from any users and if it's been broken for
>>> two
>>>>>> versions then there doesn't seem to be enough interest to maintain
>> it.
>>>>>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39132@gmail.com wrote:
>>>>>>>
>>>>>>> LOL :_0
>>>>>>> -s
>>>>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin bdbruin@gmail.com
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I agree. Maybe we should make a choice on what we consider 1st
>> class
>>>>>>>> executors and move others to contrib (I'm looking at you celery)
>>>>>>>> Sent from my iPhone
>>>>>>>>
>>>>>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor ash@apache.org wrote:
>>>>>>>>> Is anyone using the Mesos Executor? I think we should deprecate
>> and
>>>>>>>>> remove it.
>>>>>>>>> The reason I say that is I have a feeling it's been broken since
>>>>>>>>> 1.9.0
>>>>>>>
>>>>>>>> and no-one has noticed:
>>>>>>>>
>>>>>>>>> class MesosExecutor(BaseExecutor, LoginMixin):
>>>>>>>>> def start(self):
>>>>>>>>> # ...
>>>>>>>>> self.log.info(
>>>>>>>>> 'MesosFramework master : %s, name : %s, cpu : %s, mem :
>>>>>>>>> %s,
>>>>>>>
>>>>>>>> checkpoint : %s',
>>>>>>>>
>>>>>>>>> master, framework.name,
>>>>>>>>> str(task_cpu), str(task_memory),
>> str(framework.checkpoint)
>>>>>>>>> )
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> That mixin should have been LoggingMixin, not LoginMixin. But
>> given
>>>>>>>>> this
>>>>>>>>> bug has been in since
>>>>>>
>>>>>>
>>>
>> https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29
>>>>>>
>>>>>>>> which was in the 1.9.0 release I propose we simply remove this
>>> executor
>>>>>>>> from master.
>>>>>>>>
>>>>>>>>> Thoughts?
>>>>>>>>> Ash
>>>>
>>>>
>>>
>>>
>>
Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Posted by Maxime Beauchemin <ma...@gmail.com>.
From memory, I think MesosExecutor depends on pickling to get DAG
definitions to workers, which we should also deprecate.
About CeleryExecutor, we never had the intention to make it the recommended
option for production early on. The intent back in 2014 was to write a
YarnExecutor quickly (that was before the rise of k8s), but realized that
container support wasn't ready in Yarn at the time, and had hard
restrictions on what we could install on Hadoop/Yarn data nodes, so we
moved forward with Celery.
I think it's practical to have LocalExecutor around though for unit test,
local use, and for POCs.
Max
On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot <
o.girardot@lateral-thoughts.com> wrote:
> Hi, we tried to use it for a new project 1year ago, but it was too broken
> to be usable, and we didn't have the time to make it work.
> Of course we are still hoping to use it one day, but that doesn't mean it
> should not move to contrib, it was already clear from the documentation
> that it was not a first class citizen.
>
> Regards,
>
> Olivier.
>
> Le dim. 27 janv. 2019 à 12:12, Ash Berlin-Taylor <as...@apache.org> a écrit
> :
>
> > That PR pre-dates 1.9.0 being released.
> >
> > > On 27 Jan 2019, at 06:57, airflowuser <airflowuser@protonmail.com
> .INVALID>
> > wrote:
> > >
> > > The Mesos executor is being used (at least by some people) and there is
> > an open PR to extend it:
> > > https://github.com/apache/airflow/pull/3739
> > >
> > >
> > > Sent with ProtonMail Secure Email.
> > >
> > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > On Friday, January 25, 2019 7:30 PM, Kaxil Naik <ka...@gmail.com>
> > wrote:
> > >
> > >> What about Dask Executor as well.
> > >>
> > >> On Fri, Jan 25, 2019, 16:35 Daniel Imberman <
> daniel.imberman@gmail.com
> > >> wrote:
> > >>
> > >>> @sid basically my response lol
> > >>> I haven't seen any emails from any users and if it's been broken for
> > two
> > >>> versions then there doesn't seem to be enough interest to maintain
> it.
> > >>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39132@gmail.com wrote:
> > >>>
> > >>>> LOL :_0
> > >>>> -s
> > >>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin bdbruin@gmail.com
> > >>>> wrote:
> > >>>>
> > >>>>> I agree. Maybe we should make a choice on what we consider 1st
> class
> > >>>>> executors and move others to contrib (I'm looking at you celery)
> > >>>>> Sent from my iPhone
> > >>>>>
> > >>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor ash@apache.org wrote:
> > >>>>>> Is anyone using the Mesos Executor? I think we should deprecate
> and
> > >>>>>> remove it.
> > >>>>>> The reason I say that is I have a feeling it's been broken since
> > >>>>>> 1.9.0
> > >>>>
> > >>>>> and no-one has noticed:
> > >>>>>
> > >>>>>> class MesosExecutor(BaseExecutor, LoginMixin):
> > >>>>>> def start(self):
> > >>>>>> # ...
> > >>>>>> self.log.info(
> > >>>>>> 'MesosFramework master : %s, name : %s, cpu : %s, mem :
> > >>>>>> %s,
> > >>>>
> > >>>>> checkpoint : %s',
> > >>>>>
> > >>>>>> master, framework.name,
> > >>>>>> str(task_cpu), str(task_memory),
> str(framework.checkpoint)
> > >>>>>> )
> > >>>>>>
> > >>>>>>
> > >>>>>> That mixin should have been LoggingMixin, not LoginMixin. But
> given
> > >>>>>> this
> > >>>>>> bug has been in since
> > >>>
> > >>>
> >
> https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29
> > >>>
> > >>>>> which was in the 1.9.0 release I propose we simply remove this
> > executor
> > >>>>> from master.
> > >>>>>
> > >>>>>> Thoughts?
> > >>>>>> Ash
> > >
> > >
> >
> >
>
Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Posted by Olivier Girardot <o....@lateral-thoughts.com>.
Hi, we tried to use it for a new project 1year ago, but it was too broken
to be usable, and we didn't have the time to make it work.
Of course we are still hoping to use it one day, but that doesn't mean it
should not move to contrib, it was already clear from the documentation
that it was not a first class citizen.
Regards,
Olivier.
Le dim. 27 janv. 2019 à 12:12, Ash Berlin-Taylor <as...@apache.org> a écrit :
> That PR pre-dates 1.9.0 being released.
>
> > On 27 Jan 2019, at 06:57, airflowuser <ai...@protonmail.com.INVALID>
> wrote:
> >
> > The Mesos executor is being used (at least by some people) and there is
> an open PR to extend it:
> > https://github.com/apache/airflow/pull/3739
> >
> >
> > Sent with ProtonMail Secure Email.
> >
> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > On Friday, January 25, 2019 7:30 PM, Kaxil Naik <ka...@gmail.com>
> wrote:
> >
> >> What about Dask Executor as well.
> >>
> >> On Fri, Jan 25, 2019, 16:35 Daniel Imberman <daniel.imberman@gmail.com
> >> wrote:
> >>
> >>> @sid basically my response lol
> >>> I haven't seen any emails from any users and if it's been broken for
> two
> >>> versions then there doesn't seem to be enough interest to maintain it.
> >>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39132@gmail.com wrote:
> >>>
> >>>> LOL :_0
> >>>> -s
> >>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin bdbruin@gmail.com
> >>>> wrote:
> >>>>
> >>>>> I agree. Maybe we should make a choice on what we consider 1st class
> >>>>> executors and move others to contrib (I'm looking at you celery)
> >>>>> Sent from my iPhone
> >>>>>
> >>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor ash@apache.org wrote:
> >>>>>> Is anyone using the Mesos Executor? I think we should deprecate and
> >>>>>> remove it.
> >>>>>> The reason I say that is I have a feeling it's been broken since
> >>>>>> 1.9.0
> >>>>
> >>>>> and no-one has noticed:
> >>>>>
> >>>>>> class MesosExecutor(BaseExecutor, LoginMixin):
> >>>>>> def start(self):
> >>>>>> # ...
> >>>>>> self.log.info(
> >>>>>> 'MesosFramework master : %s, name : %s, cpu : %s, mem :
> >>>>>> %s,
> >>>>
> >>>>> checkpoint : %s',
> >>>>>
> >>>>>> master, framework.name,
> >>>>>> str(task_cpu), str(task_memory), str(framework.checkpoint)
> >>>>>> )
> >>>>>>
> >>>>>>
> >>>>>> That mixin should have been LoggingMixin, not LoginMixin. But given
> >>>>>> this
> >>>>>> bug has been in since
> >>>
> >>>
> https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29
> >>>
> >>>>> which was in the 1.9.0 release I propose we simply remove this
> executor
> >>>>> from master.
> >>>>>
> >>>>>> Thoughts?
> >>>>>> Ash
> >
> >
>
>
Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Posted by Ash Berlin-Taylor <as...@apache.org>.
That PR pre-dates 1.9.0 being released.
> On 27 Jan 2019, at 06:57, airflowuser <ai...@protonmail.com.INVALID> wrote:
>
> The Mesos executor is being used (at least by some people) and there is an open PR to extend it:
> https://github.com/apache/airflow/pull/3739
>
>
> Sent with ProtonMail Secure Email.
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Friday, January 25, 2019 7:30 PM, Kaxil Naik <ka...@gmail.com> wrote:
>
>> What about Dask Executor as well.
>>
>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman <daniel.imberman@gmail.com
>> wrote:
>>
>>> @sid basically my response lol
>>> I haven't seen any emails from any users and if it's been broken for two
>>> versions then there doesn't seem to be enough interest to maintain it.
>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39132@gmail.com wrote:
>>>
>>>> LOL :_0
>>>> -s
>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin bdbruin@gmail.com
>>>> wrote:
>>>>
>>>>> I agree. Maybe we should make a choice on what we consider 1st class
>>>>> executors and move others to contrib (I'm looking at you celery)
>>>>> Sent from my iPhone
>>>>>
>>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor ash@apache.org wrote:
>>>>>> Is anyone using the Mesos Executor? I think we should deprecate and
>>>>>> remove it.
>>>>>> The reason I say that is I have a feeling it's been broken since
>>>>>> 1.9.0
>>>>
>>>>> and no-one has noticed:
>>>>>
>>>>>> class MesosExecutor(BaseExecutor, LoginMixin):
>>>>>> def start(self):
>>>>>> # ...
>>>>>> self.log.info(
>>>>>> 'MesosFramework master : %s, name : %s, cpu : %s, mem :
>>>>>> %s,
>>>>
>>>>> checkpoint : %s',
>>>>>
>>>>>> master, framework.name,
>>>>>> str(task_cpu), str(task_memory), str(framework.checkpoint)
>>>>>> )
>>>>>>
>>>>>>
>>>>>> That mixin should have been LoggingMixin, not LoginMixin. But given
>>>>>> this
>>>>>> bug has been in since
>>>
>>> https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29
>>>
>>>>> which was in the 1.9.0 release I propose we simply remove this executor
>>>>> from master.
>>>>>
>>>>>> Thoughts?
>>>>>> Ash
>
>
Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Posted by airflowuser <ai...@protonmail.com.INVALID>.
The Mesos executor is being used (at least by some people) and there is an open PR to extend it:
https://github.com/apache/airflow/pull/3739
Sent with ProtonMail Secure Email.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, January 25, 2019 7:30 PM, Kaxil Naik <ka...@gmail.com> wrote:
> What about Dask Executor as well.
>
> On Fri, Jan 25, 2019, 16:35 Daniel Imberman <daniel.imberman@gmail.com
> wrote:
>
> > @sid basically my response lol
> > I haven't seen any emails from any users and if it's been broken for two
> > versions then there doesn't seem to be enough interest to maintain it.
> > On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39132@gmail.com wrote:
> >
> > > LOL :_0
> > > -s
> > > On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin bdbruin@gmail.com
> > > wrote:
> > >
> > > > I agree. Maybe we should make a choice on what we consider 1st class
> > > > executors and move others to contrib (I'm looking at you celery)
> > > > Sent from my iPhone
> > > >
> > > > > On 25 Jan 2019, at 12:46, Ash Berlin-Taylor ash@apache.org wrote:
> > > > > Is anyone using the Mesos Executor? I think we should deprecate and
> > > > > remove it.
> > > > > The reason I say that is I have a feeling it's been broken since
> > > > > 1.9.0
> > >
> > > > and no-one has noticed:
> > > >
> > > > > class MesosExecutor(BaseExecutor, LoginMixin):
> > > > > def start(self):
> > > > > # ...
> > > > > self.log.info(
> > > > > 'MesosFramework master : %s, name : %s, cpu : %s, mem :
> > > > > %s,
> > >
> > > > checkpoint : %s',
> > > >
> > > > > master, framework.name,
> > > > > str(task_cpu), str(task_memory), str(framework.checkpoint)
> > > > > )
> > > > >
> > > > >
> > > > > That mixin should have been LoggingMixin, not LoginMixin. But given
> > > > > this
> > > > > bug has been in since
> >
> > https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29
> >
> > > > which was in the 1.9.0 release I propose we simply remove this executor
> > > > from master.
> > > >
> > > > > Thoughts?
> > > > > Ash
Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Posted by Kaxil Naik <ka...@gmail.com>.
What about Dask Executor as well.
On Fri, Jan 25, 2019, 16:35 Daniel Imberman <daniel.imberman@gmail.com
wrote:
> @sid basically my response lol
>
> I haven't seen any emails from any users and if it's been broken for two
> versions then there doesn't seem to be enough interest to maintain it.
>
> On Fri, Jan 25, 2019, 8:27 AM Sid Anand <r3...@gmail.com> wrote:
>
> > LOL :_0
> > -s
> >
> > On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin <bd...@gmail.com>
> wrote:
> >
> > > I agree. Maybe we should make a choice on what we consider 1st class
> > > executors and move others to contrib (I'm looking at you celery)
> > >
> > > Sent from my iPhone
> > >
> > > > On 25 Jan 2019, at 12:46, Ash Berlin-Taylor <as...@apache.org> wrote:
> > > >
> > > > Is anyone using the Mesos Executor? I think we should deprecate and
> > > remove it.
> > > >
> > > > The reason I say that is I have a feeling it's been broken since
> 1.9.0
> > > and no-one has noticed:
> > > >
> > > > class MesosExecutor(BaseExecutor, LoginMixin):
> > > >
> > > > def start(self):
> > > > # ...
> > > > self.log.info(
> > > > 'MesosFramework master : %s, name : %s, cpu : %s, mem :
> %s,
> > > checkpoint : %s',
> > > > master, framework.name,
> > > > str(task_cpu), str(task_memory), str(framework.checkpoint)
> > > > )
> > > >
> > > >
> > > > That mixin should have been LoggingMixin, not LoginMixin. But given
> > this
> > > bug has been in since
> > >
> >
> https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29
> > > which was in the 1.9.0 release I propose we simply remove this executor
> > > from master.
> > > >
> > > > Thoughts?
> > > >
> > > > Ash
> > >
> >
>
Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Posted by Daniel Imberman <da...@gmail.com>.
@sid basically my response lol
I haven't seen any emails from any users and if it's been broken for two
versions then there doesn't seem to be enough interest to maintain it.
On Fri, Jan 25, 2019, 8:27 AM Sid Anand <r3...@gmail.com> wrote:
> LOL :_0
> -s
>
> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin <bd...@gmail.com> wrote:
>
> > I agree. Maybe we should make a choice on what we consider 1st class
> > executors and move others to contrib (I'm looking at you celery)
> >
> > Sent from my iPhone
> >
> > > On 25 Jan 2019, at 12:46, Ash Berlin-Taylor <as...@apache.org> wrote:
> > >
> > > Is anyone using the Mesos Executor? I think we should deprecate and
> > remove it.
> > >
> > > The reason I say that is I have a feeling it's been broken since 1.9.0
> > and no-one has noticed:
> > >
> > > class MesosExecutor(BaseExecutor, LoginMixin):
> > >
> > > def start(self):
> > > # ...
> > > self.log.info(
> > > 'MesosFramework master : %s, name : %s, cpu : %s, mem : %s,
> > checkpoint : %s',
> > > master, framework.name,
> > > str(task_cpu), str(task_memory), str(framework.checkpoint)
> > > )
> > >
> > >
> > > That mixin should have been LoggingMixin, not LoginMixin. But given
> this
> > bug has been in since
> >
> https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29
> > which was in the 1.9.0 release I propose we simply remove this executor
> > from master.
> > >
> > > Thoughts?
> > >
> > > Ash
> >
>
Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Posted by Sid Anand <r3...@gmail.com>.
LOL :_0
-s
On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin <bd...@gmail.com> wrote:
> I agree. Maybe we should make a choice on what we consider 1st class
> executors and move others to contrib (I'm looking at you celery)
>
> Sent from my iPhone
>
> > On 25 Jan 2019, at 12:46, Ash Berlin-Taylor <as...@apache.org> wrote:
> >
> > Is anyone using the Mesos Executor? I think we should deprecate and
> remove it.
> >
> > The reason I say that is I have a feeling it's been broken since 1.9.0
> and no-one has noticed:
> >
> > class MesosExecutor(BaseExecutor, LoginMixin):
> >
> > def start(self):
> > # ...
> > self.log.info(
> > 'MesosFramework master : %s, name : %s, cpu : %s, mem : %s,
> checkpoint : %s',
> > master, framework.name,
> > str(task_cpu), str(task_memory), str(framework.checkpoint)
> > )
> >
> >
> > That mixin should have been LoggingMixin, not LoginMixin. But given this
> bug has been in since
> https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29
> which was in the 1.9.0 release I propose we simply remove this executor
> from master.
> >
> > Thoughts?
> >
> > Ash
>
Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Posted by Bolke de Bruin <bd...@gmail.com>.
I agree. Maybe we should make a choice on what we consider 1st class executors and move others to contrib (I'm looking at you celery)
Sent from my iPhone
> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor <as...@apache.org> wrote:
>
> Is anyone using the Mesos Executor? I think we should deprecate and remove it.
>
> The reason I say that is I have a feeling it's been broken since 1.9.0 and no-one has noticed:
>
> class MesosExecutor(BaseExecutor, LoginMixin):
>
> def start(self):
> # ...
> self.log.info(
> 'MesosFramework master : %s, name : %s, cpu : %s, mem : %s, checkpoint : %s',
> master, framework.name,
> str(task_cpu), str(task_memory), str(framework.checkpoint)
> )
>
>
> That mixin should have been LoggingMixin, not LoginMixin. But given this bug has been in since https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29 which was in the 1.9.0 release I propose we simply remove this executor from master.
>
> Thoughts?
>
> Ash