You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Ryan Hatter <ry...@gmail.com> on 2021/03/22 00:32:34 UTC
[DISCUSS] Add Breeze Support for CeleryExecutor and KubernetesExecutor
I recently had some trouble trying to fix a bug in the CeleryExecutor
<https://github.com/apache/airflow/pull/14883>. The code change was small,
but it was really difficult to set up a development environment using the
CeleryExecutor. I ultimately had to muck around with the test case that
covers this situation, default_airflow.cfg, and default_celery.py.
Developing using Docker
<https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html>
would require an update to the image each time you want to make a change to
the codebase (or maybe `exec`ing into the relevant container?) which is a
pain.
This led me to two questions:
1. Would it make sense to allow developers to choose the executor for
their Breeze environment?
2. If not, how do folks test out changes they make to the CeleryExecutor
or KubernetesExecutor?
Thanks!
Ryan
Re: [DISCUSS] Add Breeze Support for CeleryExecutor and KubernetesExecutor
Posted by Jarek Potiuk <ja...@potiuk.com>.
OH and BTW - this is the very same setup that is used on CI to run the k8s
tests ;)
On Tue, Mar 23, 2021 at 12:07 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> Oh, I did not write about K8S Executor because we already have K8S
> executor support in Breeze. It does not use Breeze docker image - it uses
> KinD (Kubernetes-in-Docker) that has all the bells and whistles (and more).
>
> Here is the part of Breeze video when I showcase it:
>
> https://github.com/apache/airflow/blob/master/BREEZE.rst#running-kubernetes-tests
>
> And here the whole chapter describing how to test Airflow with K8S:
>
> https://github.com/apache/airflow/blob/master/TESTING.rst#running-tests-with-kubernetes
>
> What we have built-in with Breeze (automagically):
>
> * configures and start kind cluster (kind is a really nice
> development-focus installation of K8S that uses K8S-in-Docker approach).
> * deploys airflow with K8S executor
> * createss a k8s virtualenv which installs all necessary dependencies that
> you can run tests against and allows you to enter it with `shell` command
> * it even has nice k9s integration (which is a fantastic tool for
> interacting with k8s)
> * and it even has a step-by-step screenshot guide on how to debug the K8s
> tests with IntelliJ.
>
> So I believe K8S executor is well covered already :)
>
> J.
>
>
> On Mon, Mar 22, 2021 at 11:08 PM Ryan Hatter <ry...@gmail.com>
> wrote:
>
>> Yeah I'll give that a shot.
>>
>> Do you think something with KubernetesExecutor with something like
>> MiniKube would also make sense?
>>
>> On Mon, Mar 22, 2021 at 5:51 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> Actually it is not that strightforward, but possibly we can make it
>>> works much more easily
>>>
>>> In order to make Celery Executor works you need to do a bit more (but
>>> should be easy to add as an option to Breeze):
>>>
>>> * you need to start rabitmq or redis as integration (`--integration
>>> rabbitq --integration redis')
>>> * you need to start worker(s) (`airflow worker` in the background)
>>> * you might want to start flower optionally (the celery monitoring tool)
>>>
>>> So maybe we could add extra switch to start-airflow command
>>> (--use-celery-executor) that could set those integrations and start
>>> worker/flower additionally to running webserver/scheduler now in tmux ?
>>>
>>> WDYT? Maybe Ryan you can check if my recipe works ? I could add it then
>>> as an option.
>>>
>>> BTW. We already have a number of CeleryExecutor tests that use the
>>> integrations, so Breeze has all what's needed:
>>>
>>>
>>> https://github.com/apache/airflow/blob/master/tests/executors/test_celery_executor.py#L109
>>>
>>> J.
>>>
>>> On Mon, Mar 22, 2021 at 2:24 PM Ryan Hatter <ry...@gmail.com>
>>> wrote:
>>>
>>>> Hmm, maybe I was just getting twisted around with docker then. I’ll
>>>> have a look at what you shared.
>>>>
>>>> Thanks Bin :)
>>>>
>>>> On Mar 22, 2021, at 01:52, Xinbin Huang <bi...@gmail.com> wrote:
>>>>
>>>>
>>>> Hi Ryan,
>>>>
>>>> I believe breeze already provides tools for you to do that:
>>>>
>>>> 1.Would it make sense to allow developers to choose the executor for
>>>> their Breeze environment?
>>>>
>>>> You can set up environment variables and any other custom setup you
>>>> want in the file: */files/airflow-breeze-config/variables.env. *To set
>>>> up CeleryExecutor, you just need to put `export
>>>> AIRFLOW__CORE__EXECUTOR=CeleryExecutor` in the file.
>>>>
>>>> 2. Developing using Docker
>>>> <https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html> would
>>>> require an update to the image each time you want to make a change to the
>>>> codebase
>>>>
>>>> Breeze automatically mounts the local source to the container unless
>>>> you explicitly skip it with the flag *--skip-mounting-local-sources. *You
>>>> can find more details here
>>>> https://github.com/apache/airflow/blob/master/BREEZE.rst#mounting-local-sources-to-breeze
>>>>
>>>> Best
>>>> Bin
>>>>
>>>>
>>>> On Sun, Mar 21, 2021 at 5:32 PM Ryan Hatter <ry...@gmail.com>
>>>> wrote:
>>>>
>>>>> I recently had some trouble trying to fix a bug in the CeleryExecutor
>>>>> <https://github.com/apache/airflow/pull/14883>. The code change was
>>>>> small, but it was really difficult to set up a development environment
>>>>> using the CeleryExecutor. I ultimately had to muck around with the test
>>>>> case that covers this situation, default_airflow.cfg, and
>>>>> default_celery.py. Developing using Docker
>>>>> <https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html>
>>>>> would require an update to the image each time you want to make a change to
>>>>> the codebase (or maybe `exec`ing into the relevant container?) which is a
>>>>> pain.
>>>>>
>>>>> This led me to two questions:
>>>>>
>>>>> 1. Would it make sense to allow developers to choose the executor
>>>>> for their Breeze environment?
>>>>> 2. If not, how do folks test out changes they make to the
>>>>> CeleryExecutor or KubernetesExecutor?
>>>>>
>>>>> Thanks!
>>>>> Ryan
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>> +48 660 796 129
>>>
>>
>
> --
> +48 660 796 129
>
--
+48 660 796 129
Re: [DISCUSS] Add Breeze Support for CeleryExecutor and KubernetesExecutor
Posted by Jarek Potiuk <ja...@potiuk.com>.
Oh, I did not write about K8S Executor because we already have K8S executor
support in Breeze. It does not use Breeze docker image - it uses KinD
(Kubernetes-in-Docker) that has all the bells and whistles (and more).
Here is the part of Breeze video when I showcase it:
https://github.com/apache/airflow/blob/master/BREEZE.rst#running-kubernetes-tests
And here the whole chapter describing how to test Airflow with K8S:
https://github.com/apache/airflow/blob/master/TESTING.rst#running-tests-with-kubernetes
What we have built-in with Breeze (automagically):
* configures and start kind cluster (kind is a really nice
development-focus installation of K8S that uses K8S-in-Docker approach).
* deploys airflow with K8S executor
* createss a k8s virtualenv which installs all necessary dependencies that
you can run tests against and allows you to enter it with `shell` command
* it even has nice k9s integration (which is a fantastic tool for
interacting with k8s)
* and it even has a step-by-step screenshot guide on how to debug the K8s
tests with IntelliJ.
So I believe K8S executor is well covered already :)
J.
On Mon, Mar 22, 2021 at 11:08 PM Ryan Hatter <ry...@gmail.com> wrote:
> Yeah I'll give that a shot.
>
> Do you think something with KubernetesExecutor with something like
> MiniKube would also make sense?
>
> On Mon, Mar 22, 2021 at 5:51 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Actually it is not that strightforward, but possibly we can make it works
>> much more easily
>>
>> In order to make Celery Executor works you need to do a bit more (but
>> should be easy to add as an option to Breeze):
>>
>> * you need to start rabitmq or redis as integration (`--integration
>> rabbitq --integration redis')
>> * you need to start worker(s) (`airflow worker` in the background)
>> * you might want to start flower optionally (the celery monitoring tool)
>>
>> So maybe we could add extra switch to start-airflow command
>> (--use-celery-executor) that could set those integrations and start
>> worker/flower additionally to running webserver/scheduler now in tmux ?
>>
>> WDYT? Maybe Ryan you can check if my recipe works ? I could add it then
>> as an option.
>>
>> BTW. We already have a number of CeleryExecutor tests that use the
>> integrations, so Breeze has all what's needed:
>>
>>
>> https://github.com/apache/airflow/blob/master/tests/executors/test_celery_executor.py#L109
>>
>> J.
>>
>> On Mon, Mar 22, 2021 at 2:24 PM Ryan Hatter <ry...@gmail.com>
>> wrote:
>>
>>> Hmm, maybe I was just getting twisted around with docker then. I’ll have
>>> a look at what you shared.
>>>
>>> Thanks Bin :)
>>>
>>> On Mar 22, 2021, at 01:52, Xinbin Huang <bi...@gmail.com> wrote:
>>>
>>>
>>> Hi Ryan,
>>>
>>> I believe breeze already provides tools for you to do that:
>>>
>>> 1.Would it make sense to allow developers to choose the executor for
>>> their Breeze environment?
>>>
>>> You can set up environment variables and any other custom setup you want
>>> in the file: */files/airflow-breeze-config/variables.env. *To set up
>>> CeleryExecutor, you just need to put `export
>>> AIRFLOW__CORE__EXECUTOR=CeleryExecutor` in the file.
>>>
>>> 2. Developing using Docker
>>> <https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html> would
>>> require an update to the image each time you want to make a change to the
>>> codebase
>>>
>>> Breeze automatically mounts the local source to the container unless you
>>> explicitly skip it with the flag *--skip-mounting-local-sources. *You
>>> can find more details here
>>> https://github.com/apache/airflow/blob/master/BREEZE.rst#mounting-local-sources-to-breeze
>>>
>>> Best
>>> Bin
>>>
>>>
>>> On Sun, Mar 21, 2021 at 5:32 PM Ryan Hatter <ry...@gmail.com>
>>> wrote:
>>>
>>>> I recently had some trouble trying to fix a bug in the CeleryExecutor
>>>> <https://github.com/apache/airflow/pull/14883>. The code change was
>>>> small, but it was really difficult to set up a development environment
>>>> using the CeleryExecutor. I ultimately had to muck around with the test
>>>> case that covers this situation, default_airflow.cfg, and
>>>> default_celery.py. Developing using Docker
>>>> <https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html>
>>>> would require an update to the image each time you want to make a change to
>>>> the codebase (or maybe `exec`ing into the relevant container?) which is a
>>>> pain.
>>>>
>>>> This led me to two questions:
>>>>
>>>> 1. Would it make sense to allow developers to choose the executor
>>>> for their Breeze environment?
>>>> 2. If not, how do folks test out changes they make to the
>>>> CeleryExecutor or KubernetesExecutor?
>>>>
>>>> Thanks!
>>>> Ryan
>>>>
>>>>
>>>>
>>
>> --
>> +48 660 796 129
>>
>
--
+48 660 796 129
Re: [DISCUSS] Add Breeze Support for CeleryExecutor and KubernetesExecutor
Posted by Ryan Hatter <ry...@gmail.com>.
Yeah I'll give that a shot.
Do you think something with KubernetesExecutor with something like MiniKube
would also make sense?
On Mon, Mar 22, 2021 at 5:51 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Actually it is not that strightforward, but possibly we can make it works
> much more easily
>
> In order to make Celery Executor works you need to do a bit more (but
> should be easy to add as an option to Breeze):
>
> * you need to start rabitmq or redis as integration (`--integration
> rabbitq --integration redis')
> * you need to start worker(s) (`airflow worker` in the background)
> * you might want to start flower optionally (the celery monitoring tool)
>
> So maybe we could add extra switch to start-airflow command
> (--use-celery-executor) that could set those integrations and start
> worker/flower additionally to running webserver/scheduler now in tmux ?
>
> WDYT? Maybe Ryan you can check if my recipe works ? I could add it then as
> an option.
>
> BTW. We already have a number of CeleryExecutor tests that use the
> integrations, so Breeze has all what's needed:
>
>
> https://github.com/apache/airflow/blob/master/tests/executors/test_celery_executor.py#L109
>
> J.
>
> On Mon, Mar 22, 2021 at 2:24 PM Ryan Hatter <ry...@gmail.com> wrote:
>
>> Hmm, maybe I was just getting twisted around with docker then. I’ll have
>> a look at what you shared.
>>
>> Thanks Bin :)
>>
>> On Mar 22, 2021, at 01:52, Xinbin Huang <bi...@gmail.com> wrote:
>>
>>
>> Hi Ryan,
>>
>> I believe breeze already provides tools for you to do that:
>>
>> 1.Would it make sense to allow developers to choose the executor for
>> their Breeze environment?
>>
>> You can set up environment variables and any other custom setup you want
>> in the file: */files/airflow-breeze-config/variables.env. *To set up
>> CeleryExecutor, you just need to put `export
>> AIRFLOW__CORE__EXECUTOR=CeleryExecutor` in the file.
>>
>> 2. Developing using Docker
>> <https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html> would
>> require an update to the image each time you want to make a change to the
>> codebase
>>
>> Breeze automatically mounts the local source to the container unless you
>> explicitly skip it with the flag *--skip-mounting-local-sources. *You
>> can find more details here
>> https://github.com/apache/airflow/blob/master/BREEZE.rst#mounting-local-sources-to-breeze
>>
>> Best
>> Bin
>>
>>
>> On Sun, Mar 21, 2021 at 5:32 PM Ryan Hatter <ry...@gmail.com>
>> wrote:
>>
>>> I recently had some trouble trying to fix a bug in the CeleryExecutor
>>> <https://github.com/apache/airflow/pull/14883>. The code change was
>>> small, but it was really difficult to set up a development environment
>>> using the CeleryExecutor. I ultimately had to muck around with the test
>>> case that covers this situation, default_airflow.cfg, and
>>> default_celery.py. Developing using Docker
>>> <https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html>
>>> would require an update to the image each time you want to make a change to
>>> the codebase (or maybe `exec`ing into the relevant container?) which is a
>>> pain.
>>>
>>> This led me to two questions:
>>>
>>> 1. Would it make sense to allow developers to choose the executor
>>> for their Breeze environment?
>>> 2. If not, how do folks test out changes they make to the
>>> CeleryExecutor or KubernetesExecutor?
>>>
>>> Thanks!
>>> Ryan
>>>
>>>
>>>
>
> --
> +48 660 796 129
>
Re: [DISCUSS] Add Breeze Support for CeleryExecutor and KubernetesExecutor
Posted by Jarek Potiuk <ja...@potiuk.com>.
> Would I be crazy to ask if Breeze should be rewritten in Python?
Arguments in support, I think, might be...
No. not crazy. There is even an issue for that
https://github.com/apache/airflow/issues/12282
We just havn't gotten to that - yet.
J
.
On Sat, Mar 27, 2021 at 11:30 PM Ryan Hatter <ry...@gmail.com> wrote:
> This seems to work, with a tiny caveat: `airflow celery worker` is the
> command to start the celery worker instead of `airflow worker`:
>
> 1. Launch breeze with `--integration rabbitq` or `--integration redis`
> 2. Set the executor to celery with `export
> AIRFLOW__CORE__EXECUTOR=CeleryExecutor`
> 3. Start a celery worker with `airflow celery worker -D`
>
> Would I be crazy to ask if Breeze should be rewritten in Python? Arguments
> in support, I think, might be...
>
> 1. Easier modularization/extensibility
> 2. I'd guess that more people are comfortable in Python than Bash
> 3. At least one of the macOS issues could be resolved
>
> Thoughts?
>
>
> On Mon, Mar 22, 2021 at 5:51 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Actually it is not that strightforward, but possibly we can make it works
>> much more easily
>>
>> In order to make Celery Executor works you need to do a bit more (but
>> should be easy to add as an option to Breeze):
>>
>> * you need to start rabitmq or redis as integration (`--integration
>> rabbitq --integration redis')
>> * you need to start worker(s) (`airflow worker` in the background)
>> * you might want to start flower optionally (the celery monitoring tool)
>>
>> So maybe we could add extra switch to start-airflow command
>> (--use-celery-executor) that could set those integrations and start
>> worker/flower additionally to running webserver/scheduler now in tmux ?
>>
>> WDYT? Maybe Ryan you can check if my recipe works ? I could add it then
>> as an option.
>>
>> BTW. We already have a number of CeleryExecutor tests that use the
>> integrations, so Breeze has all what's needed:
>>
>>
>> https://github.com/apache/airflow/blob/master/tests/executors/test_celery_executor.py#L109
>>
>> J.
>>
>> On Mon, Mar 22, 2021 at 2:24 PM Ryan Hatter <ry...@gmail.com>
>> wrote:
>>
>>> Hmm, maybe I was just getting twisted around with docker then. I’ll have
>>> a look at what you shared.
>>>
>>> Thanks Bin :)
>>>
>>> On Mar 22, 2021, at 01:52, Xinbin Huang <bi...@gmail.com> wrote:
>>>
>>>
>>> Hi Ryan,
>>>
>>> I believe breeze already provides tools for you to do that:
>>>
>>> 1.Would it make sense to allow developers to choose the executor for
>>> their Breeze environment?
>>>
>>> You can set up environment variables and any other custom setup you want
>>> in the file: */files/airflow-breeze-config/variables.env. *To set up
>>> CeleryExecutor, you just need to put `export
>>> AIRFLOW__CORE__EXECUTOR=CeleryExecutor` in the file.
>>>
>>> 2. Developing using Docker
>>> <https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html> would
>>> require an update to the image each time you want to make a change to the
>>> codebase
>>>
>>> Breeze automatically mounts the local source to the container unless you
>>> explicitly skip it with the flag *--skip-mounting-local-sources. *You
>>> can find more details here
>>> https://github.com/apache/airflow/blob/master/BREEZE.rst#mounting-local-sources-to-breeze
>>>
>>> Best
>>> Bin
>>>
>>>
>>> On Sun, Mar 21, 2021 at 5:32 PM Ryan Hatter <ry...@gmail.com>
>>> wrote:
>>>
>>>> I recently had some trouble trying to fix a bug in the CeleryExecutor
>>>> <https://github.com/apache/airflow/pull/14883>. The code change was
>>>> small, but it was really difficult to set up a development environment
>>>> using the CeleryExecutor. I ultimately had to muck around with the test
>>>> case that covers this situation, default_airflow.cfg, and
>>>> default_celery.py. Developing using Docker
>>>> <https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html>
>>>> would require an update to the image each time you want to make a change to
>>>> the codebase (or maybe `exec`ing into the relevant container?) which is a
>>>> pain.
>>>>
>>>> This led me to two questions:
>>>>
>>>> 1. Would it make sense to allow developers to choose the executor
>>>> for their Breeze environment?
>>>> 2. If not, how do folks test out changes they make to the
>>>> CeleryExecutor or KubernetesExecutor?
>>>>
>>>> Thanks!
>>>> Ryan
>>>>
>>>>
>>>>
>>
>> --
>> +48 660 796 129
>>
>
--
+48 660 796 129
Re: [DISCUSS] Add Breeze Support for CeleryExecutor and KubernetesExecutor
Posted by Ryan Hatter <ry...@gmail.com>.
This seems to work, with a tiny caveat: `airflow celery worker` is the
command to start the celery worker instead of `airflow worker`:
1. Launch breeze with `--integration rabbitq` or `--integration redis`
2. Set the executor to celery with `export
AIRFLOW__CORE__EXECUTOR=CeleryExecutor`
3. Start a celery worker with `airflow celery worker -D`
Would I be crazy to ask if Breeze should be rewritten in Python? Arguments
in support, I think, might be...
1. Easier modularization/extensibility
2. I'd guess that more people are comfortable in Python than Bash
3. At least one of the macOS issues could be resolved
Thoughts?
On Mon, Mar 22, 2021 at 5:51 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Actually it is not that strightforward, but possibly we can make it works
> much more easily
>
> In order to make Celery Executor works you need to do a bit more (but
> should be easy to add as an option to Breeze):
>
> * you need to start rabitmq or redis as integration (`--integration
> rabbitq --integration redis')
> * you need to start worker(s) (`airflow worker` in the background)
> * you might want to start flower optionally (the celery monitoring tool)
>
> So maybe we could add extra switch to start-airflow command
> (--use-celery-executor) that could set those integrations and start
> worker/flower additionally to running webserver/scheduler now in tmux ?
>
> WDYT? Maybe Ryan you can check if my recipe works ? I could add it then as
> an option.
>
> BTW. We already have a number of CeleryExecutor tests that use the
> integrations, so Breeze has all what's needed:
>
>
> https://github.com/apache/airflow/blob/master/tests/executors/test_celery_executor.py#L109
>
> J.
>
> On Mon, Mar 22, 2021 at 2:24 PM Ryan Hatter <ry...@gmail.com> wrote:
>
>> Hmm, maybe I was just getting twisted around with docker then. I’ll have
>> a look at what you shared.
>>
>> Thanks Bin :)
>>
>> On Mar 22, 2021, at 01:52, Xinbin Huang <bi...@gmail.com> wrote:
>>
>>
>> Hi Ryan,
>>
>> I believe breeze already provides tools for you to do that:
>>
>> 1.Would it make sense to allow developers to choose the executor for
>> their Breeze environment?
>>
>> You can set up environment variables and any other custom setup you want
>> in the file: */files/airflow-breeze-config/variables.env. *To set up
>> CeleryExecutor, you just need to put `export
>> AIRFLOW__CORE__EXECUTOR=CeleryExecutor` in the file.
>>
>> 2. Developing using Docker
>> <https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html> would
>> require an update to the image each time you want to make a change to the
>> codebase
>>
>> Breeze automatically mounts the local source to the container unless you
>> explicitly skip it with the flag *--skip-mounting-local-sources. *You
>> can find more details here
>> https://github.com/apache/airflow/blob/master/BREEZE.rst#mounting-local-sources-to-breeze
>>
>> Best
>> Bin
>>
>>
>> On Sun, Mar 21, 2021 at 5:32 PM Ryan Hatter <ry...@gmail.com>
>> wrote:
>>
>>> I recently had some trouble trying to fix a bug in the CeleryExecutor
>>> <https://github.com/apache/airflow/pull/14883>. The code change was
>>> small, but it was really difficult to set up a development environment
>>> using the CeleryExecutor. I ultimately had to muck around with the test
>>> case that covers this situation, default_airflow.cfg, and
>>> default_celery.py. Developing using Docker
>>> <https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html>
>>> would require an update to the image each time you want to make a change to
>>> the codebase (or maybe `exec`ing into the relevant container?) which is a
>>> pain.
>>>
>>> This led me to two questions:
>>>
>>> 1. Would it make sense to allow developers to choose the executor
>>> for their Breeze environment?
>>> 2. If not, how do folks test out changes they make to the
>>> CeleryExecutor or KubernetesExecutor?
>>>
>>> Thanks!
>>> Ryan
>>>
>>>
>>>
>
> --
> +48 660 796 129
>
Re: [DISCUSS] Add Breeze Support for CeleryExecutor and KubernetesExecutor
Posted by Jarek Potiuk <ja...@potiuk.com>.
Actually it is not that strightforward, but possibly we can make it works
much more easily
In order to make Celery Executor works you need to do a bit more (but
should be easy to add as an option to Breeze):
* you need to start rabitmq or redis as integration (`--integration rabbitq
--integration redis')
* you need to start worker(s) (`airflow worker` in the background)
* you might want to start flower optionally (the celery monitoring tool)
So maybe we could add extra switch to start-airflow command
(--use-celery-executor) that could set those integrations and start
worker/flower additionally to running webserver/scheduler now in tmux ?
WDYT? Maybe Ryan you can check if my recipe works ? I could add it then as
an option.
BTW. We already have a number of CeleryExecutor tests that use the
integrations, so Breeze has all what's needed:
https://github.com/apache/airflow/blob/master/tests/executors/test_celery_executor.py#L109
J.
On Mon, Mar 22, 2021 at 2:24 PM Ryan Hatter <ry...@gmail.com> wrote:
> Hmm, maybe I was just getting twisted around with docker then. I’ll have a
> look at what you shared.
>
> Thanks Bin :)
>
> On Mar 22, 2021, at 01:52, Xinbin Huang <bi...@gmail.com> wrote:
>
>
> Hi Ryan,
>
> I believe breeze already provides tools for you to do that:
>
> 1.Would it make sense to allow developers to choose the executor for their
> Breeze environment?
>
> You can set up environment variables and any other custom setup you want
> in the file: */files/airflow-breeze-config/variables.env. *To set up
> CeleryExecutor, you just need to put `export
> AIRFLOW__CORE__EXECUTOR=CeleryExecutor` in the file.
>
> 2. Developing using Docker
> <https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html> would
> require an update to the image each time you want to make a change to the
> codebase
>
> Breeze automatically mounts the local source to the container unless you
> explicitly skip it with the flag *--skip-mounting-local-sources. *You can
> find more details here
> https://github.com/apache/airflow/blob/master/BREEZE.rst#mounting-local-sources-to-breeze
>
> Best
> Bin
>
>
> On Sun, Mar 21, 2021 at 5:32 PM Ryan Hatter <ry...@gmail.com> wrote:
>
>> I recently had some trouble trying to fix a bug in the CeleryExecutor
>> <https://github.com/apache/airflow/pull/14883>. The code change was
>> small, but it was really difficult to set up a development environment
>> using the CeleryExecutor. I ultimately had to muck around with the test
>> case that covers this situation, default_airflow.cfg, and
>> default_celery.py. Developing using Docker
>> <https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html>
>> would require an update to the image each time you want to make a change to
>> the codebase (or maybe `exec`ing into the relevant container?) which is a
>> pain.
>>
>> This led me to two questions:
>>
>> 1. Would it make sense to allow developers to choose the executor for
>> their Breeze environment?
>> 2. If not, how do folks test out changes they make to the
>> CeleryExecutor or KubernetesExecutor?
>>
>> Thanks!
>> Ryan
>>
>>
>>
--
+48 660 796 129
Re: [DISCUSS] Add Breeze Support for CeleryExecutor and KubernetesExecutor
Posted by Ryan Hatter <ry...@gmail.com>.
Hmm, maybe I was just getting twisted around with docker then. I’ll have a look at what you shared.
Thanks Bin :)
> On Mar 22, 2021, at 01:52, Xinbin Huang <bi...@gmail.com> wrote:
>
>
> Hi Ryan,
>
> I believe breeze already provides tools for you to do that:
>
> 1.Would it make sense to allow developers to choose the executor for their Breeze environment?
>
> You can set up environment variables and any other custom setup you want in the file: /files/airflow-breeze-config/variables.env. To set up CeleryExecutor, you just need to put `export AIRFLOW__CORE__EXECUTOR=CeleryExecutor` in the file.
>
> 2. Developing using Docker would require an update to the image each time you want to make a change to the codebase
>
> Breeze automatically mounts the local source to the container unless you explicitly skip it with the flag --skip-mounting-local-sources. You can find more details here https://github.com/apache/airflow/blob/master/BREEZE.rst#mounting-local-sources-to-breeze
>
> Best
> Bin
>
>
>> On Sun, Mar 21, 2021 at 5:32 PM Ryan Hatter <ry...@gmail.com> wrote:
>> I recently had some trouble trying to fix a bug in the CeleryExecutor. The code change was small, but it was really difficult to set up a development environment using the CeleryExecutor. I ultimately had to muck around with the test case that covers this situation, default_airflow.cfg, and default_celery.py. Developing using Docker would require an update to the image each time you want to make a change to the codebase (or maybe `exec`ing into the relevant container?) which is a pain.
>>
>> This led me to two questions:
>> Would it make sense to allow developers to choose the executor for their Breeze environment?
>> If not, how do folks test out changes they make to the CeleryExecutor or KubernetesExecutor?
>> Thanks!
>> Ryan
>>
>>
Re: [DISCUSS] Add Breeze Support for CeleryExecutor and KubernetesExecutor
Posted by Xinbin Huang <bi...@gmail.com>.
Hi Ryan,
I believe breeze already provides tools for you to do that:
1.Would it make sense to allow developers to choose the executor for their
Breeze environment?
You can set up environment variables and any other custom setup you want in
the file: */files/airflow-breeze-config/variables.env. *To set up
CeleryExecutor, you just need to put `export
AIRFLOW__CORE__EXECUTOR=CeleryExecutor` in the file.
2. Developing using Docker
<https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html> would
require an update to the image each time you want to make a change to the
codebase
Breeze automatically mounts the local source to the container unless you
explicitly skip it with the flag *--skip-mounting-local-sources. *You can
find more details here
https://github.com/apache/airflow/blob/master/BREEZE.rst#mounting-local-sources-to-breeze
Best
Bin
On Sun, Mar 21, 2021 at 5:32 PM Ryan Hatter <ry...@gmail.com> wrote:
> I recently had some trouble trying to fix a bug in the CeleryExecutor
> <https://github.com/apache/airflow/pull/14883>. The code change was
> small, but it was really difficult to set up a development environment
> using the CeleryExecutor. I ultimately had to muck around with the test
> case that covers this situation, default_airflow.cfg, and
> default_celery.py. Developing using Docker
> <https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html>
> would require an update to the image each time you want to make a change to
> the codebase (or maybe `exec`ing into the relevant container?) which is a
> pain.
>
> This led me to two questions:
>
> 1. Would it make sense to allow developers to choose the executor for
> their Breeze environment?
> 2. If not, how do folks test out changes they make to the
> CeleryExecutor or KubernetesExecutor?
>
> Thanks!
> Ryan
>
>
>