You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Dev Aleksander <de...@sumowski.pl> on 2018/08/21 13:25:01 UTC

updating dags best practice

Hi all,
in the place I'm currently at we're building and redeploying a new set of
containers with the latest code every time we want to update a DAG. That
doesn't feel like the fastest way.

Anyone can share their approach?

Thanks,
Aleksander

Re: updating dags best practice

Posted by Dev Aleksander <de...@sumowski.pl>.
We're currently using CeleryExecutor

On Tue, Aug 21, 2018 at 3:11 PM, Driesprong, Fokko <fo...@driesprong.frl>
wrote:

> Hi Aleksander,
>
> What kind of executor are you using? This is really important when choosing
> a strategy. For most of the Airflow deployments that I've worked with
> simply uses git to deploy the master branch. This is with a LocalExecutor.
>
> Cheers, Fokko
>
> Op di 21 aug. 2018 om 15:25 schreef Dev Aleksander <de...@sumowski.pl>
>
> > Hi all,
> > in the place I'm currently at we're building and redeploying a new set of
> > containers with the latest code every time we want to update a DAG. That
> > doesn't feel like the fastest way.
> >
> > Anyone can share their approach?
> >
> > Thanks,
> > Aleksander
> >
>

Re: updating dags best practice

Posted by Aleksander Sumowski <Al...@thetrainline.com>.
We're currently using CeleryExecutor ...

On 21/08/2018, 15:12, "Driesprong, Fokko" <fo...@driesprong.frl> wrote:

    Hi Aleksander,

    What kind of executor are you using? This is really important when choosing
    a strategy. For most of the Airflow deployments that I've worked with
    simply uses git to deploy the master branch. This is with a LocalExecutor.

    Cheers, Fokko

    Op di 21 aug. 2018 om 15:25 schreef Dev Aleksander <de...@sumowski.pl>

    > Hi all,
    > in the place I'm currently at we're building and redeploying a new set of
    > containers with the latest code every time we want to update a DAG. That
    > doesn't feel like the fastest way.
    >
    > Anyone can share their approach?
    >
    > Thanks,
    > Aleksander
    >


The information in this email (and any attachments) is confidential and is intended solely for the use of the individual or entity to whom it is addressed. If you received this email in error please tell us by reply email (or telephone the sender) and delete all electronic copies on your system or other copies known to you. Trainline Investments Holdings Limited (Registered No.5776685), Trainline.com Limited (Registered No. 3846791) and Trainline International Limited (Registered No. 6881309) are all registered in England and Wales with registered office at 3rd floor, 120 Holborn, London, EC1N 2TD.

Re: updating dags best practice

Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Hi Aleksander,

What kind of executor are you using? This is really important when choosing
a strategy. For most of the Airflow deployments that I've worked with
simply uses git to deploy the master branch. This is with a LocalExecutor.

Cheers, Fokko

Op di 21 aug. 2018 om 15:25 schreef Dev Aleksander <de...@sumowski.pl>

> Hi all,
> in the place I'm currently at we're building and redeploying a new set of
> containers with the latest code every time we want to update a DAG. That
> doesn't feel like the fastest way.
>
> Anyone can share their approach?
>
> Thanks,
> Aleksander
>

Re: updating dags best practice

Posted by Ry Walker <ry...@astronomer.io>.
We follow the strategy to re-build Docker with each deployment, because
sometimes DAG changes can include new or updated dependencies. This
Docker-first approach makes local development nice too (run same
Docker images on local machine during debug as production). This isnt
painful for us because we’ve also automated the dockerizarion, deployment,
and graceful restart of the Airflow cluster with our CLI and API, and
because we’re using a Docker repository, roll back is possible.

-Ry

Sent from mobile


On Aug 21, 2018 at 8:25 AM, Dev Aleksander <de...@sumowski.pl> wrote:


Hi all,
in the place I'm currently at we're building and redeploying a new set of
containers with the latest code every time we want to update a DAG. That
doesn't feel like the fastest way.

Anyone can share their approach?

Thanks,
Aleksander