You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Thoralf Gutierrez <th...@gmail.com> on 2017/10/03 19:33:37 UTC
Runbook to upgrade Airflow
Hey everybody!
Does anybody have some kind of runbook to upgrade airflow (with a Celery
backend) without having any downtime (i.e. tasks keep on running as you
upgrade)?
I have this in mind, but not sure if I am missing something or if I should
be careful with the order of steps (especially for upgrading the db schema):
1. run airflow upgradedb from anywhere
2. one worker at a time
2a. make sure it doesn't start any new task.
2b. wait for all tasks to be finished
2c. run pip install airflow --upgrade
2d. re-enable worker
3. one webserver at a time
3a. kill webserver
3b. run pip install airflow --upgrade
3c. start webserver
4. scheduler
4a. kill scheduler
4b. run pip intsall airflow --upgrade
4c. start scheduler
Thanks,
Thoralf
Re: Runbook to upgrade Airflow
Posted by Andrew Maguire <an...@gmail.com>.
would love to have a sort of step by step here.
upgrading to 1.9 now as want to help as best i can and figure installing
latest version one good way to help find any issues.
unless its as simple as pip install.
So far steps i'm taking (we just run local on single server as use BigQuery
for most work)
1. wget
https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0alpha1/apache-airflow-1.9.0alpha1+incubating-bin.tar.gz
2. pip install apache-airflow-1.9.0alpha1+incubating-bin.tar.gz
3. reboot server
4. airflow upgradedb
Anything i might be missing here or anything dangerous in what i'm doing?
Cheers,
Andy
On Wed, Oct 4, 2017 at 10:09 PM Thoralf Gutierrez <
thoralfgutierrez@gmail.com> wrote:
> I see - thanks for the answers!
>
> On Wed, 4 Oct 2017 at 13:10 Alex Guziel <al...@airbnb.com.invalid>
> wrote:
>
> > We pretty much just shut everything off (fence the db by changing the
> > password), upgrade the db, then turn everything back on and change the db
> > password back. We accept that there will be downtime for maintenance and
> do
> > it at a low impact time (like 20:00 utc ). Also, the thing with
> sqlalchemy
> > is a lot more things become breaking changes because of the way it writes
> > queries.
> >
> > It might be good to have downtime even without schema changes to prevent
> > weird behavior if the changeset is large.
> >
> > But for small stuff, depending on the scope of the change, workers don't
> > really need to be restarted, scheduler can be handled manually, and
> > webserver we do a rolling restart.
> >
> > On Wed, Oct 4, 2017 at 7:54 AM Thoralf Gutierrez <
> > thoralfgutierrez@gmail.com>
> > wrote:
> >
> > > Thanks for your answer Alex.
> > >
> > > I guess you mean it won't work if there is a _breaking_ schema change
> > > right? But for new patch and minor versions (i.e. without breaking
> > > changes), I'm guessing that upgrading the db first should do the trick?
> > >
> > > @Airbnb team, how do you all do it? If I remember correctly, you often
> > > update your version of airflow to test out new releases. What does your
> > > runbook for Airflow upgrades look like?
> > >
> > > Cheers,
> > > Thoralf
> > >
> > > On Tue, 3 Oct 2017 at 13:04 Alex Guziel <alex.guziel@airbnb.com
> .invalid>
> > > wrote:
> > >
> > > > You won't be able to if there's a schema change.
> > > >
> > > > On Tue, Oct 3, 2017 at 12:33 PM, Thoralf Gutierrez <
> > > > thoralfgutierrez@gmail.com> wrote:
> > > >
> > > > > Hey everybody!
> > > > >
> > > > > Does anybody have some kind of runbook to upgrade airflow (with a
> > > Celery
> > > > > backend) without having any downtime (i.e. tasks keep on running as
> > you
> > > > > upgrade)?
> > > > >
> > > > > I have this in mind, but not sure if I am missing something or if I
> > > > should
> > > > > be careful with the order of steps (especially for upgrading the db
> > > > > schema):
> > > > >
> > > > > 1. run airflow upgradedb from anywhere
> > > > >
> > > > > 2. one worker at a time
> > > > > 2a. make sure it doesn't start any new task.
> > > > > 2b. wait for all tasks to be finished
> > > > > 2c. run pip install airflow --upgrade
> > > > > 2d. re-enable worker
> > > > >
> > > > > 3. one webserver at a time
> > > > > 3a. kill webserver
> > > > > 3b. run pip install airflow --upgrade
> > > > > 3c. start webserver
> > > > >
> > > > > 4. scheduler
> > > > > 4a. kill scheduler
> > > > > 4b. run pip intsall airflow --upgrade
> > > > > 4c. start scheduler
> > > > >
> > > > > Thanks,
> > > > > Thoralf
> > > > >
> > > >
> > >
> >
>
Re: Runbook to upgrade Airflow
Posted by Thoralf Gutierrez <th...@gmail.com>.
I see - thanks for the answers!
On Wed, 4 Oct 2017 at 13:10 Alex Guziel <al...@airbnb.com.invalid>
wrote:
> We pretty much just shut everything off (fence the db by changing the
> password), upgrade the db, then turn everything back on and change the db
> password back. We accept that there will be downtime for maintenance and do
> it at a low impact time (like 20:00 utc ). Also, the thing with sqlalchemy
> is a lot more things become breaking changes because of the way it writes
> queries.
>
> It might be good to have downtime even without schema changes to prevent
> weird behavior if the changeset is large.
>
> But for small stuff, depending on the scope of the change, workers don't
> really need to be restarted, scheduler can be handled manually, and
> webserver we do a rolling restart.
>
> On Wed, Oct 4, 2017 at 7:54 AM Thoralf Gutierrez <
> thoralfgutierrez@gmail.com>
> wrote:
>
> > Thanks for your answer Alex.
> >
> > I guess you mean it won't work if there is a _breaking_ schema change
> > right? But for new patch and minor versions (i.e. without breaking
> > changes), I'm guessing that upgrading the db first should do the trick?
> >
> > @Airbnb team, how do you all do it? If I remember correctly, you often
> > update your version of airflow to test out new releases. What does your
> > runbook for Airflow upgrades look like?
> >
> > Cheers,
> > Thoralf
> >
> > On Tue, 3 Oct 2017 at 13:04 Alex Guziel <al...@airbnb.com.invalid>
> > wrote:
> >
> > > You won't be able to if there's a schema change.
> > >
> > > On Tue, Oct 3, 2017 at 12:33 PM, Thoralf Gutierrez <
> > > thoralfgutierrez@gmail.com> wrote:
> > >
> > > > Hey everybody!
> > > >
> > > > Does anybody have some kind of runbook to upgrade airflow (with a
> > Celery
> > > > backend) without having any downtime (i.e. tasks keep on running as
> you
> > > > upgrade)?
> > > >
> > > > I have this in mind, but not sure if I am missing something or if I
> > > should
> > > > be careful with the order of steps (especially for upgrading the db
> > > > schema):
> > > >
> > > > 1. run airflow upgradedb from anywhere
> > > >
> > > > 2. one worker at a time
> > > > 2a. make sure it doesn't start any new task.
> > > > 2b. wait for all tasks to be finished
> > > > 2c. run pip install airflow --upgrade
> > > > 2d. re-enable worker
> > > >
> > > > 3. one webserver at a time
> > > > 3a. kill webserver
> > > > 3b. run pip install airflow --upgrade
> > > > 3c. start webserver
> > > >
> > > > 4. scheduler
> > > > 4a. kill scheduler
> > > > 4b. run pip intsall airflow --upgrade
> > > > 4c. start scheduler
> > > >
> > > > Thanks,
> > > > Thoralf
> > > >
> > >
> >
>
Re: Runbook to upgrade Airflow
Posted by Alex Guziel <al...@airbnb.com.INVALID>.
We pretty much just shut everything off (fence the db by changing the
password), upgrade the db, then turn everything back on and change the db
password back. We accept that there will be downtime for maintenance and do
it at a low impact time (like 20:00 utc ). Also, the thing with sqlalchemy
is a lot more things become breaking changes because of the way it writes
queries.
It might be good to have downtime even without schema changes to prevent
weird behavior if the changeset is large.
But for small stuff, depending on the scope of the change, workers don't
really need to be restarted, scheduler can be handled manually, and
webserver we do a rolling restart.
On Wed, Oct 4, 2017 at 7:54 AM Thoralf Gutierrez <th...@gmail.com>
wrote:
> Thanks for your answer Alex.
>
> I guess you mean it won't work if there is a _breaking_ schema change
> right? But for new patch and minor versions (i.e. without breaking
> changes), I'm guessing that upgrading the db first should do the trick?
>
> @Airbnb team, how do you all do it? If I remember correctly, you often
> update your version of airflow to test out new releases. What does your
> runbook for Airflow upgrades look like?
>
> Cheers,
> Thoralf
>
> On Tue, 3 Oct 2017 at 13:04 Alex Guziel <al...@airbnb.com.invalid>
> wrote:
>
> > You won't be able to if there's a schema change.
> >
> > On Tue, Oct 3, 2017 at 12:33 PM, Thoralf Gutierrez <
> > thoralfgutierrez@gmail.com> wrote:
> >
> > > Hey everybody!
> > >
> > > Does anybody have some kind of runbook to upgrade airflow (with a
> Celery
> > > backend) without having any downtime (i.e. tasks keep on running as you
> > > upgrade)?
> > >
> > > I have this in mind, but not sure if I am missing something or if I
> > should
> > > be careful with the order of steps (especially for upgrading the db
> > > schema):
> > >
> > > 1. run airflow upgradedb from anywhere
> > >
> > > 2. one worker at a time
> > > 2a. make sure it doesn't start any new task.
> > > 2b. wait for all tasks to be finished
> > > 2c. run pip install airflow --upgrade
> > > 2d. re-enable worker
> > >
> > > 3. one webserver at a time
> > > 3a. kill webserver
> > > 3b. run pip install airflow --upgrade
> > > 3c. start webserver
> > >
> > > 4. scheduler
> > > 4a. kill scheduler
> > > 4b. run pip intsall airflow --upgrade
> > > 4c. start scheduler
> > >
> > > Thanks,
> > > Thoralf
> > >
> >
>
Re: Runbook to upgrade Airflow
Posted by Thoralf Gutierrez <th...@gmail.com>.
Thanks for your answer Alex.
I guess you mean it won't work if there is a _breaking_ schema change
right? But for new patch and minor versions (i.e. without breaking
changes), I'm guessing that upgrading the db first should do the trick?
@Airbnb team, how do you all do it? If I remember correctly, you often
update your version of airflow to test out new releases. What does your
runbook for Airflow upgrades look like?
Cheers,
Thoralf
On Tue, 3 Oct 2017 at 13:04 Alex Guziel <al...@airbnb.com.invalid>
wrote:
> You won't be able to if there's a schema change.
>
> On Tue, Oct 3, 2017 at 12:33 PM, Thoralf Gutierrez <
> thoralfgutierrez@gmail.com> wrote:
>
> > Hey everybody!
> >
> > Does anybody have some kind of runbook to upgrade airflow (with a Celery
> > backend) without having any downtime (i.e. tasks keep on running as you
> > upgrade)?
> >
> > I have this in mind, but not sure if I am missing something or if I
> should
> > be careful with the order of steps (especially for upgrading the db
> > schema):
> >
> > 1. run airflow upgradedb from anywhere
> >
> > 2. one worker at a time
> > 2a. make sure it doesn't start any new task.
> > 2b. wait for all tasks to be finished
> > 2c. run pip install airflow --upgrade
> > 2d. re-enable worker
> >
> > 3. one webserver at a time
> > 3a. kill webserver
> > 3b. run pip install airflow --upgrade
> > 3c. start webserver
> >
> > 4. scheduler
> > 4a. kill scheduler
> > 4b. run pip intsall airflow --upgrade
> > 4c. start scheduler
> >
> > Thanks,
> > Thoralf
> >
>
Re: Runbook to upgrade Airflow
Posted by Alex Guziel <al...@airbnb.com.INVALID>.
You won't be able to if there's a schema change.
On Tue, Oct 3, 2017 at 12:33 PM, Thoralf Gutierrez <
thoralfgutierrez@gmail.com> wrote:
> Hey everybody!
>
> Does anybody have some kind of runbook to upgrade airflow (with a Celery
> backend) without having any downtime (i.e. tasks keep on running as you
> upgrade)?
>
> I have this in mind, but not sure if I am missing something or if I should
> be careful with the order of steps (especially for upgrading the db
> schema):
>
> 1. run airflow upgradedb from anywhere
>
> 2. one worker at a time
> 2a. make sure it doesn't start any new task.
> 2b. wait for all tasks to be finished
> 2c. run pip install airflow --upgrade
> 2d. re-enable worker
>
> 3. one webserver at a time
> 3a. kill webserver
> 3b. run pip install airflow --upgrade
> 3c. start webserver
>
> 4. scheduler
> 4a. kill scheduler
> 4b. run pip intsall airflow --upgrade
> 4c. start scheduler
>
> Thanks,
> Thoralf
>