You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by John Smodic <js...@argo.ai> on 2019/09/12 18:23:41 UTC

Airflow node different versions

Hey all,

I'm looking to find an upgrade path for upgrading Airflow from 1.10.2 to 1.10.5.

But the problem is, the Airflow production cluster is pretty busy all the time.

Would it be problematic to upgrade the Webserver and Scheduler to 1.10.5 and gradually update the nodes as they become free? This would leave a period where nodes are on 1.10.2 and other nodes are on 1.10.5, but I don't know if that is expected to cause any issues.

For 1.9 -> 1.10 I did a full blue green deployment, but didn't necessarily want to go through with that for what seems like a relatively minor series of patches.

Thanks!

Re: Airflow node different versions

Posted by "Bacal, Eugene" <eb...@paypal.com.INVALID>.
Curious, how did it go?

Thanks, 
-Eugene 

On 9/14/19, 4:14 PM, "Alex Guziel" <al...@airbnb.com.INVALID> wrote:

    Agree with Bolke here. Not much is going on in worker as long as there
    aren’t breaking changes.
    
    On Sat, Sep 14, 2019 at 1:24 PM Bolke de Bruin <bd...@gmail.com> wrote:
    
    > I actually think that it is not that risky (although ymmv). Worker nodes
    > are pretty independent from the scheduler/webserver. As long as the
    > datamodel hasnt changed and nodes dont change their reporting (new
    > statusses) to the db (that hasnt happened for a long time) you are probably
    > okay.
    >
    > So the proper way to do (test ;-)) it is scheduler first, webserver next,
    > nodes.
    >
    > Bolke
    >
    > Verstuurd vanaf mijn iPad
    >
    > > Op 13 sep. 2019 om 12:49 heeft Driesprong, Fokko <fo...@driesprong.frl>
    > het volgende geschreven:
    > >
    > > Hi John,
    > >
    > > I've never tried it like you're suggesting. It feels a bit risky. I can't
    > > tell what you will encounter if you run different versions of Airflow.
    > How
    > > are you running Airflow?
    > >
    > > Cheers, Fokko
    > >
    > > Op do 12 sep. 2019 om 20:29 schreef John Smodic <js...@argo.ai>:
    > >
    > >> Hey all,
    > >>
    > >> I'm looking to find an upgrade path for upgrading Airflow from 1.10.2 to
    > >> 1.10.5.
    > >>
    > >> But the problem is, the Airflow production cluster is pretty busy all
    > the
    > >> time.
    > >>
    > >> Would it be problematic to upgrade the Webserver and Scheduler to 1.10.5
    > >> and gradually update the nodes as they become free? This would leave a
    > >> period where nodes are on 1.10.2 and other nodes are on 1.10.5, but I
    > don't
    > >> know if that is expected to cause any issues.
    > >>
    > >> For 1.9 -> 1.10 I did a full blue green deployment, but didn't
    > necessarily
    > >> want to go through with that for what seems like a relatively minor
    > series
    > >> of patches.
    > >>
    > >> Thanks!
    > >>
    >

Re: Airflow node different versions

Posted by Alex Guziel <al...@airbnb.com.INVALID>.
Agree with Bolke here. Not much is going on in worker as long as there
aren’t breaking changes.

On Sat, Sep 14, 2019 at 1:24 PM Bolke de Bruin <bd...@gmail.com> wrote:

> I actually think that it is not that risky (although ymmv). Worker nodes
> are pretty independent from the scheduler/webserver. As long as the
> datamodel hasnt changed and nodes dont change their reporting (new
> statusses) to the db (that hasnt happened for a long time) you are probably
> okay.
>
> So the proper way to do (test ;-)) it is scheduler first, webserver next,
> nodes.
>
> Bolke
>
> Verstuurd vanaf mijn iPad
>
> > Op 13 sep. 2019 om 12:49 heeft Driesprong, Fokko <fo...@driesprong.frl>
> het volgende geschreven:
> >
> > Hi John,
> >
> > I've never tried it like you're suggesting. It feels a bit risky. I can't
> > tell what you will encounter if you run different versions of Airflow.
> How
> > are you running Airflow?
> >
> > Cheers, Fokko
> >
> > Op do 12 sep. 2019 om 20:29 schreef John Smodic <js...@argo.ai>:
> >
> >> Hey all,
> >>
> >> I'm looking to find an upgrade path for upgrading Airflow from 1.10.2 to
> >> 1.10.5.
> >>
> >> But the problem is, the Airflow production cluster is pretty busy all
> the
> >> time.
> >>
> >> Would it be problematic to upgrade the Webserver and Scheduler to 1.10.5
> >> and gradually update the nodes as they become free? This would leave a
> >> period where nodes are on 1.10.2 and other nodes are on 1.10.5, but I
> don't
> >> know if that is expected to cause any issues.
> >>
> >> For 1.9 -> 1.10 I did a full blue green deployment, but didn't
> necessarily
> >> want to go through with that for what seems like a relatively minor
> series
> >> of patches.
> >>
> >> Thanks!
> >>
>

Re: Airflow node different versions

Posted by Bolke de Bruin <bd...@gmail.com>.
I actually think that it is not that risky (although ymmv). Worker nodes are pretty independent from the scheduler/webserver. As long as the datamodel hasnt changed and nodes dont change their reporting (new statusses) to the db (that hasnt happened for a long time) you are probably okay.

So the proper way to do (test ;-)) it is scheduler first, webserver next, nodes.

Bolke

Verstuurd vanaf mijn iPad

> Op 13 sep. 2019 om 12:49 heeft Driesprong, Fokko <fo...@driesprong.frl> het volgende geschreven:
> 
> Hi John,
> 
> I've never tried it like you're suggesting. It feels a bit risky. I can't
> tell what you will encounter if you run different versions of Airflow. How
> are you running Airflow?
> 
> Cheers, Fokko
> 
> Op do 12 sep. 2019 om 20:29 schreef John Smodic <js...@argo.ai>:
> 
>> Hey all,
>> 
>> I'm looking to find an upgrade path for upgrading Airflow from 1.10.2 to
>> 1.10.5.
>> 
>> But the problem is, the Airflow production cluster is pretty busy all the
>> time.
>> 
>> Would it be problematic to upgrade the Webserver and Scheduler to 1.10.5
>> and gradually update the nodes as they become free? This would leave a
>> period where nodes are on 1.10.2 and other nodes are on 1.10.5, but I don't
>> know if that is expected to cause any issues.
>> 
>> For 1.9 -> 1.10 I did a full blue green deployment, but didn't necessarily
>> want to go through with that for what seems like a relatively minor series
>> of patches.
>> 
>> Thanks!
>>

Re: Airflow node different versions

Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Hi John,

I've never tried it like you're suggesting. It feels a bit risky. I can't
tell what you will encounter if you run different versions of Airflow. How
are you running Airflow?

Cheers, Fokko

Op do 12 sep. 2019 om 20:29 schreef John Smodic <js...@argo.ai>:

> Hey all,
>
> I'm looking to find an upgrade path for upgrading Airflow from 1.10.2 to
> 1.10.5.
>
> But the problem is, the Airflow production cluster is pretty busy all the
> time.
>
> Would it be problematic to upgrade the Webserver and Scheduler to 1.10.5
> and gradually update the nodes as they become free? This would leave a
> period where nodes are on 1.10.2 and other nodes are on 1.10.5, but I don't
> know if that is expected to cause any issues.
>
> For 1.9 -> 1.10 I did a full blue green deployment, but didn't necessarily
> want to go through with that for what seems like a relatively minor series
> of patches.
>
> Thanks!
>