You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by "Maycock, Luke" <lu...@affiliate.oliverwyman.com> on 2016/09/29 10:08:18 UTC
Airflow Releases
Airflow Developers,
We were looking at writing a workflow framework in Python when we found Airflow. We have carried out some proof of concept work for using Airflow and wish to continue using it as it comes with lots of great features out-of-the-box.
We have created our own fork here:
https://github.com/owlabs/incubator-airflow
So far, the only thing we have committed back to the main repository is the following fix to the mssql_hook:
https://github.com/apache/incubator-airflow/pull/1626
Among other types of tasks, we wish to be able to run mssql tasks using Airflow. In order to do so, the above and below fixes are required:
https://github.com/apache/incubator-airflow/pull/1458<https://github.com/apache/incubator-airflow/pull/1458/commits/e7e655fde3c29742149d047028cbb21aecba86ed>
We have created a Chef cookbook for configuring VMs with Airflow and its prerequisites. As part of this cookbook, we are installing the latest release of Airflow. However, it appears that the latest release does not have the aforementioned fixes.
Do you know when the next release of Airflow is expected? Is there anything we can do to help get the next release in place?
Luke Maycock
OLIVER WYMAN
luke.maycock@affiliate.oliverwyman.com<ma...@affiliate.oliverwyman.com>
www.oliverwyman.com<http://www.oliverwyman.com/>
________________________________
This e-mail and any attachments may be confidential or legally privileged. If you received this message in error or are not the intended recipient, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained herein. Please inform us of the erroneous delivery by return e-mail. Thank you for your cooperation.
Re: Airflow Releases
Posted by George Leslie-Waksman <ge...@cloverhealth.com>.
I consider https://github.com/apache/incubator-airflow/pull/1738 critical
to get in before the release. Without it, plugins are entirely
non-functional.
We have a staging environment running off a version close to master, I can
probably push it closer to master and try to identify bugs on our end.
--George Leslie-Waksman (Clover Health)
On Fri, Sep 30, 2016 at 12:56 PM Bolke de Bruin <bd...@gmail.com> wrote:
> We will switch to master beginning next week. I think it would be nice to
> sync on git commit hashes that we are considering to be “master”.
>
> We have some operational concerns so from our point of view we probably
> need a couple of RCs.
>
>
> - Bolke
>
> > Op 30 sep. 2016, om 18:13 heeft Alex Van Boxel <al...@vanboxel.be> het
> volgende geschreven:
> >
> > I'll do the same. Nice to have the 1.8 on the horizon.
> >
> > On Fri, Sep 30, 2016 at 5:51 PM Chris Riccomini <cr...@apache.org>
> > wrote:
> >
> >>> I'm not sure how other projects do this, but I propose that we let the
> >> RC settle for a week.
> >>
> >> +1
> >>
> >>> Who's on board!?
> >>
> >> Me. :)
> >>
> >> On Fri, Sep 30, 2016 at 7:56 AM, Maxime Beauchemin
> >> <ma...@gmail.com> wrote:
> >>> As related note: the most people we can get to test this release
> >> candidate
> >>> the better. Note that Airbnb uses only a subset of the Airflow
> codebase,
> >>> there are many operators and features we do not use at all, so we
> >> encourage
> >>> everyone to get on board with testing the RC.
> >>>
> >>> At Airbnb, our process starts with sending the RC into our staging
> >> cluster,
> >>> let it settle and fix any bug over 48 hours or so. We look for changes
> >>> around DagBag processing time, memory footprint on workers, smooth
> >>> processing around our canaries.
> >>>
> >>> When the RC is ready to hit production, we take a full backup of the
> >>> metadata DB, and make sure that the whole team is fully available to
> >> babysit
> >>> the release and hotfix any issue that may pop up, with a clear, fast
> >> revert
> >>> plan.
> >>>
> >>> As all of our tasks are assumed to be idempotent the worst case
> scenarios
> >>> are around delayed SLAs and wasted resources which is all manageable.
> >>>
> >>> I'm not sure how other projects do this, but I propose that we let the
> RC
> >>> settle for a week. I suggest that we reset the clock to 7 more days for
> >> each
> >>> increment of RC, unless the hotfix is very minor.
> >>>
> >>> Who's on board!?
> >>>
> >>> Max
> >>>
> >>> On Fri, Sep 30, 2016 at 7:43 AM, Maxime Beauchemin
> >>> <ma...@gmail.com> wrote:
> >>>>
> >>>> Airbnb will cut a release candidate on the week of October 10th off of
> >>>> master.
> >>>>
> >>>> Community, if you have outstanding PRs that you want to make this
> >> release,
> >>>> now is the time! Though we do not want to rush-merge anything risky
> >> that may
> >>>> make the release process harder.
> >>>>
> >>>> Max
> >>>>
> >>>> On Fri, Sep 30, 2016 at 3:59 AM, Maycock, Luke
> >>>> <lu...@affiliate.oliverwyman.com> wrote:
> >>>>>
> >>>>> Hi Chris,
> >>>>>
> >>>>>
> >>>>> We are currently running master in our environment and we'll be sure
> to
> >>>>> report any issues.
> >>>>>
> >>>>>
> >>>>> Do you know when the 1.8 release is due?
> >>>>>
> >>>>>
> >>>>> Cheers,
> >>>>>
> >>>>> Luke Maycock
> >>>>> OLIVER WYMAN
> >>>>>
> >>>>> luke.maycock@affiliate.oliverwyman.com<mailto:
> >> luke.maycock@affiliate.oliverwyman.com>
> >>>>> www.oliverwyman.com<http://www.oliverwyman.com/>
> >>>>>
> >>>>>
> >>>>> ________________________________
> >>>>> From: Chris Riccomini <cr...@apache.org>
> >>>>> Sent: 29 September 2016 18:14:02
> >>>>> To: dev@airflow.incubator.apache.org
> >>>>> Subject: Re: Airflow Releases
> >>>>>
> >>>>> Hey Luke,
> >>>>>
> >>>>>> Is there anything we can do to help get the next release in place?
> >>>>>
> >>>>> One thing that would definitely help is running master somewhere in
> >>>>> your environment, and reporting any issues that see. Over the next
> few
> >>>>> weeks, AirBNB and a few other folks will be doing the same in an
> >>>>> effort to harden the 1.8 release.
> >>>>>
> >>>>> Cheers,
> >>>>> Chris
> >>>>>
> >>>>> On Thu, Sep 29, 2016 at 3:08 AM, Maycock, Luke
> >>>>> <lu...@affiliate.oliverwyman.com> wrote:
> >>>>>> Airflow Developers,
> >>>>>>
> >>>>>>
> >>>>>> We were looking at writing a workflow framework in Python when we
> >> found
> >>>>>> Airflow. We have carried out some proof of concept work for using
> >> Airflow
> >>>>>> and wish to continue using it as it comes with lots of great
> features
> >>>>>> out-of-the-box.
> >>>>>>
> >>>>>>
> >>>>>> We have created our own fork here:
> >>>>>>
> >>>>>> https://github.com/owlabs/incubator-airflow
> >>>>>>
> >>>>>>
> >>>>>> So far, the only thing we have committed back to the main repository
> >> is
> >>>>>> the following fix to the mssql_hook:
> >>>>>>
> >>>>>> https://github.com/apache/incubator-airflow/pull/1626
> >>>>>>
> >>>>>>
> >>>>>> Among other types of tasks, we wish to be able to run mssql tasks
> >> using
> >>>>>> Airflow. In order to do so, the above and below fixes are required:
> >>>>>>
> >>>>>>
> >>>>>> https://github.com/apache/incubator-airflow/pull/1458<
> >>
> https://github.com/apache/incubator-airflow/pull/1458/commits/e7e655fde3c29742149d047028cbb21aecba86ed
> >>>
> >>>>>>
> >>>>>>
> >>>>>> We have created a Chef cookbook for configuring VMs with Airflow and
> >>>>>> its prerequisites. As part of this cookbook, we are installing the
> >> latest
> >>>>>> release of Airflow. However, it appears that the latest release does
> >> not
> >>>>>> have the aforementioned fixes.
> >>>>>>
> >>>>>> Do you know when the next release of Airflow is expected? Is there
> >>>>>> anything we can do to help get the next release in place?
> >>>>>>
> >>>>>>
> >>>>>> Luke Maycock
> >>>>>> OLIVER WYMAN
> >>>>>>
> >>>>>> luke.maycock@affiliate.oliverwyman.com<mailto:
> >> luke.maycock@affiliate.oliverwyman.com>
> >>>>>> www.oliverwyman.com<http://www.oliverwyman.com/>
> >>>>>>
> >>>>>>
> >>>>>> ________________________________
> >>>>>> This e-mail and any attachments may be confidential or legally
> >>>>>> privileged. If you received this message in error or are not the
> >> intended
> >>>>>> recipient, you should destroy the e-mail message and any attachments
> >> or
> >>>>>> copies, and you are prohibited from retaining, distributing,
> >> disclosing or
> >>>>>> using any information contained herein. Please inform us of the
> >> erroneous
> >>>>>> delivery by return e-mail. Thank you for your cooperation.
> >>>>>
> >>>>> ________________________________
> >>>>> This e-mail and any attachments may be confidential or legally
> >>>>> privileged. If you received this message in error or are not the
> >> intended
> >>>>> recipient, you should destroy the e-mail message and any attachments
> or
> >>>>> copies, and you are prohibited from retaining, distributing,
> >> disclosing or
> >>>>> using any information contained herein. Please inform us of the
> >> erroneous
> >>>>> delivery by return e-mail. Thank you for your cooperation.
> >>>>
> >>>>
> >>>
> >>
>
>
Re: Airflow Releases
Posted by Bolke de Bruin <bd...@gmail.com>.
We will switch to master beginning next week. I think it would be nice to sync on git commit hashes that we are considering to be “master”.
We have some operational concerns so from our point of view we probably need a couple of RCs.
- Bolke
> Op 30 sep. 2016, om 18:13 heeft Alex Van Boxel <al...@vanboxel.be> het volgende geschreven:
>
> I'll do the same. Nice to have the 1.8 on the horizon.
>
> On Fri, Sep 30, 2016 at 5:51 PM Chris Riccomini <cr...@apache.org>
> wrote:
>
>>> I'm not sure how other projects do this, but I propose that we let the
>> RC settle for a week.
>>
>> +1
>>
>>> Who's on board!?
>>
>> Me. :)
>>
>> On Fri, Sep 30, 2016 at 7:56 AM, Maxime Beauchemin
>> <ma...@gmail.com> wrote:
>>> As related note: the most people we can get to test this release
>> candidate
>>> the better. Note that Airbnb uses only a subset of the Airflow codebase,
>>> there are many operators and features we do not use at all, so we
>> encourage
>>> everyone to get on board with testing the RC.
>>>
>>> At Airbnb, our process starts with sending the RC into our staging
>> cluster,
>>> let it settle and fix any bug over 48 hours or so. We look for changes
>>> around DagBag processing time, memory footprint on workers, smooth
>>> processing around our canaries.
>>>
>>> When the RC is ready to hit production, we take a full backup of the
>>> metadata DB, and make sure that the whole team is fully available to
>> babysit
>>> the release and hotfix any issue that may pop up, with a clear, fast
>> revert
>>> plan.
>>>
>>> As all of our tasks are assumed to be idempotent the worst case scenarios
>>> are around delayed SLAs and wasted resources which is all manageable.
>>>
>>> I'm not sure how other projects do this, but I propose that we let the RC
>>> settle for a week. I suggest that we reset the clock to 7 more days for
>> each
>>> increment of RC, unless the hotfix is very minor.
>>>
>>> Who's on board!?
>>>
>>> Max
>>>
>>> On Fri, Sep 30, 2016 at 7:43 AM, Maxime Beauchemin
>>> <ma...@gmail.com> wrote:
>>>>
>>>> Airbnb will cut a release candidate on the week of October 10th off of
>>>> master.
>>>>
>>>> Community, if you have outstanding PRs that you want to make this
>> release,
>>>> now is the time! Though we do not want to rush-merge anything risky
>> that may
>>>> make the release process harder.
>>>>
>>>> Max
>>>>
>>>> On Fri, Sep 30, 2016 at 3:59 AM, Maycock, Luke
>>>> <lu...@affiliate.oliverwyman.com> wrote:
>>>>>
>>>>> Hi Chris,
>>>>>
>>>>>
>>>>> We are currently running master in our environment and we'll be sure to
>>>>> report any issues.
>>>>>
>>>>>
>>>>> Do you know when the 1.8 release is due?
>>>>>
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Luke Maycock
>>>>> OLIVER WYMAN
>>>>>
>>>>> luke.maycock@affiliate.oliverwyman.com<mailto:
>> luke.maycock@affiliate.oliverwyman.com>
>>>>> www.oliverwyman.com<http://www.oliverwyman.com/>
>>>>>
>>>>>
>>>>> ________________________________
>>>>> From: Chris Riccomini <cr...@apache.org>
>>>>> Sent: 29 September 2016 18:14:02
>>>>> To: dev@airflow.incubator.apache.org
>>>>> Subject: Re: Airflow Releases
>>>>>
>>>>> Hey Luke,
>>>>>
>>>>>> Is there anything we can do to help get the next release in place?
>>>>>
>>>>> One thing that would definitely help is running master somewhere in
>>>>> your environment, and reporting any issues that see. Over the next few
>>>>> weeks, AirBNB and a few other folks will be doing the same in an
>>>>> effort to harden the 1.8 release.
>>>>>
>>>>> Cheers,
>>>>> Chris
>>>>>
>>>>> On Thu, Sep 29, 2016 at 3:08 AM, Maycock, Luke
>>>>> <lu...@affiliate.oliverwyman.com> wrote:
>>>>>> Airflow Developers,
>>>>>>
>>>>>>
>>>>>> We were looking at writing a workflow framework in Python when we
>> found
>>>>>> Airflow. We have carried out some proof of concept work for using
>> Airflow
>>>>>> and wish to continue using it as it comes with lots of great features
>>>>>> out-of-the-box.
>>>>>>
>>>>>>
>>>>>> We have created our own fork here:
>>>>>>
>>>>>> https://github.com/owlabs/incubator-airflow
>>>>>>
>>>>>>
>>>>>> So far, the only thing we have committed back to the main repository
>> is
>>>>>> the following fix to the mssql_hook:
>>>>>>
>>>>>> https://github.com/apache/incubator-airflow/pull/1626
>>>>>>
>>>>>>
>>>>>> Among other types of tasks, we wish to be able to run mssql tasks
>> using
>>>>>> Airflow. In order to do so, the above and below fixes are required:
>>>>>>
>>>>>>
>>>>>> https://github.com/apache/incubator-airflow/pull/1458<
>> https://github.com/apache/incubator-airflow/pull/1458/commits/e7e655fde3c29742149d047028cbb21aecba86ed
>>>
>>>>>>
>>>>>>
>>>>>> We have created a Chef cookbook for configuring VMs with Airflow and
>>>>>> its prerequisites. As part of this cookbook, we are installing the
>> latest
>>>>>> release of Airflow. However, it appears that the latest release does
>> not
>>>>>> have the aforementioned fixes.
>>>>>>
>>>>>> Do you know when the next release of Airflow is expected? Is there
>>>>>> anything we can do to help get the next release in place?
>>>>>>
>>>>>>
>>>>>> Luke Maycock
>>>>>> OLIVER WYMAN
>>>>>>
>>>>>> luke.maycock@affiliate.oliverwyman.com<mailto:
>> luke.maycock@affiliate.oliverwyman.com>
>>>>>> www.oliverwyman.com<http://www.oliverwyman.com/>
>>>>>>
>>>>>>
>>>>>> ________________________________
>>>>>> This e-mail and any attachments may be confidential or legally
>>>>>> privileged. If you received this message in error or are not the
>> intended
>>>>>> recipient, you should destroy the e-mail message and any attachments
>> or
>>>>>> copies, and you are prohibited from retaining, distributing,
>> disclosing or
>>>>>> using any information contained herein. Please inform us of the
>> erroneous
>>>>>> delivery by return e-mail. Thank you for your cooperation.
>>>>>
>>>>> ________________________________
>>>>> This e-mail and any attachments may be confidential or legally
>>>>> privileged. If you received this message in error or are not the
>> intended
>>>>> recipient, you should destroy the e-mail message and any attachments or
>>>>> copies, and you are prohibited from retaining, distributing,
>> disclosing or
>>>>> using any information contained herein. Please inform us of the
>> erroneous
>>>>> delivery by return e-mail. Thank you for your cooperation.
>>>>
>>>>
>>>
>>
Re: Airflow Releases
Posted by Alex Van Boxel <al...@vanboxel.be>.
I'll do the same. Nice to have the 1.8 on the horizon.
On Fri, Sep 30, 2016 at 5:51 PM Chris Riccomini <cr...@apache.org>
wrote:
> > I'm not sure how other projects do this, but I propose that we let the
> RC settle for a week.
>
> +1
>
> > Who's on board!?
>
> Me. :)
>
> On Fri, Sep 30, 2016 at 7:56 AM, Maxime Beauchemin
> <ma...@gmail.com> wrote:
> > As related note: the most people we can get to test this release
> candidate
> > the better. Note that Airbnb uses only a subset of the Airflow codebase,
> > there are many operators and features we do not use at all, so we
> encourage
> > everyone to get on board with testing the RC.
> >
> > At Airbnb, our process starts with sending the RC into our staging
> cluster,
> > let it settle and fix any bug over 48 hours or so. We look for changes
> > around DagBag processing time, memory footprint on workers, smooth
> > processing around our canaries.
> >
> > When the RC is ready to hit production, we take a full backup of the
> > metadata DB, and make sure that the whole team is fully available to
> babysit
> > the release and hotfix any issue that may pop up, with a clear, fast
> revert
> > plan.
> >
> > As all of our tasks are assumed to be idempotent the worst case scenarios
> > are around delayed SLAs and wasted resources which is all manageable.
> >
> > I'm not sure how other projects do this, but I propose that we let the RC
> > settle for a week. I suggest that we reset the clock to 7 more days for
> each
> > increment of RC, unless the hotfix is very minor.
> >
> > Who's on board!?
> >
> > Max
> >
> > On Fri, Sep 30, 2016 at 7:43 AM, Maxime Beauchemin
> > <ma...@gmail.com> wrote:
> >>
> >> Airbnb will cut a release candidate on the week of October 10th off of
> >> master.
> >>
> >> Community, if you have outstanding PRs that you want to make this
> release,
> >> now is the time! Though we do not want to rush-merge anything risky
> that may
> >> make the release process harder.
> >>
> >> Max
> >>
> >> On Fri, Sep 30, 2016 at 3:59 AM, Maycock, Luke
> >> <lu...@affiliate.oliverwyman.com> wrote:
> >>>
> >>> Hi Chris,
> >>>
> >>>
> >>> We are currently running master in our environment and we'll be sure to
> >>> report any issues.
> >>>
> >>>
> >>> Do you know when the 1.8 release is due?
> >>>
> >>>
> >>> Cheers,
> >>>
> >>> Luke Maycock
> >>> OLIVER WYMAN
> >>>
> >>> luke.maycock@affiliate.oliverwyman.com<mailto:
> luke.maycock@affiliate.oliverwyman.com>
> >>> www.oliverwyman.com<http://www.oliverwyman.com/>
> >>>
> >>>
> >>> ________________________________
> >>> From: Chris Riccomini <cr...@apache.org>
> >>> Sent: 29 September 2016 18:14:02
> >>> To: dev@airflow.incubator.apache.org
> >>> Subject: Re: Airflow Releases
> >>>
> >>> Hey Luke,
> >>>
> >>> > Is there anything we can do to help get the next release in place?
> >>>
> >>> One thing that would definitely help is running master somewhere in
> >>> your environment, and reporting any issues that see. Over the next few
> >>> weeks, AirBNB and a few other folks will be doing the same in an
> >>> effort to harden the 1.8 release.
> >>>
> >>> Cheers,
> >>> Chris
> >>>
> >>> On Thu, Sep 29, 2016 at 3:08 AM, Maycock, Luke
> >>> <lu...@affiliate.oliverwyman.com> wrote:
> >>> > Airflow Developers,
> >>> >
> >>> >
> >>> > We were looking at writing a workflow framework in Python when we
> found
> >>> > Airflow. We have carried out some proof of concept work for using
> Airflow
> >>> > and wish to continue using it as it comes with lots of great features
> >>> > out-of-the-box.
> >>> >
> >>> >
> >>> > We have created our own fork here:
> >>> >
> >>> > https://github.com/owlabs/incubator-airflow
> >>> >
> >>> >
> >>> > So far, the only thing we have committed back to the main repository
> is
> >>> > the following fix to the mssql_hook:
> >>> >
> >>> > https://github.com/apache/incubator-airflow/pull/1626
> >>> >
> >>> >
> >>> > Among other types of tasks, we wish to be able to run mssql tasks
> using
> >>> > Airflow. In order to do so, the above and below fixes are required:
> >>> >
> >>> >
> >>> > https://github.com/apache/incubator-airflow/pull/1458<
> https://github.com/apache/incubator-airflow/pull/1458/commits/e7e655fde3c29742149d047028cbb21aecba86ed
> >
> >>> >
> >>> >
> >>> > We have created a Chef cookbook for configuring VMs with Airflow and
> >>> > its prerequisites. As part of this cookbook, we are installing the
> latest
> >>> > release of Airflow. However, it appears that the latest release does
> not
> >>> > have the aforementioned fixes.
> >>> >
> >>> > Do you know when the next release of Airflow is expected? Is there
> >>> > anything we can do to help get the next release in place?
> >>> >
> >>> >
> >>> > Luke Maycock
> >>> > OLIVER WYMAN
> >>> >
> >>> > luke.maycock@affiliate.oliverwyman.com<mailto:
> luke.maycock@affiliate.oliverwyman.com>
> >>> > www.oliverwyman.com<http://www.oliverwyman.com/>
> >>> >
> >>> >
> >>> > ________________________________
> >>> > This e-mail and any attachments may be confidential or legally
> >>> > privileged. If you received this message in error or are not the
> intended
> >>> > recipient, you should destroy the e-mail message and any attachments
> or
> >>> > copies, and you are prohibited from retaining, distributing,
> disclosing or
> >>> > using any information contained herein. Please inform us of the
> erroneous
> >>> > delivery by return e-mail. Thank you for your cooperation.
> >>>
> >>> ________________________________
> >>> This e-mail and any attachments may be confidential or legally
> >>> privileged. If you received this message in error or are not the
> intended
> >>> recipient, you should destroy the e-mail message and any attachments or
> >>> copies, and you are prohibited from retaining, distributing,
> disclosing or
> >>> using any information contained herein. Please inform us of the
> erroneous
> >>> delivery by return e-mail. Thank you for your cooperation.
> >>
> >>
> >
>
Re: Airflow Releases
Posted by Chris Riccomini <cr...@apache.org>.
> I'm not sure how other projects do this, but I propose that we let the RC settle for a week.
+1
> Who's on board!?
Me. :)
On Fri, Sep 30, 2016 at 7:56 AM, Maxime Beauchemin
<ma...@gmail.com> wrote:
> As related note: the most people we can get to test this release candidate
> the better. Note that Airbnb uses only a subset of the Airflow codebase,
> there are many operators and features we do not use at all, so we encourage
> everyone to get on board with testing the RC.
>
> At Airbnb, our process starts with sending the RC into our staging cluster,
> let it settle and fix any bug over 48 hours or so. We look for changes
> around DagBag processing time, memory footprint on workers, smooth
> processing around our canaries.
>
> When the RC is ready to hit production, we take a full backup of the
> metadata DB, and make sure that the whole team is fully available to babysit
> the release and hotfix any issue that may pop up, with a clear, fast revert
> plan.
>
> As all of our tasks are assumed to be idempotent the worst case scenarios
> are around delayed SLAs and wasted resources which is all manageable.
>
> I'm not sure how other projects do this, but I propose that we let the RC
> settle for a week. I suggest that we reset the clock to 7 more days for each
> increment of RC, unless the hotfix is very minor.
>
> Who's on board!?
>
> Max
>
> On Fri, Sep 30, 2016 at 7:43 AM, Maxime Beauchemin
> <ma...@gmail.com> wrote:
>>
>> Airbnb will cut a release candidate on the week of October 10th off of
>> master.
>>
>> Community, if you have outstanding PRs that you want to make this release,
>> now is the time! Though we do not want to rush-merge anything risky that may
>> make the release process harder.
>>
>> Max
>>
>> On Fri, Sep 30, 2016 at 3:59 AM, Maycock, Luke
>> <lu...@affiliate.oliverwyman.com> wrote:
>>>
>>> Hi Chris,
>>>
>>>
>>> We are currently running master in our environment and we'll be sure to
>>> report any issues.
>>>
>>>
>>> Do you know when the 1.8 release is due?
>>>
>>>
>>> Cheers,
>>>
>>> Luke Maycock
>>> OLIVER WYMAN
>>>
>>> luke.maycock@affiliate.oliverwyman.com<ma...@affiliate.oliverwyman.com>
>>> www.oliverwyman.com<http://www.oliverwyman.com/>
>>>
>>>
>>> ________________________________
>>> From: Chris Riccomini <cr...@apache.org>
>>> Sent: 29 September 2016 18:14:02
>>> To: dev@airflow.incubator.apache.org
>>> Subject: Re: Airflow Releases
>>>
>>> Hey Luke,
>>>
>>> > Is there anything we can do to help get the next release in place?
>>>
>>> One thing that would definitely help is running master somewhere in
>>> your environment, and reporting any issues that see. Over the next few
>>> weeks, AirBNB and a few other folks will be doing the same in an
>>> effort to harden the 1.8 release.
>>>
>>> Cheers,
>>> Chris
>>>
>>> On Thu, Sep 29, 2016 at 3:08 AM, Maycock, Luke
>>> <lu...@affiliate.oliverwyman.com> wrote:
>>> > Airflow Developers,
>>> >
>>> >
>>> > We were looking at writing a workflow framework in Python when we found
>>> > Airflow. We have carried out some proof of concept work for using Airflow
>>> > and wish to continue using it as it comes with lots of great features
>>> > out-of-the-box.
>>> >
>>> >
>>> > We have created our own fork here:
>>> >
>>> > https://github.com/owlabs/incubator-airflow
>>> >
>>> >
>>> > So far, the only thing we have committed back to the main repository is
>>> > the following fix to the mssql_hook:
>>> >
>>> > https://github.com/apache/incubator-airflow/pull/1626
>>> >
>>> >
>>> > Among other types of tasks, we wish to be able to run mssql tasks using
>>> > Airflow. In order to do so, the above and below fixes are required:
>>> >
>>> >
>>> > https://github.com/apache/incubator-airflow/pull/1458<https://github.com/apache/incubator-airflow/pull/1458/commits/e7e655fde3c29742149d047028cbb21aecba86ed>
>>> >
>>> >
>>> > We have created a Chef cookbook for configuring VMs with Airflow and
>>> > its prerequisites. As part of this cookbook, we are installing the latest
>>> > release of Airflow. However, it appears that the latest release does not
>>> > have the aforementioned fixes.
>>> >
>>> > Do you know when the next release of Airflow is expected? Is there
>>> > anything we can do to help get the next release in place?
>>> >
>>> >
>>> > Luke Maycock
>>> > OLIVER WYMAN
>>> >
>>> > luke.maycock@affiliate.oliverwyman.com<ma...@affiliate.oliverwyman.com>
>>> > www.oliverwyman.com<http://www.oliverwyman.com/>
>>> >
>>> >
>>> > ________________________________
>>> > This e-mail and any attachments may be confidential or legally
>>> > privileged. If you received this message in error or are not the intended
>>> > recipient, you should destroy the e-mail message and any attachments or
>>> > copies, and you are prohibited from retaining, distributing, disclosing or
>>> > using any information contained herein. Please inform us of the erroneous
>>> > delivery by return e-mail. Thank you for your cooperation.
>>>
>>> ________________________________
>>> This e-mail and any attachments may be confidential or legally
>>> privileged. If you received this message in error or are not the intended
>>> recipient, you should destroy the e-mail message and any attachments or
>>> copies, and you are prohibited from retaining, distributing, disclosing or
>>> using any information contained herein. Please inform us of the erroneous
>>> delivery by return e-mail. Thank you for your cooperation.
>>
>>
>
Re: Airflow Releases
Posted by Maxime Beauchemin <ma...@gmail.com>.
As related note: the most people we can get to test this release candidate
the better. Note that Airbnb uses only a subset of the Airflow codebase,
there are many operators and features we do not use at all, so we encourage
everyone to get on board with testing the RC.
At Airbnb, our process starts with sending the RC into our staging cluster,
let it settle and fix any bug over 48 hours or so. We look for changes
around DagBag processing time, memory footprint on workers, smooth
processing around our canaries.
When the RC is ready to hit production, we take a full backup of the
metadata DB, and make sure that the whole team is fully available to
babysit the release and hotfix any issue that may pop up, with a clear,
fast revert plan.
As all of our tasks are assumed to be idempotent the worst case scenarios
are around delayed SLAs and wasted resources which is all manageable.
I'm not sure how other projects do this, but I propose that we let the RC
settle for a week. I suggest that we reset the clock to 7 more days for
each increment of RC, unless the hotfix is very minor.
Who's on board!?
Max
On Fri, Sep 30, 2016 at 7:43 AM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:
> Airbnb will cut a release candidate on the week of October 10th off of
> master.
>
> Community, if you have outstanding PRs that you want to make this release,
> now is the time! Though we do not want to rush-merge anything risky that
> may make the release process harder.
>
> Max
>
> On Fri, Sep 30, 2016 at 3:59 AM, Maycock, Luke <luke.maycock@affiliate.
> oliverwyman.com> wrote:
>
>> Hi Chris,
>>
>>
>> We are currently running master in our environment and we'll be sure to
>> report any issues.
>>
>>
>> Do you know when the 1.8 release is due?
>>
>>
>> Cheers,
>>
>> Luke Maycock
>> OLIVER WYMAN
>> luke.maycock@affiliate.oliverwyman.com<mailto:luke.maycock@
>> affiliate.oliverwyman.com>
>> www.oliverwyman.com<http://www.oliverwyman.com/>
>>
>>
>> ________________________________
>> From: Chris Riccomini <cr...@apache.org>
>> Sent: 29 September 2016 18:14:02
>> To: dev@airflow.incubator.apache.org
>> Subject: Re: Airflow Releases
>>
>> Hey Luke,
>>
>> > Is there anything we can do to help get the next release in place?
>>
>> One thing that would definitely help is running master somewhere in
>> your environment, and reporting any issues that see. Over the next few
>> weeks, AirBNB and a few other folks will be doing the same in an
>> effort to harden the 1.8 release.
>>
>> Cheers,
>> Chris
>>
>> On Thu, Sep 29, 2016 at 3:08 AM, Maycock, Luke
>> <lu...@affiliate.oliverwyman.com> wrote:
>> > Airflow Developers,
>> >
>> >
>> > We were looking at writing a workflow framework in Python when we found
>> Airflow. We have carried out some proof of concept work for using Airflow
>> and wish to continue using it as it comes with lots of great features
>> out-of-the-box.
>> >
>> >
>> > We have created our own fork here:
>> >
>> > https://github.com/owlabs/incubator-airflow
>> >
>> >
>> > So far, the only thing we have committed back to the main repository is
>> the following fix to the mssql_hook:
>> >
>> > https://github.com/apache/incubator-airflow/pull/1626
>> >
>> >
>> > Among other types of tasks, we wish to be able to run mssql tasks using
>> Airflow. In order to do so, the above and below fixes are required:
>> >
>> > https://github.com/apache/incubator-airflow/pull/1458<https:
>> //github.com/apache/incubator-airflow/pull/1458/commits/e7e6
>> 55fde3c29742149d047028cbb21aecba86ed>
>> >
>> >
>> > We have created a Chef cookbook for configuring VMs with Airflow and
>> its prerequisites. As part of this cookbook, we are installing the latest
>> release of Airflow. However, it appears that the latest release does not
>> have the aforementioned fixes.
>> >
>> > Do you know when the next release of Airflow is expected? Is there
>> anything we can do to help get the next release in place?
>> >
>> >
>> > Luke Maycock
>> > OLIVER WYMAN
>> > luke.maycock@affiliate.oliverwyman.com<mailto:luke.maycock@
>> affiliate.oliverwyman.com>
>> > www.oliverwyman.com<http://www.oliverwyman.com/>
>> >
>> >
>> > ________________________________
>> > This e-mail and any attachments may be confidential or legally
>> privileged. If you received this message in error or are not the intended
>> recipient, you should destroy the e-mail message and any attachments or
>> copies, and you are prohibited from retaining, distributing, disclosing or
>> using any information contained herein. Please inform us of the erroneous
>> delivery by return e-mail. Thank you for your cooperation.
>>
>> ________________________________
>> This e-mail and any attachments may be confidential or legally
>> privileged. If you received this message in error or are not the intended
>> recipient, you should destroy the e-mail message and any attachments or
>> copies, and you are prohibited from retaining, distributing, disclosing or
>> using any information contained herein. Please inform us of the erroneous
>> delivery by return e-mail. Thank you for your cooperation.
>>
>
>
Re: Airflow Releases
Posted by Maxime Beauchemin <ma...@gmail.com>.
Airbnb will cut a release candidate on the week of October 10th off of
master.
Community, if you have outstanding PRs that you want to make this release,
now is the time! Though we do not want to rush-merge anything risky that
may make the release process harder.
Max
On Fri, Sep 30, 2016 at 3:59 AM, Maycock, Luke <
luke.maycock@affiliate.oliverwyman.com> wrote:
> Hi Chris,
>
>
> We are currently running master in our environment and we'll be sure to
> report any issues.
>
>
> Do you know when the 1.8 release is due?
>
>
> Cheers,
>
> Luke Maycock
> OLIVER WYMAN
> luke.maycock@affiliate.oliverwyman.com<mailto:luke.
> maycock@affiliate.oliverwyman.com>
> www.oliverwyman.com<http://www.oliverwyman.com/>
>
>
> ________________________________
> From: Chris Riccomini <cr...@apache.org>
> Sent: 29 September 2016 18:14:02
> To: dev@airflow.incubator.apache.org
> Subject: Re: Airflow Releases
>
> Hey Luke,
>
> > Is there anything we can do to help get the next release in place?
>
> One thing that would definitely help is running master somewhere in
> your environment, and reporting any issues that see. Over the next few
> weeks, AirBNB and a few other folks will be doing the same in an
> effort to harden the 1.8 release.
>
> Cheers,
> Chris
>
> On Thu, Sep 29, 2016 at 3:08 AM, Maycock, Luke
> <lu...@affiliate.oliverwyman.com> wrote:
> > Airflow Developers,
> >
> >
> > We were looking at writing a workflow framework in Python when we found
> Airflow. We have carried out some proof of concept work for using Airflow
> and wish to continue using it as it comes with lots of great features
> out-of-the-box.
> >
> >
> > We have created our own fork here:
> >
> > https://github.com/owlabs/incubator-airflow
> >
> >
> > So far, the only thing we have committed back to the main repository is
> the following fix to the mssql_hook:
> >
> > https://github.com/apache/incubator-airflow/pull/1626
> >
> >
> > Among other types of tasks, we wish to be able to run mssql tasks using
> Airflow. In order to do so, the above and below fixes are required:
> >
> > https://github.com/apache/incubator-airflow/pull/1458<ht
> tps://github.com/apache/incubator-airflow/pull/1458/commits/
> e7e655fde3c29742149d047028cbb21aecba86ed>
> >
> >
> > We have created a Chef cookbook for configuring VMs with Airflow and its
> prerequisites. As part of this cookbook, we are installing the latest
> release of Airflow. However, it appears that the latest release does not
> have the aforementioned fixes.
> >
> > Do you know when the next release of Airflow is expected? Is there
> anything we can do to help get the next release in place?
> >
> >
> > Luke Maycock
> > OLIVER WYMAN
> > luke.maycock@affiliate.oliverwyman.com<mailto:luke.
> maycock@affiliate.oliverwyman.com>
> > www.oliverwyman.com<http://www.oliverwyman.com/>
> >
> >
> > ________________________________
> > This e-mail and any attachments may be confidential or legally
> privileged. If you received this message in error or are not the intended
> recipient, you should destroy the e-mail message and any attachments or
> copies, and you are prohibited from retaining, distributing, disclosing or
> using any information contained herein. Please inform us of the erroneous
> delivery by return e-mail. Thank you for your cooperation.
>
> ________________________________
> This e-mail and any attachments may be confidential or legally privileged.
> If you received this message in error or are not the intended recipient,
> you should destroy the e-mail message and any attachments or copies, and
> you are prohibited from retaining, distributing, disclosing or using any
> information contained herein. Please inform us of the erroneous delivery by
> return e-mail. Thank you for your cooperation.
>
Re: Airflow Releases
Posted by "Maycock, Luke" <lu...@affiliate.oliverwyman.com>.
Hi Chris,
We are currently running master in our environment and we'll be sure to report any issues.
Do you know when the 1.8 release is due?
Cheers,
Luke Maycock
OLIVER WYMAN
luke.maycock@affiliate.oliverwyman.com<ma...@affiliate.oliverwyman.com>
www.oliverwyman.com<http://www.oliverwyman.com/>
________________________________
From: Chris Riccomini <cr...@apache.org>
Sent: 29 September 2016 18:14:02
To: dev@airflow.incubator.apache.org
Subject: Re: Airflow Releases
Hey Luke,
> Is there anything we can do to help get the next release in place?
One thing that would definitely help is running master somewhere in
your environment, and reporting any issues that see. Over the next few
weeks, AirBNB and a few other folks will be doing the same in an
effort to harden the 1.8 release.
Cheers,
Chris
On Thu, Sep 29, 2016 at 3:08 AM, Maycock, Luke
<lu...@affiliate.oliverwyman.com> wrote:
> Airflow Developers,
>
>
> We were looking at writing a workflow framework in Python when we found Airflow. We have carried out some proof of concept work for using Airflow and wish to continue using it as it comes with lots of great features out-of-the-box.
>
>
> We have created our own fork here:
>
> https://github.com/owlabs/incubator-airflow
>
>
> So far, the only thing we have committed back to the main repository is the following fix to the mssql_hook:
>
> https://github.com/apache/incubator-airflow/pull/1626
>
>
> Among other types of tasks, we wish to be able to run mssql tasks using Airflow. In order to do so, the above and below fixes are required:
>
> https://github.com/apache/incubator-airflow/pull/1458<https://github.com/apache/incubator-airflow/pull/1458/commits/e7e655fde3c29742149d047028cbb21aecba86ed>
>
>
> We have created a Chef cookbook for configuring VMs with Airflow and its prerequisites. As part of this cookbook, we are installing the latest release of Airflow. However, it appears that the latest release does not have the aforementioned fixes.
>
> Do you know when the next release of Airflow is expected? Is there anything we can do to help get the next release in place?
>
>
> Luke Maycock
> OLIVER WYMAN
> luke.maycock@affiliate.oliverwyman.com<ma...@affiliate.oliverwyman.com>
> www.oliverwyman.com<http://www.oliverwyman.com/>
>
>
> ________________________________
> This e-mail and any attachments may be confidential or legally privileged. If you received this message in error or are not the intended recipient, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained herein. Please inform us of the erroneous delivery by return e-mail. Thank you for your cooperation.
________________________________
This e-mail and any attachments may be confidential or legally privileged. If you received this message in error or are not the intended recipient, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained herein. Please inform us of the erroneous delivery by return e-mail. Thank you for your cooperation.
Re: Airflow Releases
Posted by "Maycock, Luke" <lu...@affiliate.oliverwyman.com>.
Hi All,
We are running master in our environment and have noticed something new (that wasn't present in release 1.7.1.3).
I have the following DAG, which I ran a backfill on:
# ------ BuildTask6 ------ BuildTask7 ------ BuildTask8 ------ BuildTask9
# / /
# InitBuildTask ------ BuildTask1 ------ BuildTask2 ------ BuildTask3 /
# \ \ /
# ------ BuildTask4 ------------------------------------------ BuildTask5
#
When the backfill begins, the outputs gives warnings for all tasks that have not yet had their dependencies met (full output attached) i.e. all tasks except the one task that has no dependencies. This continues until the backfill has completed. Is this expected behaviour?
Cheers,
Luke Maycock
OLIVER WYMAN
luke.maycock@affiliate.oliverwyman.com<ma...@affiliate.oliverwyman.com>
www.oliverwyman.com<http://www.oliverwyman.com/>
________________________________
From: Chris Riccomini <cr...@apache.org>
Sent: 29 September 2016 18:14:02
To: dev@airflow.incubator.apache.org
Subject: Re: Airflow Releases
Hey Luke,
> Is there anything we can do to help get the next release in place?
One thing that would definitely help is running master somewhere in
your environment, and reporting any issues that see. Over the next few
weeks, AirBNB and a few other folks will be doing the same in an
effort to harden the 1.8 release.
Cheers,
Chris
On Thu, Sep 29, 2016 at 3:08 AM, Maycock, Luke
<lu...@affiliate.oliverwyman.com> wrote:
> Airflow Developers,
>
>
> We were looking at writing a workflow framework in Python when we found Airflow. We have carried out some proof of concept work for using Airflow and wish to continue using it as it comes with lots of great features out-of-the-box.
>
>
> We have created our own fork here:
>
> https://github.com/owlabs/incubator-airflow
>
>
> So far, the only thing we have committed back to the main repository is the following fix to the mssql_hook:
>
> https://github.com/apache/incubator-airflow/pull/1626
>
>
> Among other types of tasks, we wish to be able to run mssql tasks using Airflow. In order to do so, the above and below fixes are required:
>
> https://github.com/apache/incubator-airflow/pull/1458<https://github.com/apache/incubator-airflow/pull/1458/commits/e7e655fde3c29742149d047028cbb21aecba86ed>
>
>
> We have created a Chef cookbook for configuring VMs with Airflow and its prerequisites. As part of this cookbook, we are installing the latest release of Airflow. However, it appears that the latest release does not have the aforementioned fixes.
>
> Do you know when the next release of Airflow is expected? Is there anything we can do to help get the next release in place?
>
>
> Luke Maycock
> OLIVER WYMAN
> luke.maycock@affiliate.oliverwyman.com<ma...@affiliate.oliverwyman.com>
> www.oliverwyman.com<http://www.oliverwyman.com/>
>
>
> ________________________________
> This e-mail and any attachments may be confidential or legally privileged. If you received this message in error or are not the intended recipient, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained herein. Please inform us of the erroneous delivery by return e-mail. Thank you for your cooperation.
________________________________
This e-mail and any attachments may be confidential or legally privileged. If you received this message in error or are not the intended recipient, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained herein. Please inform us of the erroneous delivery by return e-mail. Thank you for your cooperation.
Re: Airflow Releases
Posted by Chris Riccomini <cr...@apache.org>.
Hey Luke,
> Is there anything we can do to help get the next release in place?
One thing that would definitely help is running master somewhere in
your environment, and reporting any issues that see. Over the next few
weeks, AirBNB and a few other folks will be doing the same in an
effort to harden the 1.8 release.
Cheers,
Chris
On Thu, Sep 29, 2016 at 3:08 AM, Maycock, Luke
<lu...@affiliate.oliverwyman.com> wrote:
> Airflow Developers,
>
>
> We were looking at writing a workflow framework in Python when we found Airflow. We have carried out some proof of concept work for using Airflow and wish to continue using it as it comes with lots of great features out-of-the-box.
>
>
> We have created our own fork here:
>
> https://github.com/owlabs/incubator-airflow
>
>
> So far, the only thing we have committed back to the main repository is the following fix to the mssql_hook:
>
> https://github.com/apache/incubator-airflow/pull/1626
>
>
> Among other types of tasks, we wish to be able to run mssql tasks using Airflow. In order to do so, the above and below fixes are required:
>
> https://github.com/apache/incubator-airflow/pull/1458<https://github.com/apache/incubator-airflow/pull/1458/commits/e7e655fde3c29742149d047028cbb21aecba86ed>
>
>
> We have created a Chef cookbook for configuring VMs with Airflow and its prerequisites. As part of this cookbook, we are installing the latest release of Airflow. However, it appears that the latest release does not have the aforementioned fixes.
>
> Do you know when the next release of Airflow is expected? Is there anything we can do to help get the next release in place?
>
>
> Luke Maycock
> OLIVER WYMAN
> luke.maycock@affiliate.oliverwyman.com<ma...@affiliate.oliverwyman.com>
> www.oliverwyman.com<http://www.oliverwyman.com/>
>
>
> ________________________________
> This e-mail and any attachments may be confidential or legally privileged. If you received this message in error or are not the intended recipient, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained herein. Please inform us of the erroneous delivery by return e-mail. Thank you for your cooperation.