You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Ash Berlin-Taylor <as...@firemirror.com> on 2017/09/08 09:31:34 UTC
Re: 1.9.0 PR mini-rundown
There are a few PRs I would also like to see in this release, I've gone through the open Github pull requests and picked some out. I've only listed ones that I would accept if I was reviewing them.
One of these two (one fixes the specific Celery setting I needed, the other allows much more general customization of Celery config):
- https://github.com/apache/incubator-airflow/pull/2572
- https://github.com/apache/incubator-airflow/pull/2542 or https://github.com/apache/incubator-airflow/pull/2558 (both address the same Jira in slightly different ways.)
https://github.com/apache/incubator-airflow/pull/2578/files looks like it might be an important fix for a change on master, though is lacking tests covering the bug.
https://github.com/apache/incubator-airflow/pull/2540 `Variable.setdefault` raises an exception when the variable already exists(!), at least when using JSON.
### Two PRs that look good but aren't quite ready yet
https://github.com/apache/incubator-airflow/pull/2327/files The `--ignore_all_dependencies` flag wasn't working. (This seems to be failing due to a missing license, but otherwise looks good and is an important fix. I have asked the author to add the missing license to the new test file.)
https://github.com/apache/incubator-airflow/pull/2531 Server-side paging of DAGS - helps with large airflow instances. (Though this is currently marked WIP, so I have asked the author for clarification on if this is ready or not)
https://github.com/apache/incubator-airflow/pull/2562 - adding the 1.8.2 changelog to master branch.
### And then some nice to haves/simple changes:
https://github.com/apache/incubator-airflow/pull/2532 Migrate the S3Hook to use boto3.
https://github.com/apache/incubator-airflow/pull/2509/files a tiny refactor to move a calculation only to where it is needed.
https://github.com/apache/incubator-airflow/pull/2479 fix syntax errors in the docs
https://github.com/apache/incubator-airflow/pull/2405 exposes a new option to `airflow worker` for `--celery_hostname` to support running multiple workers on the same node.
https://github.com/apache/incubator-airflow/pull/2507/files - properly clears hidden fields in the Connection UI when changing type.
https://github.com/apache/incubator-airflow/pull/2237/files - Fix issue with Gunicorn 19.4 and upgrade to it. Probably a good idea to not be stuck on older versions of modules? (has conflicts, but its two one-line changes so looks like it would be easy to resolve at merge time.)
https://github.com/apache/incubator-airflow/pull/2189/files - replace an instance of hardcoded `scheduled_` prefix with ID_PREFIX variable. No semantic meaning but seems to be the last instance of this string that isn't using the variable.
I hope this was helpful.
-ash
> On 7 Sep 2017, at 20:26, Chris Riccomini <cr...@apache.org> wrote:
>
> Double negative for the win. Rephrase: Highly unlikely K8s will make 1.9.0.
>
> On Thu, Sep 7, 2017 at 12:26 PM, Chris Riccomini <cr...@apache.org>
> wrote:
>
>> Highly unlikely K8s will not make 1.9.0.
>>
>> On Thu, Sep 7, 2017 at 12:19 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>>
>>> Depends on how fast it stabilizes...
>>>
>>> Sent from my iPhone
>>>
>>>> On 7 Sep 2017, at 21:11, Sergei Iakhnin <ll...@gmail.com> wrote:
>>>>
>>>> The Kubernetes features would be exciting to have. Do you think these
>>> will
>>>> make it into this release?
>>>>
>>>>> On Thu, Sep 7, 2017 at 8:22 PM Bolke de Bruin <bd...@gmail.com>
>>> wrote:
>>>>>
>>>>> Would be appreciated!
>>>>>
>>>>> Op 7 sep. 2017 8:06 p.m. schreef "Chris Riccomini" <
>>> criccomini@apache.org
>>>>>> :
>>>>>
>>>>>> @Bolke, I can wait to cut the test branch until that PR is merged, if
>>> you
>>>>>> like.
>>>>>>
>>>>>> On Thu, Sep 7, 2017 at 10:19 AM, Bolke de Bruin <bd...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>>> One of my team members is preparing a comprehensive PR that reworks
>>> all
>>>>>>> logging to use the standard framework that was recently introduced.
>>> It
>>>>> is
>>>>>>> expected to be become public tomorrow. So please don't cut too early
>>>>> ;-).
>>>>>>> In addition it would be nice to have a PR clean up? Some PRs have
>>> been
>>>>>>> waiting for some time.
>>>>>>>
>>>>>>> BTW: we will maintain 1.8.X for bug fixes until 1.10/2.0 comes
>>> along.
>>>>>>>
>>>>>>> Cheers
>>>>>>> Bolke
>>>>>>>
>>>>>>> Sent from my iPhone
>>>>>>>
>>>>>>>> On 7 Sep 2017, at 18:55, Chris Riccomini <cr...@apache.org>
>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hey all,
>>>>>>>>
>>>>>>>> The workflow is documented here:
>>>>>>>>
>>>>>>>>
>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Releasing+Airflow
>>>>>>>>
>>>>>>>> I'll probably cut the 1.9.0 test branch today or tomorrow, depending
>>>>> on
>>>>>>>> schedule. I'll let folks know when it's cut via dev@.
>>>>>>>>
>>>>>>>>> Once the branch is cut does that mean no more PRs will be included
>>>>> in
>>>>>>> the
>>>>>>>> release, or can some changes still be cherry-picked/merged in?
>>>>>>>>
>>>>>>>> We can still cherry-pick PRs in. Very quickly, though, we begin
>>>>>> rejecting
>>>>>>>> any PRs that aren't bug fixes. It's a big exercise to stabilize
>>> these
>>>>>>>> releases, so I will start rejecting merging non-bug fix PRs pretty
>>>>>>> quickly.
>>>>>>>> Probably within a couple of weeks.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Chris
>>>>>>>>
>>>>>>>> On Thu, Sep 7, 2017 at 9:31 AM, Ash Berlin-Taylor <
>>>>>>>> ash_airflowlist@firemirror.com> wrote:
>>>>>>>>
>>>>>>>>> What is the workflow? Once the branch is cut does that mean no more
>>>>>> PRs
>>>>>>>>> will be included in the release, or can some changes still be
>>>>>>>>> cherry-picked/merged in?
>>>>>>>>>
>>>>>>>>> (Is this documented somewhere I can read?)
>>>>>>>>>
>>>>>>>>> Oh, and thing someone in gitter chat pointed out: the CHANGELOG on
>>>>>>> master
>>>>>>>>> doesn't include entries for 1.8.2.
>>>>>>>>>
>>>>>>>>> -ash
>>>>>>>>>> On 7 Sep 2017, at 17:28, Andrew Chen <an...@databricks.com>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Any estimate when branch 1.9 will be cut?
>>>>>>>>>>
>>>>>>>>>> On Thu, Sep 7, 2017 at 9:25 AM, Maxime Beauchemin <
>>>>>>>>>> maximebeauchemin@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Go for it! Still trying to wrap up 1.8.2 ...
>>>>>>>>>>>
>>>>>>>>>>> Max
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Sep 7, 2017 at 8:31 AM, Bolke de Bruin <
>>> bdbruin@gmail.com
>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Awesome!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On 7 Sep 2017, at 17:04, Chris Riccomini <
>>> criccomini@apache.org
>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I want to get the ball rolling on 1.9.0, now that 1.8.2 is out
>>>>> the
>>>>>>>>>>> door.
>>>>>>>>>>>>>
>>>>>>>>>>>>> @Max, I recall you mentioning driving this, but if you're burnt
>>>>>> out
>>>>>>>>>>> after
>>>>>>>>>>>>> the 1.8.2 work, Joy and I can drive 1.9.0. Just let me know.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Chris
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>> --
>>>>
>>>> Sergei
>>>
>>
>>
Re: 1.9.0 PR mini-rundown
Posted by Bolke de Bruin <bd...@gmail.com>.
Nice run down. There is also a security patch we need to integrate (also to 1.8.3).
Bolke
Verstuurd vanaf mijn iPad
> Op 8 sep. 2017 om 11:31 heeft Ash Berlin-Taylor <as...@firemirror.com> het volgende geschreven:
>
> There are a few PRs I would also like to see in this release, I've gone through the open Github pull requests and picked some out. I've only listed ones that I would accept if I was reviewing them.
>
> One of these two (one fixes the specific Celery setting I needed, the other allows much more general customization of Celery config):
>
> - https://github.com/apache/incubator-airflow/pull/2572
> - https://github.com/apache/incubator-airflow/pull/2542 or https://github.com/apache/incubator-airflow/pull/2558 (both address the same Jira in slightly different ways.)
>
>
> https://github.com/apache/incubator-airflow/pull/2578/files looks like it might be an important fix for a change on master, though is lacking tests covering the bug.
>
> https://github.com/apache/incubator-airflow/pull/2540 `Variable.setdefault` raises an exception when the variable already exists(!), at least when using JSON.
>
>
> ### Two PRs that look good but aren't quite ready yet
>
> https://github.com/apache/incubator-airflow/pull/2327/files The `--ignore_all_dependencies` flag wasn't working. (This seems to be failing due to a missing license, but otherwise looks good and is an important fix. I have asked the author to add the missing license to the new test file.)
>
> https://github.com/apache/incubator-airflow/pull/2531 Server-side paging of DAGS - helps with large airflow instances. (Though this is currently marked WIP, so I have asked the author for clarification on if this is ready or not)
>
> https://github.com/apache/incubator-airflow/pull/2562 - adding the 1.8.2 changelog to master branch.
>
>
>
> ### And then some nice to haves/simple changes:
>
> https://github.com/apache/incubator-airflow/pull/2532 Migrate the S3Hook to use boto3.
>
> https://github.com/apache/incubator-airflow/pull/2509/files a tiny refactor to move a calculation only to where it is needed.
>
> https://github.com/apache/incubator-airflow/pull/2479 fix syntax errors in the docs
>
> https://github.com/apache/incubator-airflow/pull/2405 exposes a new option to `airflow worker` for `--celery_hostname` to support running multiple workers on the same node.
>
> https://github.com/apache/incubator-airflow/pull/2507/files - properly clears hidden fields in the Connection UI when changing type.
>
> https://github.com/apache/incubator-airflow/pull/2237/files - Fix issue with Gunicorn 19.4 and upgrade to it. Probably a good idea to not be stuck on older versions of modules? (has conflicts, but its two one-line changes so looks like it would be easy to resolve at merge time.)
>
> https://github.com/apache/incubator-airflow/pull/2189/files - replace an instance of hardcoded `scheduled_` prefix with ID_PREFIX variable. No semantic meaning but seems to be the last instance of this string that isn't using the variable.
>
> I hope this was helpful.
>
> -ash
>
>
>> On 7 Sep 2017, at 20:26, Chris Riccomini <cr...@apache.org> wrote:
>>
>> Double negative for the win. Rephrase: Highly unlikely K8s will make 1.9.0.
>>
>> On Thu, Sep 7, 2017 at 12:26 PM, Chris Riccomini <cr...@apache.org>
>> wrote:
>>
>>> Highly unlikely K8s will not make 1.9.0.
>>>
>>>> On Thu, Sep 7, 2017 at 12:19 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>>>>
>>>> Depends on how fast it stabilizes...
>>>>
>>>> Sent from my iPhone
>>>>
>>>>> On 7 Sep 2017, at 21:11, Sergei Iakhnin <ll...@gmail.com> wrote:
>>>>>
>>>>> The Kubernetes features would be exciting to have. Do you think these
>>>> will
>>>>> make it into this release?
>>>>>
>>>>>> On Thu, Sep 7, 2017 at 8:22 PM Bolke de Bruin <bd...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>> Would be appreciated!
>>>>>>
>>>>>> Op 7 sep. 2017 8:06 p.m. schreef "Chris Riccomini" <
>>>> criccomini@apache.org
>>>>>>> :
>>>>>>
>>>>>>> @Bolke, I can wait to cut the test branch until that PR is merged, if
>>>> you
>>>>>>> like.
>>>>>>>
>>>>>>> On Thu, Sep 7, 2017 at 10:19 AM, Bolke de Bruin <bd...@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>> One of my team members is preparing a comprehensive PR that reworks
>>>> all
>>>>>>>> logging to use the standard framework that was recently introduced.
>>>> It
>>>>>> is
>>>>>>>> expected to be become public tomorrow. So please don't cut too early
>>>>>> ;-).
>>>>>>>> In addition it would be nice to have a PR clean up? Some PRs have
>>>> been
>>>>>>>> waiting for some time.
>>>>>>>>
>>>>>>>> BTW: we will maintain 1.8.X for bug fixes until 1.10/2.0 comes
>>>> along.
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>> Bolke
>>>>>>>>
>>>>>>>> Sent from my iPhone
>>>>>>>>
>>>>>>>>> On 7 Sep 2017, at 18:55, Chris Riccomini <cr...@apache.org>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hey all,
>>>>>>>>>
>>>>>>>>> The workflow is documented here:
>>>>>>>>>
>>>>>>>>>
>>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Releasing+Airflow
>>>>>>>>>
>>>>>>>>> I'll probably cut the 1.9.0 test branch today or tomorrow, depending
>>>>>> on
>>>>>>>>> schedule. I'll let folks know when it's cut via dev@.
>>>>>>>>>
>>>>>>>>>> Once the branch is cut does that mean no more PRs will be included
>>>>>> in
>>>>>>>> the
>>>>>>>>> release, or can some changes still be cherry-picked/merged in?
>>>>>>>>>
>>>>>>>>> We can still cherry-pick PRs in. Very quickly, though, we begin
>>>>>>> rejecting
>>>>>>>>> any PRs that aren't bug fixes. It's a big exercise to stabilize
>>>> these
>>>>>>>>> releases, so I will start rejecting merging non-bug fix PRs pretty
>>>>>>>> quickly.
>>>>>>>>> Probably within a couple of weeks.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Chris
>>>>>>>>>
>>>>>>>>> On Thu, Sep 7, 2017 at 9:31 AM, Ash Berlin-Taylor <
>>>>>>>>> ash_airflowlist@firemirror.com> wrote:
>>>>>>>>>
>>>>>>>>>> What is the workflow? Once the branch is cut does that mean no more
>>>>>>> PRs
>>>>>>>>>> will be included in the release, or can some changes still be
>>>>>>>>>> cherry-picked/merged in?
>>>>>>>>>>
>>>>>>>>>> (Is this documented somewhere I can read?)
>>>>>>>>>>
>>>>>>>>>> Oh, and thing someone in gitter chat pointed out: the CHANGELOG on
>>>>>>>> master
>>>>>>>>>> doesn't include entries for 1.8.2.
>>>>>>>>>>
>>>>>>>>>> -ash
>>>>>>>>>>> On 7 Sep 2017, at 17:28, Andrew Chen <an...@databricks.com>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Any estimate when branch 1.9 will be cut?
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Sep 7, 2017 at 9:25 AM, Maxime Beauchemin <
>>>>>>>>>>> maximebeauchemin@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Go for it! Still trying to wrap up 1.8.2 ...
>>>>>>>>>>>>
>>>>>>>>>>>> Max
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Sep 7, 2017 at 8:31 AM, Bolke de Bruin <
>>>> bdbruin@gmail.com
>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Awesome!
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 7 Sep 2017, at 17:04, Chris Riccomini <
>>>> criccomini@apache.org
>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I want to get the ball rolling on 1.9.0, now that 1.8.2 is out
>>>>>> the
>>>>>>>>>>>> door.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> @Max, I recall you mentioning driving this, but if you're burnt
>>>>>>> out
>>>>>>>>>>>> after
>>>>>>>>>>>>>> the 1.8.2 work, Joy and I can drive 1.9.0. Just let me know.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>> --
>>>>>
>>>>> Sergei
>>>>
>>>
>>>
>